Visual short-term memory is more complex than previously assumed

For the study, Dr. Hui Zhang, Rebekka Heinen and Professor Nikolai Axmacher from the Department of Neuropsychology collaborated with the team headed by Jing Liu and Professor Gui Xue from Beijing Normal University.

The banana in short-term memory

Visual short-term memory helps us remember objects for a short period of time when these objects are no longer visible. Until now, it has been assumed that short-term memory is based on only one type of brain activity. The German-Chinese research team has now disproved this assumption. The researchers recorded brain activity in epilepsy patients using electrodes that were inserted into the brain for the purpose of surgical planning. The patients saw pictures of objects like a banana and had to remember them for a short time.

Deep neural networks help interpret brain activity

Earlier studies by other groups had shown that deep neural networks process images in similar steps as humans do. If a person or a deep neural network sees a banana, the first step is to process simple characteristics such as its yellow colour and smooth texture. Later on, the processed information becomes more and more complex. Eventually, the human and the network recognise the specific crescent shape and finally identify the banana.

The researchers compared the different processing steps of the neural network with the brain data of the patients. This enabled them to see which activity patterns belong to the processing of simple visual properties like the yellow colour of the banana and which belong to more complex properties like its name.

First simple, then complex

Based on this result, the team then showed that objects are not only represented in one form in short-term memory, as previously assumed, but in several forms simultaneously. When looking at them, initially simple properties of the banana are processed, then complex properties are added. During the memorisation phase, simple and complex information is retained together. The visual short-term memory is thus more complex than has long been assumed.