Role of inferior temporal neurons in visual memory I. Temporal encoding of information about visual images, recalled images, and behavioral context

1. Lesions of the inferior temporal (IT) cortex selectively hamper monkeys in tasks requiring visual memory. A system that recognizes images must be able to encode a current stimulus, recall the code of a previous stimulus, compare the codes of the two stimuli, and make a decision on the basis of the outcome of the comparison. Therefore, IT neurons must be involved in at least one of these processes. To determine the specific role of IT neurons in visual memory, we measured the information conveyed in the neuronal responses about current patterns, recalled patterns, and behavioral context. 2. Two monkeys were trained to perform a sequential matching task using a set of 32 black and white Walsh patterns. In the course of an experiment, each pattern was presented repeatedly in sample, match, and nonmatch behavioral contexts. While the monkeys were performing the task, we recorded the activity of 76 neurons from area TE of IT. The neuronal responses to the stimuli were converted to spike density functions, and the resultant waveforms were quantified using their principal components. The relationships between the responses and the stimuli were studied using analysis of variance and information theory. 3. The analysis of variance was applied to the neuronal response waveforms using the context (sample, match, or nonmatch) and the patterns of the stimuli as independent variables and the spike count or the coefficients of the principal components as the dependent variables. We found that the waveforms of most neurons were significantly modulated by both the pattern and the context of the stimulus presentation. 4. We also analyzed the stimulus-response relationships using information theory. The input codes were based on the pattern and context of the stimuli, and the output codes were based on the spike count or the principal components of the responses. The neuronal response waveforms were found to convey significant amounts of information about both the pattern and context of the stimuli. Transmitted information was greatest when the response of a neuron was interpreted as a message about the combination of pattern and context. Nevertheless, there was information about context independent of pattern and vice versa. 5. We also used information theory to determine whether the neuronal responses to the second, or test, stimulus conveyed information about the pattern of the first, or sample, stimulus. The input codes were based on the patterns of the sample stimuli, and the output codes were based on the responses to the nonmatch test stimuli. We found that the responses to the nonmatch stimuli carried significant amounts of information about the pattern of the previous sample stimuli. Furthermore, two thirds of this information was transmitted in the temporal distribution of the spikes and only one third in their number. 6. Area TE includes neurons in the superior temporal sulcus and the inferior temporal gyrus. On the basis of penetration depth records, we found that neurons in the gyrus appeared to convey more than twice as much information about the patterns of the current stimuli as the neurons in the sulcus. Thus there appears to be a functional subdivision within the anatomically defined area TE. 7. We hypothesize that the role of IT neurons in visual memory tasks is to compare the internal representations of current visual images with the internal representations of recalled images. The companion paper tests a hypothetical neuronal mechanism, based on multiplication of temporally modulated waveforms, for making this comparison.