Multisensory auditory-visual interactions during early sensory processing in humans: A high-density electrical mapping study

Sophie Molholm, Walter Ritter, Micah M. Murray, Daniel C. Javitt, Charles E. Schroeder, John J. Foxe

Research output: Contribution to journalArticle

603 Citations (Scopus)

Abstract

Integration of information from multiple senses is fundamental to perception and cognition, but when and where this is accomplished in the brain is not well understood. This study examined the timing and topography of cortical auditory-visual interactions using high-density event-related potentials (ERPs) during a simple reaction-time (RT) task. Visual and auditory stimuli were presented alone and simultaneously. ERPs elicited by the auditory and visual stimuli when presented alone were summed ('sum' ERP) and compared to the ERP elicited when they were presented simultaneously ('simultaneous' ERP). Divergence between the 'simultaneous' and 'sum' ERP indicated auditory-visual (AV) neural response interactions. There was a surprisingly early right parieto-occipital AV interaction, consistent with the finding of an earlier study [J. Cogn. Neurosci. 11 (1999) 473]. The timing of onset of this effect (46 ms) was essentially simultaneous with the onset of visual cortical processing, as indexed by the onset of the visual C1 component, which is thought to represent the earliest cortical visual evoked potential. The coincident timing of the early AV interaction and C1 strongly suggests that AV interactions can affect early visual sensory processing. Additional AV interactions were found within the time course of sensory processing (up to 200 ms post stimulus onset). In total, this system of AV effects over the scalp was suggestive of both activity unique to multisensory processing, and the modulation of 'unisensory' activity. RTs to the stimuli when presented simultaneously were significantly faster than when they were presented alone. This RT facilitation could not be accounted for by probability summation, as evidenced by violation of the 'race' model, providing compelling evidence that auditory-visual neural interactions give rise to this RT effect.

Original languageEnglish (US)
Pages (from-to)115-128
Number of pages14
JournalCognitive Brain Research
Volume14
Issue number1
DOIs
StatePublished - 2002
Externally publishedYes

Fingerprint

Evoked Potentials
Reaction Time
Visual Evoked Potentials
Scalp
Cognition
Brain

Keywords

  • Auditory
  • Electrophysiology
  • ERPs
  • Multisensory
  • Visual

ASJC Scopus subject areas

  • Behavioral Neuroscience
  • Cognitive Neuroscience
  • Experimental and Cognitive Psychology

Cite this

Multisensory auditory-visual interactions during early sensory processing in humans : A high-density electrical mapping study. / Molholm, Sophie; Ritter, Walter; Murray, Micah M.; Javitt, Daniel C.; Schroeder, Charles E.; Foxe, John J.

In: Cognitive Brain Research, Vol. 14, No. 1, 2002, p. 115-128.

Research output: Contribution to journalArticle

Molholm, Sophie ; Ritter, Walter ; Murray, Micah M. ; Javitt, Daniel C. ; Schroeder, Charles E. ; Foxe, John J. / Multisensory auditory-visual interactions during early sensory processing in humans : A high-density electrical mapping study. In: Cognitive Brain Research. 2002 ; Vol. 14, No. 1. pp. 115-128.
@article{c4af68388b9b4aeda67ef56f7d8da6cf,
title = "Multisensory auditory-visual interactions during early sensory processing in humans: A high-density electrical mapping study",
abstract = "Integration of information from multiple senses is fundamental to perception and cognition, but when and where this is accomplished in the brain is not well understood. This study examined the timing and topography of cortical auditory-visual interactions using high-density event-related potentials (ERPs) during a simple reaction-time (RT) task. Visual and auditory stimuli were presented alone and simultaneously. ERPs elicited by the auditory and visual stimuli when presented alone were summed ('sum' ERP) and compared to the ERP elicited when they were presented simultaneously ('simultaneous' ERP). Divergence between the 'simultaneous' and 'sum' ERP indicated auditory-visual (AV) neural response interactions. There was a surprisingly early right parieto-occipital AV interaction, consistent with the finding of an earlier study [J. Cogn. Neurosci. 11 (1999) 473]. The timing of onset of this effect (46 ms) was essentially simultaneous with the onset of visual cortical processing, as indexed by the onset of the visual C1 component, which is thought to represent the earliest cortical visual evoked potential. The coincident timing of the early AV interaction and C1 strongly suggests that AV interactions can affect early visual sensory processing. Additional AV interactions were found within the time course of sensory processing (up to 200 ms post stimulus onset). In total, this system of AV effects over the scalp was suggestive of both activity unique to multisensory processing, and the modulation of 'unisensory' activity. RTs to the stimuli when presented simultaneously were significantly faster than when they were presented alone. This RT facilitation could not be accounted for by probability summation, as evidenced by violation of the 'race' model, providing compelling evidence that auditory-visual neural interactions give rise to this RT effect.",
keywords = "Auditory, Electrophysiology, ERPs, Multisensory, Visual",
author = "Sophie Molholm and Walter Ritter and Murray, {Micah M.} and Javitt, {Daniel C.} and Schroeder, {Charles E.} and Foxe, {John J.}",
year = "2002",
doi = "10.1016/S0926-6410(02)00066-6",
language = "English (US)",
volume = "14",
pages = "115--128",
journal = "Cognitive Brain Research",
issn = "0926-6410",
publisher = "Elsevier BV",
number = "1",

}

TY - JOUR

T1 - Multisensory auditory-visual interactions during early sensory processing in humans

T2 - A high-density electrical mapping study

AU - Molholm, Sophie

AU - Ritter, Walter

AU - Murray, Micah M.

AU - Javitt, Daniel C.

AU - Schroeder, Charles E.

AU - Foxe, John J.

PY - 2002

Y1 - 2002

N2 - Integration of information from multiple senses is fundamental to perception and cognition, but when and where this is accomplished in the brain is not well understood. This study examined the timing and topography of cortical auditory-visual interactions using high-density event-related potentials (ERPs) during a simple reaction-time (RT) task. Visual and auditory stimuli were presented alone and simultaneously. ERPs elicited by the auditory and visual stimuli when presented alone were summed ('sum' ERP) and compared to the ERP elicited when they were presented simultaneously ('simultaneous' ERP). Divergence between the 'simultaneous' and 'sum' ERP indicated auditory-visual (AV) neural response interactions. There was a surprisingly early right parieto-occipital AV interaction, consistent with the finding of an earlier study [J. Cogn. Neurosci. 11 (1999) 473]. The timing of onset of this effect (46 ms) was essentially simultaneous with the onset of visual cortical processing, as indexed by the onset of the visual C1 component, which is thought to represent the earliest cortical visual evoked potential. The coincident timing of the early AV interaction and C1 strongly suggests that AV interactions can affect early visual sensory processing. Additional AV interactions were found within the time course of sensory processing (up to 200 ms post stimulus onset). In total, this system of AV effects over the scalp was suggestive of both activity unique to multisensory processing, and the modulation of 'unisensory' activity. RTs to the stimuli when presented simultaneously were significantly faster than when they were presented alone. This RT facilitation could not be accounted for by probability summation, as evidenced by violation of the 'race' model, providing compelling evidence that auditory-visual neural interactions give rise to this RT effect.

AB - Integration of information from multiple senses is fundamental to perception and cognition, but when and where this is accomplished in the brain is not well understood. This study examined the timing and topography of cortical auditory-visual interactions using high-density event-related potentials (ERPs) during a simple reaction-time (RT) task. Visual and auditory stimuli were presented alone and simultaneously. ERPs elicited by the auditory and visual stimuli when presented alone were summed ('sum' ERP) and compared to the ERP elicited when they were presented simultaneously ('simultaneous' ERP). Divergence between the 'simultaneous' and 'sum' ERP indicated auditory-visual (AV) neural response interactions. There was a surprisingly early right parieto-occipital AV interaction, consistent with the finding of an earlier study [J. Cogn. Neurosci. 11 (1999) 473]. The timing of onset of this effect (46 ms) was essentially simultaneous with the onset of visual cortical processing, as indexed by the onset of the visual C1 component, which is thought to represent the earliest cortical visual evoked potential. The coincident timing of the early AV interaction and C1 strongly suggests that AV interactions can affect early visual sensory processing. Additional AV interactions were found within the time course of sensory processing (up to 200 ms post stimulus onset). In total, this system of AV effects over the scalp was suggestive of both activity unique to multisensory processing, and the modulation of 'unisensory' activity. RTs to the stimuli when presented simultaneously were significantly faster than when they were presented alone. This RT facilitation could not be accounted for by probability summation, as evidenced by violation of the 'race' model, providing compelling evidence that auditory-visual neural interactions give rise to this RT effect.

KW - Auditory

KW - Electrophysiology

KW - ERPs

KW - Multisensory

KW - Visual

UR - http://www.scopus.com/inward/record.url?scp=0036279543&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0036279543&partnerID=8YFLogxK

U2 - 10.1016/S0926-6410(02)00066-6

DO - 10.1016/S0926-6410(02)00066-6

M3 - Article

C2 - 12063135

AN - SCOPUS:0036279543

VL - 14

SP - 115

EP - 128

JO - Cognitive Brain Research

JF - Cognitive Brain Research

SN - 0926-6410

IS - 1

ER -