关键词: 视听跨通道, 整合, 冲突控制, 注意, 神经机制

Abstract: Multisensory interaction refers to a set of processing in which signals from one sensory modality interact with signals from another sensory modality. Previous researches have focused on two different aspects of the audiovisual interaction: one is how the information from different modalities bind together to produce a multisensory event (audiovisual integration); another is how the brain monitors and resolves conflicts between different modalities. This review article focused on the interaction between auditory and visual modalities, with three parts: the psychological and neural mechanisms of audiovisual integration, and the influence of attention on audiovisual integration; the psychological and neural mechanisms of audiovisual conflict processing, and the influences of attention on audiovisual conflict processing; potential research questions for future studies in this research area. For the psychological mechanisms, previous studies found that participants responded faster to the audiovisual congruent stimuli than to the unisensory (visual or auditory) stimuli, demonstrating that multisensory integration facilitates processing efficiency. Similarly, the incongruent audiovisual stimuli delay the reaction time as compared with the congruent audiovisual stimuli, suggesting a cross-modal conflict effect. Moreover, there are differences between groups of neurological and psychiatric populations (such as autism spectrum disorder, schizophrenia) and control groups in the audiovisual integration and conflict processing, which may help explain the impairment of the cognitive or social functions. For the neural mechanisms, the activation of STC (superior temporal cortex) is stronger for the multisensory stimuli than for the unimodal stimuli, and the suppression of the STC impairs audiovisual integration, suggesting that STC plays a critical role in multisensory integration. Frontal regions show stronger activities in the audiovisual incongruent condition as compared with the congruent condition, indicating audiovisual conflict processing. For the neural dynamic mechanism, previous studies suggested that the audiovisual integration occurs at an early stage (e.g., N1/P2, 100-200 ms) of the information processing, while the audiovisual conflict processing related with the incongruent negativity (Ninc, 200-400 ms). Attention modulates audiovisual integration and conflict control. On the one hand, the salience of the stimulus from one sensory modality will induce a strong neural activity, which will combine with the activity from the other sensory modality and facilitate multisensory integration. On the other hand, the attention also influences multisensory integration in a top-down way, which can be either facilitation or distraction effect. For the audiovisual conflict processing, one of the significant factors is the perceptual load of visual or auditory processing. Previous studies have suggested that the perceptual load of the visual task affects the information processing in the auditory modality. Although lots of studies have investigated the cognitive and neural mechanisms of the audiovisual information processing, there are still questions of theoretic importance that need further investigation. First, the underlying cognitive mechanisms of audiovisual interaction need further investigation. Second, the activities of a single brain area would not be enough to uncover the neural mechanisms underlying the audiovisual interaction. Future studies need to examine the brain connectivity during audiovisual interaction. Third, further studies are needed to investigate the multisensory interaction in the neurological and psychiatric populations.

Key words: audiovisual, integration, conflict control, attention, neural mechanism