Abstract: As the forms of music performance and music aesthetics become more and more diverse, the audio-visual integration in musical aesthetics process needs more attention from musical performers and researchers. However, whether there is integration superiority in Multisensory Integration (MI) of musical emotion recognition and musical emotion experience remains unclear. The distinct characteristics of the music materials and different expertise levels of the participants in previous studies may account for the inconsistent results. The aim of the current study was to expand the related research by investigating the influence of musical emotion type and music training experience on the mechanism of audio-visual integration. In this way, this study could help music performers choose the best way to express their emotions. In the present study, participants were divided into two groups, one was musician who had been musically trained for many years, and the other was non-musician who had no musical expertise. In order to control the potential interference of material characteristics, this study compiled a database of music excerpts with various musical features including opera, symphony, folk music and other forms of music from both eastern and western cultures. Each clip lasted 30 seconds to ensure the musical integrity. The performer's facial expression and body posture was complete and clear. 95 college students were invited to identify the type and intensity of musical emotions. Based on the ratings,42 clips were chosen for the formal experiment. There are three sensory modality conditions: auditory only (AO), visual only (VO) and audio-visual (AV). The order of the exposure was identical across participants with single sensory mode first and multisensory mode second. After the presentation of music clips, subjects were instructed to identify the type and the intensity of emotion recognition, as well as the type and intensity of emotion experience. The reaction time (RT), accuracy (ACC), and intensity of every assessment were analyzed using 3 (sensory modality: AO vs. VO vs. AV) ×3 (emotion type: happy vs. sad) ×2(group: musician vs. non-musician) repeated measures ANOVAs. Results: 1) The repeated measure ANOVA showed a significant main effect of the sensory modality (p < .001). Post hoc comparisons demonstrated that there was a significant difference between AV and VO (p < 0.001), but there was no significant difference between AV and AO (p > 0.05). 2) There was a significant interaction between emotion type and group on ACC (p<.05). Simple effects tests indicated that non-musician participants had higher accuracy in identifying sadness than musician participants, while musician participants showed higher accuracy in identifying happy assessment than non-musician participants. Conclusion: 1) We found integration superiority during musical emotion recognition and experience in AV modality over VO modality, but not AO modality. 2) Although emotion type and music training experience does not directly affect the existence of audio-visual integration superiority integration, different sensory modality was characterized with different processing strategy and trait. Musicians were less affected by visual emotional cues than non-musicians, but emotion recognition and experience of sad music were disturbed by musical expertise, which made it difficult to assess. 3）In order to help the listener who has no music expertise to have better music aesthetic experience, performers could add congruence visual emotion cues to auditory music performance.

Key words: musical emotion, audio-visual, emotion type, musical expertise, integration superiority

摘要： 为了探讨视听双通道下的音乐情绪加工机制及音乐情绪类型和音乐训练背景对加工机制的影响，本研究采用表达开心和悲伤的音乐表演视频为材料，比较音乐组被试和非音乐组被试在单听觉通道、单视觉通道和视听双通道三种情境下的情绪评定速度、正确率和强度。结果发现：1）视听双通道与单视觉通道差异显著，与单听觉通道差异不显著。2）非音乐组被试对悲伤的评定正确率高于音乐组被试，对开心的评定正确率低于音乐组被试。说明音乐情绪加工的视听双通道整合优势仅相对单视觉通道存在；非音乐组被试对视觉通道情绪信息的变化更敏感，音乐组被试更依赖音乐经验；可在音乐表演时加入协调的视觉通道情绪信息帮助没有音乐训练经验的听赏者。

关键词: 音乐情绪, 视听双通道, 情绪类型, 音乐经验, 整合优势