关键词: 音乐情绪, 视听双通道, 情绪类型, 音乐经验, 整合优势

Abstract: As the forms of music performance and music aesthetics become more and more diverse, the audio-visual integration in musical aesthetics process needs more attention from musical performers and researchers. However, whether there is integration superiority in Multisensory Integration (MI) of musical emotion recognition and musical emotion experience remains unclear. The distinct characteristics of the music materials and different expertise levels of the participants in previous studies may account for the inconsistent results. The aim of the current study was to expand the related research by investigating the influence of musical emotion type and music training experience on the mechanism of audio-visual integration. In this way, this study could help music performers choose the best way to express their emotions. In the present study, participants were divided into two groups, one was musician who had been musically trained for many years, and the other was non-musician who had no musical expertise. In order to control the potential interference of material characteristics, this study compiled a database of music excerpts with various musical features including opera, symphony, folk music and other forms of music from both eastern and western cultures. Each clip lasted 30 seconds to ensure the musical integrity. The performer's facial expression and body posture was complete and clear. 95 college students were invited to identify the type and intensity of musical emotions. Based on the ratings,42 clips were chosen for the formal experiment. There are three sensory modality conditions: auditory only (AO), visual only (VO) and audio-visual (AV). The order of the exposure was identical across participants with single sensory mode first and multisensory mode second. After the presentation of music clips, subjects were instructed to identify the type and the intensity of emotion recognition, as well as the type and intensity of emotion experience. The reaction time (RT), accuracy (ACC), and intensity of every assessment were analyzed using 3 (sensory modality: AO vs. VO vs. AV) ×3 (emotion type: happy vs. sad) ×2(group: musician vs. non-musician) repeated measures ANOVAs. Results: 1) The repeated measure ANOVA showed a significant main effect of the sensory modality (p < .001). Post hoc comparisons demonstrated that there was a significant difference between AV and VO (p < 0.001), but there was no significant difference between AV and AO (p > 0.05). 2) There was a significant interaction between emotion type and group on ACC (p<.05). Simple effects tests indicated that non-musician participants had higher accuracy in identifying sadness than musician participants, while musician participants showed higher accuracy in identifying happy assessment than non-musician participants. Conclusion: 1) We found integration superiority during musical emotion recognition and experience in AV modality over VO modality, but not AO modality. 2) Although emotion type and music training experience does not directly affect the existence of audio-visual integration superiority integration, different sensory modality was characterized with different processing strategy and trait. Musicians were less affected by visual emotional cues than non-musicians, but emotion recognition and experience of sad music were disturbed by musical expertise, which made it difficult to assess. 3）In order to help the listener who has no music expertise to have better music aesthetic experience, performers could add congruence visual emotion cues to auditory music performance.

Key words: musical emotion, audio-visual, emotion type, musical expertise, integration superiority