Abstract: Auditory information can transfer emotional message through rhythm and semantic meaning. In some cases, the emotional implication from the rhythm and the semantic might even be inconsistent. If such a complicated auditory information simultaneously presented with visual information like faces, what kind of interaction among them would happen when people process them? To investigate this question is the aim of our present study. The sound materials were produced by two professional speakers, a male and a female. In Experiment 1, speakers read neutral word in a happy or angry rhythm respectively, and the participants’ task was to judge whether the emotion of the rhythm of speech was consistent with the facial expression which was presented at the same time or not. In Experiment 2, speakers read positive word in an angry rhythm or read negative word in a happy rhythm respectively, and the participants’ task was to judge either the emotion of the rhythm of speech or the semantics of speech was consistent with the facial expression which was presented at the same time. There were 43 participants (21 females) taking part in Experiment 1 and 40 participants (20 females) taking part in Experiment 2. Each experiment included 120 trails. In Experiment 1, half were consistent and the other were inconsistent. In Experiment 2, half were consistent with the semantic information and the other were consistent with the rhythm information. The key-response mapping was counterbalanced across participants. Repeated measures ANOVA with facial expression and rhythm-emotion consistency as the within-subjects factors in Experiment 1 and with facial expression and judgment cue as the within-subjects factors in Experiment 2 were performed on the participant’s mean reaction time and accuracy. The results revealed that (1) when the facial expression was positive, participants were more accurate to judge the relationship of information between the visual and auditory channels[Experiment 1:F(1，42)=15.414，p<0.001，partial?2=0.268;Experiment 2:F(1，39)=6.824，p<0.05，partial?2=0.149]. (2) In Experiment 1,when the valence of the rhythm was consistent with the facial expression, the judgment of the relationship of information between the visual and auditory channels was faster [F(1，42)=37.631，p<0.001，partial?2=0.473] and more accurate[F(1，42)=21.800，p<0.001，partial?2=0.342]. (3) In Experiment 2, when the facial expression was negative, compared with the rhythm clues of words, the semantic clues could facilitate participants’ response in judging the relationship of visual and auditory stimuli [F(1，39)=15.78，p<0.001, partial?2=0.405]. The results suggested that when the visual and auditory stimuli are presented at the same time, the visual information may be processed in advance and then affected the auditory information processing. No matter the emotional valence of visual and auditory stimulus was conflicting or not, positive facial expression promoted the cognitive judgment about the relationship between the visual and auditory information. When the emotional valence of visual and auditory stimulus was congruent, it brought out an Easy processing phenomenon. When the emotional valence of visual and auditory stimulus was conflicting, negative facial expression and the semantic information in the auditory channel could promote each other’s processing. Present study innovatively explored the separated role of the semantic emotional information and the rhythm emotional information of a word on the judgment about the relationship between the visual and auditory. It initially revealed that the semantic information had speed advantage while the rhythm information had accuracy advantage when the visual stimulus was negative facial expression.

Key words: cross-modal, facial expression, emotional voice, rhythm-semantic conflicts

摘要： 通过要求被试判断同时呈现的视听信息情绪效价的关系，考察视听情绪信息整合加工特点。实验一中词汇效价与韵律效价不冲突，实验二中词汇效价与韵律效价冲突。两个实验一致发现当面孔表情为积极时，被试对视听通道情绪信息关系判断更准确；实验二还发现，当面孔表情为消极时，相对于韵律线索，被试根据语义线索对视听信息关系判断更迅速。上述结果说明视听信息在同时呈现时，视觉信息可能先行加工，并影响到随后有关视听关系的加工。

关键词: 跨通道, 面孔表情, 声音情绪, 韵律语义冲突