Subtitle Preference While Viewing Native Language Movie: The Effect of Sound

Abstract

Abstract:

Movies and television programs are generally dubbed or subtitled in the local language recently. Subtitle preference or obligatory was found in studies of text and graphics integration and television viewing. From previous findings, researchers found that the fixation of subtitle was not affected while the soundtrack was turned off. Researchers concluded that subtitle and sound was overlapped and sound was redundant while they prefer to watch subtitles. The purpose of this study planed to investigate the sound effect on subtitle preference while viewing native language movie. Chinese and English subtitle movie clips were chose as materials, half with sound and the other was turned off. Students were recruited from a normal university as participants. Eye tracking technique was used as data collecting. After the eye movements data collecting, an inventory of the movie clip content was tested. Total eye movement, areas of interest of subtitle and video, and subtitle pieces were analyzed to find the effects of sound. Results showed that participants fixated more on subtitle area than video when the sound was turned off. Fixation number, fixation durations on subtitle of movie without sound were significantly higher than movie with sound. More generally, the purpose of subtitle preference was mainly for reading and understanding, the sound of video can inactive and help subtitle for movie content processing.

Key words: subtitle preference, eye movements, movie viewing, sound

摘要：

字幕作为电影和电视节目的重要组成部分，在视频观看过程中起到辅助加工的作用。先前的研究认为字幕是视频语言的直接转译，字幕与声音信息具有很高的重叠性，所以关闭视频的声音不影响被试对字幕的加工。研究以电影视频作为刺激材料，采用眼动技术探讨了声音对电影观看过程中字幕偏好性的影响作用。实验通过控制视频的声音信息，探讨声音对字幕偏好性加工的影响作用。结果发现：视频的声音信息会对字幕的加工产生影响，关闭声音后被试对字幕的加工时间和次数会增加，无声条件下对字幕的注视次数和注视时间占总注视的比例超过了50%，说明声音和字幕对视频的理解具有相互辅助的作用。

关键词: 字幕偏好性, 眼动, 电影观看, 声音

王福兴白学军闫国利周宗奎. 电影观看过程中的字幕偏好性：声音的影响作用[J]. 心理科学, 2012, 35(1): 44-49.

References

Boatner, E. B. (1980). Captioned Films for the Deaf. Retrieved 02-21, 2010, from http://www.cfv.org/caai/nadh84.pdf Carroll, P. J., Young, J. R., & Guertin, M. S. (1992). Visual analysis of cartoons: A view from the far side. In K. Rayner (Ed.), Eye movements and visual cognition: Scene perception and reading (pp. 444-461). New York: Springer-Verlag. Chapdelaine, C., Gouaillier, V., Beaulieu, M., & Gagnon, L. (2007). Improving video captioning for deaf and hearing-impaired people based on eye movement and attention overload. Paper presented at the Conference on Human Vision and Electronic Imaging XII, San Jose,CA. d'Ydewalle, G., & Gielen, I. (1992). Attention allocation with overlapping sound, image, and text. In K. Rayner (Ed.), Eye movements and visual cognition: Scene perception and reading (pp. 415-427). New York: Springer-Verlag. d'Ydewalle, G., Praet, C., Verfaillie, K., & Rensbergen, J. V. (1991). Watching subtitled television: Automatic reading behavior. Communication Research, 18(5), 650-666. d’Ydewalle, G., & De Bruycker, W. (2007). Eye movements of children and adults while reading television subtitles. European Psychologist, 12(3), 196-205. d’Ydewalle, G., Van Rensbergen, J., & Pollet, J. (1987). Reading a message when the same message is available auditorily in another language: The case of subtitling. In O. R. J. K. & A. Levy-Schoen (Eds.), Eye movements: From physiology to cognition (pp. 313-321). Amsterdam: Norht-Holland. De Bruycker, W., & d'Ydewalle, G. (2003). Reading native and foreign language television subtitles in children and adults. In J. Hy?n?, R. Radach & H. Deubel (Eds.), The mind's eye: Cognitive and applied aspects of eye movement research (pp. 671-684): North-Holland. Hannus, M., & Hyona, J. (1999). Utilization of illustrations during learning of science textbook passages among low-and high-ability children. Contemporary Educational Psychology, 24(2), 95-123. Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32, 717-742. Jensema, C. J., Danturthi, R. S., & Burch, R. (2000). Time spent viewing captions on television programs. American Annals of the Deaf, 145(5), 464-468. Linebarger, D. L. (2001). Learning to read from television: The effects of using captions and narration. Journal of Educational Psychology, 93(2), 288-298. Norwood, M. J. (1988). Captioning for deaf people: An historical overview. Paper presented at the Speech to Text: Today and Tomorrow. Proceedings of a Conference at Gallaudet University, Washington, D.C. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372-422. Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62(8), 1457-1506. Rayner, K., Rotello, C. M., Stewart, A. J., Keir, J., & Duffy, S. A. (2001). Integrating text and pictorial information: Eye movements when looking at print advertisements. Journal of Experimental Psychology: Applied, 7(3), 219-226. Takeda, M. (1999). Eye movements while viewing a foreign movie with subtitles. In W. Becker, H. Deubel & T. Mergner (Eds.), Current Oculomotor Research: Physiological and Psychological Aspects (pp. 313-316). New York: Kluwer Academic. Vatikiotis-Bateson, E., Eigsti, I. M., Yano, S., & Munhall, K. G. (1998). Eye movement of perceivers during audiovisual speech perception. Perception and Psychophysics, 60(6), 926-940. Wang, F., Zhou, Z., Zhao, X., Bai, X., & Yan, G. (Under Review). Subtitle preference while viewing native language movie: The effect of subtitle familiarity. 俞自萍, 曹愈, & 曹凯. (1996). 色盲检查图(第五版). 北京: 人民卫生出版社.

[1]	Yi Jiaqi, Liu Zhifang, Tong Wen, Chen Chaoyang. Aging Enhances the Effect of Word Segmentation Cues in Chinese Reading: Evidences from an Eye Movement Study [J]. Journal of Psychological Science, 2023, 46(4): 779-786.
[2]	. Automatic Processing Advantages of High Sensation Seekers for Risk Sounds: Evidence from N1、P2 and MMN [J]. Journal of Psychological Science, 2022, 45(6): 1344-1351.
[3]	. Using Citespace-based Bibliometric Analysis to Examine the Hotspots and Trends from 20 Years of Research on Eye Movements in Reading [J]. Journal of Psychological Science, 2022, 45(4): 811-819.
[4]	. Attention bias to drug related cues of heroin abstainer: Evidence from eye movements [J]. Journal of Psychological Science, 2022, 45(2): 481-490.
[5]	. The Influence of Saccade Launch Site on Preview Processing During Reading Chinese [J]. Journal of Psychological Science, 2021, 44(5): 1081-1088.
[6]	Jia-Wei ZHAO. Lexicalization of Chinese New Words: Evidence from Eye Movements [J]. Journal of Psychological Science, 2021, 44(4): 779-785.
[7]	. Eye Movement Evidence for the Licensing Effect of Predictive Inference in Moral Text Reading [J]. Journal of Psychological Science, 2021, 44(4): 947-953.
[8]	. The effects of the word predictability on eye movements of skilled and less-skilled developing readers in Chinese Sentnces reading [J]. , 2020, 43(6): 1369-1375.
[9]	. Effects of Inter-character Spacing on the Reading of Chinese Young and Older Adults: An Eye Movement Study [J]. , 2020, 43(1): 68-74.
[10]	. Inserting Spaces Benefits Chinese Developmental Dyslexic Children for Reducing their Visual Crowding: Evidence from Eye Movements [J]. , 2019, 42(4): 834-840.
[11]	. The differences of the contextual predictability effect between highly-skilled and low-skilled developing readers during parafoveal processing: Evidences from eye movements [J]. , 2019, 42(4): 848-853.
[12]	. Parafoveal Processing of Character N+1 of Deaf Readers [J]. , 2019, 42(4): 997-1003.
[13]	Jian GUAN Wen-Rui Li. The Competition between Inhibition of Return and Emotional Attention Bias: Evidence from Eye Movements [J]. Psychological Science, 2018, 41(6): 1353-1358.
[14]	. The perceptual span of second graders in Chinese primary school [J]. Psychological Science, 2018, 41(4): 849-855.
[15]	. Hazard Perception in Young Drivers: the Role of Hazard Types [J]. Psychological Science, 2018, 41(2): 447-452.