Abstract: New HSK is the official Chinese proficiency test for speakers of other languages, which is administered by China’s Hanban (i.e. National Office for Teaching Chinese as a Foreign Language). It was first launched in 2010, and held five times annually. Being a rather new test, research and development focusing on an improvement in the test’s quality is essential. Writing test scores of the New HSK, similar to those of any other language proficiency tests, are most vulnerable to reliability criticism. The study collected writing samples (n=52) from two sets of mock tests of the writing part of the New HSK, level 5. Data analysis was performed from the perspective of Generalizability Theory using Genova and mGenova software. Variance components were estimated for possible influencing effects of item, raters, rating speed, and their interactions. Dependability coefficient (Phi) was estimated for current test settings. Moreover, D studies with various rating patterns were conducted to explore approaches promoting the Phi coefficient. Major findings are: (a) According to current allocation of quantity for each item type, the descending order of the Phi coefficient for each item type is: writing based on keywords given, writing based on the given photo, ordering of the inner-sentence components; (b) For the Phi coefficient to be at least 0.8 for each item type, ordering items needs to increase from 8 items to 20 while the other two need to increase one item each; (c) If item quantities are increased as specified in b, then, the ordering items needs only one rater, writing based on keywords needs two, and writing based on photo needs three. However, if writing based on photo has three items, two raters will be enough to serve the need of keeping Phi coefficient as least 0.8; (d) With current allocation of item quantities for each item type, if calculation of the comprehensive score of writing uses weight proportional to the raw scores, then the Phi coefficient for the writing test could marginally reach the level of 0.8. But the study explored various approaches reaching a Phi coefficient at least 0.85 with relatively lower costs (for details, please refer to section 3.2.2 of this paper). For such estimations, the analysis applied solver functions of Microsoft Excel. (e) The study did not find effect of rating speed significant. However, this conclusion was limited to the two different speeds investigated by this study: a speed that each individual rater feels comfortable, and a speed that raters feel a little pushy but still have confidence about the rating quality under that speed. Furthermore, more evidence is needed to support such a conclusion, especially real data from professional raters. The study suggested to investigate effect of rating speeds on generalizability of writing test scores with more rigorous designs and more mature data collection techniques, and called for more attention to reliability issues of writing tests.

Key words: new HSK, writing assessment, reliability, Generalizability Theory

摘要： 本研究以概化理论为视角，搜集了新HSK五级模拟书写题的作答和评分数据，估算了题型、题量、评卷员人数、评阅速度等潜在影响效应的方差分量，考察了新HSK书写成绩的可靠性，并探索了改善该分数可靠性的途径。基于概化理论和规划求解的数据分析发现了题量的调整方案以及题型、题量、评卷员人数的最优组合方案。本研究对评阅速度进行的分析属于前沿性的理论探索，而其他数据分析结果，则可能有益于旨在改进该测试质量的决策实践。

关键词: 新HSK，写作测评，信度，概化理论