Abstract: Classification consistency and accuracy are respectively deemed as significant criterions for evaluating the reliability and validity of test property in cognitive diagnostic assessment (CDA). Some indices, such as tetrachoric correlation, pattern-level classification consistency and accuracy indices, and attribute- and pattern-level classification consistency and accuracy indices have been introduced by several Psychometrians. However, all the existing indices focus on the scenario with dichotomous attributes. A CDA with polytomous attributes has received more and more attention in recent years due to additional diagnostic information that polytomous attributes can provide. This article first points out that the posterior probability distributions of attribute profiles and marginal posterior probability distributions of individual attributes under polytomous attribute scenario are rather different from those under dichotomous attribute scenario. Due to these distinctions, new classification consistency and accuracy indices that can represent the reliability and validity for a CDA with polytomous attributes are needed. This article proposes the attribute- and pattern-level classification consistency and accuracy indices based on binary method and develops procedures for the computation of the new indices specifically designed for CDAs with polytomous attributes. These indices make it possible to calculate the reliability and validity of CDAs with polytomous attributes. Simulation studies are conducted to evaluate the performance of the new indices using RPa-DINA model. In the simulation, the authors investigate the effects of five different factors: the hightest level of polytomous attributes, item quality, test length, number of polytomous attributes, and cutoff scores. Simulation results indicate that: (1) different values of cutoff score affect the performance of the reliability and validity indices. The RMSEs are almost similar when Ck and Cp are bounded between 0.1 and 0.6. Especially, the RMSE values are lowest when Ck and Cp are around 0.5 in most experiment conditions. So, the cutoff score taking 0.5 is recommended in practice. (2) the RMSEs of attribute- and pattern-level classification consistency and accuracy increase with the hightest level of polytomous attributes and the number of polytomous attributes. (3) the RMSEs decrease with item quality and test length. Finally, the authors discuss the essence of difference about the posterior probability distributions and marginal posterior probability distributions between dichotomous and polytomous attribute scenarios. Some practical suggestions for test construction are given and future research directions are proposed.

Key words: cognitive diagnostic assessment, polytomous attribute, classification consistency, classification accuracy

摘要： 分类一致性和准确性是认知诊断评估中的重要指标，前者反映信度问题，后者反映效度问题。已有研究提出的指标均是基于二分属性，而多分属性的后验概率分布和属性边际概率分布均不同于二分属性，需要构建新指标来衡量多分属性情景下的信效度。本研究基于二分思想，构建出二元式信息指标用于计算多分属性测验中的信效度，并通过实验设计考察了新指标在多种影响因素中的表现，验证了新指标的有效性。最后，为多分属性诊断测验的编制提供了建议，并提出未来研究方向。

关键词: 认知诊断评估, 多分属性, 分类一致性, 分类准确性