A Comparative Study of Item Selection Methods in CD-CAT based on Nominal Response Model

doi:10.16719/j.cnki.1671-6981.20230628

Abstract

Abstract:

Cognitive Diagnostic Computerized Adaptive Testing (CD-CAT) combines the advantages of cognitive diagnosis and CAT, which could improve the efficiency and accuracy of CD-CAT. CD-CAT can be divided into two types, including dichotomous and polytomous. Presently, the majority of researches on CD-CAT are based on dichotomous CD-CAT. However, among the practical tests in psychology and education, there are many polytomous items, which can be further divided into nominal polytomous and ordinal polytomous items according to whether there is an order or grade between every response category. Nominal polytomous items are items whose response categories are independent and without orders or grades between every response category. Although researchers have developed (ordinal) polytomous CDMs and corresponding CD-CAT, few nominal CDMs and CD-CAT are based on nominal responses.

This study introduces seven commonly used item selection methods in dichotomous CD-CAT into NCD-CAT (CD-CAT based on nominal response models). PMR (pattern match ratio) and test efficiency index are evaluated under different conditions between these item selection methods. Here are details of two simulation studies below.

Study 1 compared the performance of NR_PWKL, NR_PWCDI, NR_PWACDI, NR_MPWKL, NR_SHE, NR_MI, and NR_GDI methods under different test lengths (5, 10, 15, 20) and item pool qualities (high and low) in NCD-CAT. Results showed that: (1) the PMRs of NR_PWCDI, NR_PWACDI, NR_MPWKL, NR_SHE, and NR_MI are higher than or equal to that of NR_PWKL, especially in short tests. (2) As test length gets longer, that PMR advantage is missing, which is the same as the results of Zheng and Chang (2016). (3) Compared to test length, item quality has a greater impact on PMR. For instance, with item quality descending, the PMR declined about 30% among all conditions.

Study 2 was an experiment study on variable-length NCD-CAT that was conducted to compare the performance of each item selection method under the conditions of three maximum posterior probabilities ( .8, .85, .9) and two item qualities (high and low). The results showed that: (1)under all experimental conditions the average test lengths of NR_PWCDI, NR_PWACDI, NR_MPWKL, NR_SHE, and NR_MI are shorter than that of NR_PWKL; the difference is more than .738; (2)affected by item quality, the average length of NR_GDI is smaller than that of NR_PWKL under high-quality conditions and larger than it under low-quality conditions.

To sum up, this study compared the performance of 7 commonly used item selection methods of dichotomous CD-CAT in NCD-CAT with different conditions (fixed and variable length). The simulation study showed that, under most conditions, the NR_PWCDI, NR_PWACDI, NR_MPWKL, NR_SHE, and NR_MI methods performed well when compared to baseline algorithm NR_PWKL. This study has expanded the alternatives of item selection methods in NCD-CAT.

Key words: CD-CAT, nominal responses, NR-cRUM, item selection methods, multiple-choice items

摘要： 当前大多数CD-CAT有关的研究都是基于0-1计分的数据资料展开的,而在实际的教育与心理测验中,还包含大量称名反应数据。本文基于称名反应认知诊断模型（NR-cRUM）开发了适用于称名反应数据的CD-CAT（以下简称NCD-CAT）,并将7种0-1计分CD-CAT的选题方法引入NCD-CAT中。比较不同条件下,不同选题方法对被试判准率和测验效率的影响。结果表明NR_PWCDI、NR_MPWKL等PWKL系新方法和NR_SHE/MI方法能较好地适用于NCD-CAT,且在大多数条件下优于基线方法NR_PWKL。研究拓展了称名多级计分CD-CAT的选题方法。

关键词: CD-CAT, 称名反应, NR-cRUM, 选题方法, 多项选择题

Zhang Jie, Luo Zhaosheng, Yu Xiaofeng, Qin Chunying. A Comparative Study of Item Selection Methods in CD-CAT based on Nominal Response Model[J]. Journal of Psychological Science, 2023, 46(6): 1495-1503.

张杰, 罗照盛, 喻晓锋, 秦春影. 基于称名反应模型的CD-CAT选题方法比较研究^*[J]. 心理科学, 2023, 46(6): 1495-1503.

References

[1] 陈平, 李珍, 辛涛. (2011). 认知诊断计算机化自适应测验的题库使用均匀性初探. 心理与行为研究, 9(2), 125-132, 153.
[2] 高椿雷, 罗照盛, 郑蝉金, 喻晓锋, 彭亚风, 郭小军. (2017). CD-CAT初始阶段项目选取方法. 心理科学, 40(2), 485-491.
[3] 郭磊, 郑蝉金, 边玉芳, 宋乃庆, 夏凌翔. (2016). 认知诊断计算机化自适应测验中新的选题策略: 结合项目区分度指标. 心理学报, 48(7), 903-914.
[4] 郭磊, 周文杰. (2021). 基于选项层面的认知诊断非参数方法. 心理学报, 53(9), 1032-1043.
[5] 李佳, 丁树良, 况天昊. (2021). 区分度与测验进程相匹配的CAT选题策略. 江西师范大学学报(自然科学版), 45(4), 384-389.
[6] 李瑜. (2014). 多选题认知诊断测验编制及多策略的多选题认知诊断模型的开发 (博士学位论文). 江西师范大学,南昌.
[7] 刘拓, 张佳慧, 辛涛. (2015). 多项选择题中干扰项信息的利用. 心理学探新, 35(3), 261-265.
[8] 罗照盛, 杭丹丹, 秦春影, 喻晓锋. (2020). 可以处理补偿作用的认知诊断模型: CDINA模型. 江西师范大学学报(自然科学版), 44(5), 441-453.
[9] 罗照盛, 喻晓锋, 高椿雷, 李喻骏, 彭亚风, 王睿, 王钰彤. (2015). 基于属性掌握概率的认知诊断计算机化自适应测验选题策略. 心理学报, 47(5), 679-688.
[10] 涂冬波, 蔡艳, 戴海琦. (2013). 认知诊断CAT选题策略及初始题选取方法. 心理科学, 36(2), 469-474.
[11] 涂冬波, 郑蝉金, 戴步云, 汪文义. (2017). 计算机化自适应测验: 理论与方法. 北京师范大学出版社.
[12] 王晓庆, 罗芬, 丁树良, 熊建华. (2016). 多级评分计算机化自适应测验动态调和平均选题策略. 心理学探新, 36(3), 270-275.
[13] 夏梦连, 毛秀珍, 杨睿. (2018). 属性多级和项目多级评分的认知诊断模型. 江西师范大学学报(自然科学版), 42(2), 134-138.
[14] Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29-51.
[15] Cheng, Y. (2009). When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika, 74(4), 619-632.
[16] de la Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163-183.
[17] Gao X. L., Ma W. C., Wang D. X., Cai Y., & Tu D. B. (2021). A class of cognitive diagnosis models for polytomous data. Journal of Educational and Behavioral Statistics, 46(3), 297-322.
[18] Gao X. L., Wang D. X., Cai Y., & Tu D. B. (2020). Cognitive diagnostic computerized adaptive testing for polytomously scored items. Journal of Classification, 37(3), 709-729.
[19] Guo, L., & Zheng, C. J. (2019). Termination rules for variable-length CD-CAT from the information theory perspective. Frontiers in Psychology, 10, Article 1122.
[20] Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29(4), 262-277.
[21] Henson R., Roussos L., Douglas J., & He X. M. (2008). Cognitive diagnostic attribute-level discrimination indices. Applied Psychological Measurement, 32(4), 275-288.
[22] Huebner, A., & Wang, C. (2011). A note on comparing examinee classification methods for cognitive diagnosis models. Educational and Psychological Measurement, 71(2), 407-419.
[23] Kaplan M., de la Torre J., & Barrada J. R. (2015). New item selection methods for cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 39(3), 167-188.
[24] Leighton J. P.,& Gierl, M. J. (2007). Cognitive diagnostic assessment for education: Theory and applications Cambridge University Press Theory and applications. Cambridge University Press.
[25] Ma, W. C. (2021). A higher-order cognitive diagnosis model with ordinal attributes for dichotomous response data. Multivariate Behavioral Research. Advance online publication.
[26] Ma, W. C, & de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275.
[27] Mellenbergh, G. J. (1995). Conceptual notes on models for discrete polytomous item responses. Applied Psychological Measurement, 19(1), 91-100.
[28] Templin J., Henson R., Rupp A., Jang E., & Ahmed M.(2008). Cognitive diagnosis models for nominal response data. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.
[29] von Davier, M., & Lee, Y.-S. (2019). Handbook of diagnostic classification models. Springer.
[30] Wang, C. (2013). Mutual information item selection method in cognitive diagnostic computerized adaptive testing with short test length. Educational and Psychological Measurement, 73(6), 1017-1035.
[31] Wang S. Y., Fellouris G., & Chang H.-H. (2017). Computerized adaptive testing that allows for response revision: Design and asymptotic theory. Statistica Sinica, 27(4), 1987-2010.
[32] Wang S. Y., Fellouris G., & Chang H. H. (2019). Statistical foundations for computerized adaptive testing with response revision. Psychometrika, 84(2), 375-394.
[33] Yigit H. D., Sorrel M. A., & de la Torre, J. (2019). Computerized adaptive testing for cognitively based multiple-choice data. Applied Psychological Measurement, 43(5), 388-401.
[34] Yu X. F., Cheng Y., & Chang H.-H. (2019). Recent developments in cognitive diagnostic computerized adaptive testing (CD-CAT): A comprehensive review. In M. von Davier, & Y. S. Lee (Eds.), Handbook of diagnostic classification models (pp. 307-331). Springer.
[35] Zheng, C. J., & Chang, H. H. (2016). High-efficiency response distribution-based item selection algorithms for short-length cognitive diagnostic computerized adaptive testing. Applied Psychological Measurement, 40(8), 608-624.
[36] Zheng, C. J., & Wang, C. (2017). Application of binary searching for item exposure control in cognitive diagnostic computerized adaptive testing. Applied Psychological Measurement, 41(7), 561-576.