融入能力信息的认知诊断模型开发与应用*

doi:10.16719/j.cnki.1671-6981.20240423

摘要/Abstract

摘要： 新时代教育评价改革对评价的科学性、客观性和专业性提出了更高的要求。以一般（高阶）能力作为连接项目反应理论模型与认知诊断模型的桥梁,相互利用不同试题上的作答信息,开发出融入能力信息的认知诊断模型。模拟研究考查了新模型在四种参数分布和五种题量下的表现,并与DINA模型与2PLM进行了比较。结果显示：在相同题量下,新模型能力参数的均方误差优于2PLM的均方误差;新模型的模式判准率优于DINA模型的模式判准率;属性斜率越大,新模型的属性判准率和能力精度更高。在英语水平证书考试的实测数据上,新模型相对拟合指标优于DINA和HO-DINA等模型;新模型虽稍逊于2PLM,但两者所得能力与测验总分相关较高;新模型在属性2上分类准确性高于其他模型,分析发现新模型可利用能力信息提高Q阵中考查次数较少属性的分类准确性。

关键词: 能力, 知识状态, 2PL模型, DINA模型, MCMC算法

Abstract: The item response theory (IRT) is an important model to estimate the ability parameters, and the cognitive diagnosis model (CDM) is a vital model to diagnose the cognitive structure of the examinees. Because test item responses contain the information about the ability parameters and the knowledge states and their relationship, how to fully utilize the information to further enhance the accuracy of the knowledge states, becomes very important for helping to decrease the number of tests and test lengths.
Considering the strong relationship between higher-order DINA (HO-DINA) model and two-parameter logistic model (2PLM), this new statistical measurement model was proposed by regarding the general ability as higher-order ability and estimated by Markov Chain Monte Carlo. In the estimation of ability, the knowledge states are considered as response patterns in establishing the relationship between the ability and knowledge states. Five attributes were considered in the study. Test consists of two parts of items, one is fitted by 2PLM and the other is fitted by the DINA model. The simulation study examined the performance of the new model with four model parameter distributions and five different test lengths, and compared it with the DINA model and 2PLM respectively. In the analysis of Examination for the Certificate of Proficiency in English (ECPE) test data, the absolute fit indices, -2LL, AIC, BIC and DIC were provided as an example to evaluate the model-data fit.
The results show that: (a) The new proposed model in this study can obtain the ability parameters and cognitive structure parameters of the examinees in one test, and it has a good recovery for the estimation of attribute patterns, item parameters, and ability parameters. In particular, the accuracy of ability parameters has been greatly improved, and the MCMC algorithm is feasible; (b) The longer the CDM-based test length is, the higher the correct classification rate of knowledge state is, and the same is true for the ability estimation based on 2PLM; (c) Compared with lognormal distribution, the error of ability estimation is slightly smaller when the item discriminations followed from uniform distribution. When the slip parameters and the guessing parameters are small, the correct classification rate for attributes or attribute patterns are higher. In addition, the analysis of the ECPE data shows that the estimation accuracy of the new model is reasonable and has practical implications.
Because there is always a relationship between the levels of ability and the mastery of knowledge, the ability of estimation enriched in test items fitting by an item response theory model can provide information for the classification of knowledge state, and item only fitting the CDM can also indirectly provide the information to improve the precision of ability estimation. In addition, it does not need calibrate the attribute vector for all test items and not require all items fitted two models at the same time. The new model uses a general or high-order ability as bridge between item response theory and cognitive diagnostic models, utilizing information from different test items to improve the accuracy of abilities and knowledge, tates estimation.

Key words: ability, knowledge state, 2PLM, the DINA Model, MCMC

宋丽红, 胡海洋, 汪文义, 丁树良, 袁思玉. 融入能力信息的认知诊断模型开发与应用^*[J]. 心理科学, 2024, 47(4): 947-958.

Song Lihong, Hu Haiyang, Wang Wenyi, Ding Shuliang, Yuan Siyu. Development of a Cognitive Diagnostic Model with Ability Covariate and Its Applications[J]. Journal of Psychological Science, 2024, 47(4): 947-958.

参考文献

[1] 丁树良, 罗芬, 汪文义, 熊建华. (2019). 0-1评分认知诊断测验设计. 江西师范大学学报(自然科学版), 43(5), 441-447.
[2] 丁树良, 汪文义, 杨淑群. (2011). 认知诊断测验蓝图的设计. 心理科学, 34(2), 258-265.
[3] 罗芬, 王晓庆, 蔡艳, 涂冬波. (2020). 基于基尼指数的双目标CD-CAT选题策略. 心理学报, 52(12), 1452-1465.
[4] 涂冬波, 蔡艳, 戴海琦, 丁树良. (2011). HO-DINA模型的MCMC参数估计及模型性能研究. 心理科学, 34(6), 1476-1481.
[5] 汪大勋, 涂冬波. (2021). 认知诊断计算机化自适应测量技术在心理障碍诊断与评估中的应用. 江西师范大学学报(自然科学版), 45(2), 111-117.
[6] 王璞珏, 刘红云. (2019). 让自适应测验更知人善选——基于推荐系统的选题策略. 心理学报, 51(9), 1057-1067.
[7] 韦来生. (2016). 贝叶斯统计. 高等教育出版社.
[8] 詹沛达, 潘艳方, 李菲茗. (2021). 面向“为学习而测评”的纵向认知诊断模型. 心理科学, 44(1), 214-222.
[9] 中共中央, 国务院. (2020-10-14). 中共中央国务院印发深化新时代教育评价改革总体方案. 人民日报.
[10] Akbay, L., & Kaplan, M. (2017). Transition to multidimensional and cognitive diagnosis adaptive testing: An overview of CAT. The Online Journal of New Horizons in Education, 7(1), 206-214.
[11] Ames, A. J., & Penfield, R. D. (2015). An NCME instructional module on item-fit statistics for item response theory models. Educational Measurement: Issues and Practice, 34(3), 39-48.
[12] Baker F. B.& Kim, S. H. (2004). Item response theory: Parameter estimation techniques CRC Press Parameter estimation techniques. CRC Press.
[13] Bao Y., Shen Y. W., Wang S. Y., & Bradshaw L. (2021). Flexible computerized adaptive tests to detect misconceptions and estimate ability simultaneously. Applied Psychological Measurement, 45(1), 3-21.
[14] Bradshaw, L., & Templin, J. (2013). Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika, 79(3), 403-425.
[15] Chen, J. S., & de la Torre, J. (2018). Introducing the general polytomous diagnosis modeling framework. Frontiers in Psychology, 9, Article 1474.
[16] Dai B. Y., Zhang M. Q., & Li G. M. (2016). Exploration of item selection in dual-purpose cognitive diagnostic computerized adaptive testing: Based on the RRUM. Applied Psychological Measurement, 40(8), 625-640.
[17] de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333-353.
[18] Gelman A., Carlin J. B., Stern H. S., Dunson D. B., Vehtari A., & Rubin, D. B. (2013). Bayesian data analysis. CRC Press.
[19] Gitomer, D. H., & Yamamoto, K. (1991). Performance modeling that integrates latent trait and class theory. Journal of Educational Measurement, 28(2), 173-189.
[20] Givens, G. H., & Hoeting, J. A. (2013). Computational statistics. John Wiley & Sons, Inc.
[21] Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26(4), 301-321.
[22] Henson R. A., Templin J. L., & Willse J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191-210.
[23] Hong H., Wang C., Lim Y. S., & Douglas J. (2015). Efficient models for cognitive diagnosis with continuous and mixed-type latent variables. Applied Psychological Measurement, 39(1), 31-43.
[24] Hsu, C. L., & Wang, W. C. (2015). Variable-length computerized adaptive testing using the higher order DINA model. Journal of Educational Measurement, 52(2), 125-143.
[25] Huang, H. Y. (2017). Multilevel cognitive diagnosis models for assessing changes in latent attributes. Journal of Educational Measurement, 54(4), 440-480.
[26] Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258-272.
[27] Kang H. A., Zhang S. S., & Chang H. H. (2017). Dual-objective item selection criteria in cognitive diagnostic computerized adaptive testing. Journal of Educational Measurement, 54(2), 165-183.
[28] Lee, S. Y. (2017). Growth curve cognitive diagnosis models for longitudinal assessment(Unpublished doctoral dissertation). University of California, Berkeley.
[29] Ma, W. C., & de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275.
[30] Ma, W. C., & de la Torre, J. (2020). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1-26.
[31] McGlohen, M., & Chang, H. H. (2008). Combining computer adaptive testing technology with cognitively diagnostic assessment. Behavior Research Methods, 40(3), 808-821.
[32] Patz, R. J. & Junker, B. W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146-178.
[33] Rupp A. A., Templin J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications Guilford Press Theory, methods, and applications. Guilford Press.
[34] Sorrel M. A., Abad F. J., & Nájera P. (2021). Improving accuracy and usage by correctly selecting: The effects of model selection in cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 45(2), 112-129.
[35] Wang C., Chang H. H., & Douglas J. (2012). Combining CAT with cognitive diagnosis: A weighted item selection approach. Behavior Research Methods, 44(1), 95-109.
[36] Wang C., Zheng C. J., & Chang H. H. (2014). An enhanced approach to combine item response theory with cognitive diagnosis in adaptive testing. Journal of Educational Measurement, 51(4), 358-380.
[37] Wang S. Y., Yang Y., Culpepper S. A., & Douglas J. A. (2018). Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden Markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57-87.
[38] Wang W. Y., Song L. H., Chen P., Meng Y. R., & Ding S. L. (2015). Attribute-level and pattern-level classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52(4), 457-476.
[39] Yan D. L., Mislevy R. J., & Almond R. G. (2003). Design and analysis in a cognitive assessment. ETS Research Report Series, 2003(2), i-47.
[40] Zhan P. D., Jiao H., Liao D. D., & Li F. M. (2019). A longitudinal higher-order diagnostic classification model. Journal of Educational and Behavioral Statistics, 44(3), 251-281.
[41] Zhan P. D., Ma W. C., Jiao H., & Ding S. L. (2020). A sequential higher order latent structural model for hierarchical attributes in cognitive diagnostic assessments. Applied Psychological Measurement, 44(1), 65-83.
[42] Zhang, S. S., & Chang, H. H. (2016). From smart testing to smart learning: How testing technology can assist the new generation of education. International Journal of Smart Technology and Learning, 1(1), 67-92.
[43] Zheng C. J., He G. R., & Gao C. L. (2018). The information product methods: A unified approach to dual-purpose computerized adaptive testing. Applied Psychological Measurement, 42(4), 321-324.

融入能力信息的认知诊断模型开发与应用^*

Development of a Cognitive Diagnostic Model with Ability Covariate and Its Applications

PDF

评审附件

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	谢庆斌, 孙娟娟, 王英杰, 李燕. 幼儿行为自我调节能力的发展：母亲、同伴、教师的作用^*[J]. 心理科学, 2024, 47(5): 1080-1087.
[2]	黄量杰成, 闫艳, 胡寅凤, 孙鸿莉, 焦岚, 尹军. 低年段小学生的学业能力性别刻板印象及其对兴趣选择的影响：自我肯定的作用^*[J]. 心理科学, 2024, 47(4): 855-862.
[3]	段锦云, 吴珏彧, 徐婷婷, 梁凤华. 建议者的权力和地位对决策者建议采纳的影响^*[J]. 心理科学, 2024, 47(4): 910-917.
[4]	邹艳春, 陈启瑶, 彭坚, 曾晓晴. 问题导向领导人际情绪管理的收益与代价:基于员工与领导的差异化视角^*[J]. 心理科学, 2024, 47(3): 630-638.
[5]	史滋福, 陈枭豪. 嵌套集合建模能力和数字表征对儿童贝叶斯推理的影响^*[J]. 心理科学, 2024, 47(2): 274-280.
[6]	王允宏, 罗俊龙. 直觉的逻辑性：冲突检测和个体差异的作用机制^*[J]. 心理科学, 2024, 47(1): 52-60.
[7]	朱芷滢, 王烨晖, 杜长宏, 王发明, 杨朝兮. 师生关系对学生数学成绩的影响：行为参与和社会情感能力的链式中介作用^*[J]. 心理科学, 2024, 47(1): 97-104.
[8]	宋萌, 姜傲雪, 王震. 配偶对员工绿色行为的影响：基于溢出-交叉模型的分析^*[J]. 心理科学, 2024, 47(1): 196-204.
[9]	周蔓, 刘彦楼, 滕雅茹. 认知诊断模型中被试参数估计方法的选择^*[J]. 心理科学, 2024, 47(1): 229-236.
[10]	方浩宇, 祝孝亮, 赵鑫. 睡眠质量对10~12岁儿童学业成绩的影响：刷新能力的中介作用 ^*[J]. 心理科学, 2023, 46(5): 1090-1097.
[11]	杜秀芳袁晓倩徐政. 建议者印象和基于动机的个体差异特征对建议采纳的影响[J]. 心理科学, 2023, 46(3): 719-725.
[12]	闫宜人刘宁. 热情还是能力？合作关系中的热情-能力偏好[J]. 心理科学, 2023, 46(3): 594-602.
[13]	郑天鹏周文杰郭磊 . 基于题目作答时间信息的认知诊断模型[J]. 心理科学, 2023, 46(2): 478-490.
[14]	张文洁戴文纯何茹钟毅平. 幼儿对受助者的能力评价及任务难度感知的作用[J]. 心理科学, 2023, 46(2): 339-346.
[15]	杨彩霞王露莼张竞扬王伟皮忠玲李西营. 基于日常事件的情绪标注任务对老年人主观生活质量的影响[J]. 心理科学, 2022, 45(2): 331-338.