Journal of Psychological Science ›› 2023, Vol. 46 ›› Issue (2): 478-490.

Previous Articles     Next Articles

Cognitive Diagnosis Modelling Based on Response Times

Zheng Tianpeng1, 2 ,Zhou Wenjie1,Guo Lei1,3   

  1. 1Faculty of Psychology, Southwest University, Chongqing, 400715
    2Hanhong College, Southwest University, Chongqing, 400715
    3Southwest University Branch, Collaborative Innovation Center of Assessment toward Basic Education Quality, Chongqing, 400715

  • Received:2020-11-21 Revised:2021-05-11 Online:2023-03-20 Published:2023-03-20

基于题目作答时间信息的认知诊断模型

郑天鹏1, 2 周文杰1 郭 磊1,3   

  1. 1 西南大学心理学部,重庆,400715 2 西南大学含弘学院,重庆,400715 3 中国基础教育质量监测协同创新中心西南大学分中心,重庆,400715
  • 通讯作者: 郭磊

Abstract: The development of computer-based assessments can collect process data for different test evaluation purposes. In current psychological measurement modeling, response time is the most frequently studied process data in log files. The use of response time in cognitive diagnosis assessment can further improve the accuracy of diagnosis. However, almost all of the cognitive diagnosis models (CDMs), particularly those involve response time, pay little attention to process data. Recently, a few CDMs, concerning item response times, have been developed, such as the JRT-DINA model (Zhan, Jiao, & Liao, 2018). In this paper, a more concise CDM, named the RRT-DINA (Reduced Response Times DINA) model was proposed via constructing a direct relationship between the probability of correct responses and person speed parameter. And the MCMC algorithm was applied to estimate both item and person parameters of the RRT-DINA model. Firstly, an empirical study was conducted to compare the performances, which are evaluated by the PISA2012 data, of the RRT-DINA model and JRT-DINA model respectively. The results showed that comparing with JRT-DINA model, the new model had lower -2LL, AIC, BIC and DIC values, which indicated that the new model fitted this empirical data better. Secondly, to further verify the performance of the new model, four factors were investigated in the simulation study, i.e., the sample size (500 examines, 1000 examines, 2000 examines), the test length(15 test items,30 test items), the number of cognitive attributes (3, 5, 7) and the types of data generation models (RRT-DINA and JRT-DINA). The results of simulation studies revealed that: (1) When using the RRT-DINA model to generate the data, the recovery of item parameters and person speed parameters of the new model was decent. And the accuracy of parameter estimation in both CDMs will be higher with the increase of the test length, on the whole. Meanwhile, the AACCR and PCCR values obtained from the RRT-DINA model were much more precise than those got from the JRT-DINA model. Under the biggest gap, the AACCR and PCCR of RRT-DINA model were 11.3% and 20.01% higher than JRT-DINA model respectively. Furthermore, the AACCR and PCCR values of the two models were more accurate as the test length increased ; (2) When using the JRT-DINA model to generate the data, the recovery of item parameters and person speed parameters of the new model was slightly inferior to the JRT-DINA model, but the differences in terms of the recovery between the two were very close to those got from the JRT-DINA model. Under the biggest gap, the AACCR and PCCR of JRT-DINA model were only 2.88% and 11% higher than RRT-DINA model respectively. Similarly, the AACCR and PCCR values of the two models were more accurate as the test length increased. In addition, the authors discussed the reasons for choosing response time as a representative of process data, the influence of Q matrix on model parameter estimation and the question of how to choose between the RRT-DINA model and JRT-DINA model. Finally, this paper ends with the prospects of future researches: (1) Introduction of more types of process data; (2)Extension of the speed parameter to multiple dimensions. In general, this paper proposed a simplified and better-performed CDM which could utilize the response times information. And the new model is based on a more direct modeling method rather than the hierarchical modelling framework.

Key words: process data, response times, cognitive diagnosis model, MCMC algorithm, PISA2012

摘要: 在认知诊断评估中利用过程性数据,如作答时间信息,能进一步提升诊断精度。通过建立被试正确作答概率与个体速度参数之间的回归模型,开发了更简洁的新模型:RRT-DINA模型。采用实证与模拟研究,与JRT-DINA模型比较,探讨了新模型的性能。PISA2012数据研究表明,RRT-DINA模型的拟合效果更好。模拟研究结果表明:(1)RRT-DINA模型可采用MCMC算法实现参数估计,估计精度较高。(2)当以RRT-DINA生成数据时,RRT-DINA的题目参数估计精度优于JRT-DINA;当以JRT-DINA生成数据时,JRT-DINA的题目参数估计精度稍微优于RRT-DINA。(3)当以RRT-DINA生成数据时,RRT-DINA的判准率优于JRT-DINA模型;当以JRT-DINA生成数据时,JRT-DINA的判准率稍微优于RRT-DINA,且差距较小。

关键词: 过程性数据, 题目作答时间, 认知诊断模型, MCMC算法, PISA2012