Key-Action Coding Incorporating Misconceptions and Its Application in Diagnostic Classification Analysis of Process Data

Zhan Peida; Gao Fangfang; Chen Qipeng

doi:10.16719/j.cnki.1671-6981.20250220

PDF(2464 KB)

Journal of Psychological Science ›› 2025, Vol. 48 ›› Issue (2) : 481-494. DOI: 10.16719/j.cnki.1671-6981.20250220

Psychological statistics, Psychometrics & Methods

Key-Action Coding Incorporating Misconceptions and Its Application in Diagnostic Classification Analysis of Process Data

Zhan Peida¹, Gao Fangfang¹, Chen Qipeng^1,2

Author information +

History +

Abstract

Process data captures the nuances of human-computer interaction within computer-based learning and assessment systems, reflecting the intricacies of participants’ problem-solving behaviors. Among the various forms of process data, action sequences are particularly critical as they meticulously outline each step of a participant's problem-solving journey. However, the inherent non-standardization of action sequences, characterized by variation in data length across participants, poses significant challenges to the direct application of conventional psychometric models, such as item response theory models and diagnostic classification models (DCM). These conventional psychometric models are typically appropriate for structured data, necessitating adaptations for analyzing process data. One common adaptation is the key-action coding method, which involves identifying whether each participant's data includes critical problem-solving actions, and coding this presence or absence numerically (e.g., “1” for "contains" and “0” for "does not contain"). Zhan and Qiao (2022) introduced a key-action coding method to facilitate the application of DCMs to process data, aiming to determine participants'proficiency in problem-solving skills. However, their method did not consider the detrimental effects of misconceptions on problem-solving performance. Misconceptions, defined as false understandings based on personal experiences, often lead to incorrect responses in problem-solving scenarios. Recognizing and addressing these misconceptions, alongside assessing problem-solving skills, can provide deeper insights into the root causes of errors, enabling the implementation of targeted educational interventions. Despite the critical role of misconceptions in shaping problem-solving outcomes, few if any studies have integrated misconception analysis into the problem-solving process.
To address this gap, this study introduces a novel key-action coding method that incorporates both problem-solving skills and misconceptions, thereby enhancing the utility of DCMs in process data analysis. This method was evaluated using an illustrative example involving the "Tickets" assessment item from PISA 2012, comparing our approach with that of Zhan and Qiao (2022). Our model defined eight attributes—four related to problem-solving skills and four to misconceptions—and included 28 phantom items based on the assessment's scoring rules. This is in contrast to the original four attributes and ten phantom items in Zhan and Qiao’s method. Our analysis used four different DCMs: DINA, DINO, ACDM, and GDINA, with model-data fit assessed using metrics such as AIC, BIC, CAIC, and SABIC. A chi-square test evaluated the statistical differences in model fit, while item quality and classification reliability were measured using the item differentiation index and the classification accuracy index, respectively.
The findings demonstrate that: (1) The GDINA model exhibited the best relative fit, suggesting that the relationship between problem-solving skills and misconceptions in determining item responses is intricately complex and transcends simple conjunctive relationships (as shown in Table 3). (2) The integration of both problem-solving skills and misconceptions allows for a more detailed classification of participants, enabling the identification of specific factors that influence problem-solving success and failure, thereby facilitating targeted remedial interventions tailored to individual needs (as illustrated in Figure 4). (3) The incorporation of misconceptions modestly enhances the reliability of diagnostic classifications (as detailed in Table 4). (4) There is a moderate to high negative correlation between participants'mastery of misconceptions and their raw scores, underscoring that misconceptions adversely affect students’ overall problem-solving performance (as shown in Figure 3). In summary, this study introduces a key-action coding approach that incorporates misconceptions and examines its application in the diagnostic classification analysis of process data, specifically focusing on action sequences. This approach enables researchers to pinpoint specific factors that influence problem-solving outcomes and offers methodological support for targeted interventions. Enhancing participants’ problem-solving performance requires not only improving their skills but also addressing the negative impacts of misconceptions.
The innovation of this paper is primarily reflected in three aspects: (1) It marks the first integration of misconceptions into the methodology of process data analysis, significantly broadening the applicability of the technique. (2) It pioneers the investigation of how misconceptions negatively influence the problem-solving process, thereby enriching our understanding of the mechanisms underlying process data generation. (3) It expands the application of diagnostic classification models and underscores their practical value in analyzing process data, thereby enhancing the efficacy and scope of educational assessments.

Key words

cognitive diagnosis / process data / problem solving / misconception / action sequence

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

Zhan Peida, Gao Fangfang, Chen Qipeng. Key-Action Coding Incorporating Misconceptions and Its Application in Diagnostic Classification Analysis of Process Data[J]. Journal of Psychological Science. 2025, 48(2): 481-494 https://doi.org/10.16719/j.cnki.1671-6981.20250220

References

[1] 丁树良, 杨淑群, 汪文义. (2010). 可达矩阵在认知诊断测验编制中的重要作用. 江西师范大学学报(自然科学版), 34(5), 490-494.
[2] 付颜斌, 陈琦鹏, 詹沛达. (2023). 问题解决任务中行动序列的二分类建模: 单/两参数行动序列模型. 心理学报, 55(8), 1383-1396.
[3] 韩雨婷, 肖悦, 刘红云. (2022). 问题解决测验中过程数据的特征抽取与能力评估. 心理科学进展, 30(6), 1393-1409.
[4] 李美娟, 刘玥, 刘红云. (2020). 计算机动态测验中问题解决过程策略的分析: 多水平混合IRT模型的拓展与应用. 心理学报, 52(4), 528-540.
[5] 刘红云, 韩雨婷, 肖悦, 袁建林, 李美娟. (2023). 复杂问题解决能力的过程性测评与测量模型发展. 中国考试, 11, 9-20.
[6] 刘耀辉, 徐慧颖, 陈琦鹏, 詹沛达. (2022). 基于过程数据的问题解决能力测量及数据分析方法. 心理科学进展, 30(3), 522-535.
[7] 孙鑫, 黎坚, 符植煜. (2018). 利用游戏log-file预测学生推理能力和数学成绩——机器学习的应用. 心理学报, 50(7), 761-770.
[8] Akaike, H. (1981). Likelihood of a model and information criteria. Journal of Econometrics, 16(1), 3-14.
[9] Bergner, Y., & von Davier, A. A. (2019). Process data in NAEP: Past, present, and future. Journal of Educational and Behavioral Statistics, 44(6), 706-732.
[10] Bozdogan, H. (1987). Model selection and Akaike's Information Criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52(3), 345-370.
[11] Bradshaw, L., & Templin, J. (2014). Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika, 79(3), 403-425.
[12] Brault Foisy L. M., Potvin P., Riopel M., & Masson S. (2015). Is inhibition involved in overcoming a common physics misconception in mechanics? Trends in Neuroscience and Education, 4(1-2), 26-36.
[13] Chen F., Lu C., & Cui Y. (2024). Using learners'problem-solving processes in computer-based assessments for enhanced learner modeling: A deep learning approach. Education and Information Technologies, 29(11), 13713-13733.
[14] de la Torre, J. (2008). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343-362.
[15] de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199.
[16] Eggen, P., & Kauchak, D. (2004). Educational psychology: Windows, classrooms. Pearson Prentice Hall.
[17] Fu Y. B., Zhan, P. D, Chen, Q. P., & Jiao H. (2024). Joint modeling of action sequences and action time in computer-based interactive tasks. Behavior Research Methods, 56(5), 4293-4310.
[18] Greiff S., Niepel C., Scherer R., & Martin R. (2016). Understanding students'performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files. Computers in Human Behavior, 61, 36-46.
[19] Gu, Y. Q., & Xu, G. J. (2020). Partial identifiability of restricted latent class models. The Annals of Statistics, 48(4), 2082-2107.
[20] Han Y. T., Liu H. Y., & Ji F. (2022). A sequential response model for analyzing process data on technology-based problem-solving tasks. Multivariate Behavioral Research, 57(6), 960-977.
[21] Hao J., Shu Z., & von Davier A. (2015). Analyzing process data from game/scenario-based tasks: An edit distance approach. Journal of Educational Data Mining, 7(1), 33-50.
[22] Hao J. G., Smith L., Mislevy R., von Davier A., & Bauer M. (2016). Taming log files from game/simulation-based assessments: Data models and data analysis tools. ETS Research Report Series, 2016(1), 1-17.
[23] He Q. W., Borgonovi F., & Paccagnella M. (2021). Leveraging process data to assess adults'problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks. Computers and Education, 166, Article 104170.
[24] Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55.
[25] Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258-272.
[26] Kerr, D., & Chung, G. K. W. K. (2012). Identifying key features of student performance in educational video games and simulations through cluster analysis. Journal of Educational Data Mining, 4(1), 144-182.
[27] Kreitchmann R. S., de la Torre J., Sorrel M. A., Nájera P., & Abad F. J. (2023). Improving reliability estimation in cognitive diagnosis modeling. Behavior Research Methods, 55(7), 3446-3460.
[28] Kuo B. C., Chen C. H., & De La Torre, J. (2018). A cognitive diagnosis model for identifying coexisting skills and misconceptions. Applied Psychological Measurement, 42(3), 179-191.
[29] Kuo B. C., Chen C. H., Yang C. W., & Mok, M. M. C. (2016). Cognitive diagnostic models for tests with multiple-choice and constructed-response items. Educational Psychology, 36(6), 1115-1133.
[30] Levy, R. (2019). Dynamic Bayesian network modeling of game-based diagnostic assessments. Multivariate Behavioral Research, 54(6), 771-794.
[31] Li M. J., Liu H. Y., Cai M. F., & Yuan J. L. (2024). Estimation of individuals'collaborative problem solving ability in computer-based assessment. Education and Information Technologies, 29(1), 483-515.
[32] Liu H. Y., Liu Y., & Li M. J. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, Article 1372.
[33] Ma, W. C., & de la Torre, J. (2020). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1-26.
[34] Ma W. C., Sorrel M. A., Zhai X. M., & Ge Y. (2024). A dual-purpose model for binary data: Estimating ability and misconceptions. Journal of Educational Measurement, 61(2), 179-197.
[35] Martin R. E., Sexton C. M., & Gerlovich J. A. (2001). Teaching science for all children. Allyn and Bacon.
[36] Maydeu-Olivares, A. (2013). Goodness-of-fit assessment of item response theory models. Measurement, 11(3), 71-101.
[37] Mislevy, R. J. (2019). Advances in measurement and cognition. The ANNALS of the American Academy of Political and Social Science, 683(1), 164-182.
[38] OECD. (2014). PISA 2012 results: Creative problem solving: Students'skills in tackling real-life problems. PISA, OECD Publishing.
[39] Qiao, X., & Jiao, H. (2018). Data mining techniques in analyzing process data: A didactic. Frontiers in Psychology, 9, Article 2231.
[40] Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461-464.
[41] Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52(3), 333-343.
[42] Smith J. P., III, diSessa A. A., & Roschelle J. (1994). Misconceptions reconceived: A constructivist analysis of knowledge in transition. Journal of the Learning Sciences, 3(2), 115-163.
[43] Stavy R., Babai R., Tsamir P., Tirosh D., Lin F. L., & McRobbie C. (2006). Are intuitive rules universal? International Journal of Science and Mathematics Education, 4(3), 417-436.
[44] Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345-354.
[45] Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287-305.
[46] van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287-308.
[47] Xia S. L., Zhan P. D., Chan K. K. H., & Wang L. J. (2024). Assessing concept mapping competence using item expansion-based diagnostic classification analysis. Journal of Research in Science Teaching, 61(7), 1516-1542.
[48] Xiao, Y., & Liu, H. Y. (2023). A state response measurement model for problem-solving process data. Behavior Research Methods, 56(1), 258-277.
[49] Zhai, X. M., & Li, M. (2021). Validating a partial-credit scoring approach for multiple-choice science items: An application of fundamental ideas in science. International Journal of Science Education, 43(10), 1640-1666.
[50] Zhan P. D., Jiao H., & Liao D. D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262-286.
[51] Zhan, P. D., & Qiao, X. (2022). Diagnostic classification analysis of problem-solving competence using process data: An item expansion method. Psychometrika, 87(4), 1529-1547.