关键词: 认知诊断, 过程数据, 问题解决, 迷思概念, 行动序列

Abstract: Process data captures the nuances of human-computer interaction within computer-based learning and assessment systems, reflecting the intricacies of participants’ problem-solving behaviors. Among the various forms of process data, action sequences are particularly critical as they meticulously outline each step of a participant's problem-solving journey. However, the inherent non-standardization of action sequences, characterized by variation in data length across participants, poses significant challenges to the direct application of conventional psychometric models, such as item response theory models and diagnostic classification models (DCM). These conventional psychometric models are typically appropriate for structured data, necessitating adaptations for analyzing process data. One common adaptation is the key-action coding method, which involves identifying whether each participant's data includes critical problem-solving actions, and coding this presence or absence numerically (e.g., “1” for "contains" and “0” for "does not contain"). Zhan and Qiao (2022) introduced a key-action coding method to facilitate the application of DCMs to process data, aiming to determine participants'proficiency in problem-solving skills. However, their method did not consider the detrimental effects of misconceptions on problem-solving performance. Misconceptions, defined as false understandings based on personal experiences, often lead to incorrect responses in problem-solving scenarios. Recognizing and addressing these misconceptions, alongside assessing problem-solving skills, can provide deeper insights into the root causes of errors, enabling the implementation of targeted educational interventions. Despite the critical role of misconceptions in shaping problem-solving outcomes, few if any studies have integrated misconception analysis into the problem-solving process.
To address this gap, this study introduces a novel key-action coding method that incorporates both problem-solving skills and misconceptions, thereby enhancing the utility of DCMs in process data analysis. This method was evaluated using an illustrative example involving the "Tickets" assessment item from PISA 2012, comparing our approach with that of Zhan and Qiao (2022). Our model defined eight attributes—four related to problem-solving skills and four to misconceptions—and included 28 phantom items based on the assessment's scoring rules. This is in contrast to the original four attributes and ten phantom items in Zhan and Qiao’s method. Our analysis used four different DCMs: DINA, DINO, ACDM, and GDINA, with model-data fit assessed using metrics such as AIC, BIC, CAIC, and SABIC. A chi-square test evaluated the statistical differences in model fit, while item quality and classification reliability were measured using the item differentiation index and the classification accuracy index, respectively.
The findings demonstrate that: (1) The GDINA model exhibited the best relative fit, suggesting that the relationship between problem-solving skills and misconceptions in determining item responses is intricately complex and transcends simple conjunctive relationships (as shown in Table 3). (2) The integration of both problem-solving skills and misconceptions allows for a more detailed classification of participants, enabling the identification of specific factors that influence problem-solving success and failure, thereby facilitating targeted remedial interventions tailored to individual needs (as illustrated in Figure 4). (3) The incorporation of misconceptions modestly enhances the reliability of diagnostic classifications (as detailed in Table 4). (4) There is a moderate to high negative correlation between participants'mastery of misconceptions and their raw scores, underscoring that misconceptions adversely affect students’ overall problem-solving performance (as shown in Figure 3). In summary, this study introduces a key-action coding approach that incorporates misconceptions and examines its application in the diagnostic classification analysis of process data, specifically focusing on action sequences. This approach enables researchers to pinpoint specific factors that influence problem-solving outcomes and offers methodological support for targeted interventions. Enhancing participants’ problem-solving performance requires not only improving their skills but also addressing the negative impacts of misconceptions.
The innovation of this paper is primarily reflected in three aspects: (1) It marks the first integration of misconceptions into the methodology of process data analysis, significantly broadening the applicability of the technique. (2) It pioneers the investigation of how misconceptions negatively influence the problem-solving process, thereby enriching our understanding of the mechanisms underlying process data generation. (3) It expands the application of diagnostic classification models and underscores their practical value in analyzing process data, thereby enhancing the efficacy and scope of educational assessments.

Key words: cognitive diagnosis, process data, problem solving, misconception, action sequence