关键词: 认知诊断, Q矩阵, 非参数方法, GDI方法

Abstract: Different from the item response models that postulate a single underlying proficiency, cognitive diagnostic assessments (CDAs) can provide fine-grained diagnostic information on students' knowledge states to support classroom training. The Q-matrix, which links each item to a set of cognitive skills, is necessary to infer students' knowledge states and serves as the foundation for cognitive diagnosis. In reality, domain experts often construct the Q-matrix, which is undoubtedly influenced by their subjective tendencies and, to a significant extent, may contain certain misspecifications. In order to guarantee the accuracy of the Q-matrix, several Q-matrix validation methods have been put forth to find and fix incorrect entries in the known Q-matrix supplied by experts. However, the majority of the currently used methods are parametric methods that are constrained by the sample size or cognitive diagnosis model (CDM).To address this concern, this work developed a generalized nonparametric method to validate the Q-matrix based on the response data, which can be applied to a broad class of cognitive diagnosis models(CDMs).
A general nonparametric classification approach (GNPC) has been proposed by Chiu et al.(2018) and can be applied when the model is saturated, and the sample size is limited. Besides, Chiu (2013) also proposed a nonparametric Q-matrix validation method by minimizing the residual sum of squares (RSS) between the observed responses and ideal responses, which can only be used with the deterministic input, noisy and gate (DINA) and deterministic input, noisy or gate (DINO) models. Inspired by these two studies, a generalized nonparametric Q-validation method has been proposed in this paper, and the steps of the method are as follows. First, using the GNPC approach, it is possible to estimate the attribute patterns of each examinee. The ideal response of every examinee on each item can therefore be calculated in the saturated model using the GNPC method. For each item, the residual sum of squares (RSS) of the ideal response and the observed response can be calculated and the item with the highest RSS will be validated first. For the q-vector of an item, a statistic L was constructed and penalized with the sample size and the number of attributes. Finally, the q-vector corresponding to the smallest penalty L is selected as the q-vector of the item.
The feasibility and effectiveness of the proposed method were evaluated by simulated data generated under various conditions and an application example in real data. The performance of the method in this research was compared with the RSS method, GDI method (de la Torre & Chiu, 2016), and stepwise method(Ma & de la Torre, 2020). In the simulation studies, a number of pertinent variables were taken into account, such as the percentage of misspecifications, cognitive diagnostic models, sample sizes, and attribute distributions. Results demonstrate that (1) In general, the proposed nonparametric method outperforms the GDI and the stepwise method, and can be used with a broad class of cognitive diagnosis models (CDMs). (2)In the small sample (N<200), the performance of the proposed nonparametric method is the best of the three methods, followed by the stepwise method, and the GDI method performs the worst. (3) The empirical data analysis shows that the Q-matrix, validated by the nonparametric method, has better relative fitting and absolute fitting results.

Key words: cognitive diagnosis, Q-matrix, nonparametric method, gdi method