A Generalized Nonparametric Q-Matrix Validation Method

doi:10.16719/j.cnki.1671-6981.20230526

Abstract

Abstract: Different from the item response models that postulate a single underlying proficiency, cognitive diagnostic assessments (CDAs) can provide fine-grained diagnostic information on students' knowledge states to support classroom training. The Q-matrix, which links each item to a set of cognitive skills, is necessary to infer students' knowledge states and serves as the foundation for cognitive diagnosis. In reality, domain experts often construct the Q-matrix, which is undoubtedly influenced by their subjective tendencies and, to a significant extent, may contain certain misspecifications. In order to guarantee the accuracy of the Q-matrix, several Q-matrix validation methods have been put forth to find and fix incorrect entries in the known Q-matrix supplied by experts. However, the majority of the currently used methods are parametric methods that are constrained by the sample size or cognitive diagnosis model (CDM).To address this concern, this work developed a generalized nonparametric method to validate the Q-matrix based on the response data, which can be applied to a broad class of cognitive diagnosis models(CDMs).
A general nonparametric classification approach (GNPC) has been proposed by Chiu et al.(2018) and can be applied when the model is saturated, and the sample size is limited. Besides, Chiu (2013) also proposed a nonparametric Q-matrix validation method by minimizing the residual sum of squares (RSS) between the observed responses and ideal responses, which can only be used with the deterministic input, noisy and gate (DINA) and deterministic input, noisy or gate (DINO) models. Inspired by these two studies, a generalized nonparametric Q-validation method has been proposed in this paper, and the steps of the method are as follows. First, using the GNPC approach, it is possible to estimate the attribute patterns of each examinee. The ideal response of every examinee on each item can therefore be calculated in the saturated model using the GNPC method. For each item, the residual sum of squares (RSS) of the ideal response and the observed response can be calculated and the item with the highest RSS will be validated first. For the q-vector of an item, a statistic L was constructed and penalized with the sample size and the number of attributes. Finally, the q-vector corresponding to the smallest penalty L is selected as the q-vector of the item.
The feasibility and effectiveness of the proposed method were evaluated by simulated data generated under various conditions and an application example in real data. The performance of the method in this research was compared with the RSS method, GDI method (de la Torre & Chiu, 2016), and stepwise method(Ma & de la Torre, 2020). In the simulation studies, a number of pertinent variables were taken into account, such as the percentage of misspecifications, cognitive diagnostic models, sample sizes, and attribute distributions. Results demonstrate that (1) In general, the proposed nonparametric method outperforms the GDI and the stepwise method, and can be used with a broad class of cognitive diagnosis models (CDMs). (2)In the small sample (N<200), the performance of the proposed nonparametric method is the best of the three methods, followed by the stepwise method, and the GDI method performs the worst. (3) The empirical data analysis shows that the Q-matrix, validated by the nonparametric method, has better relative fitting and absolute fitting results.

Key words: cognitive diagnosis, Q-matrix, nonparametric method, gdi method

摘要： Q矩阵是认知诊断的基础,Q矩阵标定错误会影响被试分类的准确性。本研究基于非参数角度开发了不受模型限制的Q矩阵修正方法,并与已有参数化方法进行了比较。研究结果发现：（1）非参数的PLM方法可实现各模型下的Q矩阵修正,方法具有计算简单、方便使用且不受模型限制的特征。（2）非参数的PLM方法表现明显优于stepwise方法,而GDI方法和RSS方法的表现最差。（3）实证数据分析表明,PLM方法修正后的Q矩阵具有更好的相对拟合和绝对拟合结果。

关键词: 认知诊断, Q矩阵, 非参数方法, GDI方法

Wang Daxun, Xiao Qingwen, Tan Qingrong, Cai Yan, Tu Dongbo. A Generalized Nonparametric Q-Matrix Validation Method[J]. Journal of Psychological Science, 2023, 46(5): 1237-1245.

汪大勋, 肖清文, 谭青蓉, 蔡艳, 涂冬波. 一种广义的非参数化Q矩阵修正方法 ^*[J]. 心理科学, 2023, 46(5): 1237-1245.

References

[1] 郭磊, 杨静, 宋乃庆. (2018). 谱聚类算法在不同属性层级结构诊断评估中的应用. 心理科学, 41(3), 735-742.
[2] 康春花, 任平, 曾平飞. (2015). 非参数认知诊断方法: 多级评分的聚类分析. 心理学报, 47(8), 1077-1088.
[3] 涂冬波, 蔡艳, 戴海琦. (2012). 基于DINA模型的Q矩阵修正方法. 心理学报, 44(4), 558-568.
[4] 汪大勋, 高旭亮, 蔡艳, 涂冬波. (2018). 一种非参数化的Q矩阵估计方法: ICC-IR方法开发. 心理科学, 41(2), 466-474.
[5] 王立君, 唐芳, 詹沛达. (2020). 基于认知诊断测评的个性化补救教学效果分析: 以“一元一次方程”为例. 心理科学, 43(6), 1490-1497.
[6] Chen, J. S. (2017). A residual-based approach to validate Q-Matrix specifications. Applied Psychological Measurement, 41(4), 277-293.
[7] Chiu C. Y., Douglas J., & Li X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633-665.
[8] Chiu, C. Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598-618.
[9] Chiu, C. Y., & Douglas, J. (2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns. Journal of Classification, 30(2), 225-250.
[10] Chiu C. Y., Sun Y., & Bian Y. H. (2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83(2), 355-375.
[11] Cui, Y., & Li, J. (2015). Evaluating person fit for cognitive diagnostic assessment. Applied Psychological Measurement, 39(3), 223-238.
[12] de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343-362.
[13] de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115-130.
[14] de la Torre, J., & Chiu, C. Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253-273.
[15] Hansen M., Cai L., Monroe S., & Li Z. (2016). Limited-information goodness-of-fit testing of diagnostic classification item response models. British Journal of Mathematical and Statistical Psychology, 69(3), 225-252.
[16] Liu J. C., Xu G. J., & Ying Z. L. (2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548-564.
[17] Liu R., Huggins-Manley A. C., & Bulut O. (2018). Retrofitting diagnostic classification models to responses from IRT-based assessment forms. Educational and Psychological Measurement, 78(3), 357-383.
[18] Liu Y. L., Tian W., & Xin T. (2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41(1), 3-26.
[19] Liu Y. L., Xin T., Andersson B., & Tian W. (2019). Information matrix estimation procedures for cognitive diagnostic models. British Journal of Mathematical and Statistical Psychology, 72(1), 18-37.
[20] Ma, W. C., & de la Torre, J. (2020). An empirical Q-matrix validation method for the sequential generalized DINA model. British Journal of Mathematical and Statistical Psychology, 73(1), 142-163.
[21] Maydeu-Olivares, A., & Joe, H. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49(4), 305-328.
[22] Nájera P., Sorrel M. A., & Abad F. J. (2019). Reconsidering cutoff points in the general method of empirical Q-matrix validation. Educational and Psychological Measurement, 79(4), 727-753.
[23] Nájera P., Sorrel M. A., de la Torre J., & Abad F. J. (2021). Balancing fit and parsimony to improve Q-matrix validation. British Journal of Mathematical and Statistical Psychology, 74(1), 110-130.
[24] Philipp M., Strobl C., de la Torre J., & Zeileis A. (2018). On the estimation of standard errors in cognitive diagnosis models. Journal of Educational and Behavioral Statistics, 43(1), 88-115.
[25] Ren H., Xu N. N., Lin Y. X., Zhang S. M., & Yang T. (2021). Remedial teaching and learning from a cognitive diagnostic model perspective: Taking the data distribution characteristics as an example. Frontiers in Psychology, 12, Article 628607.
[26] Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52(3), 333-343.
[27] Wang D. X., Cai Y., & Tu D. B. (2020). Q-matrix estimation methods for cognitive diagnosis models: Based on partial known Q-matrix. Multivariate Behavioral Research, 56(3), 514-526.
[28] Wang W. Y., Song L. H., Ding S. L., Meng Y. R., Cao C. X., & Jie Y. J. (2018). An EM-based method for Q-matrix validation. Applied Psychological Measurement, 42(6), 446-459.
[29] Xu, G. J., & Shang, Z. R. (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284-1295.