Compare the Diagnostic Assessment Classification Accuracy when the Q-Matrix Contains Error

Journal of Psychological Science ›› 2014, Vol. 37 ›› Issue (6) : 1478-1484.

PDF(5638 KB)
PDF(5638 KB)
Journal of Psychological Science ›› 2014, Vol. 37 ›› Issue (6) : 1478-1484.

Compare the Diagnostic Assessment Classification Accuracy when the Q-Matrix Contains Error

Author information +
History +

Abstract

In recent years, cognitive diagnostic assessment is an area of research that has attracted widespread attention. As we all know, one of the important components in cognitive diagnosis is Q-matrix, because Q-matrix reflects the design of the assessment instrument and is the core element that determines the quality of the diagnostic feedback for the instrument. At present, there are some researches about classification accuracy in DINA model with error existed in Q-matrix. These studies indicate that the quality of the Q-matrix has a great influence on the diagnostic accuracy rate, and also indicate that cognitive diagnosis models constructed around Q-matrix are sensitive to the accuracy of Q-matrix, greatly influenced by Q-matrix, and mostly, the starting point of these research are “if the Q-matrix contains errors, how does it affect the accuracy of parameters estimation and classification accuracy”. Up to now, the most problem is that we haven’t an effective method for validating the Q-matrix at hand. Different diagnostic models have different diagnostic classification accuracy rate, and affected by factors that are not the same. Bayesian networks is one of a widespread concerned model, it has strong processing capacity to uncertainly problem. Starting from another perspective view, uses Bayesian network model which less affected by Q-matrix as diagnosis classification model. Compares Bayesian network with the DINA model in cognitive diagnostic classification accuracy on the base of a Q-matrix which contains errors. Bayesian network classification model is less affected by the Q-matrix than DINA model. Then, two simulation studies are carried out. The first is to study the performance of DINA and Bayesian network classification model when the Q-matrix contains error items, the data is generated under DINA model. To be fair, the data generated in the second research doesn’t base on any specified models, adopts the method introduced by Leighton, Gierl & Hunka(2004). Investigates the effect of different type of Q-matrix (contains a reachable matrix or not), contain different type of error (contain 0, 5, 7, 10, 13, 15 items which have 0, 1, 2, 3 erred calibrate attributes) during classification in different models. The performance of Bayesian network classification model was superior in many cases than DINA model. When Q matrix contained a reachable matrix and 5(or less) error specified items, the performance of DINA model was slightly better than the Bayesian network classification model; but when Q matrix didn’t contained a reachable matrix, or contained more than 5 error specified items, the Bayesian network classification model is better than DINA model.

Key words

cognitive diagnosis / Q matrix / Bayesian Networks / reachable matrix / DINA model

Cite this article

Download Citations
Compare the Diagnostic Assessment Classification Accuracy when the Q-Matrix Contains Error[J]. Journal of Psychological Science. 2014, 37(6): 1478-1484
PDF(5638 KB)

Accesses

Citation

Detail

Sections
Recommended

/