Abstract: The purposes of cognitive diagnosis (CD) were to detect the internal psychological rules and processing mechanism of human. Through cognitive diagnosis assessment, it would report the features about individuals, such as his/her currently cognitive status, the cognitive procedure, and the processing skills and knowledge structures etc., which would help the teachers and students learn the advantages and weaknesses specifically, and choose a more effective way to teach or learn. As a word, CD would play an important role on individual’s development. More and more attentions should be paid on cognitive diagnosis assessment. With more and more applications of CD in practice, researchers and practitioners found out that the task to identify a test Q matrix was very hard. Even a test Q matrix was established, it was also difficult to evaluate its correctness. Virtually, there might be several possible test Q matrices provided by the experts at the same time. Practitioners always confused about which might be the right. Corresponding to this phenomenon, it needed to find a method to determine which one was the most appropriate one. This study attempted to propose some indexes which might be used for the selection of test Q matrix. In order to verify the effectiveness of the indexes, a comparison study was done. It expected to know the advantages and disadvantages of each indexes, and get an idea that which indexes might be used to choose Q matrices. All these kind of information could be very useful for the applicants. In this study, some test Q matrices with different degree of correctness were provided, and the DINA model was applied. Both Monte Carlo simulation method and real data analysis were used, all the results indicated that: Firstly, according to these eight indexes, the average correct identification rate (ACIR) of the BIC and BIC2 was greater than 95 percent. The ACIRs of the other indexes were less than 90 percent, some indexes were even less than 50 percent. Secondly, the ACIRs varied according to different categories of incorrect Q matrices. Considered with the E error type, the ACIRs of these indexes were relatively low, most of them were less than 50 percent, except from the BIC2 index. Thirdly, weighted by both the size of samples and the number of parameters, the performances of BIC and BIC2 were better than those of the rest. From the above consequences, it showed that the BIC and BIC2 might be the best indexes for the selection of test Q matrix when multiple test Q matrices were supposed.

Key words: cognitive diagnosis, test Q matrix, correct identification rate, DINA model

摘要： 本研究对多个测验Q矩阵的相对合理性的比较与选用开展研究，采用Monte Carlo模拟与实证研究相结合的范式，探讨R_square、HCI、-2LL、AIC、BIC、residual、ABS_residual及本研究新开发的BIC2等八项指标在测验Q矩阵合理性侦查效果及其比较。研究发现：八项指标中，除BIC和BIC2两项指标的对测验Q矩阵相对合理性的平均正确识别率在95%以上，其余指标的平均正确识别率不足90%，整体而言，考虑样本容量及参数个数双重加权的 BIC和BIC2两项指标的表现总体上优于其它几项指标；各项指标在不同Q矩阵错误类型下其正确识别率也不尽相同。

关键词: 认知诊断, 测验Q矩阵, 正确识别率, DINA模型