Abstract:

Abstract Multi-stage Testing is a computerized testing which aims to improve the disadvantages of CAT, it incorporates most of the advantages from CAT and linear testing. It has some advantages, such as reducing the test length, efficient for measurement, allowing test takers item review, easier to implement, easier to assemble, flexible test schedule for test takers and so on. Cognitive Diagnosis Assessment aims to determine whether or not examinees have each of many attributes or skills underlying responses to items. In contrast to IRT, CDA provides a more detailed evaluation of the strengths and weaknesses of students. Zheng & Chang (2014) presented a new adaptive testing design called on-the-fly assembled multistage adaptive testing (OMST), which combined the benefits of CAT and MST and offsets their limitations. Like MST, OMST is administered in stages and only adapts between stages. But unlike MST, where the modules in every stage are all pre-assembled before administration, the stages in OMST are assembled on the fly. This study presents Cognitive Diagnostic Multi-stage Testing based on the thoughts of MST、CDA、CD-CAT and OMST, in order to increase the application of CDA. CD-MST is a new idea which combines CD with MST. CD-MST has many advantages. It has the functions of cognitive diagnosis and adaptive; compared with CD-CAT, it has speed superiority, and allows the examinees to go back to check and revise; it is also a flexible testing, the test takers can arrange the testing designs according to their demand. CD-MST can solve some practical problems, it has theoretical and practical values. Among the main components of CD-MST, stage item selection strategies can influent the estimation of the examinee’s attribute vectors. Here we use some CD-CAT’s item selection indices to choose stage items of CD-MST. In CD-CAT, these indices are used to choose item, the largest one or the smallest one is chosen. But in CD-MST, if the stage has five items, it is needed to choose the largest five items or the smallest five items, so it is called stage item selection strategies. The research used two simulations to illustrate the CD-MST. First, we tried to find out if the testing designs of CD-MST could influence the results of CD-MST, two contributory factors were considered, the quality of item bank and the item selection strategy. Second, we wanted to contract the CD-MST and the CD-CAT, the following value indexes were considered, the ACCR and the PCCR. We also investigated the speed of CD-MST and CD-CAT. The results showed that compared with MPWKL, GDI and SHE, the recovery rate of PWKL was poorer. This is because PWKL relies on the estimate of knowledge state α_i^((t)). α_i^((t)) is inaccurate in initial stage of testing, PWKL would be influenced. CD-MST has fewer adaptive frequencies, so PWKL is not fit CD-MST. Compared with CD-CAT, CD-MST was more influenced by the quality of item bank. When the item bank had high quality, the recovery rate of CD-MST was as high as CD-CAT. Form the results of the examinees’ average time consuming, we could conclude that CD-CAT cost more time than CD-MST, CD-MST can save the time of 2/3.

摘要：

MST结合了纸笔测验和CAT的优势，现阶段在美国的许多大型考试中得到了应用。本文结合MST、认知诊断、CD-CAT和OMST的思想对CD-MST的可行性进行研究。CD-MST具有认知诊断和自适应的功能，能够使用较少的题目为被试提供即时的、准确的、丰富的诊断信息；同时它计算速度较快，允许考生返回检查和修改，更符合实际考试情境，且在测验的编制上更容易控制。本研究考察了选题策略和题库质量对不同测验设计的CD-MST的影响，并同CD-CAT进行了比较。通过模拟研究发现：MPWKL、GDI和SHE选题策略同样也适用于CD-MST的选题，在题库质量好的情况下这三种选题策略的判准率同CD-CAT持平。CD-MST的测验时间要比CD-CAT缩短2/3以上。