Item Selection Methods for Balancing Test Efficiency with Item Bank Usage Efficiency in CD-CAT

Journal of Psychological Science ›› 2014, Vol. 37 ›› Issue (1) : 212-216.

PDF(282 KB)
PDF(282 KB)
Journal of Psychological Science ›› 2014, Vol. 37 ›› Issue (1) : 212-216.

Item Selection Methods for Balancing Test Efficiency with Item Bank Usage Efficiency in CD-CAT

Author information +
History +

Abstract

Cognitive diagnostic computerized adaptive testing (CD-CAT) is a popular mode of online testing of cognitive diagnostic assessment (CDA). The key to a CD-CAT program is the item selection methods. Three of the most popular methods are developed based on Kullback–Leibler information (KL), Shannon entropy (SHE) and expected discrimination method (ED) to utilize selecting items in CD-CAT. These methods can achieve much better test efficiency, however, they often lead to unbalanced item usage within a pool. Diagnostic test would not be a high–stake test, so the item overexposure problem may not be a major concern. Whereas the item underexposure problem lead to wasting time and money invested in developing each item on it and the high test overlap rate problem lead to the effects of intense exercise . Although,restrictive progressive method (RP-PWKL) and restrictive threshold method (RT-PWKL)are proposed to balance item exposure control with measurement accuracy, RP-PWKL and RT-PWKL suppress overexposure is to add a restriction so that the maximum exposure rate will be kept under a predetermined value. The rationale for the maximum exposure rate should be further consideration. For above consideration, the article proposes two item selection methods for CD-CAT based on the Deterministic Input, Noisy ‘And’ Gate” (DINA) model. First, using KL information as a discrimination function of ED, KLED is proposed to handle other cognitive diagnostic models, besides the DINA model. Second, according to the idea of randomization strategies, in which the selection of the item is always made at random among the most informative items, randomization halving algorithm (RHA) is proposed. For RHA,all items within the specified range are available for selection rather than an arbitrary or only one number. Moreover, we show the connection between KLED based on KL, HA, and RHA; KLED can be regard as a weighted HA method, weighted by the corresponding item parameters; HA can be regard as RHA without adding a random component between different item attribute vectors in the Q matrix of item pool. Then, two simulation studies are carried out, one using a simulated item bank, and the other based on items calibrated from real data. Eight item selection strategies are taken into consideration in these studies, including random, posterior–weighted KL (PWKL) , RP-PWKL, RT-PWKL, ED, halving algorithm (HA), KLED and RHA. In addition, VRP-PWKL and VRT-PWKL are proposed for variable-length CD-CAT as an extended version of RP-PWKL and RT-PWKL.Simulation studies for fixed or variable-length CD-CAT are conducted based on the eight methods, and the results are compared in terms of pattern or attribute classification correct rate, error classification rate, item exposure rate, test overlap rate. The simulation results show that: RHA, HA, RP-PWKL, VRP-PWKL and VRT-PWKL have more balanced usage of the item bank and slight decrease in correct classification rate of knowledge state; RHA, HA, VRP-PWKL and VRT-PWKL can be used for variable-length CD-CAT. Though the results from the simulation study are encouraging, further studies of CD-CAT are proposed for the future investigations such as different cognitive diagnostic models.

Key words

CD-CAT, item selection methods, item bank usage, halving algorithm

Cite this article

Download Citations
Item Selection Methods for Balancing Test Efficiency with Item Bank Usage Efficiency in CD-CAT[J]. Journal of Psychological Science. 2014, 37(1): 212-216
PDF(282 KB)

Accesses

Citation

Detail

Sections
Recommended

/