Abstract:

From a psychological perspective, cognitive diagnostic models (CDMs) are divided into two basic categories called compensatory and noncompensatory diagnostic models depending on the interaction of skills. The interaction of skills in examinee’ item response behavior may sometimes be better modeled as disjunctive (e.g., psychological assessment) and other times be better modeled as conjunctive (e.g., mathematical assessment). In other words, the choice of mode of attribute interaction clearly depends on the diagnostic setting. The selection of an appropriate CDM is based on analyses of the cognitive interaction between the skills and the items on the test. It almost always requires consultation of the literature and close collaboration among psychometric and substantive experts, in addition to empirical checking and confirmation. In conjunctive condensation rule, hierarchy consistency index (HCI) or item consistency index (ICI) can be directly used to assess whether actual examinees’ or items’ response patterns match the expected response patterns. It should be noted that the HCI or the ICI cannot be used with disjunctive CDMs where the mastery of all the attributes measured by an item is not necessary for successful performance because of the assumption that high ability on one attribute can compensate for low ability on other attributes. This leads us to propose the new indices that are specifically designed to identify misfits of item response vectors relative to disjunctive models. HCI or ICI requires the assumption that, from a correct or incorrect response, one can infer that the examinee has or has not mastered all attributes required by the item. Such an inference is often unreasonable and it is likely to draw the possibly incorrect inference. Considering statistical inference is generally more precise than everyday inference, we introduce a consistency index based on hypothesis testing to help detect misfitting item response vectors under the disjunctive condensation rule. The new consistency index with the original ICI can be used in the selection and evaluation of conjunctive or disjunctive model for data analyses. We also proposed a method to estimate slipping and guessing parameters based on comparison of item responses. To investigate whether the new indices can work well under certain conditions, simulated data are generated with an independent structure using five attributes. Four important factors were included in the design of the simulation study: (a) cognitive diagnostic model including two conjunctive models and a disjunctive model; (b) the quality of test Q-matrix with the four error rates from 0 to .4 with step .1, (c) the quality of items, and (d) the number of examinees with N = 500 or 1,000. The results show that these indices can be used to evaluate cognitive assumptions, to assess the quality of test Q-matrix, to identify poor items with attribute misspecification and to estimate noise rate in response data. The new consistency index with original indices may provide better understanding of the nature of cognitive assumptions and they will help determine which psychometric model is most appropriate and interpretable for the intended diagnostic assessment.

摘要：

在认知诊断评估中，评价认知模型与作答数据的拟合非常重要。已有的层级相合性指标(HCI)仅能用于评价连接规则下模型与数据的拟合情况，有必要研究分离规则下相合性指标。HCI假设某项目上正确作答，便推断其子项目上的错误作答为失拟。由于作答反应的随机性，提出基于假设检验的项目相合性指标。该指标可用于区分连接规则和分离规则的作答数据、评价Q矩阵质量和衡量作答数据中的噪音、还可为评价认知模型和选择认知诊断模型提供参考。