Psychological tests mainly include cognitive tests and personality tests. In cognitive tests, participants can make a correct choice when they know the correct answer, regardless of the guessing factor. However, in personality tests, participants are free to improve their scores. Previous studies have shown that traditional personality tests are prone to acquiescence response, halo effect, impression management, and other abnormal responses.
Many methods used to detect abnormal responses are only proposed for ability tests, but are lacking for personality tests. With the wide application of personality tests in the field of talent assessment, it becomes more urgent to detect whether there is an abnormal response in the test. The development of forced-choice questionnaires avoids the disadvantages of traditional personality tests to some extent. However, personality test is still affected by response style and random response, especially by back random response. Because the length of personality tests is long or the motivation of the participants is low, the participants are prone to random response in the latter part of the test (BRR). BRR is a common abnormal phenomenon in psychological tests. It can increase the error of potential trait estimation, which cannot reflect the real trait level of the participants. At the same time, it can seriously reduce the reliability and validity of the test.
Change point analysis is a popular method for detecting abnormal responses in psychological tests. The advantage of CPA is that it can identify not only the abnormal response of a particular participant, but also detect the specific location of the change point (Shao, 2016). Therefore, the CPA method can help researchers clean up the abnormal part of the data independently without deleting all the data of the participants during data analysis. In this way, the influence of abnormal response can be reduced, the valid data can be retained to the maximum extent and the accuracy of parameter estimation can be improved.
On the basis of previous studies and in combination with the special nature of CPA and BRR, the study applied the existing methods of CPA to forced-choice questionnaires for the first time. Under the framework of MUPP-2PL, the existing methods Lmax, Rmax, and Wmax of CPA were compared and verified through simulation study. This was to provide an effective and reasonable method for detecting abnormal participants in forced-choice questionnaires.
Monte Carlo simulation was used in this study. Firstly, under the framework of MUPP-2PL, the distribution characteristics of Lmax, Rmax, and Wmax in different test length and dimension correlation were discussed, and the 95th percentile of their respective experience distribution was obtained as the critical value (i. e. the criterion for judging whether BRR existed in the process of the test). Secondly, the detection effects of Lmax, Rmax, and Wmax on BRR were verified under different BRR prevalence, dimension correlation, BRR severity and test length.
In all the experimental conditions, the type-I errors of Lmax, Rmax, and Wmax were close to the level of significance ( α = .05). The power of is much higher than that of the other two methods. The absolute lag of is relatively the most accurate. The results of potential trait estimation showed that the accuracy of potential trait estimation was significantly improved after the CPA method was used to clean the abnormal response data. In general, the results of all CPA methods for BRR detection in forced-choice questionnaires were satisfactory and the results of the three methods were highly consistent ( κ > .61, p < .001). In addition, the results of empirical data have also reached similar conclusions. The labeling overlap rate of Wmax and Lmax for abnormal subjects reached 46. 7%, but the labeling overlap rate of Rmax, Wmax, and Lmax were lower respectively.
Key words
forced-choice questionnaires /
change point analysis /
back random response /
multi-unidimensional pairwise-preference two-parameter logistic model
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
References
[1] 安葳鹏, 程小博, 刘雨. (2020). Fleiss' Kappa系数在贝叶斯决策树算法中的应用. 计算机工程与应用, 56(7), 137-140.
[2] 罗薇. (2020). “大五”人格对大学生学业成绩、心理健康的影响: 成就目标的中介作用 (硕士学位论文). 南京师范大学.
[3] 佘梦华. (2021). MIRT框架下后程随机作答侦查方法: 一种优化的变点分析法 (硕士学位论文). 江西师范大学,南昌.
[4] 张龙飞, 王晓雯, 蔡艳, 涂冬波. (2020). 心理与教育测验中异常反应侦查新技术: 变点分析法. 心理科学进展, 28(9), 1462-1477.
[5] Andrews, D. W. K. (1993). Tests for parameter instability and structural change with unknown change point. Econometrica, 61(4), 821-856.
[6] Baker F. B.,& Kim, S. H. (2004). Item response theory: Parameter estimation techniques CRC Press Parameter estimation techniques. CRC Press.
[7] Berry D. T. R., Wetter M. W., Baer R. A., Larsen L., Clark C., & Monroe K. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4(3), 340-345.
[8] Cho S., Drasgow F., & Cao M. Y. (2015). An investigation of emotional intelligence measures using item response theory. Psychological Assessment, 27(4), 1241-1252.
[9] Clark M. E., Gironda R. J., & Young R. W. (2003). Detection of back random responding: Effectiveness of MMPI-2 and Personality Assessment Inventory validity indices. Psychological Assessment, 15(2), 223-234.
[10] De Francisco Carvalho L., De Oliveira Filho, A. Q., Pessotto F., & Bortolotti, S. L. V. (2015). Application of the unfolding model to the aggression dimension of the dimensional clinical personality inventory (IDCP). Revista Colombiana de Psicología, 23(2), 339-349.
[11] Horváth, L., & Csörgö, M. (1997). Limit theorems in change-point analysis. Wiley.
[12] Huang, J. L., & Mead, A. D. (2014). Effect of personality item writing on psychometric properties of ideal-point and Likert scales. Psychological Assessment, 26(4), 1162-1172.
[13] Morillo D., Leenen I., Abad F., Hontangas P., De La Torre J., & Ponsoda V. (2016). A dominance variant under the multi-unidimensional pairwise-preference framework. Applied Psychological Measurement, 40(7), 500-516.
[14] Shao, C. (2016). Aberrant response detection using change-point analysis (Unpublished doctorial dissertation). University of Notre Dame.
[15] Shao C., Li J., & Cheng Y. (2016). Detection of test speedness using change-point analysis. Psychometrika, 81(4), 1118-1141.
[16] Sinharay, S. (2016). Person fit analysis in computerized adaptive testing using tests for a change point. Journal of Educational and Behavioral Statistics, 41(5), 521-549.
[17] Worsley, K. J. (1979). On the likelihood ratio test for a shift in location of normal populations. Journal of the American Statistical Association, 74(366a), 365-367.
[18] Yamamoto, K., & Everson, H. T. (1997). Modeling the effects of test length and test time on parameter estimation using the HYBRID model IRT. In J. Rost & R. Langeheine (Eds.), Applications of latent trait class models in the social sciences (pp. 89-98). Waxmann Verlag GmbH.
[19] Yu, X. F., & Cheng, Y. (2019). A change-point analysis procedure based on weighted residuals to detect back random responding. Psychological Methods, 24(5), 658-674.
PDF(2722 KB)
