Detection of Back Random Responding in Forced-Choice Questionnaires: The Change Point Analysis Method

doi:10.16719/j.cnki.1671-6981.20240622

Abstract

Abstract: Psychological tests mainly include cognitive tests and personality tests. In cognitive tests, participants can make a correct choice when they know the correct answer, regardless of the guessing factor. However, in personality tests, participants are free to improve their scores. Previous studies have shown that traditional personality tests are prone to acquiescence response, halo effect, impression management, and other abnormal responses.
Many methods used to detect abnormal responses are only proposed for ability tests, but are lacking for personality tests. With the wide application of personality tests in the field of talent assessment, it becomes more urgent to detect whether there is an abnormal response in the test. The development of forced-choice questionnaires avoids the disadvantages of traditional personality tests to some extent. However, personality test is still affected by response style and random response, especially by back random response. Because the length of personality tests is long or the motivation of the participants is low, the participants are prone to random response in the latter part of the test (BRR). BRR is a common abnormal phenomenon in psychological tests. It can increase the error of potential trait estimation, which cannot reflect the real trait level of the participants. At the same time, it can seriously reduce the reliability and validity of the test.
Change point analysis is a popular method for detecting abnormal responses in psychological tests. The advantage of CPA is that it can identify not only the abnormal response of a particular participant, but also detect the specific location of the change point (Shao, 2016). Therefore, the CPA method can help researchers clean up the abnormal part of the data independently without deleting all the data of the participants during data analysis. In this way, the influence of abnormal response can be reduced, the valid data can be retained to the maximum extent and the accuracy of parameter estimation can be improved.
On the basis of previous studies and in combination with the special nature of CPA and BRR, the study applied the existing methods of CPA to forced-choice questionnaires for the first time. Under the framework of MUPP-2PL, the existing methods L_max, R_max, and W_maxof CPA were compared and verified through simulation study. This was to provide an effective and reasonable method for detecting abnormal participants in forced-choice questionnaires.
Monte Carlo simulation was used in this study. Firstly, under the framework of MUPP-2PL, the distribution characteristics of L_max, R_max, and W_maxin different test length and dimension correlation were discussed, and the 95th percentile of their respective experience distribution was obtained as the critical value (i. e. the criterion for judging whether BRR existed in the process of the test). Secondly, the detection effects of L_max, R_max, and W_maxon BRR were verified under different BRR prevalence, dimension correlation, BRR severity and test length.
In all the experimental conditions, the type-I errors of L_max, R_max, and W_maxwere close to the level of significance ( α = .05). The power of is much higher than that of the other two methods. The absolute lag of is relatively the most accurate. The results of potential trait estimation showed that the accuracy of potential trait estimation was significantly improved after the CPA method was used to clean the abnormal response data. In general, the results of all CPA methods for BRR detection in forced-choice questionnaires were satisfactory and the results of the three methods were highly consistent ( κ > .61, p < .001). In addition, the results of empirical data have also reached similar conclusions. The labeling overlap rate of W_maxand L_max for abnormal subjects reached 46. 7%, but the labeling overlap rate of R_max, W_max, and L_max were lower respectively.

Key words: forced-choice questionnaires, change point analysis, back random response, multi-unidimensional pairwise-preference two-parameter logistic model

摘要： 变点分析法是目前在心理与教育测验中实用性非常高的侦测异常反应的手段,变点分析方法不仅能够识别被试在测验过程中是否存在异常作答反应,还能检测异常变点发生的具体位置。本研究首次将变点分析方法拓展到迫选测验中,并对变点分析法的各指标在迫选测验中的异常反应侦测效果进行了验证。结果表明,基于似然比检验的个人拟合统计量（L_max）、基于加权残差的个人拟合统计量（R_max）和基于Wald检验的个人拟合统计量（W_max）在所有实验条件下,I-类错误率接近显著性水平(α = .05);在统计检验力方面,L_max的检测力高于另外两种指标;在变点定位绝对偏差方面,W_max定位的变点位置相对来说最准确;潜在特质估计结果表明,通过变点分析方法对被试的异常作答数据进行清洗后,被试的潜在特质估计精度有了显著改善。总体来说,变点分析法的各指标在迫选测验中对后程随机作答的检测结果比较理想且三种方法的侦测结果具有高度一致性（κ > .61,p < .001）。

关键词: 迫选测验, 变点分析, 后程随机作答, MUPP-2PL模型

Wang Xue, Luo Fen, Cai Yan, Tu Dongbo. Detection of Back Random Responding in Forced-Choice Questionnaires: The Change Point Analysis Method[J]. Journal of Psychological Science, 2024, 47(6): 1507-1518.

王雪, 罗芬, 蔡艳, 涂冬波. 迫选测验中后程随机作答的侦查：基于变点分析法^*[J]. 心理科学, 2024, 47(6): 1507-1518.

References

[1] 安葳鹏, 程小博, 刘雨. (2020). Fleiss' Kappa系数在贝叶斯决策树算法中的应用. 计算机工程与应用, 56(7), 137-140.
[2] 罗薇. (2020). “大五”人格对大学生学业成绩、心理健康的影响: 成就目标的中介作用 (硕士学位论文). 南京师范大学.
[3] 佘梦华. (2021). MIRT框架下后程随机作答侦查方法: 一种优化的变点分析法 (硕士学位论文). 江西师范大学,南昌.
[4] 张龙飞, 王晓雯, 蔡艳, 涂冬波. (2020). 心理与教育测验中异常反应侦查新技术: 变点分析法. 心理科学进展, 28(9), 1462-1477.
[5] Andrews, D. W. K. (1993). Tests for parameter instability and structural change with unknown change point. Econometrica, 61(4), 821-856.
[6] Baker F. B.,& Kim, S. H. (2004). Item response theory: Parameter estimation techniques CRC Press Parameter estimation techniques. CRC Press.
[7] Berry D. T. R., Wetter M. W., Baer R. A., Larsen L., Clark C., & Monroe K. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4(3), 340-345.
[8] Cho S., Drasgow F., & Cao M. Y. (2015). An investigation of emotional intelligence measures using item response theory. Psychological Assessment, 27(4), 1241-1252.
[9] Clark M. E., Gironda R. J., & Young R. W. (2003). Detection of back random responding: Effectiveness of MMPI-2 and Personality Assessment Inventory validity indices. Psychological Assessment, 15(2), 223-234.
[10] De Francisco Carvalho L., De Oliveira Filho, A. Q., Pessotto F., & Bortolotti, S. L. V. (2015). Application of the unfolding model to the aggression dimension of the dimensional clinical personality inventory (IDCP). Revista Colombiana de Psicología, 23(2), 339-349.
[11] Horváth, L., & Csörgö, M. (1997). Limit theorems in change-point analysis. Wiley.
[12] Huang, J. L., & Mead, A. D. (2014). Effect of personality item writing on psychometric properties of ideal-point and Likert scales. Psychological Assessment, 26(4), 1162-1172.
[13] Morillo D., Leenen I., Abad F., Hontangas P., De La Torre J., & Ponsoda V. (2016). A dominance variant under the multi-unidimensional pairwise-preference framework. Applied Psychological Measurement, 40(7), 500-516.
[14] Shao, C. (2016). Aberrant response detection using change-point analysis (Unpublished doctorial dissertation). University of Notre Dame.
[15] Shao C., Li J., & Cheng Y. (2016). Detection of test speedness using change-point analysis. Psychometrika, 81(4), 1118-1141.
[16] Sinharay, S. (2016). Person fit analysis in computerized adaptive testing using tests for a change point. Journal of Educational and Behavioral Statistics, 41(5), 521-549.
[17] Worsley, K. J. (1979). On the likelihood ratio test for a shift in location of normal populations. Journal of the American Statistical Association, 74(366a), 365-367.
[18] Yamamoto, K., & Everson, H. T. (1997). Modeling the effects of test length and test time on parameter estimation using the HYBRID model IRT. In J. Rost & R. Langeheine (Eds.), Applications of latent trait class models in the social sciences (pp. 89-98). Waxmann Verlag GmbH.
[19] Yu, X. F., & Cheng, Y. (2019). A change-point analysis procedure based on weighted residuals to detect back random responding. Psychological Methods, 24(5), 658-674.