Estimating Reliability of Unidimensional Test in Two-Level Studies

YE Bao-Juan; YUN Zhong-Lin

PDF(481 KB)

Journal of Psychological Science ›› 2013, Vol. 36 ›› Issue (3) : 728-733.

Estimating Reliability of Unidimensional Test in Two-Level Studies

¹,Zhong-Lin WEN

Author information +

History +

Abstract

In the studies of psychology, education and management, we often face data with a two-level structure. For example, students are nested within schools, and employees are nested within enterprises. In such two-level studies, subjects (e. g., students, employees) do not perform independently. Subjects within the same group are usually correlated each other. The independence assumption on individuals in such two-level studies is usually not true. Estimating test reliability is an important step in data analysis. If test reliability is overestimated, the statistical results based on the test are misleading. Reliability is not an intrinsic property of a test, rather, it varies depending on the population in which it is used. Previous research showed that test reliability would be overestimated if the nested relationship was not considered. Hence, test reliability estimation methods that were proposed under the frame of single-level designs are not appropriate for the two-level designs. Raykov and du Toit (2005) deduced a formula for estimating reliability of unidimensional test in the two-level designs based on a two-level confirmatory factor analysis model in which factor loadings of the beween-group part were constrained to be equal to their counterparts of the within-group part. Their formula is only suitable for a rather special situation when the above constraints are correct for the model. Morerover, their method is difficult to be understood, and their program is complicated to be imitated. Till now, most empirical researchers still estimate test reliability as in a single-level design even if the study is a two-level design. So it is necessary to study how to estimate reliability in two-level design and propose a simpler program for computation. We deduced a new formula to estimate the test reliability of the unidimensional test in two-level designs based on two-level confirmatory factor analysis. Whether factor loadings of the beween-group part are fixed or not, the formula is appropriate to estimate the test reliability. An example was illustrated how to estimate test reliability in two-level designs by using our proposed formula with a simple Mplus program. For the purpose of comparison, we also estimated test reliability by using Raykov and du Toit’s (2005) method and simply calculate the composite reliability by treating the data set as if it came from a single-level design. Both of the latter results overestimated the test reliability. Therefore, our new method was recommended. Before we use the new method to compute the reliability of unidimensional test in two-level design, we should analyze a two-level confirmatory factor analysis model and test the goodness of fit. The estimated test reliability is meaningful only when the model is accepted.

Key words

two-level study / unidimensional test / reliability / confirmatory factor analysis

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

Zhong-Lin WEN. Estimating Reliability of Unidimensional Test in Two-Level Studies[J]. Journal of Psychological Science. 2013, 36(3): 728-733

References

温忠麟, 叶宝娟. (2011). 测验信度估计：从系数到内部一致性信度. 心理学报, 43, 821–829.
叶宝娟, 温忠麟. (2011). 单维测验合成信度三种区间估计的比较. 心理学报, 43, 453–461.
张雷, 雷雳, 郭伯良. (2003). 多层线性模型应用. 北京: 教育科学出版社.
Bentler, P. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, 137–143.
Biemer, P. P., Christ, S. L., & Wiesen, C. A. (2009). A general approach for estimating scale score reliability for panel survey data. Psychological Methods, 14, 400–412.
Brincks, A. M., Feaster, D. J., & Mitrani, V. B. (2010). A Multilevel mediation model of stress and coping for women with HIV and their families. Family Process, 49, 517–530.
Choi. J. N. (2007). Change-oriented organizational citizenship behavior: Effects of work environment characteristics and intervening psychological processes. Journal of Organizational Behavior, 28, 467–484.
Dedrick, R. F., & Greenbaum, P. E. (2011). Multilevel confirmatory factor analysis of a scale measuring interagency collaboration of children’s mental health agencies. Journal of Emotional and Behavioral Disorders, 19, 27-40.
D’Haenens, E. (2007, September). The construction of class and school process variables based on individual questionnaires. Paper presented at the Annual European Conference on Educational Research Conference, Ghent.
Green, S. B., & Yang, Y. (2009). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74, 121–135.
Greenbaum, P. E., Wang, W., Boothroyd, R., Kutash, K., & Friedman, R. M. (2011). Multilevel confirmatory factor analysis of the systems of care implementation survey (SOCIS). Journal of Behavioral Health Services & Research, 38, 303–326.
Heck, R. H., & Thomas, S. L. (2009). An introduction to multilevel modeling techniques. New York: Routledge.
Hogan, T. P., Benjamin, A, & Brezinski, K. L. (2000). Reliability methods: A note on the frequency of use various types. Educational and Psychological Measurement, 60, 523-531.
Kamata, A., Bauer, D. J., & Miyazaki, Y. (2008). Multilevel measurement modeling. In A. A. O’Connell & D. B. McCoach (Eds.), Multilevel Modeling of Educational Data (pp. 345–388). Charlotte, NC: Information Age Publishing.
Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd ed.), Thousand Oaks, CA: Sage.
Kim, J. (2010). Influence of neighbourhood collective efficacy on adolescent sexual behaviour: Variation by gender and activity participation. Child: Care, Health and Development, 36, 646–654.
Lachin, J. M. (2004). The role of measurement reliability in clinical trials. Clinical Trials, 1, 553–566.
Laenen, A. & Alonso, A. (2010). The functional living index-cancer: Estimating its reliability based on clinical trial data. Quality of Life Research, 19, 103–109.
Laenen, A., Alonso, A., Molenberghs, G., & Vangeneugden. T. (2009). A family of measures to evaluate scale reliability in a longitudinal setting. Journal of the Royal Statistical Society, 172, 237–253.
Laenen, A., Vangeneugden, T., Geys, H., & Molenberghs. G. (2006). Generalized reliability estimation using repeated measurements. British Journal of Mathematical and Statistical Psychology, 59, 113–131.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading: Addison-Wesley.
Muthén, B. (1991). Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 28, 338–354.
Raykov, T. (2011). Intraclass correlation coefficients in hierarchical designs: Evaluation using latent variable modeling. Structural Equation Modeling, 18, 73–90.
Raykov, T., & du Toit, S. H. C. (2005). Estimation of reliability for multiple-component measuring instruments in hierarchical designs. Structural Equation Modeling, 12, 536–550.
Raykov, T. & Penev, S. (2009). Estimation of maximal reliability for multiple-component instruments in multilevel designs. British Journal of Mathematical Psychology, 62, 129–142.
Sexton, J. B., Helmreich, R. L., Neilands, T. B., Rowan, K., Vella, K., & Boyden, J. et al. (2006). The safety attitudes questionnaire: Psychometric properties benchmarking data, and emerging research. BMC health Services Research, 6, 44–53.
Siminoff, L. A., Wilson-Genderson, M. & Baker, S. (2010), Depressive symptoms in lung cancer patients and their family caregivers and the influence of family environment. Psycho-Oncology, 19, 1285–1293.
Walumbwa, F., Peterson, S., Avolio, B., & Hartnell, C. (2010). An investigation of the relationships among leader and follower psychological capital, service climate, and job performance. Personnel Psychology, 63, 937–963