Abstract
Missing data are easily find in psychological surveys and experiments. For example, in performance assessment, a certain group of raters rated a certain group of examinees. By this token, the data from performance assessment compose a sparse data matrix. Researchers are always concerned about how to make good use of the observed data. Brennan(2001) provided the estimating formulas of p×i design of sparse data. But in practice, there are always more than one factor which effect the experiment. Especially the factor of rater. This factor is the one which cannot be ignored in the performance assessment.
The aim of this article is to find a way which can estimate the variance component of sparse data rapidly and effectively. In China, many studies only analyzed complete data. There are two demerit as following. Firstly, If missing data were encountered, researchers usually deleted incomplete records or used imputation before analysis. But using these methods to analyze performance assessment will reduce the data which can use to analyze. Secondly, the estimated value will differ along with different imputation methods. This article provided the estimating formulas of p×i×r design of sparse data, which are on the basis of the estimating formulas of p×i design of sparse data provided by Brennan(2001).
This article used matlab7.0 to simulate data which were usually encountered in examination, then used GT theory to estimate variance components. We simulated two conditions respectively, small size with 200 students and large-sized with 10000 students. And then used the estimating formulas of p×i×r design of sparse data to estimate variance components, in order to test the formulas’ validity.
The research showed that: These formulas could provided a good estimation of variance components. The estimated variance components approach to set values. The accuracy rates of item and rater were highest. The accuracy rates of interaction of student and item was low. The maximum bias of interaction could reach 1.5. The number of items had the most important effect on the estimation. The number of item increased only a little, the accuracy rate would increased by a big margin. These formulas could provided a good estimation when the amount of item was moderate. We also found that these formulas could used in either small or large amount of data. Either kind of data could get little bias. In performance assessment, we can increase the number of item to enhance the accuracy rate of variance components. If researchers cannot increase the number of item, they can increase the number of rater instead, this way can also enhance the accuracy rate. The number of rater cannot be to large. It can get little bias when the number of rater reach 5.
Key words
Test score /
Sparse data /
Generalizability Theory /
Two-faceted random cross design(p×i×r) /
Estimating variance components
Cite this article
Download Citations
Analysis of Estimating Variance Components for Sparse Data of Test score in Generalizability Theory[J]. Journal of Psychological Science. 2014, 37(3): 742-747
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}