Abstract Estimating variance component, which is the essential technique for generalizability theory, is constrained by sampling. Different sampling may cause different estimated variance component. Therefore, estimating the variability of estimated variance components needs to be further explored. The variability of estimated variance components mainly includes standard error and confidence interval. In the past studies, there were some problems as follows. Fist, it was often that some researchers only focused on normal distribution data and neglected non-normal distribution data. In fact, non-normal distribution data could be always seen in such tests as TOEFL test. Second, the previous studies didn’t compare the variability of estimated variance components using traditional, bootstrap, jackknife and Markov Chain Monte Carlo method (MCMC) at the same time. The study adopts Monte Carlo data simulation technique to compare the variability of estimated variance components for generalizability theory based on three non-normal distribution data using four methods that include traditional method, bootstrap method, jackknife method and MCMC method. As for traditional method, ANOVA is used to estimate the variance components and their standard error and TBJGL are used to estimate the confidence intervals. As for bootstrap method, twelve bootstrap strategies are considered. But jackknife method only considered three strategies. Moreover, two strategies, informative and noninformative priors, are considered in MCMC method. Three non-normal distribution data are simulated by some techniques. To compare these four methods in two variabilities, the criterion is made and it is the bias. The smaller bias, the more reliable the results are.Some programs are made by us in R statistical programming environment. To link R program with WinBUGS, R2winGUGS and Code package are used. To simulate skewed data, HyperbolicDist package is used. The simulation results are as follows. First, data distribution has an effect on the variability of estimated variance components. Different estimation procedures have different results. For normal data, traditional, bootstrap and MCMC procedure are accurate for estimating the variability of estimated variance components, but jackknife procedure isn’t. For dichotomous data, bootstrap procedure is accurate, but MCMC procedure is either accurate or inaccurate. Traditional and jackknife procedure aren’t accurate. For polytomous data, bootstrap procedure is accurate, but traditional, jackknife and MCMC procedure aren’t. For skewed data, bootstrap procedure is accurate, but jackknife procedure isn’t. Traditional and MCMC procedure are either accurate or inaccurate.