PDF(571 KB)
Regularized Structural Equation Modeling: Balancing Exploratory and Confirmatory Analysis
Wang Siyi, Deng Yating, Zhang Lijin, Zheng Shufang, Pan Junhao
Journal of Psychological Science ›› 2026, Vol. 49 ›› Issue (2) : 463-472.
PDF(571 KB)
PDF(571 KB)
Regularized Structural Equation Modeling: Balancing Exploratory and Confirmatory Analysis
Currently, most researchers rely on confirmatory structural equation modeling (SEM) for their analyses, which involves constructing relationships between variables or constraining specific parameters based on established theory and prior knowledge. While imposing constraints on parameters can produce a simple and interpretable model, excessive constraints may lead to model misspecification due to the inherent complexity of human behavior, potentially resulting in poor model fit and biased parameter estimates. The lack of theoretical support in new research domains also hinders the effectiveness of confirmatory SEM. Thus, researchers should consider adopting a data-driven approach to identify models that better fit the data. Data-driven exploratory approaches are associated with a high risk of Type I errors, making them prone to including redundant parameters in the model. Regularized SEM not only supports the exploratory process but also delivers a simpler model with a good fit.
In this study, we summarized the value and methods of regularized SEM, including frequentist and Bayesian regularized SEM. First, we reviewed frequentist approaches to regularized SEM. We introduced various regularization methods, including Ridge, Lasso, and Elastic Net. We also listed the penalty functions of commonly used regularization methods in Appendix 1. Additionally, we summarized the applications of frequentist regularized SEM in exploratory factor analysis, mediating model, MIMIC model, and multi-group model. We then discussed Bayesian regularization with shrinkage priors (see Appendix 2 for details), outlining their developments and applications in factor analysis, measurement invariance across time or groups, and variable selection. Second, we compared the advantages and disadvantages of frequentist and Bayesian SEM. Compared to frequentist regularization methods, Bayesian regularization offers several benefits in SEM: (1) It allows for effective standard error estimation and interval estimation of parameters; (2) It provides greater modeling flexibility by relaxing certain constraints imposed by frequentist methods; (3) It performs better in complex models, reducing non-convergence issues; (4) It allows the direct application of hyperpriors to tuning parameters, avoiding cumbersome cross-validation processes. Some researchers have argued that frequentist regularization methods were preferable for simple models, as these methods demonstrated superior performance and required fewer computational resources. It should be noted that the performance of some Bayesian regularization shrinkage priors in variable selection and parameter estimation can be influenced by the choice of priors, and there is currently no consensus on the specific settings of these priors. In contrast, the adjustment parameter λ in frequentist approaches can be selected based on objective criteria, thus avoiding the subjectivity of prior selection in Bayesian regularization. Finally, we discussed the advantages of applying regularized SEM: (1) regularized SEM enhances the reproducibility of psychological research by improving generalization; (2) it effectively balances exploratory and confirmatory methods, making it particularly valuable for scale development.; (3) it performs dimensionality reduction on high-dimensional data, simplifying variable sets and identifying statistically significant variables. Moreover, if researchers want to focus on both the presence of the effect and the magnitude of the effect during the exploratory phase, they can refer to the logic of relaxed Lasso in XMed. In the first stage, regularization can be employed for model selection, while in the second stage, traditional methods can be used to obtain parameter estimates. Regularized SEM also provides a novel approach to model selection by enabling model refinement and variable selection through parameter shrinkage, rather than relying solely on traditional evaluation metrics. However, it remains unclear whether parameter shrinkage methods are superior or inferior to traditional approaches. Notably, researchers should be aware of the bias-variance trade-off when using the regularization methods. We have also compiled a list of currently available software and packages for implementing regularized SEM in Appendix 3. We hope to advance the application of regularized SEM in psychological research.
bayesian regularization / frequentist regularization / regularization / structural equation modeling
| [1] |
宋琼雅, 张沥今, 潘俊豪. (2021). 贝叶斯多组比较——渐近测量不变性. 心理学探新, 41(1), 69-75.
|
| [2] |
张沥今. (2022) 贝叶斯正则化结构方程模型:以MIMIC模型和并行中介模型为例 (硕士学位论文).中山大学,广州.
|
| [3] |
张沥今, 魏夏琰, 陆嘉琦, 潘俊豪. (2020). Lasso回归:从解释到预测. 心理科学进展, 28(10), 1777-1791.
传统的最小二乘回归法关注于对当前数据集的准确估计, 容易导致模型的过拟合, 影响模型结论的可重复性。随着方法学领域的发展, 涌现出的新兴统计工具可以弥补传统方法的局限, 从过度关注回归系数值的解释转向提升研究结果的预测能力也愈加成为心理学领域重要的发展趋势。Lasso方法通过在模型估计中引入惩罚项的方式, 可以获得更高的预测准确度和模型概化能力, 同时也可以有效地处理过拟合和多重共线性问题, 有助于心理学理论的构建和完善。
|
| [4] |
张沥今, 陆嘉琦, 魏夏琰, 潘俊豪. (2019). 贝叶斯结构方程模型及其研究现状. 心理科学进展, 27(11), 1812-1825.
在心理学研究中结构方程模型(Structural Equation Modeling, SEM)被广泛用于检验潜变量间的因果效应, 其估计方法有频率学方法(如, 极大似然估计)和贝叶斯方法两类。近年来由于贝叶斯统计的流行及其在结构方程建模中易于处理小样本、缺失数据及复杂模型等方面的优势, 贝叶斯结构方程模型发展迅速, 但其在国内心理学领域的应用不足。主要介绍了贝叶斯结构方程模型的方法基础和优良特性, 及几类常用的贝叶斯结构方程模型及其应用现状, 旨在为应用研究者介绍新的研究工具。
|
| [5] |
温聪聪. (2025). 惩罚对齐法:测量不变性检验的新方法. 心理科学进展, 33(1), 176-196.
Asparouhov和Muthén在2023年提出了一种全新的惩罚结构方程模型框架。惩罚对齐法是该模型框架在测量不变性检验领域的应用范例。惩罚对齐法继承了多组探索性因子分析在探索性因子分析框架内进行测量不变性检验和可以估计交叉载荷等优点, 继承了对齐法使用成分损失函数、允许模型中存在一定量不等参数等优点, 同时继承了贝叶斯结构方程模型对模型参数设置先验分布、通过检验模型参数的近似测量不变性达到潜因子均值比较的目的等优点。此外, 惩罚对齐法还克服了传统测量不变性检验方法的一些不足。本文以大学生职业价值观研究为例, 比较了多组验证性因子分析、基于验证性因子分析的惩罚对齐法分析、基于探索性结构方程模型的惩罚对齐法分析拟合样本数据的效果, 演示了如何使用惩罚对齐法进行测量不变性检验和多组比较。
|
| [6] |
温忠麟, 刘方, 郑渊丹, 廖心怡, 黄亦南. (2024). 为何调节效应或中介效应的实证文章那么多? 应用心理学, 30(4), 291-297.
|
| [7] |
|
| [8] |
Suicide is a major public health concern. One consistently cited risk factor for suicide is childhood maltreatment, which also may play a role in the transition from suicidal ideation to suicidal behavior.
|
| [9] |
|
| [10] |
|
| [11] |
|
| [12] |
|
| [16] |
|
| [17] |
|
| [18] |
Item factor analysis (IFA), also known as Multidimensional Item Response Theory (MIRT), is a general framework for specifying the functional relationship between respondents’ multiple latent traits and their responses to assessment items. The key element in MIRT is the relationship between the items and the latent traits, so-called item factor loading structure. The correct specification of this loading structure is crucial for accurate calibration of item parameters and recovery of individual latent traits. This paper proposes a regularized Gaussian Variational Expectation Maximization (GVEM) algorithm to efficiently infer item factor loading structure directly from data. The main idea is to impose an adaptive L1\\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{mathrsfs}\n\\usepackage{upgreek}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}$$L_1$$\\end{document}-type penalty to the variational lower bound of the likelihood to shrink certain loadings to 0. This new algorithm takes advantage of the computational efficiency of GVEM algorithm and is suitable for high-dimensional MIRT applications. Simulation studies show that the proposed method accurately recovers the loading structure and is computationally efficient. The new method is also illustrated using the National Education Longitudinal Study of 1988 (NELS:88) mathematics and science assessment data.
|
| [19] |
|
| [21] |
We introduce the network model as a formal psychometric model, conceptualizing the covariance between psychometric indicators as resulting from pairwise interactions between observable variables in a network structure. This contrasts with standard psychometric models, in which the covariance between test items arises from the influence of one or more common latent variables. Here, we present two generalizations of the network model that encompass latent variable structures, establishing network modeling as parts of the more general framework of structural equation modeling (SEM). In the first generalization, we model the covariance structure of latent variables as a network. We term this framework latent network modeling (LNM) and show that, with LNM, a unique structure of conditional independence relationships between latent variables can be obtained in an explorative manner. In the second generalization, the residual variance-covariance structure of indicators is modeled as a network. We term this generalization residual network modeling (RNM) and show that, within this framework, identifiable models can be obtained in which local independence is structurally violated. These generalizations allow for a general modeling framework that can be used to fit, and compare, SEM models, network models, and the RNM and LNM generalizations. This methodology has been implemented in the free-to-use software package lvnet, which contains confirmatory model testing as well as two exploratory search algorithms: stepwise search algorithms for low-dimensional datasets and penalized maximum likelihood estimation for larger datasets. We show in simulation studies that these search algorithms perform adequately in identifying the structure of the relevant residual or latent networks. We further demonstrate the utility of these generalizations in an empirical example on a personality inventory dataset.
|
| [23] |
|
| [24] |
|
| [26] |
Studies of the neural basis of intelligence have focused on comparing brain imaging variables with global scales instead of the cognitive domains integrating these scales or quotients. Here, the relation between mean tract-based fractional anisotropy (mTBFA) and intelligence indices was explored. Deterministic tractography was performed using a regions of interest approach for 10 white-matter fascicles along which the mTBFA was calculated. The study sample included 83 healthy individuals from the second wave of the Cuban Human Brain Mapping Project, whose WAIS-III intelligence quotients and indices were obtained. Inspired by the "Watershed model" of intelligence, we employed a regularized hierarchical Multiple Indicator, Multiple Causes model (MIMIC), to assess the association of mTBFA with intelligence scores, as mediated by latent variables summarizing the indices. Regularized MIMIC, used due to the limited sample size, selected relevant mTBFA by means of an elastic net penalty and achieved good fits to the data. Two latent variables were necessary to describe the indices: Fluid intelligence (Perceptual Organization and Processing Speed indices) and Crystallized Intelligence (Verbal Comprehension and Working Memory indices). Regularized MIMIC revealed effects of the forceps minor tract on crystallized intelligence and of the superior longitudinal fasciculus on fluid intelligence. The model also detected the significant effect of age on both latent variables.© 2019 The Authors. Human Brain Mapping published by Wiley Periodicals, Inc.
|
| [27] |
Exploratory factor analysis (EFA) is widely used in psychological (assessment) research. Due to its exploratory nature, several researcher degrees of freedom exist on how to conduct the analysis. While simulation studies can provide meaningful insights into which factor retention methods to use to determine the number of latent factors, or which estimation methods recover parameter values most precisely given certain data characteristics, the issue of rotational indeterminacy makes it very difficult to decide which rotation method to apply. An alternative to the two-stage approach of extracting factors and subsequently rotating them to foster interpretability is the so-called regularized EFA. In this paper, we contrast both approaches and demonstrate how regularized EFA can be applied. In doing so, we want to encourage researchers to try out the approach themselves and help them find a way of EFA that appears less arbitrary compared to classical factor rotation.
|
| [28] |
|
| [29] |
|
| [30] |
|
| [31] |
Multilevel structural equation models are increasingly applied in psychological research. With increasing model complexity, estimation becomes computationally demanding, and small sample sizes pose further challenges on estimation methods relying on asymptotic theory. Recent developments of Bayesian estimation techniques may help to overcome the shortcomings of classical estimation techniques. The use of potentially inaccurate prior information may, however, have detrimental effects, especially in small samples. The present Monte Carlo simulation study compares the statistical performance of classical estimation techniques with Bayesian estimation using different prior specifications for a two-level SEM with either continuous or ordinal indicators. Using two software programs (Mplus and Stan), differential effects of between- and within-level sample sizes on estimation accuracy were investigated. Moreover, it was tested to which extent inaccurate priors may have detrimental effects on parameter estimates in categorical indicator models. For continuous indicators, Bayesian estimation did not show performance advantages over ML. For categorical indicators, Bayesian estimation outperformed WLSMV solely in case of strongly informative accurate priors. Weakly informative inaccurate priors did not deteriorate performance of the Bayesian approach, while strong informative inaccurate priors led to severely biased estimates even with large sample sizes. With diffuse priors, Stan yielded better results than Mplus in terms of parameter estimates.
|
| [32] |
|
| [34] |
|
| [35] |
|
| [39] |
Methodological innovations have allowed researchers to consider increasingly sophisticated statistical models that are better in line with the complexities of real world behavioral data. However, despite these powerful new analytic approaches, sample sizes may not always be sufficiently large to deal with the increase in model complexity. This poses a difficult modeling scenario that entails large models with a comparably limited number of observations given the number of parameters. We here describe a particular strategy to overcoming this challenge, called. Regularization, a method to penalize model complexity during estimation, has proven a viable option for estimating parameters in this small n, large p setting, but has so far mostly been used in linear regression models. Here we show how to integrate regularization within structural equation models, a popular analytic approach in psychology. We first describe the rationale behind regularization in regression contexts, and how it can be extended to regularized structural equation modeling (Jacobucci, Grimm, & McArdle, 2016). Our approach is evaluated through the use of a simulation study, showing that regularized SEM outperforms traditional SEM estimation methods in situations with a large number of predictors and small sample size. We illustrate the power of this approach in two empirical examples: modeling the neural determinants of visual short term memory, as well as identifying demographic correlates of stress, anxiety and depression. We illustrate the performance of the method and discuss practical aspects of modeling empirical data, and provide a step-by-step online tutorial.
|
| [40] |
|
| [42] |
|
| [43] |
The problem of penalized maximum likelihood (PML) for an exploratory factor analysis (EFA) model is studied in this paper. An EFA model is typically estimated using maximum likelihood and then the estimated loading matrix is rotated to obtain a sparse representation. Penalized maximum likelihood simultaneously fits the EFA model and produces a sparse loading matrix. To overcome some of the computational drawbacks of PML, an approximation to PML is proposed in this paper. It is further applied to an empirical dataset for illustration. A simulation study shows that the approximation naturally produces a sparse loading matrix and more accurately estimates the factor loadings and the covariance matrix, in the sense of having a lower mean squared error than factor rotations, under various conditions.
|
| [44] |
|
| [45] |
Traditionally, two distinct approaches have been employed for exploratory factor analysis: maximum likelihood factor analysis and principal component analysis. A third alternative, called regularized exploratory factor analysis, was introduced recently in the psychometric literature. Small sample size is an important issue that has received considerable discussion in the factor analysis literature. However, little is known about the differential performance of these three approaches to exploratory factor analysis in a small sample size scenario. A simulation study and an empirical example demonstrate that regularized exploratory factor analysis may be recommended over the two traditional approaches, particularly when sample sizes are small (below 50) and the sample covariance matrix is near singular.
|
| [46] |
|
| [47] |
|
| [51] |
|
| [52] |
|
| [53] |
|
| [54] |
|
| [55] |
|
| [56] |
Scalar invariance is an unachievable ideal that in practice can only be approximated; often using potentially questionable approaches such as partial invariance based on a stepwise selection of parameter estimates with large modification indices. Study 1 demonstrates an extension of the power and flexibility of the alignment approach for comparing latent factor means in large-scale studies (30 OECD countries, 8 factors, 44 items, N = 249,840), for which scalar invariance is typically not supported in the traditional confirmatory factor analysis approach to measurement invariance (CFA-MI). Importantly, we introduce an alignment-within-CFA (AwC) approach, transforming alignment from a largely exploratory tool into a confirmatory tool, and enabling analyses that previously have not been possible with alignment (testing the invariance of uniquenesses and factor variances/covariances; multiple-group MIMIC models; contrasts on latent means) and structural equation models more generally. Specifically, it also allowed a comparison of gender differences in a 30-country MIMIC AwC (i.e., a SEM with gender as a covariate) and a 60-group AwC CFA (i.e., 30 countries × 2 genders) analysis. Study 2, a simulation study following up issues raised in Study 1, showed that latent means were more accurately estimated with alignment than with the scalar CFA-MI, and particularly with partial invariance scalar models based on the heavily criticized stepwise selection strategy. In summary, alignment augmented by AwC provides applied researchers from diverse disciplines considerable flexibility to address substantively important issues when the traditional CFA-MI scalar model does not fit the data. (PsycINFO Database Record(c) 2018 APA, all rights reserved).
|
| [57] |
|
| [58] |
Ordinary least squares and stepwise selection are widespread in behavioral science research; however, these methods are well known to encounter overfitting problems such that R(2) and regression coefficients may be inflated while standard errors and p values may be deflated, ultimately reducing both the parsimony of the model and the generalizability of conclusions. More optimal methods for selecting predictors and estimating regression coefficients such as regularization methods (e.g., Lasso) have existed for decades, are widely implemented in other disciplines, and are available in mainstream software, yet, these methods are essentially invisible in the behavioral science literature while the use of sub optimal methods continues to proliferate. This paper discusses potential issues with standard statistical models, provides an introduction to regularization with specific details on both Lasso and its related predecessor ridge regression, provides an example analysis and code for running a Lasso analysis in R and SAS, and discusses limitations and related methods.
|
| [59] |
|
| [60] |
|
| [61] |
This article proposes a new approach to factor analysis and structural equation modeling using Bayesian analysis. The new approach replaces parameter specifications of exact zeros with approximate zeros based on informative, small-variance priors. It is argued that this produces an analysis that better reflects substantive theories. The proposed Bayesian approach is particularly beneficial in applications where parameters are added to a conventional model such that a nonidentified model is obtained if maximum-likelihood estimation is applied. This approach is useful for measurement aspects of latent variable modeling, such as with confirmatory factor analysis, and the measurement part of structural equation modeling. Two application areas are studied, cross-loadings and residual correlations in confirmatory factor analysis. An example using a full structural equation model is also presented, showing an efficient way to find model misspecification. The approach encompasses 3 elements: model testing using posterior predictive checking, model estimation, and model modification. Monte Carlo simulations and real data are analyzed using Mplus. The real-data analyses use data from Holzinger and Swineford's (1939) classic mental abilities study, Big Five personality factor data from a British survey, and science achievement data from the National Educational Longitudinal Study of 1988.
|
| [62] |
|
| [63] |
|
| [64] |
As a commonly used tool for operationalizing measurement models, confirmatory factor analysis (CFA) requires strong assumptions that can lead to a poor fit of the model to real data. The post hoc modification model approach attempts to improve CFA fit through the use of modification indexes for identifying significant correlated residual error terms. We analyzed a 28-item emotion measure collected for n = 175 participants. The post hoc modification approach indicated that 90 item-pair errors were significantly correlated, which demonstrated the challenge in using a modification index, as the error terms must be individually modified as a sequence. Additionally, the post hoc modification approach cannot guarantee a positive definite covariance matrix for the error terms. We propose a method that enables the entire inverse residual covariance matrix to be modeled as a sparse positive definite matrix that contains only a few off-diagonal elements bounded away from zero. This method circumvents the problem of having to handle correlated residual terms sequentially. By assigning a Lasso prior to the inverse covariance matrix, this Bayesian method achieves model parsimony as well as an identifiable model. Both simulated and real data sets were analyzed to evaluate the validity, robustness, and practical usefulness of the proposed procedure. (PsycINFO Database Record(c) 2017 APA, all rights reserved).
|
| [65] |
Ansari et al. (Psychometrika 67:49-77, 2002) applied a multilevel heterogeneous model for confirmatory factor analysis to repeated measurements on individuals. While the mean and factor loadings in this model vary across individuals, its factor structure is invariant. Allowing the individual-level residuals to be correlated is an important means to alleviate the restriction imposed by configural invariance. We relax the diagonality assumption of residual covariance matrix and estimate it using a formal Bayesian Lasso method. The approach improves goodness of fit and avoids ad hoc one-at-a-time manipulation of entries in the covariance matrix via modification indexes. We illustrate the approach using simulation studies and real data from an ecological momentary assessment.
|
| [66] |
|
| [67] |
|
| [68] |
|
| [69] |
Measurement invariance assesses the psychometric equivalence of a construct across groups or across time. Measurement noninvariance suggests that a construct has a different structure or meaning to different groups or on different measurement occasions in the same group, and so the construct cannot be meaningfully tested or construed across groups or across time. Hence, prior to testing mean differences across groups or measurement occasions (e.g., boys and girls, pretest and posttest), or differential relations of the construct across groups, it is essential to assess the invariance of the construct. Conventions and reporting on measurement invariance are still in flux, and researchers are often left with limited understanding and inconsistent advice. Measurement invariance is tested and established in different steps. This report surveys the state of measurement invariance testing and reporting, and details the results of a literature review of studies that tested invariance. Most tests of measurement invariance include configural, metric, and scalar steps; a residual invariance step is reported for fewer tests. Alternative fit indices (AFIs) are reported as model fit criteria for the vast majority of tests; χ is reported as the single index in a minority of invariance tests. Reporting AFIs is associated with higher levels of achieved invariance. Partial invariance is reported for about one-third of tests. In general, sample size, number of groups compared, and model size are unrelated to the level of invariance achieved. Implications for the future of measurement invariance testing, reporting, and best practices are discussed.
|
| [70] |
|
| [71] |
|
| [72] |
|
| [73] |
|
| [74] |
|
| [75] |
|
| [76] |
|
| [77] |
An important challenge in statistical modeling is to balance how well our model explains the phenomenon under investigation with the parsimony of this explanation. In structural equation modeling (SEM), penalization approaches that add a penalty term to the estimation procedure have been proposed to achieve this balance. An alternative to the classical penalization approach is Bayesian regularized SEM in which the prior distribution serves as the penalty function. Many different shrinkage priors exist, enabling great flexibility in terms of shrinkage behavior. As a result, different types of shrinkage priors have been proposed for use in a wide variety of SEMs. However, the lack of a general framework and the technical details of these shrinkage methods can make it difficult for researchers outside the field of (Bayesian) regularized SEM to understand and apply these methods in their own work. Therefore, the aim of this paper is to provide an overview of Bayesian regularized SEM, with a focus on the types of SEMs in which Bayesian regularization has been applied as well as available software implementations. Through an empirical example, various open-source software packages for (Bayesian) regularized SEM are illustrated and all code is made available online to aid researchers in applying these methods. Finally, reviewing the current capabilities and constraints of Bayesian regularized SEM identifies several directions for future research.
|
| [78] |
|
| [79] |
|
| [80] |
|
| [81] |
Model misspecifications may have a systematic effect on parameters, causing biases in their estimates. In the application of structural equation models, every interesting model is fallible. When simultaneously evaluating a model, it is of interest to study whether all parameters are affected by a misspecification. This paper provides three procedures for evaluating such an effect: (1) analyzing the path, (2) using a functional relationship, and (3) using a significance test. Analyzing the path is illustrated through a confirmatory factor model. This method is ad hoc but intuitive. A more rigorous approach is built upon the concept of orthogonality of two sets of parameters. When parameter a is orthogonal to parameter b, omitting parameter b will not affect the estimation of parameter a. The functional relationship of two sets of parameters is used to check their orthogonality. The distribution of the difference between estimates based on different models is obtained, which provides a Hausman-like way to check significant parameter differences that are due to biases. Examples illustrate that these procedures can provide valuable information on identifying parameter estimates that are systematically affected by a model misspecification.
|
| [82] |
|
| [83] |
|
/
| 〈 |
|
〉 |
