PDF(704 KB)
Simulating Human Participants with Large Language Models: Principles, Limitations, and Recommendations
Xie Tian, Qiu Lin
Journal of Psychological Science ›› 2026, Vol. 49 ›› Issue (4) : 770-782.
PDF(704 KB)
PDF(704 KB)
Simulating Human Participants with Large Language Models: Principles, Limitations, and Recommendations
The rapid advancement of Large Language Models (LLMs) is catalyzing a profound transformation in psychological research. This review advances a dual-view framework for critically assessing LLMs: as powerful simulators of human participants and as novel non-human agents for scientific inquiry. As simulators, LLMs demonstrate considerable potential in replicating human responses. Empirical studies have shown that models can mimic human cognitive biases in decision-making tasks, achieve performance on the Theory of Mind (ToM) tasks comparable to that of young children, and successfully replicate a significant portion of main and interaction effects from classic psychology experiments. However, this paper moves beyond cataloging these successes to deconstruct the simulator concept itself, revealing two fundamental, principle-level limitations. First, the model's core compression principle, while effective at capturing group averages, inherently struggles to replicate the internal structural associations of individual differences. Second, a profound asymmetry exists between the model's training phase (knowledge acquisition) and its alignment phase (strategic expression), which fundamentally determines that any simulation is a strategically optimized expression rather than a faithful representation of the model's internal knowledge state. Consequently, their inherent limitations and systematic biases - such as reflecting a predominantly Western, Educated, Industrialized, Rich, and Democratic (WEIRD) perspective - are not merely data-level flaws, but deep-seated mechanistic properties. This transforms them into valuable data for investigating the mechanisms of non-human intelligence. This critical perspective aligns with recent scholarly calls to systematically analyze the theoretical fallacies, such as anthropomorphism and identity essentialization, that arise from uncritically substituting LLMs for human participants.
Within this theoretical framework, the paper systematically organizes the current methodologies into two primary technical paths, distinguished by their depth of intervention. The first is the path of prompt engineering, which guides the model's output without altering its internal parameters. This path encompasses techniques for individual-agent simulation, where methods like simple prompts are used to assign specific personas to the model, as well as techniques for group-level simulation, where in-context learning allows the model to predict aggregate public opinion by generalizing from a few examples. The second, more intensive path is that of model fine-tuning, which directly modifies the model's parameters by retraining it on large-scale, individual-level datasets to enhance its predictive accuracy for specific populations. To rigorously assess the outputs from these methods, the review scrutinizes two key evaluation frameworks. The first, algorithmic fidelity, evaluates the degree to which an LLM can mirror the complex relationships between thoughts, attitudes, and socio-cultural backgrounds within specific human subpopulations. The second, the Turing Experiment, assesses an LLM's ability to replicate the behavioral outcomes of human subjects in classic social science experiments, focusing on functional equivalence in contexts like the Ultimatum Game or Milgram's obedience studies, while also critiquing the fallacy of “perfect alignment” as a goal.
Finally, this review provides crucial suggestions for leveraging LLMs in psychological research. It strongly emphasizes that LLMs cannot replace human participants due to fundamental limitations, such as the lack of embodied cognition, subjective experience, and genuine understanding. This is particularly evident in their inability to natively generate non-textual data crucial for many fields, such as reaction times in cognitive psychology or physiological indicators in neuroscience. Instead, the paper argues that future research should proceed along dual paths: (1) Deepening the role of LLMs as a tool and (2) Pioneering the study of LLMs as a subject. As a tool, LLMs can serve as invaluable simulators for hypothesis generation, pre-testing study materials, and conducting exploratory research on sensitive topics where human participation poses ethical risks. As a subject, researchers can investigate the LLM itself as a novel agent. For instance, they can systematical map the ideological spectrums of different models, analyze their "hallucinations" as a form of computational creativity, or design "implicit association tests" to uncover their deep-seated biases. This dual-path approach ensures that this powerful technology is leveraged rigorously and responsibly for knowledge discovery. Ultimately, it highlights the irreplaceable value of human researchers in proposing profound questions, interpreting complexity, and defending scientific integrity.
large language models (LLMs) / simulation / human participants / non-human agents / research automation
| [1] |
谢天, 邱林, 李雨曈, 罗殷, 刘盼. (2023). 大模型时代的社会科学, 何去何从?图书情报知识, 40(6), 6-9, 30.
|
| [2] |
|
| [3] |
|
| [4] |
We propose and explore the possibility that language models can be studied as effective proxies for specific human subpopulations in social science research. Practical and research applications of artificial intelligence tools have sometimes been limited by problematic biases (such as racism or sexism), which are often treated as uniform properties of the models. We show that the “algorithmic bias” within one such tool—the GPT-3 language model—is instead both fine-grained and demographically correlated, meaning that proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups. We term this propertyalgorithmic fidelityand explore its extent in GPT-3. We create “silicon samples” by conditioning the model on thousands of sociodemographic backstories from real human participants in multiple large surveys conducted in the United States. We then compare the silicon and human samples to demonstrate that the information contained in GPT-3 goes far beyond surface similarity. It is nuanced, multifaceted, and reflects the complex interplay between ideas, attitudes, and sociocultural context that characterize human attitudes. We suggest that language models with sufficient algorithmic fidelity thus constitute a novel and powerful tool to advance understanding of humans and society across a variety of disciplines.
|
| [5] |
|
| [6] |
|
| [7] |
|
| [8] |
|
| [9] |
Large language models (LLMs) offer new research possibilities for social scientists, but their potential as “synthetic data” is still largely unknown. In this paper, we investigate how accurately the popular LLM ChatGPT can recover public opinion, prompting the LLM to adopt different “personas” and then provide feeling thermometer scores for 11 sociopolitical groups. The average scores generated by ChatGPT correspond closely to the averages in our baseline survey, the 2016–2020 American National Election Study (ANES). Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data. We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a 3-month period. Altogether, our findings raise serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.
|
| [10] |
|
| [11] |
|
| [12] |
|
| [13] |
|
| [14] |
|
| [15] |
Recent work suggests that language models such as GPT can make human-like judgments across a number of domains. We explore whether and when language models might replace human participants in psychological science. We review nascent research, provide a theoretical model, and outline caveats of using AI as a participant.Copyright © 2023 Elsevier Ltd. All rights reserved.
|
| [16] |
|
| [17] |
|
| [18] |
Careful bias management and data fidelity are key.
|
| [19] |
|
| [20] |
|
| [21] |
|
| [22] |
|
| [23] |
|
| [24] |
The basis of science is the hypothetico-deductive method and the recording of experiments in sufficient detail to enable reproducibility. We report the development of Robot Scientist "Adam," which advances the automation of both. Adam has autonomously generated functional genomics hypotheses about the yeast Saccharomyces cerevisiae and experimentally tested these hypotheses by using laboratory automation. We have confirmed Adam's conclusions through manual experiments. To describe Adam's research, we have developed an ontology and logical language. The resulting formalization involves over 10,000 different research units in a nested treelike structure, 10 levels deep, that relates the 6.6 million biomass measurements to their logical description. This formalization describes how a machine contributed to scientific knowledge.
|
| [25] |
|
| [26] |
|
| [27] |
|
| [28] |
|
| [29] |
|
| [30] |
|
| [31] |
|
| [32] |
|
| [33] |
|
| [34] |
|
| [35] |
|
| [36] |
|
| [37] |
|
| [38] |
|
| [39] |
|
| [40] |
|
| [41] |
|
| [42] |
|
| [43] |
In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤.05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
|
| [44] |
|
| [45] |
|
| [46] |
|
| [47] |
|
| [48] |
|
| [49] |
|
| [50] |
|
| [51] |
In what sense does a large language model (LLM) have knowledge? We answer by granting LLMs 'instrumental knowledge': knowledge gained by using next-word generation as an instrument. We then ask how instrumental knowledge is related to the ordinary, 'worldly knowledge' exhibited by humans, and explore this question in terms of the degree to which instrumental knowledge can be said to incorporate the structured world models of cognitive science. We discuss ways LLMs could recover degrees of worldly knowledge and suggest that such recovery will be governed by an implicit, resource-rational tradeoff between world models and tasks. Our answer to this question extends beyond the capabilities of a particular AI system and challenges assumptions about the nature of knowledge and intelligence.Copyright © 2024 Elsevier Ltd. All rights reserved.
|
| [52] |
|
| [53] |
|
/
| 〈 |
|
〉 |