Why it matters
Language models can be conditioned to answer as members of particular groups. Conditioned on thousands of sociodemographic backstories from real survey participants, one model reproduced response distributions from many human subgroups, a property its authors called algorithmic fidelity[1]. Simulations have replicated classic findings from economic, psycholinguistic and social-psychology experiments, and have also exposed consistent distortions[2]. Agents built around a record of their own experiences produce believable individual and social behaviour in a sandbox town[3]. Agents built from two-hour interviews, structured surveys or both predicted participants' held-out General Social Survey answers at 82 to 86 per cent of the participants' own two-week test-retest consistency[4].
If simulations of this kind hold up, questions about people could be explored before people are asked: an instrument could be drafted and tested, or an interview rehearsed, at little cost and with no burden on respondents. That prospect is why the methods for checking simulated people matter as much as the methods for building them.
What is hard
Agreement at the level of averages is weak evidence. Across four datasets, model predictions tracked each item's average human response closely, but once item means were removed they explained 3.05 per cent of the remaining respondent-specific variation, against a human test-retest benchmark of 53.6 per cent[5]. Synthetic survey responses have shown less variation than real ones, regression coefficients that differ from those estimated on survey data, sensitivity to small changes in prompt wording and different results from the same prompt over three months[6]. Model opinions can be misaligned with those of many demographic groups even when the model is steered towards them[7], and models can misportray and flatten identity groups[8].
Agreement can also be manufactured. Matching published marginals does not discriminate simulating respondents from estimating the population directly, and the format in which answers are elicited can dominate measured fidelity[9]. A model may have seen the survey it is asked to reproduce. Contamination of evaluation data is a recognised problem, and methods exist to detect it at the level of single instances[10].
A simulated person also moves. Chat models have been shown to drift from their system instructions within eight rounds of conversation[11]. In experiments run on simulated users, an intervention can shift the population's latent attributes so that the arms of the experiment no longer hold the same people, which can inflate or hide an effect[12].
Current directions
- Recall against estimate. Measuring how much of an instrument a model has seen, and asking whether agreement with published results rises with that exposure.
- State that holds still. Separating what a simulated person is, what it may learn and what it has revealed, so that a procedure is credited for learning about a person, not for changing one.
- Baselines that bite. Scoring simulations against each item's mean and against the spread between real replications of the same instrument, so that a generic average answer cannot pass as fidelity.
How the lab approaches it
Echo or estimate? plans to use published survey instruments released before and after the training cut-off of the models tested. It would measure each model's exposure to each instrument with a recall probe and ask whether closeness to published results grows with measured exposure, beyond what each item's average answer explains. An interviewer that changes the person it studies would compare four ways of holding a simulated interviewee on synthetic profiles. It asks whether keeping fixed traits, permitted learning and revealed answers in separate channels stops a question planner from favouring questions that work by changing the person.
The stages described here would use published instruments or authored profiles only and involve no human respondents. Their limits follow from that: a result on published surveys or synthetic profiles would say nothing directly about a fresh population or a real interviewee. Both inquiries are written designs, with margins and stopping conditions to be written down before any data is read; neither has run.
The longer aim is a measurement discipline for simulated people: tests that say what a simulation's agreement with human data is evidence of, and what it is not.
Open problems in this programme
- When a language model reproduces published survey results, how much of the agreement comes from having seen those results in training?
- Can a simulated sample reproduce the variance and the relationships between variables found in human survey data, not only the averages?
- How far does a model conditioned on one person's interview or survey answers predict that person's answers to new questions, and for whom does it fail?
- When a simulated person is exposed to an intervention or to a sequence of adaptive questions, does the simulator keep that person's fixed attributes fixed?
References
- Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., Wingate, D. (2023). Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis 31(3), 337–351.
- Aher, G., Arriaga, R. I., Kalai, A. T. (2023). Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies. International Conference on Machine Learning (ICML 2023).
- Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., Bernstein, M. S. (2023). Generative Agents: Interactive Simulacra of Human Behavior. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST 2023), 1–22.
- Park, J. S., Zou, C. Q., Kamphorst, J., Egan, N., Shaw, A., Hill, B. M., et al. (2024). LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals. arXiv preprint 2411.10109.
- Ahn, D., Mao, C., Lee, D. (2026). Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates. arXiv preprint 2608.29455.
- Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., Larson, J. M. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis 32(4), 401–416.
- Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., Hashimoto, T. (2023). Whose Opinions Do Language Models Reflect? arXiv preprint 2303.17548.
- Wang, A., Morgenstern, J., Dickerson, J. P. (2025). Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence 7(3), 400–411.
- Doudkin, A. (2026). Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure. arXiv preprint 2609.07305.
- Golchin, S., Surdeanu, M. (2024). Time Travel in LLMs: Tracing Data Contamination in Large Language Models. International Conference on Learning Representations (ICLR 2024); arXiv preprint 2308.08493.
- Li, K., Liu, T., Bashkansky, N., Bau, D., Viégas, F., Pfister, H., et al. (2024). Measuring and Controlling Instruction (In)Stability in Language Model Dialogs. Conference on Language Modeling (COLM 2024).
- Lin, V., Yun, T., Matarić, M., Canny, J., Gretton, A., D'Amour, A. (2026). The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study. arXiv preprint 2605.20767.
