Why it matters
A world model is a learned simulator: given a state and an action, it predicts what follows. An agent can be trained inside one, rehearsing in the model's imagined world and carrying the policy back to the real environment[1]. Related designs now include reinforcement-learning systems that learn a model of their environment and improve by imagining future scenarios[2], generative environments that a user can act in frame by frame from a text or image prompt[3], and a neural model that runs an interactive game in real time[4]. In robotics, world models are increasingly used to evaluate policies by simulating the consequences of their actions and returning a success or safety verdict[5].
Each of these uses asks the model to stand in for a world. A verdict from a simulator is only as good as the simulator, and the studies below show how weak some current checks of a learned simulator are. This area works on better ones.
What is hard
In video world models, fidelity metrics such as Fréchet Video Distance reward visual realism and ignore whether the world responds correctly to an action; on two driving models, the one ranked higher for visual quality ranked lower on the paper's action-following checks[5]. A benchmark of physical principles found physical understanding in current video models severely limited and unrelated to visual realism[6]. Video models trained on scenes governed by classical mechanics generalised within their training distribution and failed outside it, imitating the closest training case instead of abstracting the rule[7].
Language models raise the same question in another form. Where the true world is a finite automaton, generative models can pass the usual diagnostics while holding a world model far less coherent than it appears, and the incoherence shows up as failure on related tasks[8].
Consistency over time is a second difficulty. A generated world has to keep what has gone out of view and return it intact on a later visit. A limited context window makes that hard, and explicit memory is one current response[9].
The third difficulty is the reference. Simulation science treats validity as relative to a purpose: a model may be valid under one set of experimental conditions and invalid under another, and the accuracy required should be fixed before the model is built[10]. By that standard, a learned world model needs a stated domain of use, and checking it against that domain needs a reference that holds the facts of the world it imitates.
Current directions
- Consistency against a written reference. Tests built from a world whose facts are recorded in advance, so that persistence, occlusion, irreversibility and counterfactual consequence can be checked by machine instead of judged by eye.
- What a policy acts to keep steady. Information that can be decoded from a network is not thereby used by it[11]. Control theory offers a different test: disturb a system and record what it resists, a method developed to identify the variables an organism controls[12].
- Validity as a domain of use. Results stated as the conditions under which a model holds, with error bounds, in place of a single score.
How the lab approaches it
Two inquiries carry the area. A hand-written world as a test for world models plans to record the facts of a hand-authored reference world in advance, in a ledger of entities, relations and dated events, to generate questions from it in five classes, and to ask whether those questions separate a competent world model from a deliberately weak one where a fidelity score does not. What a learned policy holds still plans to apply graded disturbances to a frozen robot policy in simulation and to ask whether the directions it resists predict which held-out conditions it survives, better than a probe on its internal state does.
Both designs require their comparisons, thresholds and stopping conditions to be written down before any data is read, and both state what a pass would not show. Consistency with one written world is not understanding of a world, and resistance to disturbance in simulation is not a goal held by a robot. Both inquiries are written designs; neither has run.
The longer aim is a set of instruments that let a world model be accepted or rejected for a stated use. Such instruments would matter wherever a learned simulator is asked to judge a policy, train an agent or answer a counterfactual question.
Open problems in this programme
- Does a learned world model keep the state of a place it has generated consistent when an agent leaves and returns many steps later?
- Do the fidelity scores used to rank world models predict whether a model's dynamics respond correctly to actions?
- When a learned world model is used to evaluate robot policies, under what conditions do its success verdicts rank policies in the same order as real-world trials?
- Do the variables a probe can decode from a robot policy's internal state coincide with the variables the policy actively holds steady under disturbance?
References
- Ha, D., Schmidhuber, J. (2018). World Models. arXiv preprint 1803.10122.
- Hafner, D., Pasukonis, J., Ba, J., Lillicrap, T. (2025). Mastering diverse control tasks through world models. Nature 640(8059), 647–653.
- Bruce, J., Dennis, M., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., et al. (2024). Genie: Generative Interactive Environments. arXiv preprint 2402.15391.
- Valevski, D., Leviathan, Y., Arar, M., Fruchter, S. (2025). Diffusion Models Are Real-Time Game Engines. International Conference on Learning Representations (ICLR 2025); arXiv preprint 2408.14837.
- Oefinger, C., Schäfer, F. R., Moller, K., Piccinini, M., Betz, J. (2026). Validate the Dream Before You Trust Its Verdict: Admissibility for World-Model Simulators. arXiv preprint 2607.07196.
- Motamed, S., Culp, L., Swersky, K., Jaini, P., Geirhos, R. (2025). Do generative video models understand physical principles? arXiv preprint 2501.09038.
- Kang, B., Yue, Y., Lu, R., Lin, Z., Zhao, Y., Wang, K., et al. (2025). How Far is Video Generation from World Model: A Physical Law Perspective. International Conference on Machine Learning (ICML 2025); arXiv preprint 2411.02385.
- Vafa, K., Chen, J. Y., Rambachan, A., Kleinberg, J., Mullainathan, S. (2024). Evaluating the World Model Implicit in a Generative Model. arXiv preprint 2406.03689.
- Xiao, Z., Lan, Y., Zhou, Y., Ouyang, W., Yang, S., Zeng, Y., et al. (2025). WorldMem: Long-term Consistent World Simulation with Memory. Advances in Neural Information Processing Systems 38 (NeurIPS 2025).
- Sargent, R. G. (2008). Verification and validation of simulation models. Proceedings of the 2008 Winter Simulation Conference, 157–169.
- Elazar, Y., Ravfogel, S., Jacovi, A., Goldberg, Y. (2021). Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics 9, 160–175.
- Marken, R. S. (2014). Testing for controlled variables: a model-based approach to determining the perceptual basis of behavior. Attention, Perception, & Psychophysics 76(1), 255–263.
