When one fact changes

A language model stores what was true when it was trained. Facts then change, and an update that corrects one statement while leaving the statements built on it untouched leaves the model contradicting itself. A superseded software version, a revised standard or a repealed rule each leaves dependents behind: other facts whose answer should change too.

The tests described below ask whether the edited fact was learned and whether its consequences follow, often along a chain of reasoning. Breadth is a different axis: how many facts depend on one change. This inquiry isolates it, on a public record of superseded facts, and asks what happens to each editing method as the count rises.

In people, a retraction can be remembered and still leave later inferences resting on the retracted statement^[1]⁠^[2]; this inquiry tests the model side only.

Editing methods change a model's answer to a particular question, usually by altering weights or adding a small trained adapter, without retraining from scratch. Yao and colleagues survey the methods and the criteria used to judge them^[3]. Early criteria asked whether the edit took and whether unrelated facts were spared. Later work asked about consequences.

Cohen and colleagues built RippleEdits, 5,000 factual edits scored on the logical consequences of each edit. Prominent methods failed to make the related changes consistently, and a simple in-context baseline scored best^[4]. The paper calls the number of facts an edit affects its severity. As read for this page, it uses the popularity of the edited entity as a stand-in for that count and does not report results by the count itself.

Zhong and colleagues' MQuAKE poses multi-hop questions that depend on an edited fact. Methods recalled the edit but did poorly on the questions that follow from it, while a method that stores edits outside the model and prompts it did better^[5]. Hua and colleagues tested input augmentation, fine-tuning and locate-and-edit methods across six reasoning schemes and report low performance for all of them, most of all in some schemes^[6]. Luo and colleagues generate controlled interventions at several levels of a knowledge graph and several data scales, and study propagation and consistency^[7].

These studies vary chain length, reasoning scheme or graph level. None of the sources read for this page reports performance against the number of facts that depend on one change. The search is repeated at the corpus gate.

The inquiry compares weight-based editing methods on one question. As the number of facts that depend on a changed fact grows, does the model still update them, and does it begin to change facts that should stay as they are?

Material. A public record of superseded items with a machine-readable map of what depends on what, under a licence that permits publishing derived material. Candidate kinds include software deprecations and standards supersessions; none is chosen. Events are grouped on a ladder by number of dependents: 1, 2, 4 and 8 or more. At least 150 events with three or more dependents are required, confirmed by hand count.

Methods. Two weight-based methods, one direct edit and one adapter. Each runs at two stated update budgets, on two open-weight base models, over several random seeds. Budgets, floors and the ladder are written down before any model output is read.

Controls. Two sets of facts that should not change: facts that look like a dependent, and facts that sit near the edit in the dependency graph.

Ground truth. The dependency map proposes the dependents. Two raters, one from outside the lab, audit a 100-item sample of them on whether the answer must change, plus a second sample drawn without the graph to check what the graph misses. The audit's error rate is reported with every propagation figure.

Measures. At each rung, the share of dependents updated correctly and the share of control facts changed wrongly. For each method and budget, the result is the number of dependents above which propagation falls below the floor.

Order. Three gates come in sequence: the corpus gate (count and licence), the labelling gate (rater agreement) and the pilot gate (a small pilot of both methods). A failed gate ends the inquiry.

Three conditions are fixed before anything runs. The pilot gate is a fourth; its pass condition will be written down with the budgets and floors before any model output is read.

1. Corpus gate. One named public record holds at least 150 events with three or more dependents by direct count, and its licence permits publishing derived material. If either fails, the inquiry stops. The literature search is repeated at this gate: if a study already grades propagation by number of dependents, the inquiry stops or is narrowed to a stated difference from it.
2. Labelling gate. Two raters reach Cohen's kappa, a measure of agreement corrected for chance, of 0.70 or better on “must this answer change” over the 100-item sample. If not, the inquiry stops, because the labels cannot carry the result.
3. Result. For each method and budget, propagation to dependents and wrong changes on both control sets are reported across the ladder. Propagation falling with the number of dependents, by a margin and below a floor both written in advance, is the positive answer. Each method and budget is classified on its own. No such fall, with wrong changes not rising, is the negative answer for that method and budget. No such fall, with wrong changes rising, will be reported as a third outcome: damage to facts that should stay as they are, without a propagation limit. Inconclusive: for a method and budget whose interval on the propagation fall includes both no fall and the margin written in advance, the result is reported as inconclusive, not as a positive or a negative answer. The negative answer for the inquiry as a whole requires the negative answer for every method and budget.

A fall in propagation as dependents increase could come from something that rises with the count rather than from the count. Events with many dependents may concern more prominent items, and RippleEdits used the popularity of an entity as a stand-in for how many facts an edit affects^[4]; each rung will therefore be reported with the frequency of its items in public text as a covariate. High-count events may also reach their dependents through longer chains, and questions that need several hops are already hard for edited models^[5], so propagation will be reported by hop distance within each rung. General damage can pass for a propagation limit. Editing many facts in sequence makes a model forget earlier edits and lose ability on other tasks^[8], so each event will be edited from a fresh copy of the base model, and the two control sets will show whether wrong changes rise alongside. Labels may also be harder to agree on at high counts, so rater agreement will be reported for each rung. A flat curve has its own false explanation. Models trained on a snapshot of text hold facts that expire^[9], and a base model may already know the newer state from pretraining. Each dependent will therefore be put to the base model before the edit, and those it already answers with the new value will be reported apart.

A pass would show how two named methods behave at two named budgets, on two base models, on one record. It would not show a limit on knowledge editing in general. It would not cover methods that keep changed facts outside the weights, such as an external store or an edit placed in the prompt, which did well in some of the work above^[4]⁠^[5].

One record has one dependency structure, and another record may give a different curve. The labelling check covers a sample, not every dependent. A public record may already sit in the training data of the base models.

Nothing here tests people. The comparison with human memory is the reason for asking, and it stays a question. Whether a memory can anticipate what will break it is taken up in a separate inquiry.