Far apart, possibly akin

One problem in three places

Learning machines, human minds and simulated worlds are mostly studied in separate fields, by people with different training and different journals. They share an awkward property. In each, the part that does the work cannot be opened and read. The weights and activations of a trained network can all be recorded, but what they encode cannot be read off by inspection. A person offers reports, and a report is not the process behind it. The rules of a simulated world reach an agent inside it only as consequences.

The pairing is not new. Wiener's Cybernetics (1948) carried the subtitle Control and Communication in the Animal and the Machine^[1], which put the animal and the machine under one theory of control and communication. Five years earlier, Kenneth Craik had argued that an organism carries a “small-scale model” of external reality and of its own possible actions, and uses it to try options before acting^[2]. Tolman inferred a map in rats from where they ran when a familiar route was blocked^[3]. Ha and Schmidhuber (2018) trained an agent inside a generative model of its environment and moved the resulting policy back into the real one^[4]. The vocabulary changes across the decades and the structure recurs: a system holds a model of its world and acts on what the model predicts.

That a description recurs suggests that the description is natural. It does not show that the things described are the same.

What “akin” would have to mean

The lab's name states a relation and leaves it unproved. To be a claim and not a mood, the relation needs content. Three kinds of content are available, and each can fail.

Shared mechanism. The systems compute the same thing in the same way. Conant and Ashby showed that every good regulator of a system must be a model of that system^[5], a result about regulators whatever they are made of. Whether a given brain or a given trained network meets the conditions of the proof is an empirical matter. Marr's three levels mark where such claims go wrong: the problem a system solves, the representation and algorithm that solve it, and the hardware that runs them^[6]. Two systems can be akin in what they compute and unlike in how. A kinship claim has to name its level, and a claim made at the first level fails if it is read at the second.

Shared method. The same test, applied to both, returns answers that mean the same thing. This fails when a test is valid for one system and empty for the other, for instance when it presumes a quantity the second system does not defend.

Shared failure. The systems break under the same conditions for the same reason. This fails when the breaks look alike and have different causes. One candidate is already in the literature. A probe trained to read a variable out of a network's activations may have learned the task itself, which is why probes need control tasks^[7]. Decoding a variable from neural activity invites a related step, from “a reader can extract it” to “the system represents it”, and a philosophical analysis argues that decodability is a poor guide to what is represented^[8]. Whether these are one problem or two resemblances with different causes is the kind of question the name asks.

At a fixed level, the claim fails if the resemblance disappears under a stricter control, if the test that works on one system has no counterpart in the other, or if the failures share a name and not a cause.

Two interiors, no assumed equivalence

Human inner life and the hidden state of a machine are both treated as serious territories. Neither is a metaphor for the other. A person has first-person access, a developmental history and more than a century of experimental constraint. A network has state that can in principle be read exactly and altered at will, and no first-person report whose standing is settled. These differences inform the comparison as much as any similarity does.

The stance is symmetric in seriousness and silent on equivalence. It does not assume that a model has an inner life, that a mind is a computation of the kind a network performs, or that either is the other under another name. It assumes that each interior has structure that outside evidence bears on, and that likeness is measured case by case, with the cases where it fails kept on record.

Knowing an interior from outside

Neither interior can be inspected directly, so the evidence comes from outside. Behaviour, probes and disturbances recur across the fields above.

Behaviour is the oldest. In the experiment Tolman reports, rats trained to run down one alley found it blocked and faced a fan of new paths; the preponderant choice was the path pointing towards where the food had been^[3]. That separates a stored route from a stored layout without opening anything. Probes read a variable from hidden state and need controls, as above.

Disturbance comes from control theory. Powers argued that behaviour is the control of input: a system produces whatever output holds a sensed quantity steady against disturbances^[9]. The corollary is a test. If a quantity is defended, pushing on it should draw an opposing output, and a quantity the system ignores should not. A system that holds a quantity steady shows nothing varying in that quantity while undisturbed, so watching it at rest says little about what it holds. A disturbance is the behavioural twin of an intervention on hidden state: both push on the inside and record what holds.

Two inquiries build on these methods. What a learned policy holds still takes the disturbance test to robot policies and asks whether the variables a policy defends predict where it fails better than a probe does. Instruments for a workspace asks whether the measures of a model's supposed internal workspace can be trusted before anything is concluded from them. Both are designed, with written stop rules. Neither has run.

Old questions, new instruments

Some precise questions stalled for want of an instrument. Fechner's Elemente der Psychophysik (1860) gave psychophysics its name and its methods for locating thresholds^[10]. Powers's test needs a system that can be pushed continuously and a record fine enough to show both the push and the answer. Both demand many controlled trials. A human observer tires and learns the experiment. A frozen model can be asked the same question thousands of times with one stimulus varied in exact steps, and a policy in a simulator can be disturbed on every step of a trajectory.

Generated worlds supply the other half. A world model can be graded exactly only against a world whose rules are known, and a world written down in full is such a world. A written world as a test designs that grading and asks whether questions drawn from one written world can tell a strong world model from a weak one; whether a grade carries over to a world someone else wrote needs a second world and is left open. A policy trained inside a model's own rollouts, as in Ha and Schmidhuber's agent^[4], is only as good as what the generated world preserves, and checking what it preserves needs a reference whose rules are known.

None of this is a result. The premise is that unlike systems may share a mechanism, a method or a failure. The test is to state the claim at a level and put it to behaviour, probe and disturbance against a world whose rules are known. A kinship that survives that is worth the name. One that breaks leaves a record of where, which is also an answer.