An interviewer that changes the person it studies

Asking is an intervention. A respondent who is asked a question may attend to something new, form an opinion or answer the next question differently. In one study, merely asking people about purchase intent changed their later buying^[1]. An interviewer therefore never observes a fixed object.

A planner that rehearses questions on a simulated person meets the same problem under different rules. Some chat models have been shown to drift from their system instructions within eight rounds of dialogue^[2], so a simulated person can change as a conversation proceeds. If the planner scores a candidate question by how the rehearsal turns out, it can prefer questions that push the simulation somewhere easy to read. The planner then looks good because of the simulator, not because it learned about anyone.

The inquiry asks whether the simulated person's own change can be separated cleanly enough to tell which part of a result belongs to the question and which to the simulator.

Three bodies of published work bear on the question.

Planning by rehearsal. SparkMe treats LLM interviewing as an optimisation problem and simulates possible conversation paths before choosing the next question; it reports better coverage of a topic guide than earlier systems^[3]. STaR-GATE trains a question-asking model against simulated people whose preferences are hidden from it^[4]. Choosing questions against a simulated person has precedent.

Drift in simulated people. Chat models can lose adherence to their instructions over a dialogue^[2]. CustomerSim reports that models playing shoppers drift from assigned personas and can be swayed by a salesperson's tone, and proposes UserGRPO, a training method that raises persona adherence^[5]. That is the closest rival to the design here: it keeps the person stable by training the simulator.

Drift under intervention. Lin and colleagues argue that an experiment on simulated users is observational if the treatment shifts the implied population between conditions. They propose negative control outcomes, attributes that should not change, to detect the shift, and extra persona detail to reduce it^[6]. Liu and He address a related weakness, simulated users that are too cooperative, with a disclosure gate that releases information according to the agent's behaviour^[7].

The design compares four ways of holding a simulated interviewee, on a fixed set of synthetic profiles.

Fixtures. 100 synthetic profiles, each with hidden attributes, a menu of questions drawn from at least six topic families, and written rules for what a question may reveal, what it may teach the person and what it may only suggest. At least 80 must qualify at the qualification gate. Qualification includes a leakage check: if hidden attributes can be read off the visible profile, the questions do not matter and the profile is unusable.

Arms. (1) Free-form rollouts. (2) Rollouts in which fixed attributes, permitted learning and revealed answers are held as separate state. (3) Targeted persona augmentation, of the kind Lin and colleagues propose^[6]. (4) A persona-aligned simulator rebuilt from the published description of CustomerSim and UserGRPO^[5]. The arms run at matched model-call cost.

Measure. A separate process holds the hidden attributes and scores how many of them each arm's chosen questions recover within a fixed question budget. The planner never sees them. The pre-registration will set the size of the budget, the same for every arm. Violations of negative-control attributes are reported as a diagnostic only, because an arm that reveals nothing would score perfectly on them.

Order. The qualification gate and the written margin come first, then the build of the four arms, then the freeze of the question families, so that nothing is tuned to them afterwards, then scoring. The design allows four weeks for these steps. No human data is involved.

Written before any scoring.

The inquiry passes only if the state-separated arm recovers more hidden attributes than the persona-aligned arm on the frozen families, by a margin fixed in writing before the freeze, under a one-sided test at the 0.05 level whose form and unit of resampling (profiles or question families) the pre-registration will name.

It stops, and no further comparison is sought, if any of these hold:

• at the qualification gate, fewer than 80 of the 100 profiles qualify, or the leakage check fails;
• on the pilot question families, an oracle that knows the hidden attributes recovers fewer than five percentage points more of the hidden attributes than the persona-aligned arm;
• the state-separated arm does not clear the margin over the persona-aligned arm. A tie, or a lead below the margin, counts against the idea on this fixture and stops the inquiry.

A result whose interval includes both no effect and the margin is inconclusive and is reported as such.

If the targeted-persona arm matches or beats the state-separated arm, the result is reported as ‘persona detail suffices’ and the separated design is not pursued.

A result is also rejected if one arm is denied learning that another is allowed.

The pre-registration will set the margin at the qualification gate, from the spread of recovery across the pilot question families and before the question families are frozen. It will not be chosen after the scores are seen.

An apparent win for the state-separated arm has several cheaper explanations, and the design is meant to remove most of them. Hidden attributes may leak through the visible profile, so recovery reflects reading rather than asking; the leakage check at the qualification gate will remove such profiles before any arm runs. The arm may win by blocking learning the rules allow, freezing the person so that answers stay easy to read; every arm will run under the same reveal, teach and suggest rules, and a result in which one arm is denied learning another is allowed will be rejected. It may win on compute; the arms will run at matched model-call cost. The scorer may favour it; scoring will run in a separate process that holds the hidden attributes and that no planner reads. Choices may be tuned to the test; question families will be frozen before scoring, and the margin will be fixed at the qualification gate from the pilot-family spread, before the freeze. The comparator may be weak: a loss for the persona-aligned arm can reflect the lab's rebuild rather than the method^[5], which is why the lab asks for an independent reimplementation to check the rebuild. Negative-control attributes, the drift check Lin and colleagues propose^[6], will be reported only as a diagnostic, because an arm that reveals nothing would score perfectly on them. One explanation the design reduces but cannot remove: the fixtures will be written in the same three-channel terms the treatment uses, so the treatment may fit its own fixture. A second, independently written rule set, which the lab also asks for, would be the check for that.

A pass would show an effect on synthetic profiles, scored against attributes that were authored by hand. Real respondents' traits are less tidy. The inquiry says nothing about how real people change when asked, or whether the simulated change resembles it. A ranking of planners on this fixture is not a ranking on people; testing that needs human interviews under a separate consent design, which this inquiry does not include.

The persona-aligned comparator is a rebuild from a paper, not the authors' system, so a loss for it may reflect the rebuild. Attribute recovery within a budget is also only one aim of an interview. It does not measure whether a planner surfaces themes nobody planned for, which SparkMe is designed to do^[3]. Six or so topic families in one interview setting do not support a general claim about simulated people.