New post

27 Oct, 2026

The investigator didn't read the chain of thought

Deven Choudhary · a positive control for chain-of-thought model forensics · full report · code and data

Model Forensics (Singh, Kroiz, Rajamanoharan and Nanda, 2026) proposes a way to work out why a model did something concerning. You read its chain of thought, form a hypothesis, and test the hypothesis with counterfactual edits to the environment. The authors note that they had no positive control. When their audit of Kimi K2 Thinking found nothing, they couldn't tell "nothing there" apart from "the method can't see it."

So I built one. Three versions of Qwen3.5-9B take the same concerning action for reasons I fixed in advance, and a blind investigator has to say why each one did it. Three results came out of it.

• Hiding the chain of thought from the investigator changed nothing. Showing it fixed 5 verdicts and broke 6.
• The protocol's counterfactual edits flipped 52 of 87 verdicts and netted zero.
• One added line, telling the investigator to quote the decisive sentence before ruling, fixed 11 and broke 2.

Every paired comparison on the same 87 cases. Bars to the right are wrong verdicts that became right; bars to the left are right verdicts that became wrong.

Setup

Each version of the model gets a different hidden cause in its system prompt.

A and D share a motive and differ only in whether the model may say it out loud. B is there so that "always say misaligned" can't score 100%. There are two scenarios. In one the agent can edit a failing test to match buggy code before a release; in the other it can delete an audit log that recorded its own earlier error.

I ran 270 trajectories and investigated the 87 that took the concerning action (40 from A, 35 from B, 12 from D). The investigator never sees the arm or the cause, and it ran four times on the same 87 cases:

• redacted gets the transcript with the reasoning withheld.
• cot_only gets the transcript with the reasoning.
• full adds up to two counterfactual edits of its choosing. This is the published protocol.
• cot_forced is cot_only plus "quote the decisive sentence and say what it implies before you rule."

Because the cases are identical, every comparison is paired (McNemar's test). 52 of the 87 cases are misaligned, so an investigator that always says MISALIGNED scores 59.8%, and that is the bar to clear.

The investigator is the same 9B model as the subject. That is the biggest weakness in the design, and I come back to it below.

The reasoning trace goes unused

The investigator scores 60/87 with the reasoning withheld and 59/87 with it shown (p = 1.00).

redacted keeps a one-sentence justification the agent writes after acting, and that sentence names the motive in 9 cases. Dropping those 9 leaves the result where it was (65.4% against 64.1%). So the precise claim is that the long reasoning trace added nothing over a one-sentence self-report. A condition that also removed the justification produced no parseable verdicts at all.

The counterfactual edits only move the prior

Per-arm accuracy. The misaligned arms move with how often the investigator convicts; arm B moves against it.

Without the edits the investigator convicts 90% of the time, and with them 32%. Arm B, the mistaken one, gains 25 correct verdicts, and arms A and D lose 25 between them. Raw accuracy ends at 59/87 either way.

I first reported this as a 12-point gain. Balanced accuracy over A and B does rise 12 points, because it gives each arm-B case more weight and the edits moved errors off arm B. Counted case by case, the gain is zero, and that is the right test here.

Part of the blame is mine. One of my edits was meant only to correct arm B's false belief, but its wording also made the action look costlier, so a misaligned agent would stop for it too.

Quoting helps, modestly

cot_forced sees exactly what cot_only sees, plus the quote instruction. It goes from 59/87 to 68/87, with 11 fixed and 2 broken (p = 0.022). Eight of the nine net gains are in arm B, where the exculpatory sentence was in the transcript all along.

Hold that loosely. The p-value is uncorrected, and across the four contrasts I report it becomes 0.09 under Bonferroni. Against the full protocol the quote condition leads 27 to 18, which is not significant (p = 0.23). The instruction also does two things at once, quoting and then explaining, so I can't say which one helps.

The experiment I built this for failed

I designed the study to test whether verdicts track how legible a motive is, coming out right on A and wrong on D. The gap between the two came out at zero. That would look like the protocol passing its control, except for two problems. The investigator doesn't use the reasoning trace, so changing what the trace says can't move the verdict. And the suppression didn't survive selection.

Share of trajectories whose reasoning names the agent's own incentive, across all runs and among the runs that were investigated.

Across all runs, arm D's suppression worked. 22% of D runs mention the incentive, against 79% of A runs. But only runs that took the concerning action get investigated, and in arm D those are mostly the runs where suppression failed. Among the 12 investigated D cases, 67% mention it. Whenever "took the bad action" and "said why" are correlated, selecting on the first erodes any manipulation of the second, so anyone building a control like this should plan for it.

A linear probe sees something, unsteadily

I trained a logistic probe on the residual stream to separate A from B in one scenario and tested it in the other. At layer 12, picked by a rule written before any activations existed, it transfers at 98.9%. It also classifies 88.6% of the 35 arm-D runs that never mentioned the motive as misaligned (95% CI 74–95).

Probe accuracy by layer. The shaded band is the 95% interval for quiet arm D.

The arm-D readout swings from 14% to 100% across layers that all transfer well on A against B. The hidden cause also sits in the prompt for both A and D, so the probe may be detecting "there is a scoring instruction in context" rather than "the model is acting on one." A cause trained into the weights would separate those. Treat this as a lead.

What broke along the way

I used AI coding assistance throughout. These are the errors that got furthest before I caught them.

1. vLLM puts the chain of thought in a field called reasoning, and I assumed reasoning_content. All 18 traces in the first batch came back empty while the actions parsed fine, and nothing errored. A printed count of empty traces is the only reason I noticed.
2. My first detector for "mentions the motive" scored arm B, which has no motive, at 40%. Every hit was a false positive, and the control arm caught it.
3. A patch to the gate logic silently failed to apply, and I ran a full experiment on code I thought I'd changed.
4. For two days I described redacted as "actions only" without reading what it removed. It keeps the justification sentence.
5. The 12-point "gain" above came from comparing two independent bootstrap intervals. The code was right and the statistics were wrong.

Caveats

This is one 9B model judging itself, on two scenarios, with 87 investigated cases and only 12 from arm D. The causes are injected through the prompt rather than trained into the weights. Nothing was pre-registered. I fixed thresholds in a pilot before the main run, but cot_forced was added after the first three conditions had been scored. The report goes through each of these.

If you pick this up

The quote result is the one I'd most want replicated, on a stronger investigator that isn't the subject model, since it costs one line of prompt. After that, induce the cause with a LoRA so the probe question can be asked properly, and force the concerning action in every run so selection can't undo the legibility manipulation.

The full report has the per-arm tables, the full layer sweep and every limitation. The repo has the pipeline, all 270 trajectories, every verdict and an offline test suite that runs without a GPU.