Project Write-Up
NLAs make things up 40% of the time, and their own confidence can't tell
Anthropic's Natural Language Autoencoders read a model's internal state and describe in plain English what it's thinking. People are starting to use them to audit models, including for whether a model is hiding something. The problem, which Anthropic states openly, is that they confabulate: they assert specific things that are false.
I measured how often (40% of quoted claims) and whether you can catch it (mostly, with a second independent instrument). This post is the result. The mechanism details are folded into the expandable sections, click them if you want the how, skip them if you want the what.
What an NLA is, if you want the mechanism
Two fine-tuned copies of the model. One reads the activation and writes a description. The other reads the description and tries to rebuild the activation. Training rewards descriptions that rebuild well, not descriptions that are true. Invertible and true aren't the same thing, and the gap between them is this whole post.
The other tool I use is a sparse autoencoder, an SAE. Different tool, same activation, trained separately. It breaks the vector into a handful of concepts that each have a human-readable label. Because it was trained a completely different way, it fails differently. That's the entire trick.
Forty percent
I wrote 52 trap sentences. Each one sets you up to guess wrong. A nurse named Bruno. A midwife named Hassan. A fruit called a klemvar. Then I fed them to the NLA and read what it claimed the model was thinking.
The lucky break is that the NLA quotes your own sentence back at you. "The opening phrase 'My daughter, a'." I wrote "My sister." So checking it is dumb and free: the quoted phrase is either in my sentence or it isn't. No AI judge, no labels, just string matching. 52 sentences, 8 retries each, 3 spots per sentence. 1,248 explanations.
Of the 4,072 phrases it quoted from my prompts, 1,651 weren't there. Forty percent. It quoted "My daughter" for "My sister." It turned forty-three euros into fifteen. On "The midwife, Hassan, delivered twins" it wrote "she delivered twins." Every one of those, all eight retries, same each time, and the model's own confidence sat above 0.99 the whole while.
The traps did it, and I can prove it, because I threw in boring control sentences too. "The dog slept under the table." Those only broke 12% of the time. The traps broke 54%. When the model's gut fought the sentence, the gut won.
Asking again doesn't save you
The obvious fix is to just ask it a few more times and keep what it agrees with itself on. And honestly, that works better than I thought. It sorts the true quotes from the fake ones pretty well.
But it has a wall it can't get past. Seven percent of the claims it agreed on unanimously, eight retries out of eight, were still false. And that seven percent isn't random. It's the model's biases. "She" for Hassan. "Fifteen" for forty-three. You cannot retry your way out of a mistake the model makes every single time with full confidence. It just says the wrong thing eight times and looks certain.
That seven percent is the part I actually care about. It's exactly the kind of lie an audit would swallow.
The other instrument sees it
The SAE reads the same vector but breaks differently, so I asked it the obvious question. When the NLA says "she" about Hassan, is there actually a female concept lit up in the model? Or did the NLA make it up?
I rented a GPU, ran the model, grabbed the activations, and fed them through the SAE. The check needs a control, so I ran it both ways: on positions where the NLA quoted a real word, and on positions where it invented one. If the SAE lit up equally on both, it would be firing on everything and proving nothing.
Across 212 positions: real words matched 72% of the time, invented words 17%. That is a 55-point gap, and it is not luck (z = 7.84). Read as a detector, flagging "no SAE support" catches 83% of the fabrications.
Hassan is the clean one. At his name, the concepts firing are "people's first and last names" and "Muslim male names." Nothing female, anywhere. The model knows Hassan is a man. The NLA is the thing that decided he's a she. The bias is in the describer, not the model, and you can watch it happen.
And then there's vantril, my favorite. Vantril is a word I made up. When the model hits it, the concepts that fire are sleeping, puppets, and epilepsy. It's flailing at the sounds because there's no meaning underneath. The NLA looked at that pile of nonsense and confidently reported "car."
Where this falls apart
62% should be higher. It isn't, partly because SAEs genuinely miss things, and partly because my matcher is crude, it checks whether a concept's label literally contains the word, which misses plenty of real matches. So the gap is real but the exact numbers are soft. Don't quote them to three decimals.
Numbers are a mess at this layer. "Twelve applicants" fired no clean number concept at all, so the SAE just shrugged. It's a noisy second opinion, not an oracle. Which is fine, because that was never the claim. The claim is it catches things retrying can't, and it does.
The effect held when I grew the sample: it started at 50 points on 164 positions and came out at 55 on 212, which is the direction you want when you scale something up.
What's next
The honest limit of all this is that I only checked claims about the input, because that's the one place a free answer key exists. The claims that actually matter for an audit are about the model's own thinking, and I haven't touched those yet. That's the next thing. After that, run it on the 12B model and see if the 40% is the method or just this particular NLA.
I packaged the check as a small library called crosswise (pip install crosswise, needs a GPU). It takes an NLA explanation and flags the quotes no SAE feature supports. It is a first-pass flag, not an oracle, and the matcher is deliberately crude, but it catches the confabulations that matter. Code and the full method are on github.
But the short version I'm confident in: this thing invents details 40% of the time, its own confidence has no idea, and a second instrument reading the same activation catches the lies that matter, the ones it tells the same way every time without blinking.
Code and data: github.com/potatochoudhary/crosswise. Model gemma-3-27b-it, NLA source kitft-l41, SAE 41-gemmascope-2-res-262k, all on Neuronpedia.