Systems
The architecture of self-explaining systems
Six months inside a lab building AI that narrates its own reasoning.
I spent March watching a model explain why it chose one answer over another. Most of the time it was wrong — not about the answer, but about the reason. The explanation was a confabulation: plausible, articulate, and entirely disconnected from the weights that produced the output.
The confabulation problem
When a language model generates an explanation, it is not introspecting. It is sampling from a distribution conditioned on the prompt and the answer — a separate generative act with no causal link to the original computation.
The explanation was plausible, articulate, and entirely disconnected from the weights.
We instrumented the forward pass at every transformer layer, logging attention patterns for two thousand queries across five model sizes. The circuits that produce output and the circuits that explain it do not share an information channel.