Do circuits explain model errors?

AlphaZero learns superhuman chess from nothing but the rules, yet by the accounts of Shannon and Kolmogorov it has learned almost nothing: a short program can't create information. This episode follows that puzzle through a 2026 paper that puts the observer's computing budget into the definition of information. We start with Shannon entropy (information as surprise) and Kolmogorov complexity (the shortest program), find the crack in both with pseudorandom generators, and then meet epiplexity: the size of the best model a time-limited observer can find, set against time-bounded entropy, what still looks random to it. We cover how to estimate it from a training run (the area under the loss curve, and a stricter teacher-student code), three puzzles it untangles (computation creating information, order mattering, and models learning more than the process that made their data, with cellular automata, chess, induction and emergence), what it says about text versus images and about choosing training data, and where it is still on shaky ground. An aigs pod extra, not tied to a journal-club meeting, and an unofficial summary, not produced by the authors. Narrated by a synthetic voice (Kokoro, "Bella"). Paper: Finzi, Qiu, Jiang, Izmailov, Kolter and Wilson, "From Entropy to Epiplexity" (2026), https://arxiv.org/abs/2601.03220

What does a transformer's attention actually do with measurements from a physical system? This paper takes the smallest transformer, one layer and mostly one attention head, and reads it with classical dynamical-systems theory. For a linear system, attention is an autoregressive model whose lag weights come from softmax, so they are positive and add up to one: a single attention-only head can only average its past, and a damped spring that needs weights of opposite sign comes out too smooth. A second head supplies the minus sign. For nonlinear systems seen through a single sensor, attention works like an adaptive delay embedding, in the spirit of Takens' theorem. We walk through the sign problem, how a short history stands in for a missing sensor (on the Van der Pol oscillator, the best transformer's error is more than a thousand times lower than that of an MLP that sees only the present value), why the model needs enough internal dimensions to unfold what it rebuilds (Chafee-Infante), and why flow past a cylinder goes better when the model is told the Reynolds number (without it, rebuilding the velocity field from the model's internal state is about 70 times worse). Then we ask whether good next-step forecasts mean a sound internal state, and what that suggests for machine-learned climate emulators (our extrapolation). Narrated by a synthetic voice (Kokoro, "Bella"). This continues the journal club's interpretability series (Nov 11). Paper: https://arxiv.org/abs/2512.21113

A circuit is a small piece of a neural network that is supposed to explain how the whole network does a task. The usual test asks whether the circuit still does the task, on average. This paper asks a sharper question: when the model gets an answer wrong, does the circuit get it wrong in the same way? Mostly, it doesn't. On indirect object identification in GPT-2 small, under mean ablation, the circuits tested agree with the model on 97 to 99.5 percent of the prompts it gets right, but on only 11 to 42 percent of the ones it gets wrong. We walk through the measurement, three attempts to explain the gap away (errors are rare, errors are close calls, the circuit is just too sure), a case study that adds 63 head positions to the classic circuit and reproduces about three quarters of the held-out errors, and the two omitted computations that make the wrong name win. Then we ask what a circuit is an explanation of, and what the same test would mean for climate emulators and their biases (our extrapolation). Narrated by a synthetic voice (Kokoro, "Bella"). This continues the journal club's circuits series (Nov 4). Paper: https://arxiv.org/abs/2609.35686

A small transformer predicts radar from motion capture, and when the motion is changed in physically controlled ways, it responds the way the Doppler law says it should. LAWFUL (Law-Aligned Witness for Faithful Use of Latents) turns that into a test: it scores how a model's response to counterfactual inputs tracks a known law, then uses activation patching to find the circuit that carries the response. We walk through what its 0.988 consistency score does and doesn't constrain (a network off by a constant factor would still score perfectly), how far the tested range reaches, the circuit and its weak necessity control, the probes, and the idea that attention heads pick which moments to compare. Then we ask what evidence would justify saying a model has learned the physics, including for climate emulators. Narrated by a synthetic voice (Kokoro, "Bella"). This continues the journal club's circuits series (Oct 28). Paper: https://arxiv.org/abs/2607.28672

A fast, enthusiastic two-host pass through LAWFUL, the framework that checks a neural network against a known physical law: a transformer that turns motion capture into radar, the Doppler law it was never shown, counterfactual changes in speed, activation patching, and a circuit of temporal attention heads that seems to compare early and late moments. A companion to episode 25, which takes the same paper slowly and more skeptically. Listen critically: this episode swaps the two attention-patching numbers (patching only the attention pattern gives the smaller drop, 0.35; patching only the values gives 1.31), and it presents as proven several things the paper offers as suggestions, including that the model reinvented adaptive differentiation. Good material to argue about at the meeting (Oct 28). Paper: https://arxiv.org/abs/2607.28672

Finding that a network encodes something is not the same as knowing how it uses it. Edge-based circuits have been defined for large language models, but work on vision models has so far only used neuron-based circuits, which show what is encoded but not how it is routed. The authors introduce Automatic Visual Circuit Discovery (Vi-CD), which finds sparse, edge-based computational subgraphs in vision transformers. They apply it to class-specific classification circuits, to the circuits behind typographic attacks on CLIP, and to circuits that can be steered to correct harmful behavior: on the RoCOCO benchmark, steering the discovered circuits reduces safety violations more than random ablations while keeping retrieval performance close to baseline. This opens the journal club's circuits series (Oct 21). Paper: https://arxiv.org/abs/2604.14477

Could it be that AI weather models are solving physical equations, although they may not be the equations used by conventional NWP models? The authors compute correlations of forecast skill and Centered Kernel Alignment, providing evidence that different AI weather models represent the atmosphere in similar ways, despite differences in architecture and capacity. They argue that the architecture and training of the AI models constrains the form of the physical laws that they might simulate. In particular, they propose that the models implement a particle description of the atmosphere, where the latent variables at each mesh point correspond to the position of a particle in the high dimensional latent space. They hypothesize that the movement of the particles follows a gradient flow in the latent space towards a minimum of a learned free energy functional. Analysis of the GraphCast and Aurora models show that they make changes on large spatial scales in the early processor layers and move to smaller scale with increasing layer depth, consistent with the gradient flow hypothesis. Paper: https://arxiv.org/abs/2605.23778

Artificial Intelligence (AI) weather models are improving rapidly, and their forecasts are already competitive with long-established traditional Numerical Weather Prediction (NWP). To build confidence in this new methodology, it is critical to understand how these predictions are generated. This is a huge challenge as these AI weather models remain largely black boxes. In other areas of Machine Learning (ML), mechanistic interpretability has emerged as a framework for understanding ML predictions by analysing the building blocks responsible for them. Here the authors present an open-source, highly adaptable tool which incorporates concepts from mechanistic interpretability. The tool organises internal latent representations from the model processor and allows for initial analyses, including cosine similarity and Principal Component Analysis (PCA), enabling the user to identify directions in latent space potentially associated with meteorological features. Applying the tool to the graph neural network GraphCast, they present preliminary case studies for mid-latitude synoptic-scale waves and specific humidity. These demonstrate the tool's ability to identify linear combinations of latent channels that appear to correspond to interpretable features. Paper: https://arxiv.org/abs/2604.20467

Artificial Intelligence (AI) weather models are improving rapidly, and their forecasts are already competitive with long-established traditional Numerical Weather Prediction (NWP). To build confidence in this new methodology, it is critical to understand how these predictions are generated. This is a huge challenge as these AI weather models remain largely black boxes. In other areas of Machine Learning (ML), mechanistic interpretability has emerged as a framework for understanding ML predictions by analysing the building blocks responsible for them. Here the authors present an open-source, highly adaptable tool which incorporates concepts from mechanistic interpretability. The tool organises internal latent representations from the model processor and allows for initial analyses, including cosine similarity and Principal Component Analysis (PCA), enabling the user to identify directions in latent space potentially associated with meteorological features. Applying the tool to the graph neural network GraphCast, they present preliminary case studies for mid-latitude synoptic-scale waves and specific humidity. These demonstrate the tool's ability to identify linear combinations of latent channels that appear to correspond to interpretable features. Paper: https://arxiv.org/abs/2604.20467

Deep learning weather prediction models achieve remarkable predictive skill yet remain largely opaque: little is known about how they represent physical climate phenomena internally. Mechanistic interpretability through Sparse Autoencoders (SAEs) offers a principled route to decomposing these representations, but existing SAEs assume strictly linear feature superposition - a constraint ill-suited for the highly nonlinear atmospheric dynamics encoded in modern transformers. The authors introduce KAN-SAE, a sparse autoencoder whose encoder replaces the standard ReLU with learnable per-feature B-spline activations drawn from Kolmogorov-Arnold Networks (KANs), allowing each latent dimension to develop its own nonlinear gating profile. Applied to Sonny, KAN-SAE discovers 975 alive features (vs. 566 for a linear baseline, a 72% improvement) with 20% lower inter-feature redundancy and comparable reconstruction fidelity. Without any climate supervision, KAN-SAE identifies an interpretable European heatwave feature spatially concentrated over western Europe, and a western Pacific typhoon tracker confirmed by causal steering experiments. The results demonstrate that nonlinear activations are essential for mechanistic interpretability of deep learning weather prediction models, recovering climate features that remain invisible to linear baselines. Paper: https://arxiv.org/abs/2605.17493