Mechanistic interpretability

What it is

Mechanistic interpretability is an approach to explaining model behaviour by identifying the internal computations and features that produce it, rather than by testing only inputs and outputs from outside^[1][2]. Its unit of analysis can be a computation, a direction in activation space, or a feature that is active in particular contexts. The goal is to say what inside the model is doing the work.

The problem: polysemanticity and superposition

A central problem is polysemanticity: a single neuron can be involved in many unrelated concepts. In September 2022, Elhage et al. published Toy Models of Superposition, which addressed this phenomenon^[1]. The paper builds a toy model in which polysemanticity arises because the model stores additional sparse features in superposition^[1]. It demonstrates a phase change, a connection to the geometry of uniform polytopes, and evidence of a link to adversarial examples^[1]. The toy setting makes the packing problem visible as something that can be studied directly.

How it works: sparse autoencoders

A later study applied dictionary learning to a real transformer. Bricken et al. published Towards Monosemanticity on 4 October 2023^[2]. They applied sparse autoencoders to a one-layer transformer with a 512-neuron MLP layer trained on 100 billion tokens of The Pile^[2]. They trained autoencoders with expansion factors from 1x (512 features) to 256x (131,072 features) on 8 billion data points of MLP activations^[2]. In their formulation, a feature is a direction in activation space rather than a neuron, and the feature directions are the columns of the decoder weight matrix^[2]. The autoencoder is trained with an L2 reconstruction loss plus an L1 penalty on the hidden layer activation to encourage sparsity^[2]. The L2 term asks the autoencoder to reconstruct the activations; the L1 term pushes the hidden code to use few active units, so that the learned directions are more likely to correspond to interpretable features.

What they found

The authors reported that most learned features were interpretable^[2]. One feature for Hebrew script was not active in the top dataset examples of any neuron^[2]. When they activated a base64 feature, the model generated base64 text^[2]. They also reported that 512 neurons can represent tens of thousands of features, and that features split into finer, still interpretable families as the dictionary grows^[2]. These results illustrate the difference between a neuron and a feature: a feature can be present in the model's activations without being the top example of any single neuron, and activating it can change behaviour in a specific way^[2].

Why it matters

The 2023 authors describe the work as an existence proof for extracting interpretable features from superposition^[2]. For mechanistic interpretability, that matters because it gives a procedure for moving from neuron-level descriptions to feature-level descriptions, and for testing a feature by activating it^[2]. The 2022 toy model supplied a way to study why features might be packed together in the first place^[1]. Together, the two lines of work connect a conceptual problem—polysemanticity—with a concrete dictionary-learning method and with interventions on learned features^[1][2].

Limits

The authors of the 2023 study state that they remain agnostic about whether models are fundamentally composed of features^[2]. They present the work as an existence proof, not as a complete decomposition of a model^[2]. The 2022 paper is explicitly a toy model^[1]. These limits matter for how the results should be read: the studies show that interpretable features can be extracted and manipulated in the settings they examine, while leaving open how far the picture generalizes^[1][2].