Life sciences · Preprint

Through the Looking Glass: Directly Reading and Writing Transformers

arXiv · September 9, 2026

Hypothesis-Generating Why this matters to you Expert discussion

Raises a question worth testing. It does not answer one.

AI Summary

This preprint describes a method for attributing transformer predictions to specific components and weights through post-hoc analysis of activations and gradients. The work is mechanistic rather than empirical: it characterises which parts of a model's parameters and activations correlate with predictions, and demonstrates that sparse subsets (ranging from 2 to 16 components across models) account for most of a single prediction's output. The findings are preliminary observations from an uncontrolled, exploratory analysis without peer review.

What was studied

Post-hoc mechanistic analysis of transformer models. Multiple transformer models ranging from 124M to 7B parameters; domains not specified; eighteen models analysed for component contribution; twelve models analysed for sufficient component set.. Intervention: Post-hoc reading of model weights and activations; identification of sparse sufficient component sets; installation of attention heads and units with key-value read from weights.

Key findings

Across eighteen models, median mass pushing away from predicted token is seven times the mass supporting it On baseline, 53 components carry 90% of a prediction; 13 are necessary; 8 sufficient alone Across twelve models (124M to 7B parameters), sufficient component set ranges from two to sixteen

Safety

Safety was not reported in the material analysed. Check the source before drawing any conclusion about harm.

Who this may help

The source did not state who this applies to in practice.

Plain-language summary

This is a mechanistic analysis of transformer internals using post-hoc interpretability methods; it raises questions about component interactions and causal roles rather than testing clinical or practical hypotheses with controlled comparisons.

Study design

As stated by the source record.

MethodsAbsolute value contributions of units and channels to logits were counted and ranked. Weights and activations were read directly from trained models without fitting or training. Sparse subsets were identified as sufficient to produce predictions. Attention head installations were performed and loss measured. Attribution and read accuracy computed using model parameters and activations alone.

DesignPost-hoc mechanistic analysis of transformer models

PopulationMultiple transformer models ranging from 124M to 7B parameters; domains not specified; eighteen models analysed for component contribution; twelve models analysed for sufficient component set.

InterventionPost-hoc reading of model weights and activations; identification of sparse sufficient component sets; installation of attention heads and units with key-value read from weights

Reported figures

Quoted from the source exactly as published.

Median ratio, mass away to mass toward prediction7

Components carrying 90% of prediction (baseline)53

Components prediction cannot survive losing13

Components sufficient to produce prediction alone8

Range of sufficient component set across twelve models2 to 16

Fraction of model traced from single prediction1 to 3%

Component read accuracy above chance on eight strongest inputs58.9%

Caveats

This is a preprint and has not undergone peer review
No experimental causal validation; findings are correlational and based on attribution metrics
Details on model selection, training procedures, and domains not provided
Claims about component necessity and sufficiency are inferred from ablation-like metrics, not validated with held-out causal intervention

Evidence assessment

Graded across the dimensions that decide whether you should act, each from what the source actually supports. There is no single score, and where a dimension was not assessed it says so.

Study design

limited

Post-hoc analysis of trained models without experimental controls or causal intervention validation.

Directness

limited

Infers component function from attribution metrics and weight inspection; causal claims rest on correlational evidence.

Consistency

moderate

Patterns reported across eighteen models (sizes 124M to 7B); absolute consistency figures not systematically reported.

Risk of bias

limited

No blinding, prespecified analysis plan, or independent replication reported; findings are exploratory.

Sample adequacy

moderate

Multiple models examined; no justification for model selection or sample size rationale given.

Source abstract

How many of a transformer's components decide a token? Counted by the absolute value of each unit's and channel's contribution to the logit, one prediction rests on thousands to hundreds of thousands of them. But contributions are signed, and across eighteen models the mass pushing away from the predicted token is a median of seven times the mass carrying it. Divide by the net and the count is dozens: on the baseline, 53 components carry ninety percent of a prediction, 13 it cannot survive losing, and 8 suffice to produce it alone. Across twelve models trained elsewhere, 124M to 7B parameters, the sufficient set runs from two components to sixteen, and what a prediction draws on, followed all the way back, is one to three percent of the model, a share that does not grow with size. Three quarters of a layer's update is a fixed linear map of the state it received. Everything is read from the model's own parameters and activations, with nothing trained or fitted, and it names a component on both sides: what it writes, from the predictions it drives, reaching close to half of every model; what it reads, from its weights in the frame of its own layer, at 58.9 percent above chance over its eight strongest inputs. Sorting the remainder by upstream source yields grammatical categories the embedding cannot see. A name can be acted on. An association the model does not hold installs into one spare unit, key and value read from the weights, for a quarter of a percent of held-out loss, a fortieth of what a rank-one update costs. An installed attention head and a unit two layers above it make an edit fire only where a token occurred earlier in the context, and a unit the model trained for itself is driven from two layers upstream, 86 percent of the effect passing through it. An order-preserving activation puts a unit's inputs at the instrument's ceiling, at the price of a two-part install.

Follow from this

Taken from the source record, never inferred. Follow any of these and new work involving them reaches your briefing.

Mark Oskin