Loading
Life sciences · Preprint
arXiv · September 8, 2026
Raises a question worth testing. It does not answer one.
This theoretical paper proves that frozen transformers can implement closed-form diffusion and energy-based samplers via in-context learning, and reports geometric regularities in hidden states during semantic-topic sampling that align with predicted energy patterns. The work establishes a mechanistic hypothesis for how transformer layers might perform data generation, but provides no empirical validation that transformers actually generate samples or comparison of generation quality.
Theoretical analysis with post-hoc analysis of pretrained transformer hidden states. Pretrained transformer language models; semantic-topic sampling used to probe hidden-state geometry. Intervention: Prompts consisting of words from common semantic categories (animals, foods, cities).
Transformers can realize closed-form and smoothed closed-form diffusion samplers from in-context examples without parameter updates Softmax attention computes responsibility weights and weighted empirical averages; feedforward layers implement Euler updates Normalized hidden states exhibit two-stage geometry: move toward uniform spherical reference in intermediate layers, then return to structured topic-dependent representations
Safety was not reported in the material analysed. Check the source before drawing any conclusion about harm.
The source did not state who this applies to in practice.
A theoretical construction proving transformers can simulate generative samplers, with empirical observations of geometric patterns in hidden states, but no direct demonstration of sampling capability or comparison to established generative methods.
As stated by the source record.
Graded across the dimensions that decide whether you should act, each from what the source actually supports. There is no single score, and where a dimension was not assessed it says so.
What is missing. This record has no reported figures. That is a gap in the analysis, not a judgement about the study.
A growing body of work establishes that large language models are not mere statistical memorizers, but are capable of in-context learning: performing inference at test time using only examples provided in the prompt, without any parameter updates. Prior theoretical work has shown that this capability extends to supervised learning tasks such as linear regression. We prove that in-context learning extends further to \emph{data generation}: frozen transformers can simulate iterative generative samplers from in-context samples. We first show that transformers can realize closed-form and smoothed closed-form diffusion samplers. The construction identifies a concrete generative role for softmax attention: it computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates. To empirically relate these constructions to pretrained language models, we study \emph{semantic-topic sampling}: prompts consisting of words drawn from a common semantic category, such as animals, foods, or cities. Across transformer layers, the normalized hidden states exhibit a two-stage geometry: they move toward a uniform spherical reference in intermediate layers and then return to structured, topic-dependent representations near the output. We further measure an interacting-particle energy on these hidden-state clouds and observe the same U-shape pattern. We then prove that transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.
Taken from the source record, never inferred. Follow any of these and new work involving them reaches your briefing.