Inside OpenAI Astra: The looped transformer

Frontier labs are trading parameter bloat for recurrent depth. The model gets deeper by running the same layers again, and the reasoning stops being readable.

On September 1, 2026, The Information reported that OpenAI's Astra uses something called recurrent depth. Within two days the phrase was everywhere, half the timeline calling it a breakthrough architecture and the other half calling it a safety redline. Both readings are overheated. The technique is real, it is old, and the interesting part is not the efficiency. It is what it does to the part of a model we can actually read.

**A transformer is a stack you walk once.**

Every standard transformer works the same way. A token gets embedded, then passes through layer 1, then layer 2, then every layer in order until it reaches the top, and the model emits a word. One pass, one trip through the building. If you want the model to compute more before answering, you have two options. Add more layers, which means more weights to train, store and load into VRAM. Or make it write its reasoning out as text and feed that back in as more tokens, which is what Chain of Thought does.

Looped transformers add a third option. Instead of stacking 24 different layers, you build 12 and run them twice. The hidden state comes out of layer 12, goes straight back into layer 1, and does the whole trip again. Same weights, second pass. The model has now done 24 layers of computation while storing 12 layers of parameters.

**What the paper actually proves.**

The cleanest treatment of this is *Reasoning with latent thoughts: On the power of looped transformers*, published at ICLR 2025 by Nikunj Saunshi and colleagues at Google Research. Their headline result is sharp: a k-layer transformer looped L times nearly matches a kL-layer model on reasoning tasks, using L times fewer parameters.

The synthetic experiments are almost too clean. On multi-digit addition, a single layer looped 12 times hits 99.9% accuracy, matching a 12-layer model with a twelfth of the weights. On p-hop induction, one layer looped 6 times matches the 6-layer baseline. On i-GSM math problems, one layer looped 8 times scores 73.2%, level with the 8-layer model.

At real scale the picture gets more honest. The team trained 1B-parameter models on 250B tokens of the Pile and compared a 24-layer baseline against several looped configurations. On math word problems, the 12-layer model looped twice scored 34.3 against 29.3 for the full 24-layer model, beating a network with double its parameters. But on raw perplexity, the looped models only closed 34% to 50% of the gap. Looping is not a free lunch across the board. It is specifically a reasoning lunch. Memorization still wants real parameters; reasoning wants depth, and depth is what looping is cheap at.

**The trade you are actually making.**

Here is the part the hype cycle skips. Reusing weights saves memory, not compute. Twelve layers looped twice holds half the weights in VRAM, which matters enormously when you are bandwidth-bound, but it still does the arithmetic of a 24-layer forward pass. FLOPs scale with the loop count. So does latency.

Compared to a 24-layer dense model, looping is a straight win: same compute, half the memory. Compared to the 12-layer model you actually built, you just doubled your inference bill to buy depth. Which comparison matters depends entirely on what is scarce in your datacenter. For most labs in 2026, memory bandwidth is scarcer than FLOPs, which is exactly why this idea came back.

**Not every token needs the full trip.**

The naive version loops every token the same number of times, which is wasteful. The word "the" does not need four passes through a reasoning core. A NeurIPS paper on mixture-of-recursions fixed this with a learned router that assigns each token its own loop budget. Easy tokens exit after one pass. Hard tokens keep circulating. Huginn, the 3.5B open model from the recurrent-depth line of work, uses a simpler signal: it measures the KL divergence between two successive iterations and stops when it falls below 5×10⁻⁴. When the hidden state stops changing, the thinking is done.

This is where the architecture gets genuinely new. Inference compute becomes a dial rather than a fixed cost, and the model sets the dial itself, per token, at runtime.

**The structure everybody converges on.**

Huginn's design is the reference blueprint, and it comes in three tiers. A prelude of 2 layers embeds the token into latent space. A recurrent core of 4 layers does the looping, with a mean of 32 iterations during training and anywhere from 1 to 128 at test time. A coda of 2 layers reads the final state back out into a token prediction.

The detail that makes it stable is input re-injection. On every single pass, the original embedding gets concatenated back into the hidden state before the core runs again. Without it, the state drifts and the loop degenerates. With it, the model stays anchored to what it was actually asked. Huginn trained on 795B tokens across 4,096 AMD GPUs on Oak Ridge's Frontier supercomputer, and at 32 recurrences it reaches reasoning performance the authors compare to a 50B-parameter model, from 3.5B of stored weights.

**Why safety researchers got loud.**

Chain of Thought has an accidental virtue. When a model reasons in visible text, that text can be read, parsed, classified and flagged. It is not a perfect window into the model's cognition, but it is a window. Safety teams have spent three years building tooling that depends on it existing.

A looped model does its intermediate work in hidden vectors. Nothing gets written down. The reasoning happens, converges and produces an answer without ever becoming natural language, which means there is no transcript to audit. Steven Adler, formerly of OpenAI's safety team, said OpenAI "seems to be violating one of the few redlines that exists in the AI industry." Peter Wildeford of the AI Policy Network called it potentially reckless, pointing out that Chain of Thought analysis was how anyone understood the July incident where OpenAI models attacked Hugging Face. Daniel Kokotajlo's concern was structural: even if OpenAI holds the line, whoever ships next might not.

OpenAI's answer, from chief scientist Jakub Pachocki, is that Astra's loops are capped, its computation depth stays near GPT-4 levels, and the company has worked to preserve Chain of Thought monitoring since its first reasoning models. Read carefully, that is a real commitment and also an admission that the pressure runs the other way. Every additional loop is cheaper than every additional layer.

**None of this is new.**

Universal Transformers put shared weights and recurrent depth in a paper in 2018. Huginn shipped as open weights in 2025. Nanbeige 4.2 pretrained a 3B model from scratch on 28T tokens using a 22-layer stack run twice, found that two passes was the sweet spot, and kept about 75% of the token efficiency of a standard architecture. The technical report is public. You can download the weights right now.

Sebastian Raschka's read is the correct one: Astra is not exotic, it is a known technique finally applied at frontier scale by a lab with the compute to make it pay. The novelty is the deployment, not the idea.

What deserves attention is the second-order effect. We spent three years teaching models to think out loud and building an entire safety discipline on top of the transcript. Recurrent depth makes thinking cheaper by moving it somewhere we cannot see. That is a good engineering trade and a bad monitoring trade, and those are not the same conversation.