The text encoder was optional all along: Halcyon beats pretrained T2I at 275x less compute

What they actually did

The paper, from Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas, Ehsan Adeli, Jiajun Wu, Juan Carlos Niebles, Li Fei-Fei and Michael Poli, was posted October 15, 2026, with model weights on Hugging Face the same day. The setup is deliberately simple: one decoder-only Transformer (Qwen3 architecture, randomly initialized) that reads text tokens and image tokens as a single sequence and trains on next-token prediction. Every component is built from scratch on GPIC, a 100-million-pair image-text dataset at 256x256: the image tokenizer (a Cosmos-style encoder-decoder with finite scalar quantization, mapping each image to 1,024 tokens from a 64,000-code book), the text tokenizer (a byte-level BPE with 16,384 tokens, trained on GPIC captions and OpenWebText), and the generator itself. One joint vocabulary of 80,640 entries, one epoch, 97,632 steps at batch size 1,024.

The comparison design is what makes the result legible. T2I systems are usually benchmarked against each other despite being pretrained on different data, so nobody can tell whether a win comes from architecture or dataset. The team instead retrained the two representative late-fusion baselines - LlamaGen (autoregressive, FLAN-T5-XL encoder) and JiT-T2I (pixel diffusion, Qwen3-1.7B encoder) - on the same 100 million pairs with the authors' official code, covering 256 and 1,024 image tokens, with and without perceptual loss. And when they count compute, they count all of it: language model, image tokenizer and image generator. That accounting choice is the quiet core of the paper. A "0.9B" generator conditioned on a 1.22B FLAN-T5-XL that consumed a trillion text tokens is not a 0.9B system; the field's habit of reporting the generator's size and ignoring the encoder's training cost is what makes 18x and 275x gaps possible to hide.

The three ablations are the result

The headline numbers could still be a data or tuning artifact, so the convincing part is the trio of controlled experiments asking directly: does language pretraining help? First, swapping the frozen encoder from FLAN-T5-Base to FLAN-T5-XL inside the same LlamaGen setup - 13.5x more language pretraining compute - improves gFD only from 66.7 to 63.3. Second, giving Halcyon-0.52B a text-only pretraining phase before T2I training, at 10, 20 or 50 percent of a fixed budget, monotonically worsens gFD from 48.0 to 51.6, 52.8 and 65.1; running text-first pretraining on top of a full T2I epoch at 1.5x total compute still does not help (49.2 vs 48.0). Third, initializing from Qwen3-0.6B-Base - a real pretrained language model with 36 trillion text tokens behind it - gives no gain over the identical architecture trained from scratch (47.8 vs 46.6). Whatever internet text teaches a model, little of it transfers to mapping captions to pixels in this regime, and at fixed budget it actively crowds out the pairs that do.

There is also a genuinely elegant side result. Classifier-free guidance needs unconditional image examples, and the standard recipe gets them by dropping captions from 10 percent of training examples - throwing away text supervision. Halcyon instead moves those captions after the image tokens, turning each dropped example into an image-to-text one. The model learns image captioning at zero extra pretraining cost (image-conditioned text NLL at 0.52B drops from 18.41 to 1.10) while T2I quality stays statistically unchanged across all five sizes. Free captioning as a byproduct of a bookkeeping change is the kind of trick that ends up everywhere.

What I make of it

The honest scope: this is 256x256, one epoch, models up to 4B, measured largely on GPIC's own test split. Nobody should read it as "frontier image models don't need language" - the question it settles is narrower and, I think, more useful: at fixed pretraining budget, language pretraining is not the free lunch the field assumed, and the assumption was mostly protected by accounting that never charged for it. That pattern - a component everyone includes by default turns out to be optional once you count its cost - shows up far beyond image generation. It is the same move that fixed how we compare inference costs of reasoning models. I also note the motivation is not purely academic: Radical Numerics builds multimodal biological models (their Omnii line spans twenty-plus modalities), and "how should pretraining compute be allocated across modalities" is their production question, with T2I chosen because it has public data and benchmarks. That does not invalidate the result - the baselines were retrained on shared data, which is more than most claims of this kind offer - but it is worth knowing why this particular question got answered this well.