Can a Trillion-Parameter Diffusion LLM Actually Work? Inside Nvidia's Bet on Parallel Text Generation

· 26 min read

Diffusion language models write a whole block of text at once instead of word by word. Nvidia's reported deal for Inception Labs suggests it thinks they can scale. Here's what the math, the silicon, and the research actually say.

A field of masked placeholder tokens resolving, block by block, into readable text across a grid of GPU-like tiles Diffusion LLMs start from a canvas of blanks and refine many tokens in parallel. The open question is whether that trick survives at a trillion parameters. (Illustration: author)


The deal that put diffusion on the map

On November 19, 2026, The Information reported that Nvidia is close to acquiring Inception Labs, a small Bay Area startup, for more than $500 million plus performance-based earnouts. Neither company has commented, and no filings have appeared, so treat it as reported, not confirmed. The timing is suggestive, though. A week earlier, customers reported an email from Inception saying its hosted Mercury API would remain available only through December 28, 2026.

Inception is not a typical AI startup. Its founders are three professors with deep roots in diffusion and generative-model research: Stefano Ermon (Stanford), whose lab helped pioneer score-based diffusion, Aditya Grover (UCLA), and Volodymyr Kuleshov (Cornell). Its product, Mercury, is the most commercially advanced diffusion language model (dLLM). Nearly every other chatbot you've used, including ChatGPT, Claude, and Gemini, is autoregressive (AR): it writes one token at a time, left to right. Mercury drafts a whole chunk of text at once and refines it over a few passes, the way image generators like Stable Diffusion turn noise into a picture.

The deal fits a pattern. In December 2025, Nvidia paid roughly $17–20 billion to license Groq's inference technology and hire its core team. In March 2026 it unveiled the result, the Groq 3 LPX rack, designed to sit beside its flagship Vera Rubin NVL72 GPU racks. Nvidia is collecting ways to make inference faster. Inception adds an architectural option: change how the model generates text, not only the chip it runs on.

That raises the question this article tries to answer honestly: could you build a one-trillion-parameter (1T) diffusion LLM, roughly the scale of today's frontier open models, and would it be worth it?


Where diffusion LLMs stand today

How a dLLM writes

An autoregressive model is like a typist. It predicts token 1, appends it, predicts token 2, and so on. Each new token requires reading the model's entire set of weights from memory, so the GPU spends most of its time waiting on memory, not doing math. In single-user decoding, an H100 uses well under 1% of its peak arithmetic throughput.

A diffusion LLM is more like an editor working on a fill-in-the-blanks page:

  1. Start with a block of, say, 64 [MASK] tokens after the prompt.
  2. Run the model once and predict every blank simultaneously.
  3. Keep the predictions it's most confident in, re-mask the rest.
  4. Repeat for a handful of steps until the block is filled, then move on to the next block.

Side-by-side diagram: autoregressive decoding emits one token per forward pass; block diffusion fills an 8-token block in three parallel denoising passes Autoregression emits one token per pass. Block diffusion fills a whole block over a few passes, so each expensive weight-read produces several tokens. (Diagram: author)

The key metric is tokens per forward pass (TPF). If a 64-token block takes 12 passes, TPF is about 5.3. Each pass still reads all the weights once, but now that read produces five tokens instead of one. Diffusion turns the GPU's idle arithmetic into speed.

Almost every serious system now uses block diffusion: blocks are produced left-to-right (so previous blocks can be cached, as in AR), while tokens inside a block are refined in parallel. It's a hybrid, and that matters later. (If you want a gentler introduction first, my earlier primer, What Is a Diffusion LLM and Why Does It Matter?, covers the basics from when Inception's first model, Mercury Coder, launched in early 2025.)

The current leaders

ModelWhoScaleHeadline speedMain catch
Mercury 3 (Oct 2026)Inception Labs, closedUndisclosed ("40x" Mercury 1)1,068 tok/s (vendor)~880 tok/s measured; trails Claude Haiku as an agent; API shutting down
LLaDA 2.1 Flash (Feb 2026)Ant Group, open100B MoE892 tok/s on code (quantized)32K context; needs a multi-GPU node
DiffusionGemma (Jun 2026)Google DeepMind, open25B MoE, 3.8B active~1,500 tok/s on H10069.1% vs. 84.2% for AR Gemma 4 on AIME 2026
Seed Diffusion (2025)ByteDance, demoDense, code-only2,146 tok/s on H20Code only, no long reasoning
Gemini Diffusion (2025)Google, previewUndisclosed1,479 tok/sClosed, little independent testing

Mercury 3: impressive, but read the fine print

Mercury 3 shows what diffusion can do in production. It grew more than 40 times larger than Mercury 1, supports a 1-million-token context, and lists at $0.20 per million input tokens and $0.70 per million output tokens (half that during a launch promotion). On Inception's own charts it beat speed-tier rivals GPT-6 Luna and Claude Haiku 5.5 on 10 of 12 benchmarks, including 89.2% on GPQA Diamond, while claiming 1,068 tokens per second on unspecified Nvidia GPUs.

Launch-week OpenRouter telemetry looked bad: median throughput of 160–254 tok/s. Those numbers include routing and queuing, though, so I ran my own benchmark the week of launch. It made about 63,000 API calls across 13 model configurations, covering two jobs: code generation (637 HumanEval+, MBPP+, and LiveCodeBench problems) and a multi-step SQL agent (496 BIRD questions). Everything was measured end to end over public APIs. Four findings matter for this story:

  • The speed is real. Called directly, Mercury 3 decoded ~850–900 visible tokens per second, close to Inception's 1,068 claim. That's roughly 2x Gemini 3.5 Flash-Lite, 2.7x Claude Haiku 5.5, and 5–6x GPT-6 Luna. The OpenRouter gap was mostly the service, not the model.
  • But diffusion isn't the only route to speed. My control was an autoregressive model, gpt-oss-120b, served on Cerebras's SRAM-heavy wafer-scale chips. It decoded at ~2,400 tok/s, about 2.7x faster than Mercury 3, and at medium effort it answered coding problems in 0.35 s at the median. Fast hardware can beat a fast architecture.
  • Quality is close but not ahead. On code, Mercury 3 (low effort) scored 86.0% pass@1, versus 87.9% for Claude Haiku 5.5 and 88.4% for GPT-6 Luna at medium effort. As a SQL agent, it trailed Haiku by about 5 points (52.5% vs. 57.8%), and in its fastest "instant" mode it broke a strict JSON schema on 6 of 150 tasks.
  • Reasoning can run away. At medium and high effort, 7.5–8.6% of Mercury 3's coding calls hit the 32K-token budget without producing an answer, pushing 95th-percentile latency to 52–54 seconds. No autoregressive model in the test did this. Autoregressive models fail by committing to a wrong answer; Mercury 3 with reasoning fails by not stopping. Runaways, plus regular list prices above Haiku's and Luna's, mean it often costs more per right answer: on code, Mercury 3 at low effort cost $0.59 per 1,000 solved problems even at promo pricing, versus $0.14 for Claude Haiku 5.5 with thinking off.

Bar chart of measured visible decode speed: gpt-oss-120b on Cerebras about 2,400 tokens per second, Mercury 3 about 880, Gemini 3.5 Flash-Lite 444, Claude Haiku 5.5 323, GPT-6 Luna 164, with Inception's 1,068 claim marked Mercury 3 nearly hits its claimed speed and beats the big labs' fast models, but an autoregressive model on fast hardware is faster still. Median visible decode rate on ~2,000-token answers over public APIs, October 2026. (Chart: author, from my Mercury 3 benchmark)

The other caveats are commercial. Mercury 3 has no weights, no paper, no architecture details, and text-only input. Apart from independent tests like mine, its benchmark scores are vendor-reported, which is why BenchLM declined to rank it. And the API ends December 28.

The four structural weaknesses

Beyond Mercury, the research identifies four problems every large dLLM has to solve:

  1. Memory pressure at long context. Each denoising pass queries the entire cached prompt. With a 1M-token context, that cache gets read many times per block, so memory bandwidth becomes the ceiling.
  2. Wasted compute at high load. Diffusion spends roughly S times more arithmetic per token than AR (8–12x for 8–12 passes). That's free when the GPU is idle, and expensive when it's busy. For an 8B model on an H100, diffusion is about 5x faster for one user but loses its lead around 32 concurrent users.
  3. Reasoning quality. Predicting many interdependent tokens at once assumes they're independent, which they often aren't. More on this below; it's the most important weakness.
  4. Immature software. AR has vLLM, SGLang, and TensorRT-LLM, refined over years. Diffusion serving still needs custom code for re-masking, stopping criteria, and scheduling. Mercury 3's runaway reasoning shows what happens when stopping isn't solved.

Will scaling make diffusion models smarter?

This is the real question. Speed is worthless if a 1T dLLM is dumber than a 1T AR model.

What the scaling studies say

The short answer: diffusion LLMs do improve with scale, following power laws like AR models, but they have historically needed more compute to reach the same quality. The gap is shrinking.

  • 2024–25: a scaling study by Nie et al. found masked diffusion needed about 16x more training compute than AR to reach comparable loss, a roughly constant gap across scales. The same group's LLaDA-8B was nonetheless competitive with LLaMA 3 8B on in-context learning, which proved the approach worked at real scale.
  • Data-constrained regimes: several 2025 papers ("Diffusion Beats Autoregressive in Data-Constrained Settings" and "Diffusion Language Models are Super Data Learners") found that when you have to repeat training data for many epochs, diffusion keeps improving after AR saturates. The random masking acts like built-in data augmentation. As high-quality text runs out, this could matter more than raw compute efficiency.
  • December 2025: von Rütte et al. scaled diffusion to 10B parameters and 10²² FLOPs. With properly tuned learning rates and batch sizes, the compute-optimal recipe shifted substantially and the gap between diffusion variants narrowed with scale. Direct AR comparisons were still inconclusive because datasets and tokenizers differed.
  • August 2026: LLaDA MoE v2 published practical scaling laws for Mixture-of-Experts diffusion models. Its 30B-parameter model (3B active) approached Alibaba's autoregressive Qwen3 while using about 65% as many pretraining tokens.

So the scaling curve exists and slopes in the right direction. Nobody has shown it crossing AR's curve on reasoning at frontier scale. The largest open dLLM is 100B parameters; Mercury 3's size is secret.

The Flexibility Trap

The most important dLLM paper of 2026 may be the ICML Outstanding Paper winner, The Flexibility Trap (Ni et al.). Its finding is counterintuitive: letting the model fill in tokens in any order makes it worse at reasoning.

Given freedom, a dLLM fills in the easy, low-uncertainty tokens first and postpones hard "fork" words like therefore, since, and thus, which decide where an argument goes. By the time it gets to them, the surrounding text has already locked in a direction, and the model no longer explores alternatives. On HumanEval (sampling 1,024 attempts with LLaDA), 21.3% of problems were solved only with strict left-to-right order, versus 0.6% solved only with arbitrary order.

Two-panel chart: left shows 21.3 percent of HumanEval problems solved only by left-to-right order vs 0.6 percent only by arbitrary order; right shows DiffusionGemma at 69.1 percent vs AR Gemma 4 at 84.2 percent on AIME 2026 Left: generation order matters for reasoning. Right: a diffusion fine-tune still trails its autoregressive sibling on hard math. (Chart: author, data from Ni et al. 2026 and Google DeepMind)

A separate 2026 study (Li et al.) found that dLLMs tend to fall back on left-to-right decoding anyway, because their training data, including long chains of thought, is written sequentially. Training on independent, parallel reasoning paths helps.

The same paper proposes a fix, and it's telling. JustGRPO trains dLLMs with reinforcement learning using left-to-right rollouts, then lets them generate in parallel at inference. It reached 89.1% on the GSM8K math benchmark. In other words, the best way to make diffusion reason is to train it partly like an autoregressive model.

Verdict on scaling: a 1T dLLM would almost certainly be smarter than a 100B dLLM. Whether it would match a 1T AR model trained on the same budget is unproven. The likely answer is "close, on most tasks, if you borrow AR's training tricks."


What would it take to build and serve a 1T dLLM?

Assume the training problem is solvable. Could you run one fast? Here, physics gives clearer answers.

Lesson 1: it must be Mixture-of-Experts

A dense 1T model uses all trillion parameters for every token. A Mixture-of-Experts (MoE) model splits the network into many "expert" sub-networks and routes each token to a few of them. Kimi K2.6 (1T total, 32B active) and Thinking Machines' Inkling (975B, 41B active) work this way. For a dense 1T model, diffusion's extra arithmetic is fatal: it hits the GPU's compute ceiling almost immediately.

Lesson 2: MoE has a hidden tax for diffusion

MoE creates a subtle problem. In AR decoding for a single user, each step activates 8 experts, so the GPU reads only a small slice of the weights. A diffusion block of 64 tokens routes each token independently, and together they touch almost every expert.

The math is simple. With E experts, top-k routing, and N tokens processed per pass, the expected fraction of experts that must be loaded is:

fraction loaded = 1 − (1 − k/E)^N

With 160 experts and top-8 routing, one token loads 5% of experts. A 64-token diffusion block loads 96%. That assumes tokens pick experts independently; real routing is correlated, so treat it as a worst case, but the direction holds.

Line chart of fraction of MoE experts loaded versus tokens processed per forward pass, rising from 5 percent at one token to 96 percent at 64 tokens Why diffusion's speed advantage shrinks for one user on an MoE model: a parallel block touches nearly every expert. (Chart: author)

Lesson 3: a roofline estimate

To make this concrete, I built a back-of-the-envelope roofline model: a standard way to estimate whether a workload is limited by memory bandwidth or arithmetic. The setup is a single Vera Rubin NVL72 rack using NVIDIA's current spec sheet: 1,400 TB/s of HBM4 bandwidth, 20.7 TB of memory, and 1,800 PFLOPS of dense 4-bit compute (half of NVIDIA's 3,600 PFLOPS headline, which is a sparse figure). It assumes a 32K-token context, efficiency discounts of 55% for compute and 70% for bandwidth, and a 0.15 ms communication floor per forward pass. It compared a 1T MoE model (50B active) against a hypothetical 1T dense model, under AR, conservative diffusion (TPF 5.3), and aggressive few-step diffusion (TPF 16).

These are model outputs, not measurements. Using the headline sparse compute and earlier bandwidth figures instead would inflate diffusion's best case to about 14x, so treat these conservative numbers as the ones to quote. The shape of the results is the point:

  • 1 user, MoE: diffusion is only 1.4x faster than AR (7,568 vs. 5,445 tok/s), because of the expert-loading tax.
  • 8–128 users, MoE: this is the sweet spot. Concurrent AR requests already activate most experts, so AR loses its advantage, while diffusion puts the GPU's spare compute to work. Conservative diffusion is 2.6–4.6x faster; aggressive few-step diffusion reaches 4.6–11x. The lead peaks around 32 users. By 128 users, diffusion starts hitting the compute ceiling and the gap narrows (2.9x conservative). At an illustrative $150 per hour for the rack, modeled cost at 128 users drops from about $0.33 to $0.07–0.11 per million tokens.
  • Dense 1T: diffusion hits the compute wall by 8 users. By 32 users AR has caught conservative diffusion, and at 128 users AR wins outright (127K vs. 39–59K aggregate tok/s).

Two-panel chart: left, per-user tokens per second for a 1T MoE model at batch 1, 32, 128 and 512 under AR and two diffusion settings; right, aggregate throughput at batch 128 for MoE versus dense models Modeled throughput on one Vera Rubin NVL72 rack with NVIDIA's current specs and dense compute. Diffusion wins big for sparse models at moderate load and loses for dense models at high load. (Chart: author, first-principles roofline estimate; assumptions in text)

So a 1T dLLM is feasible to serve, but only as a sparse MoE with block diffusion, on hardware with lots of spare compute. It isn't a free lunch either: about 1.4x for a single user, and 3–11x for a moderately busy API.

The software stack it needs

Hardware is the easy part. A production 1T dLLM also needs:

  • A diffusion-native serving engine with continuous batching, block-level KV caching, adaptive "stop when confident" schedulers, and variable-length outputs. Early results are promising: an October 2026 paper paired its dlmrt engine with new training to serve a 4B diffusion model about 6.9x faster on one H100 than the original model on SGLang, and ran a small LLaDA 2.1 model at 5,046 tok/s on a single B200.
  • Smarter routing. Expert-Choice routing (each expert picks its tokens, instead of each token picking experts) balances load by construction and halved training time in a 2026 study. Block-coherent routing, where tokens in a block share experts, would directly cut the expert-loading tax.
  • Long-context KV tricks such as CPU-assisted sparse KV retrieval (HERALD reports up to 3.7x higher decode throughput than GPU-only serving).
  • Numerically stable 4-bit math. A November 2026 study found MXFP4 training changes 18x more output tokens than BF16 when you merely change how the model is split across GPUs. At 1T scale, that's a reproducibility problem.

How Nvidia's stack fits in

Vera Rubin: built for diffusion, almost by accident

The key number for any accelerator is its ridge point: peak FLOPs divided by memory bandwidth. Below it, you're waiting on memory; above it, you're limited by arithmetic. Single-user AR decoding sits at roughly 3–10 FLOPs per byte. Block diffusion sits in the hundreds to thousands.

Horizontal bar chart of roofline ridge points: Vera Rubin about 2,600, Blackwell B200 2,250, H100 1,181, and Groq 3 LPX 8.2 FLOPs per byte, with shaded regions marking where AR decode and block diffusion operate Each GPU generation adds math faster than memory bandwidth. AR decode can't use that surplus; diffusion can. GPU figures use NVIDIA's peak (sparse) tensor throughput; dense math roughly halves them without changing the order. (Chart: author, specs from NVIDIA)

Wide rendering of a row of NVIDIA Vera Rubin rack-scale systems seen from the front The Vera Rubin platform deploys as rows of rack-scale systems. A single NVL72 rack holds 72 GPUs and 20.7 TB of HBM4. (Image: NVIDIA)

Using NVIDIA's peak figures, Rubin's ridge point (~2,600 FLOPs/byte) is more than double H100's and about 300 times Groq's. Each generation, Nvidia adds compute faster than memory bandwidth, so AR decoding wastes a growing share of the chip. Diffusion is the workload that puts that wasted compute to work. A 1T model in 4-bit NVFP4 is about 560 GB, which fits on 3 of a rack's 72 GPUs and leaves ~20 TB for user caches.

Groq LPX: brilliant at the opposite job

Groq's LPU makes the opposite trade-off. Each chip has 500 MB of ultra-fast on-chip SRAM and no HBM, giving a ridge point of only ~8 FLOPs/byte. That's ideal for AR decoding and poor for diffusion's arithmetic-heavy passes. Storing a 1T model's weights alone would take 6–10 full LPX racks (1,536–2,560 chips).

Don't underestimate this class of hardware, though. In my Mercury 3 benchmark, an AR model on Cerebras's SRAM-heavy chips decoded 2.7x faster than Mercury 3. For models that fit, fast AR hardware competes directly with diffusion. That may be exactly why Nvidia is betting on both: it licensed Groq's chip technology for one route to speed and is reportedly buying Inception's architecture for the other.

So Groq doesn't host a 1T dLLM. It complements one. Nvidia already pitches LPX for fast decode alongside Rubin for heavy parallel work. In that setup, a diffusion model can act as a parallel drafter for an AR model (DFlash-style speculative decoding is on TensorRT-LLM's Q3 2026 roadmap), or the two can split prefill, drafting, and verification across different racks.

Rendering of the NVIDIA Groq 3 LPX rack, a black rack with its front panel open showing stacked gold-colored compute trays The Groq 3 LPX rack packs 256 LPU chips with on-chip SRAM. It's built for fast sequential decoding, not for holding trillion-parameter weights. (Image: NVIDIA)

Future generations and CUDA

The roadmap removes some of the bottlenecks that limit dLLMs today:

LayerToday (Rubin)2027–28 (Rubin Ultra, Feynman)Why it matters for dLLMs
Compute50 PFLOPS NVFP4 per GPU (peak)3D-stacked Feynman diesBigger blocks at higher batch before hitting the compute wall
Memory288 GB HBM4/GPUHBM4E, custom "NVHBM"; research on High Bandwidth FlashCheaper long-context KV caches; room for 5–10T MoE weights
InterconnectNVLink 6 copper, 72 GPUsNVLink over co-packaged optics, planned to scale to 1,152 GPUsFaster all-to-all for experts spread across racks
SoftwareTensorRT-LLM, NeMo AutoModel dLLM fine-tuningNative block-diffusion kernels and schedulersCloses the serving maturity gap

The underrated piece is software. Nvidia's real moat is CUDA and its libraries, and diffusion's biggest practical weakness is immature serving software. Bringing Inception's team into TensorRT-LLM and NeMo, plus open-weight Nemotron models, is the fastest way to give diffusion the tooling AR has had for years. It also helps Nvidia's robotics work: Cosmos and GR00T already use diffusion and flow-matching for world simulation and motor control.


What comes next in dLLM research

If you're tracking whether 1T dLLMs will arrive, these are the developments to watch:

  1. Few-step distillation and self-correction. LLaDA 2.1's token editing lets a model draft aggressively and fix earlier mistakes, a plausible path from TPF ≈ 5 to TPF ≈ 16 without garbage output.
  2. Order-aware training. JustGRPO and parallel-reasoning fine-tuning (NAP) address the Flexibility Trap. Expect reasoning dLLMs that are trained left-to-right and decode in parallel.
  3. Converting AR models instead of training from scratch. Fast-dLLM v2 turned pretrained AR models into block-diffusion models with about 1B fine-tuning tokens. The cheapest route to a 1T dLLM may be to convert an existing 1T AR model such as Kimi K2.6.
  4. Diffusion-native MoE routing. Expert-Choice and block-coherent routing attack the expert-loading tax head-on.
  5. Hybrids. Diffusion as drafter and AR as verifier (DFlash, DiffuSpec) may ship at scale before pure dLLMs do. Diffusion is also finding a role outside generation: single-step diffusion classifiers like DJev compete with fast decision models such as TypeSafe's Jev, and Inception's Mercury Decide targets the same niche. In a production stack, these would act as cheap gatekeepers in front of a large generative model, not replace it.
  6. Memory beyond HBM. High Bandwidth Flash and in-network compute could make 5–10T sparse models and 10M-token contexts practical.
  7. Independent benchmarks. The field badly needs third-party evaluations of large dLLMs on reasoning, not vendor charts and code demos.

The honest conclusion

Promises. Diffusion LLMs are the best available match between model architecture and where GPU hardware is going. Each generation adds compute faster than memory bandwidth; AR decoding can't use that surplus, and diffusion can. For a sparse 1T model serving a busy API, back-of-the-envelope physics suggests roughly 3–11x faster generation at moderate load and tokens about a third to a fifth the cost. Nvidia buying the team that built Mercury, if the deal closes, is a sensible bet on that trend.

Feasibility. Serving a 1T dLLM is feasible on Vera Rubin-class hardware if it's a sparse MoE with block diffusion. A dense 1T dLLM loses to AR under load, and SRAM-only chips like Groq's LPU are the wrong home for it. Training one is the bigger open question. No lab has disclosed a dLLM larger than 100B parameters, and no independent evidence shows diffusion matching AR on hard reasoning at the frontier.

Reality check. The most commercially advanced dLLM does deliver its speed: in my tests it decoded close to its claimed 1,000 tokens per second. But an autoregressive model on fast hardware was 2.7x faster still, Mercury 3 trailed Claude Haiku as an agent, its reasoning sometimes didn't stop, and its API shuts down next month. Reasoning research keeps finding that diffusion works best when it borrows from autoregression.

The most likely 2027–28 outcome isn't diffusion replacing AR. It's convergence: block-diffusion hybrids that generate left-to-right between blocks and in parallel within them, trained with AR-style reasoning signals, served on GPUs whose spare compute finally has a use. A trillion-parameter diffusion model will probably exist within two years. Whether it is the smartest model, and not just the fastest, is still an open question.


References

Image credits

  • Hero illustration — author (generated with Python)
  • AR vs. block diffusion diagram — author
  • Mercury 3 decode-speed chart — author; data from the author's Mercury 3 benchmark (run R3) and Inception Labs
  • Reasoning gap chart — author; data from Ni et al. (2026) and Google DeepMind
  • MoE expert-loading curve — author
  • Modeled throughput chart — author; first-principles roofline estimate using NVIDIA's Vera Rubin NVL72 spec page
  • Ridge-point chart — author; specs from NVIDIA
  • Vera Rubin platform rendering — NVIDIA press kit (nvidianews.nvidia.com), cropped
  • Groq 3 LPX rack rendering — NVIDIA press kit (nvidianews.nvidia.com)