Mercury 3 vs. the Fast Autoregressive Models: A Technical Report

How fast is a diffusion LLM in practice, what does it get right, and what does it cost per right answer? Two use cases, 13 model configurations, about 63,000 jobs, measured end to end over public APIs in October 2026.

Where each model is when the first one finishes, by median time to answer a coding problem (cars drawn at constant speed). 637 problems × 5 samples, run r2-uc1-main-20261021T034053Z, as of 2026-10-21.

Contents

1. Summary
2. Background: the claim and what changed since 2025
3. Diffusion in practice: what the stream looks like
4. Study design
5. Use case 1: code generation
6. Use case 2: a multi-step SQL agent
7. Cost per right answer
8. Vendor and API observations
9. Scoring the claims
10. Which model for which job
11. Threats to validity
12. Future work
13. Reproducing this study

1. Summary

Inception Labs launched Mercury 3 on October 19, 2026, claiming that it outperforms GPT-6 Luna and Claude Haiku 5.5 while running 4–8× faster, at over 1,000 tokens per second (Inception blog). This report tests that claim on two workloads where latency matters to a person waiting: a coding assistant and a database agent. The short, general-audience version of this study is the companion article Is Diffusion LLM the speed demon as promised?; this is the long version with the methods, tables, and failure analysis.

The comparison set is the "fast tier" of each major lab (Claude Haiku 5.5, GPT-6 Luna, Gemini 3.5 Flash-Lite) plus a deliberately chosen control: gpt-oss-120b served on Cerebras, an ordinary autoregressive model on specialised inference hardware. Mercury 2.5 is included for lineage.

Key findings:

The verdict, in one line: Mercury 3 beats the big labs' fast models on speed, roughly matches them on code quality and trails Claude Haiku by a few points as an agent, but fast autoregressive serving hardware gets there too, so "diffusion" is one road to speed rather than the only one.

2. Background: the claim and what changed since 2025

2.1 The claim

Inception's announcement ("Diffusion at Scale: Introducing Mercury 3", dated October 19, 2026) makes four claims this study can test:

1. Quality: Mercury 3 outperforms GPT-6 Luna and Claude Haiku 5.5 across a range of benchmarks.
2. Relative speed: 4–8× faster than those models.
3. Absolute speed: 1,000+ tokens per second (1,068 tok/s on NVIDIA GPUs).
4. Implicitly, causation: the speed comes from diffusion, which generates many tokens in parallel per pass.

It also lists a 1M-token context window (up from 260K for Mercury 2.5) and launch pricing of $0.10 / $0.35 per million input / output tokens, half the list price of $0.20 / $0.70.

2.2 What changed since February 2025

When I wrote What is Diffusion LLM and why it matters in February 2025, Mercury Coder was a code-only model and its speed was a vendor claim backed by one animated demo. Twenty months later, three things are different:

• The product grew up. Mercury 3 is a general reasoning model with a 1M context, four reasoning-effort levels (instant, low, medium, high), tool calling, and structured outputs, all behind an OpenAI-compatible API.
• Reasoning changes the speed story. Hidden reasoning tokens cost time for any architecture. A fast decoder that "thinks" for 2,000 tokens is no longer instant.
• Fast autoregressive serving became ordinary. Specialised inference hardware (Cerebras-, Groq-class chips) now serves open AR models at thousands of tokens per second. That makes an honest comparison impossible without an AR control on fast hardware.

3. Diffusion in practice: what the stream looks like

An autoregressive model emits one token per forward pass, left to right. A diffusion LLM starts from a masked or noisy canvas and refines many positions per pass. Mercury exposes this through a diffusing: true streaming mode in which every chunk carries the full current text, so you can watch an answer come into focus.

Mercury 3 instant in diffusing mode, prompted to write a Python function that splits an image into left and right halves (the same prompt style as the 2025 article). Run r5-diffusion-capture-20261021T181518Z, as of 2026-10-21.

To go beyond one pretty GIF, I captured 30 diffusing-mode calls: 5 prompts in the original article's style (split an image into halves, top/bottom, quarters, grayscale, centre crop) × 3 repeats × Mercury 3 instant and low, at temperature 0.5 with a 4,096-token cap. All 30 completed normally, for a total of $0.0079.

Median (range) per call. Run r5-diffusion-capture-20261021T181518Z.

Three observations:

• Answers resolve in 5–11 visible steps, in well under a second at the median.
• The text grows in blocks. Each call has 4–7 distinct text lengths: the canvas is extended a few times while earlier blocks are still being refined. It is neither a left-to-right fill nor a single fixed-size canvas.
• Refinement is not monotonic. All 30 calls have at least one step that ends further from the final text than the step before. That is consistent with remasking, where tokens revealed early are replaced later. At low, the first visible frame is further from the final answer (median similarity 0.18 vs 0.34), plausibly because the first visible frame of a post-reasoning answer is mostly noise; I didn't test that.

One caveat: the diffusing stream is a visualisation the API provides. Without confirmation from Inception, it shouldn't be read as the literal internal denoising schedule. All latency numbers in this report come from normal streaming mode, never from diffusing mode.

4. Study design

4.1 Models and settings

Prices as listed on 2026-10-20 (configs/models.yaml).

Notes on the choices:

• Reasoning parity. Every Mercury effort level is swept. Each AR model runs at its "off/minimal" setting plus one "on" setting. Gemini runs minimal only, because its output price made a reasoning-on arm expensive.
• Temperature. Mercury only accepts 0.5–1.0 and silently resets anything else to 1.0, adding a warning field. Every model runs at 0.5 where the provider allows it. Claude Haiku 5.5 rejects non-default sampling values, and Luna's support was untested, so both run at their defaults. Every response's warning field is logged and checked.
• Luna in the agent. On OpenAI's Chat Completions endpoint, GPT-6 Luna refuses function tools unless reasoning is none (§8.4), so UC2 has no Luna reasoning-on arm.
• Same output budget for everyone: 32,768 tokens per call (including reasoning). A call that spends the whole budget without answering counts as a failure ("budget out").

4.2 Measurement

• Timing uses a monotonic clock around the raw HTTP stream, recording request sent, first byte, first non-empty content (or tool-call) delta, last content, and done. TTFT is time to the first non-empty content.
• Token counts come from the server (stream_options.include_usage), never from chunk counts. Mercury streams blocks, not tokens, so chunk counting would be meaningless.
• Throughput is reported two ways: end-to-end (all generated tokens, including hidden reasoning, over total time) and visible decode (visible tokens over the time after the first visible token).
• Order is randomised across (model, item) jobs; no model runs all its jobs back to back.
• Exclusions from latency (not from accuracy): one warm-up call per arm, any call that needed a retry, and any call in flight across a pause of more than 60 s (a stop/resume or a machine sleep). Excluded calls still count toward accuracy and cost.
• Statistics: medians and p95 with bootstrap 95% confidence intervals over problems or questions. Accuracy differences between two models use paired bootstraps over the same questions.
• Network baseline: every run probes each provider with a free GET /models before starting and periodically during the run, logging connect, TLS and time to first byte.
• Cost is a frozen price snapshot multiplied by server-reported usage, recomputed offline.
• Safety: model-generated code runs only in a locked-down Docker sandbox (no network, resource limits, non-root). SQL runs on a read-only SQLite connection with a timeout and a row cap.

4.3 Runs

Total: 62,975 jobs and about $56 at Mercury's promo price (Mercury usage was mostly covered by Inception's 100M-token free credit; R4c went past it). UC2 jobs are multi-turn tasks, so the number of model calls is higher. All runs from one macOS arm64 client in the US West on standard (non-priority) API tiers, 2026-10-20/21.

Every main run (R2–R5) passed the same pre-flight: a clean git tree (so the code version is recorded) and a network probe. Network baselines were stable throughout: median connect times were 12–35 ms everywhere. Time to first byte on GET /models was 14–22 ms for Inception, 93–159 ms for Cerebras, 128–208 ms for Google, 155–224 ms for Anthropic, and 818–1,816 ms for OpenAI (§8.1).

5. Use case 1: code generation

Scenario. A developer asks an assistant for a function and waits. Each model gets a problem and must answer with one complete Python code block; the code runs against hidden tests in a sandbox. Pass or fail, no partial credit.

Workload. All 164 HumanEval+ and 378 MBPP+ problems (EvalPlus versions, with stronger tests than the originals), plus 95 LiveCodeBench v6 easy and medium problems (AtCoder stdin and LeetCode function-style, contest dates January to April 2025). That's 637 problems × 5 samples × 13 arms = 41,405 calls. pass@1 is the mean of per-problem pass rates.

LiveCodeBench caveat: no public LiveCodeBench problems are newer than these models' mid-2026 training cutoffs, so it serves as a harder set, equally exposed for every model, not a contamination-free one.

Before any live run, the grader was validated on every EvalPlus canonical solution (all 542 pass in the sandbox) and on all 3,866 LiveCodeBench tests. R2 graded 41,405 answers with zero harness errors.

5.1 The speed–quality frontier

pass@1 vs median end-to-end latency, one point per model and setting, with 95% bootstrap CIs. Up and to the left is better. Run r2-uc1-main-20261021T034053Z, as of 2026-10-21.

Ranked by pass@1. Mercury cost at promo / regular price. Run R2. The gpt-oss medium median is 0.345 s, shown as 0.35 here and 0.34 in the charts.

Precision targets were met: every pass@1 CI half-width is ≤ 2.6 points and every median-latency CI half-width ≤ 4.5%.

What the frontier says:

• At matched quality, the Cerebras control is the fastest. gpt-oss medium is about a point below Mercury 3 low (within the CIs) at 42% of its median latency.
• Mercury 3 low ties Claude Haiku with thinking off on accuracy and takes 15% less time at the median (0.83 vs 0.97 s). It edges out Luna none (83.2% at 1.28 s).
• The most accurate models are the frontier labs' reasoning settings (Luna medium 88.4%, Haiku medium 87.9%), which no Mercury 3 setting matches.
• Mercury 3's own curve bends backwards. medium and high are slower and less accurate than low (§5.5).

5.2 By dataset

pass@1 (%) per dataset. Run R2.

The interesting column is LiveCodeBench. Mercury 3 low scores 91.4%, among the best of any arm, then collapses to 67.4% at medium and 66.1% at high. The cause isn't wrong answers: Mercury 3 high ran out of budget on 155 of its 475 LiveCodeBench calls and only 6 of its LiveCodeBench answers were wrong. When it answers, it's nearly always right; it often doesn't answer.

5.3 The latency tail

Medians hide what users remember: the slow calls.

Empirical CDF of end-to-end latency. Left: every latency-eligible code-generation call. Right: every latency-eligible SQL-agent task. Runs R2 and R4c, as of 2026-10-21.

• On code, Mercury 3 instant and gpt-oss on Cerebras are ahead through most of the distribution.
• Mercury 3 low has a 0.83 s median but a 10.4 s p95, close to GPT-6 Luna's (9.9–10.6 s). That tail comes from the minority of calls where it reasons at length.
• Mercury 3 medium and high have by far the longest tails (p95 of 52–54 s): hidden reasoning that runs to the token cap.
• In the agent (right panel), Mercury 3 instant/low and gpt-oss medium lead up to about the 90th percentile. Gemini has the tightest distribution: its p95 (6.9 s) is below theirs (8.2–9.6 s), because it rarely takes long but rarely finishes fast either.

5.4 Decode throughput and output length

Does diffusion's parallel decoding pull further ahead on long outputs? The R3 sweep sent 30 timing-only prompts (10 each asking for ~256, ~2,000, and ~8,000 output tokens; the long ones request ~800-line programs) × 3 repeats to the lowest setting of each model, plus Mercury 3 low with and without the realtime flag. 630 calls, $2.03.

Median visible decode rate for ~2k-token and ~8k+-token answers. Run r3-uc1-length-20261021T152238Z, as of 2026-10-21.

Medians per length bucket. Run R3.

Findings:

• Decode rates are flat with length for every model. Diffusion's speed-up over the frontier labs' fast models is real (2–6×) but it's a constant factor; it doesn't grow on longer outputs.
• The Cerebras control decodes ~2.7× faster than Mercury 3 at every length.
• Short Mercury answers arrive in one block. At instant, total time (0.30 s) is essentially TTFT (0.29 s): the whole ~93-token answer lands at once. On short outputs, TTFT is the latency.
• Hidden reasoning delays the first token. At low, Mercury 3's TTFT grows with the task (0.57 → 2.31 → 2.54 s) because it reasons before the first visible block (median ~740–940 reasoning tokens on medium and long prompts). At instant, TTFT stays at 0.3–0.5 s.
• realtime gave no measurable gain for code. TTFT stayed within noise of plain low in every bucket, and decode was the same. It may matter for voice-style workloads.
• Verbosity varies a lot. Asked for ~800 lines, Haiku and Luna wrote ~15k tokens, Mercury 9–10k, and gpt-oss ~6k. That's why total seconds aren't comparable across models and tokens per second are.
• All 27 length stops in R3 were verbosity, not runaways: long prompts hitting the 16,384-token cap of this sweep (Luna 13, Mercury 14), with Mercury low's reasoning at a median of ~1.1k tokens.

5.5 The runaway-reasoning failure mode

This is the most distinctive behaviour in the study, and it only shows up in Mercury 3.

Run R2. "Budget out" = stopped at the 32,768-token cap with no visible text. "Think-aloud runaway" = stopped at the cap while writing visible text, which is how the same failure shows up at instant, where Mercury 3 has no hidden reasoning.

The cost of this is large:

• Runaway calls consumed 61% of all Mercury 3 output tokens in R2.
• They set the tail latency: p95 of 51.9 s at medium and 53.6 s at high.
• They concentrate on hard or ill-posed problems. 119 problems account for every medium/high budget-out.

Would a bigger budget help? The R1b probe re-ran the two problems that ran away in the pilot, at Mercury 3's maximum of 131,072 tokens (4× the cap):

Run R1b, 12 calls, $0.29.

Quadrupling the budget rescued 1 of 12 calls (one that converged at 36,515 tokens, just above the normal cap). The rest either converged quickly anyway or reasoned to the new cap, taking 3–4 minutes each. So budget-out is mostly non-converging reasoning on hard problems, not a cap that's too tight. A curiosity I can't explain: several runaways stopped at identical token counts (128,982 twice, 129,565 twice).

Four examples from the failure gallery (all from run R2):

1. More effort, no answer (abc388_d, LiveCodeBench). Mercury 3 high spent 32,567 tokens on hidden reasoning in 62.2 s and returned nothing. The same problem is solved 5 of 5 times at low and at instant, and 0 of 10 at medium/high. Claude Haiku, Gemini, Luna medium and gpt-oss medium solve it 5 of 5.
2. An ill-posed easy problem (Mbpp/430, "find the directrix of a parabola"). Its one example (5, 3, 2 → −198) matches no textbook formula. Most AR arms write the standard formula and fail; Claude Haiku medium reverse-engineers the example and passes 5 of 5. Mercury 3 reasons until the cap on all 10 samples at medium/high and 4 of 5 at low, apparently trying to reconcile the example with the math: same score as a wrong guess, but ~32k tokens and close to a minute per call.
3. The runaway in plain sight (arc194_a, instant). With no hidden reasoning, Mercury 3 wrote ~138,000 characters of visible analysis ("Key observation…") and hit the cap before writing any code.
4. A repetition loop inside a code literal (Mbpp/440, instant). Asked to find adverbs, it started a hard-coded list of adverbs and never closed it: ~110,000 characters of repeated words until the cap. Repetition loops aren't unique to diffusion, but in R2 only Mercury 3 instant ever hit the cap while writing visible text.

Karpathy wondered in 2025 whether diffusion LLMs would show a "new, unique psychology". This is a candidate: autoregressive models fail by committing to a wrong answer; Mercury 3 with reasoning fails by not stopping. Whether that's diffusion itself or Mercury's reasoning training, these runs can't say.

5.6 Mercury lineage

Mercury 3 is a large step from Mercury 2.5. At low: 86.0% vs 76.9% pass@1 (+9.1 points) and 0.83 s vs 1.36 s median. Mercury 2.5 medium (83.1%) is still below Mercury 3 low, at 4.6× the latency (3.82 s).

6. Use case 2: a multi-step SQL agent

6.1 Design

Scenario. An analyst asks a question in plain English about a database. The agent can't see the data up front: it has to explore the schema, test queries, and submit one final SQL query whose result answers the question. Each turn is a full model round trip, so per-call latency multiplies.

Workload. BIRD Mini-Dev (SQLite). The release lists 500 questions, of which 498 are unique; 2 gold queries can't be scored within 60 s, leaving 496. The main run, R4c, uses all 496 (148 simple, 247 moderate, 101 challenging) × 3 repeats × 10 arms = 14,880 tasks. An earlier run, R4, used a stratified 150-question sample; R4c confirmed its rankings and speed ratios with error bars about 2.5× tighter, so this report quotes R4c. Example question (bird/1380, simple): "What is the total amount of money spent for food?" with the hint "total amount of money spent refers to SUM(spent); spent for food refers to category = 'Food'".

The loop, identical for every model:

• Tools: list_tables, describe_table(name), run_sql(query) (read-only, timeout, row cap), and submit_answer(sql).
• One frozen system prompt asking the model to explore with the tools, check its query's result before answering, and be efficient, with at most 8 turns.
• A 32,768-token cap per turn; temperature 0.5 where accepted.
• If a model replies without a tool call, it gets one nudge to use a tool.

Scoring. Execution accuracy (EX): the submitted query's result rows must match the gold query's as a multiset (ordered only when the gold query has a top-level ORDER BY, floats rounded to 6 decimals). The official BIRD rule (set equality) is computed alongside and differs by at most 2 points per arm. Task latency is wall time from the first request to the final answer, including local SQL, which is negligible (median 7–16 ms per task). Every task's full trace can be replayed: 14,869 of 14,880 R4c traces reproduce byte for byte. The 11 others ran SQL that changes on every run (10 use the clock, 1 uses ORDER BY RANDOM()).

6.2 Results

Median latency per model call (faint) and per whole task (solid), relative to Mercury 3 low, with mean turns per task. Run r4c-uc2-full-20261021T214456Z, as of 2026-10-21.

EX is the mean over 496 questions of each question's pass rate across 3 repeats. Paired differences are bootstrapped over questions with the repo's paired_ex_delta (seed 4, the run's seed, as in the generated decision matrix). Mercury cost at promo / regular price. Run R4c, 14,880 tasks, $21.78 ($26.80 at Mercury's regular price).

Speed: turn count is the multiplier. Task latency is roughly per-call latency × number of turns, and the number of turns is a property of the model:

Ratios of median latency. Run R4c.

• Against the frontier labs' fast models, Mercury 3's per-call speed mostly carries through: 1.9–2.8× faster per task.
• Claude Haiku is the efficient explorer: it needs about 3.2 turns, so its 3.2× per-call deficit becomes 1.9× per task.
• Gemini goes the other way: fast per call, but it explores one table at a time, so a 1.7× per-call gap widens to 2.2× per task.
• The Cerebras control's per-call lead mostly evaporates. gpt-oss low is 1.7× faster per call but only 1.07× faster per task; at medium it's slower than Mercury 3 per task.

These ratios are within a few hundredths of the 150-question R4 run, so the speed story is stable.

Accuracy: Haiku leads, gpt-oss trails, the rest are level. Each per-arm CI is about ±4 points, because questions vary a lot in difficulty; differences measured on the same questions are tighter (±2–3 points). On that paired basis:

• Claude Haiku is clearly ahead of Mercury 3 low, at both settings: +5.3 points at medium and +4.2 thinking off.
• gpt-oss on Cerebras is clearly behind, by 6.3 (low) and 7.8 points (medium). It often writes its answer as prose instead of submitting it (§6.3).
• Mercury 3 instant is slightly behind low (−2.3, upper bound −0.2: borderline).
• Luna, Gemini, and Mercury 3 medium/high are level with Mercury 3 low. As in code, more Mercury reasoning doesn't buy accuracy.

By difficulty (simple / moderate / challenging EX, %): Claude Haiku medium 70 / 55 / 48; Haiku thinking off 68 / 53 / 48; Mercury 3 low 65 / 48 / 45; Gemini 69 / 48 / 34; Luna 65 / 48 / 38; Mercury 3 instant 64 / 48 / 36; gpt-oss low 59 / 44 / 32. Mercury 3 low holds up best of the non-Claude models on the challenging questions.

A note on absolute numbers: some BIRD gold labels are wrong or ambiguous. In the 10-question pilot, 3 questions failed for every model, and on inspection each was a label problem (an odd join path, a question asking for IDs that the gold query doesn't return, an inner vs left join). That lowers every model's EX equally, so comparisons between models hold, but absolute EX understates all of them.

6.3 Reliability: how agents fail

Valid calls: known tool, JSON arguments that match the schema. Invalid SQL: run_sql calls that returned an SQLite error. Malformed: a reply with no tool call (nudged) or an invalid call; "recovered" means the task still submitted an answer. Run R4c.

Tool-call syntax is essentially solved: every arm except Gemini made at least 99.99% schema-valid calls. The differences are in finishing: answering within 8 turns, and calling submit_answer rather than writing the answer as prose. When turn-capped tasks are classified, most are still exploring rather than circling: Gemini 392 exploring vs 37 repeating a query; gpt-oss medium 327 vs 35; Mercury 3 instant 116 vs 19.

Three cases from the gallery (run R4c):

• One answer expected, many found (bird/866, Mercury 3 instant). Asked which driver set a 0:01:27 lap in race 161, it found that the hint's pattern matches 82 laps by several drivers and that an exact match returns nothing. Expecting a single answer, it kept reasoning in visible text in turn 6 ("Let me reconsider…", 93 times) until the 32,768-token cap: ~127,000 characters and 54 s for the task. 31 of 2,976 Mercury 3 instant/low tasks ended in a budget-out turn; no AR arm had any. It's the same "doesn't stop" pattern as in code, much rarer here because SQL turns are short.
• The right answer, written as prose (bird/824, gpt-oss low). It found the right query (it returns exactly the gold rows), then replied with it as JSON text instead of calling submit_answer, twice, ignoring the nudge, and ran out of turns. This is why the fastest model per call loses most of its lead per task, and much of its accuracy.
• Exploring one table at a time (bird/240, Gemini). Four describe_table calls in separate turns plus a command the read-only sandbox refuses; it reaches the right query in turn 7, re-runs a variant in turn 8, and hits the cap without submitting. Gemini hit the cap on 28.8% of tasks.

6.4 Strict structured output (R4b)

A sub-run asked for the final answer as the reply text under strict response_format: json_schema, instead of a submit_answer call, with tools still offered for exploration. R4's 150-question sample × 1 repeat × the lowest setting per model (gpt-oss on Cerebras is excluded: it rejects requests that combine tools and response_format).

Run r4b-uc2-strict-20261021T170547Z, 600 tasks, $0.61.

Two behaviours appear only in this mode:

• Mercury 3 broke the strict schema on 6 of 150 tasks. Four replies were valid JSON followed by </|inc_tool_call_ml|parameter>, which looks like its internal tool-call template; one ended in a bare </; one was empty. Every other model had zero violations.
• Claude Haiku stops exploring. With a schema set, it answered all 150 tasks in the first turn with no tool calls, writing SQL blind from the question and hint. That still scored 53.3% EX, versus 58.9% with exploration on the same 150 questions in R4. It's the fastest arm here for the wrong reason.

7. Cost per right answer

Per-token prices are misleading for two reasons: models differ in how many tokens they spend (reasoning, verbosity, turns), and the answers that matter are the correct ones. So cost here is all spend in the run, including warm-up and retried calls, divided by successes.

Total spend divided by successes; Mercury's hatched bar is the regular (non-promo) price. Runs R2 and R4c, prices as listed 2026-10-20, as of 2026-10-21.

Mercury at promo / regular price. Runs R2 and R4c.

What drives the ranking:

• List prices. At its regular price, Mercury 3 ($0.20 / $0.70) costs more per token than Claude Haiku 5.5 and GPT-6 Luna (both $0.10 / $0.50). Even the 50% promo only brings its output price to $0.35, below theirs, with input at parity.
• Runaway reasoning makes Mercury 3 medium/high the most expensive code arms per solved problem: they burn ~32k tokens on calls that return nothing.
• Turns cost input tokens. Every agent turn re-sends the growing conversation. gpt-oss and Gemini take the most turns, which, with their higher input prices, makes them the most expensive agent arms ($5.43–$7.07 per 1,000 correct).
• Caching wasn't equal. Anthropic caches only with explicit cache_control markers, which this study didn't send (so Haiku's cache hit rate was 0%), and Gemini's OpenAI-compatible endpoint doesn't report cached tokens. Both could be cheaper in a tuned production setup.

Bottom line: Mercury 3 is mid-pack per right answer at the promo price and roughly doubles at the regular price. In the agent, Mercury 3 low ($1.38) is second only to Luna at the promo price, but at the regular price ($2.77) Claude Haiku thinking off ($1.55) beats every Mercury setting. Cheap per token isn't cheap per answer.

8. Vendor and API observations

Running five providers side by side surfaces things that rarely make it into model announcements.

8.1 OpenAI's slow front door

GPT-6 Luna's first token was consistently slow, and the cause turned out to be on OpenAI's side, before the model even runs:

• A free GET /v1/models request took 814–1,209 ms of server processing at OpenAI, with ~20 ms connect and ~40 ms TLS through a nearby CDN edge. The same probe came back from the other providers in 14–224 ms.
• On real requests, OpenAI's openai-processing-ms header showed a median of 596 ms.
• OpenAI documents its usage tiers as changing rate limits and caps, not latency. To test whether paying helps, I ran 20 short prompts once on the standard tier and once on Fast mode (service_tier: "fast", at 2× the price), in random order:

Run oai-priority-ab-20261021T033428Z, 40 calls, $0.0022.

Fast mode was quicker on 13 of 20 prompts, with a median gain of 0.146 s; the big change is in the tail (worst case 6.85 s → 1.80 s). Even with Fast mode, about 450 ms is spent server-side before the first token. All study runs used the standard tier for every provider; no one got a paid speed tier.

8.2 Cerebras throttled below its documented limit

In R2, 921 calls needed retries, 765 of them Cerebras 429 errors, even though the client stayed under Cerebras's documented 1,000 requests per minute. Retried calls are excluded from latency, so Cerebras's latency numbers come from the ~2,800 of 3,185 calls per arm that weren't throttled. Its speed is real, but so is the rate limiting.

8.3 Mercury API quirks

• Silent temperature reset. Any temperature outside 0.5–1.0 is reset to 1.0, with only a top-level warning field to tell you (verified with the exact message in the M0 smoke test). A benchmark that doesn't check warning would silently run Mercury at a different temperature.
• Blocks, not tokens. A 5-token answer arrived in 3 chunks. Throughput must come from server usage counts.
• Defaults depend on effort. The default max_completion_tokens changes with reasoning effort (8,192 at instant up to 131,072 at high), and the docs disagree on the default effort itself, so both should always be set explicitly.
• Text preamble with tool calls. Mercury 3 may emit a short text preamble alongside tool_calls in the same turn; agent loops need to accept both.
• No seed, so runs aren't bit-for-bit reproducible. Every number here uses repeated samples and confidence intervals instead.

8.4 Other providers

• GPT-6 Luna won't call tools with reasoning on via Chat Completions: every request with function tools and reasoning_effort other than none got a 400 pointing to the Responses API. I dropped that arm rather than build a second adapter.
• Gemini's OpenAI-compatible endpoint streams parallel tool calls in an unusual shape (each call arriving whole at index 0, or with "index": null). My stream assembler initially merged two calls into one, producing a corrupted history. It also reports no reasoning or cached token details. Both were handled, with regression tests, before the main run.
• OpenAI signals an empty balance as HTTP 429 (insufficient_quota), the same status as a rate limit. A naive "retry on 429" loops forever on a billing problem.
• gpt-oss on Cerebras adds ~64 prompt tokens of fixed overhead per request and uses a few reasoning tokens even at low.
• BIRD Mini-Dev has 498 unique questions, not 500 (two are listed twice), and two gold queries can't be scored within 60 s.

9. Scoring the claims

The fast-hardware control is what makes the last row possible. Without it, a reader would reasonably conclude that diffusion is the only way to get sub-second answers. With it, the conclusion is narrower: diffusion lets Inception beat the big labs' fast models on (presumably) ordinary GPUs, while specialised hardware gets an ordinary AR model further still.

10. Which model for which job

Judgements as of 2026-10-21, from runs R2, R3, R4c and R4b.

11. Threats to validity

• Products, not architectures. Every comparison includes each vendor's hardware, batching, network path, and API gateway. The Cerebras control bounds how much of Mercury's lead could be "just serving", but it doesn't isolate diffusion.
• One client, one region, one day. All runs came from a single macOS arm64 host in the US West on 2026-10-20/21, on standard API tiers. Time-of-day load, other regions, and dedicated capacity could all change latency. The network probes rule out the client's network as a major factor, not the providers' load.
• LiveCodeBench is pre-cutoff. Every model has had the same chance to see these problems; it's a harder set, not a fresh one.
• UC2 per-arm accuracy is ±4 points. That's the honest limit of a 496-question benchmark; paired differences (±2–3 points) are the numbers to compare.
• R4c's tail wasn't interleaved. The other providers finished first, so the last ~2,900 R4c tasks were Mercury-only. Per-task latency isn't affected, but those tasks didn't share time windows with the other providers.
• BIRD labels are imperfect. Absolute EX understates every model; comparisons are on the same questions.
• Promo pricing. Mercury's cost numbers double at the regular price; both are shown.
• No seeds. Mercury doesn't support one, so runs are statistically, not bit-for-bit, reproducible.
• Fixed prompts, not tuned ones. One prompt per use case for every model, by design. A model-specific prompt might help any of them, especially in the agent loop (Gemini's exploration habits, gpt-oss's prose answers).
• Token caps shape results. The uniform 32,768-token cap is lower than Mercury's own defaults at medium and high. The R1b probe suggests raising it rarely helps, but on 12 calls.
• Reasoning tokens aren't visible for Gemini through its OpenAI-compatible endpoint, so its visible-vs-reasoning split is unknown.

12. Future work

• More regions and times of day. Every number here came from one client on one day.
• Mercury on fast hardware. If Inception ever serves Mercury on Cerebras- or Groq-class chips, that's the clean diffusion-vs-hardware experiment.
• Code editing. Inception's separate Mercury Edit model handles fill-in-the-middle and next-edit prediction, which this study didn't test.
• Real-time and voice-style workloads, where Mercury's realtime flag might matter.
• Controllability. Research on diffusion LLMs (LLaDA) suggested they handle reversal tasks that trip up left-to-right models. A small probe would test that on a commercial model.
• Responses-API adapter for Luna, to add a reasoning-on Luna arm to the agent.
• Prompt caching done right for every provider, to see how much it moves cost per correct task.

13. Reproducing this study

Everything is in github.com/zbruceli/mercury3: the harness, configs for every run, the grading sandbox, the analysis code, and the scripts that build every chart and table here.

uv sync                                                        # Python 3.12
make check                                                     # lint, types, offline tests (no API calls)
uv run dllm-eval fetch-data                                    # download and checksum the datasets
uv run dllm-eval run configs/experiments/r2_uc1_main.yaml --dry-run   # cost estimate, no spend
uv run --env-file .env dllm-eval run configs/experiments/r2_uc1_main.yaml --live
uv run dllm-eval score <run_id>                                # grade in Docker (UC1) or score EX (UC2)
uv run dllm-eval report                                        # rebuild every chart and table

Design choices that make the numbers checkable:

• Raw logs are append-only. Every call is recorded with its request (secrets redacted), response, timings, server usage, model id, and git commit.
• Analysis is offline. Charts and tables are rebuilt from the run directories without any API call, and two consecutive builds are byte-identical.
• Live runs refuse a dirty git tree, so every run records exactly which code produced it.
• Every figure carries a footnote with its run id, commit, model ids, service tier, client region, price-list date, and as-of date.

Benchmark data stays under its owners' licenses (HumanEval+ and MBPP+ via EvalPlus, LiveCodeBench, BIRD Mini-Dev) and is downloaded from the official sources, not redistributed. The code is MIT-licensed.

If you find a mistake in these numbers, I want to hear about it.

Data notes. All numbers come from runs on 2026-10-20/21: UC1 pilot r1-pilot-uc1-20261021T020019Z; output-cap probe r1b-cap-probe-20261021T021543Z; OpenAI Fast-mode test oai-priority-ab-20261021T033428Z; code generation r2-uc1-main-20261021T034053Z; output-length sweep r3-uc1-length-20261021T152238Z; first SQL agent run r4-uc2-main-20261021T165313Z; SQL agent on all 496 questions r4c-uc2-full-20261021T214456Z; strict-JSON sub-run r4b-uc2-strict-20261021T170547Z; diffusion capture r5-diffusion-capture-20261021T181518Z. Prices as listed 2026-10-20. Client: one macOS arm64 host, US West; standard API tiers.