Give me JSON, Hold the Mustard

When you ask an LLM for a structured response (JSON, XML), you generally expect to receive similar answers as when you ask for plain text. However, one simple experiment shows that is not always what you get.

In the One-Word Census we asked 44 language models questions with a hundred right answers — “Name a tree.” “Pick a word — any word” — and found a monoculture: 94% oak, 41% serendipity. We also found the exceptions: models with stable, off-consensus defaults. Claude Fable 5 says gouda where the field says cheddar, mustard where it says ketchup. LLM character, measurable for a dollar.

Then we changed exactly one thing. Same questions, same models, same everything — but now: “Reply with JSON only, in the form {“word”: ”…”}.”

The mode gets hungrier

In plain chat, serendipity takes 41% of the “pick a word” pool. Inside JSON it takes 64%, and the number of distinct words collapses from 52 to 36. Fruit: apple goes from 76% to 95%.¹

And the collapse isn’t uniform — it lands almost entirely on the models that had personality to lose.

	plain chat	in JSON	Δ
DeepSeek v3.2	2.63	1.32	−1.31
Hermes 4	3.21	2.30	−0.91
GPT-4o mini	2.10	1.18	−0.92
GPT-4 Turbo	2.06	1.19	−0.87
GPT-5.6 Sol	2.02	1.37	−0.65
…
Claude Opus 4.8	1.25	1.09	−0.16
Grok 4.5	1.26	1.19	−0.07
Claude Sonnet 5	1.06	1.00	−0.05

(Scores are the Mustard Quotient — bits of surprisal against the rest of the field; higher = answers the crowd doesn’t give.)

The conformist floor doesn’t move — there’s nowhere lower to go. The divergent tail falls toward it. DeepSeek v3.2, the one genuine explorer among frontier-lab models, loses half its measured personality the moment you ask for JSON. The field average drops from 1.80 to 1.58 bits, and the distinctive models up top fall the farthest:

Each model’s answer-choice surprisal — open chat (gray) versus JSON (colored), all 44 models. Most get more generic (blue); for a minority (amber) the relative score rises instead — mostly register-invariant models that hold their own answers while the field collapses around them. The distinctive models at the top fall the farthest; the conformist floor can’t.

Across the panel, models hold 144 stable off-consensus picks in chat — the same non-crowd answer, four runs out of four, like Fable’s mustard and gouda. Ask in JSON and only about half of them survive; the rest revert to the crowd. Some models hold their ground (Fable keeps the mustard), most don’t — so a model’s character is partly a property of the register you asked it in.

Maybe it’s the format?

Maybe any formatting instruction does this — extra constraint, less freedom? No — the opposite, in fact. We ran four control formats across the full battery. Asking for the answer in plain square brackets — structure with no data format — makes the field measurably more varied, not less (+0.13 bits, p=.009). And the compression itself is specific: JSON and XML compress the field hard (−0.22 bits, p≈.0002), while YAML and CSV — data formats models mostly just read, not answer in — show no reliable battery-wide effect at all.

That specificity is telling. JSON and XML are the formats models are trained to answer in — function calling, tool use, structured-output modes — and those are exactly the two that collapse the field. So most of this looks trained-in: recently, and on purpose. Not all of it — on the pick-a-word prompt every serialization concentrates the pool (even CSV pushes serendipity from 41% to 52%, YAML to 65%), which JSON-specific training doesn’t explain — but the weight of the evidence moved. The training data taught models that text inside a data structure is the canonical value; tool-use tuning turned that instinct into a reflex.²

And it isn’t the decoder’s doing either. We ran one more column with the schema actually enforced — the API’s response_format parameter, constrained decoding and all — on the 36 models whose providers support it. Enforcement compresses almost nothing beyond the request: −0.03 bits on top of the request’s −0.22. Merely asking for JSON does the damage; making the sampler guarantee it adds a rounding error. The collapse lives in the model, not the machinery.

The register even has its own tastes. In three categories where the chat crowd was undecided, JSON changed the winner: insects flip from ant to butterfly (43% → 60%), board games from chess to monopoly, dances from salsa to tango. Butterfly, monopoly, tango — that’s not conversational English, that’s the example-payload an API tutorial.

Why this matters

Chat is how people try models. Structured output is how software uses them — every agent, every tool call, every pipeline that extracts, classifies, routes, or fills a schema. That traffic is growing faster than chat, and it is being served by a measurably more collapsed model than the one you meet in the app.

Every diversity number we published in the census was, if anything, an understatement for the deployment surface that increasingly matters. The monoculture has a production tier, and the production tier is worse.

The structured-output literature measures whether format hurts accuracy (contested), whether enforcement breaks correctness (it does), whether grammar-constrained decoding can be made more diverse (it can).³ Whether the format changes which answer you get — the thing your users actually receive — is what this study answers.

Try it yourself

As always, everything is open — the data, prompts, and explorer are below.

• Paper — arXiv:2607.18476
• Data Explorer — tap2k.github.io/modelun/structured
• Raw replies — format_register.json
• Analysis — analyze.py

---

Note about AI usage: A collaboration throughout — Claude (Opus 4.8, and Fable 5) helped run the study, build the analysis, draft the figures, and edit the text. The research questions, the methodology calls, and the argument are mine. Both models are also subjects of the study.

Footnotes

1. Method identical to the census: 44 models × 31 categories × 4 runs, temperature 1.0, no system prompt; the JSON wrapper is stripped mechanically (regex, then the census junk guard) before exact-match scoring. Parse survival is 99%+ per model, so none of this is malformed-output artifact. And it isn’t a hidden temperature change either: within-model run-to-run variety stays nearly flat across formats (0.42 → 0.39) while the field-level convergence jumps — the register moves where the probability mass sits, not how much the model samples around. ↩

2. One methodological gift from the controls: some models simply can’t produce the format (one open model emitted a CSV wrapper on 1% of attempts), and their mangled replies masquerade as creativity if you don’t check. ↩

3. The nearest work: format restrictions vs. reasoning accuracy (Let Me Speak Freely, EMNLP 2024, disputed by the dottxt rebuttal); the 2026 “constraint tax” papers on decoder-level enforcement breaking correctness (validity–correctness tradeoffs, tool-calling suppression); automata-based steering for diversity within grammars. ↩