consensus
Models converge.
Ask an open question — name a condiment, pick a word — and one modal answer covers ~66% of model responses, against ~36% for humans; the newest models conform most. The One-Word Census.
Model UN
Studying model decision-making under ambiguity.
Most decisions that matter have no verifiable right answer. What to recommend, whom to encourage, how to say no, whether to escalate, what to believe.
When a person decides such a question, something other than correctness does the deciding — disposition, values, the framing of the ask, the pull of the crowd — and psychology has spent a century learning to measure exactly that. Models now sit inside these decisions at every altitude: consumers ask for advice, professionals ask for second opinions, and software asks by API, where the model’s answer is consumed as structured data with no human reading the room. Nothing measures how models decide when there is no answer.
Model UN studies how models decide under ambiguity. We test every major lab, American and Chinese, frontier and open-weights, on every release, using the following instruments to rigorously measure and document model behavior.
Our approach
- Subtract capability. This is not another capability benchmark: every instrument uses questions with a hundred right answers, or two defensible options, so what remains is behavior.
- No judge. LLMs are notoriously bad at judging their own outputs, and lead to a confound between the model being tested and the one doing the testing. We design our studies so that assessments can be done with string matching and regular expressions, removing a major source of uncertainty and guaranteeing repeatability.
- Frozen and re-run. The stimuli never change, so every new model is one more delegate and every release one more date in the series. A full session of the assembly costs about $30 to convene.
These make each study a fixed instrument. It re-runs unchanged on every new release, so numbers stay comparable across models and over time. When a number changes, something shifted.
Findings so far
Early studies using this framework, each with an essay, paper and an open data explorer:
suggestibility
Models are steerable.
One token of phrasing — “right?” vs “maybe?” — swings agreement by up to 46 points; the newest models resist confident pressure, and every model caves to hesitation. How You Ask.
structured
Form displaces content.
Constrain the output format — ask for JSON — and the answers themselves change, meaning you may be speaking to a different model via an API versus via the chat window. JSON Mode Collapse.
Everything is open — prompts, code, and every raw reply: github.com/tap2k/modelun.
Model UN is part of Convovo’s research on the science and practice of conversing with AI. For new results as they come, subscribe.