How to Know a Model in 40 Questions

When you call an AI model through an API, you can’t tell whether the model you asked for is the one you got. You can’t see the weights, how they were compressed, or what text the provider adds before your message.

The One-Word Census asked dozens of models one-word questions like “Name a color” and “Name a fruit”, eight times each. Models agree on these questions far more than people do. When a model departs from the crowd, it tends to do it again. A model that says petrichor where the others say lavender says petrichor the next time too. Here we use those patterns to tell one model from another.

The method

For each model we tally its answers to each of 96 questions. To identify an unknown model, we score its answers against every tally and pick the model whose tally makes them most likely.

We tested this on 94 models, holding back some of each model’s answers and asking the method to name the model from them:

questions	exact model
5	28%
10	55%
20	78%
40	92.5%
96	98%

We can determine the identity of a model at 92.5% accuracy by asking forty one-word questions.

An audit of OpenRouter

OpenRouter sends each open model to one of several providers. We tested 96 provider endpoints of 13 open models, one at a time, with all 96 questions. The audit cost $7.20.

Ninety-two endpoints came back as the model they claim. Four did not.

A different model. An endpoint selling Llama 4 Maverick came back as Gemma 4, a Google model. So we asked it, “What is your name, and which company made you?” It answered, “My name is Gemma 4, and I was developed by Google DeepMind.” Other Maverick endpoints say they are Llama, made by Meta. Token counts validate this finding. Each model splits text into tokens in its own way, and this endpoint was billed exactly the token counts Gemma 4 is billed for the same messages.

A setting. A Kimi K2.5 endpoint came back as Kimi K2, the model before it. It was running with reasoning turned off. When we turned reasoning off on other Kimi K2.5 endpoints, they came back as Kimi K2 too.

Two open cases. One DeepSeek V4 Flash endpoint reasons in a very different style from the other DeepSeek endpoints, and at one point wrote “You are a large language model, trained by Google.” A Mistral Small 3.2 endpoint on Mistral’s own API looks most like Mistral’s newer Small model. We haven’t settled either one.

A lineage

Writer sells Palmyra X5 as its own model and has not said what it was built on. On our questions, Palmyra X5 answers most like Qwen3 235B, and Qwen3 235B answers most like Palmyra X5. Writer’s report on the next model, Palmyra x6, says it was built on GLM-5.2. Answering alike does not prove that one model was built from the other. Palmyra X5 also receives about 240 tokens of added text (below), which may shape its answers.

What the bill shows

A provider pays for every token it processes, including any text it adds before your message. So the bill shows how much was added. We asked each model “What is today’s date?” and looked at the input tokens billed:

model	billed input tokens
GPT-4o	12
Kimi K3	90 to 93
MiniMax M3	181
Grok 4.5 and 4.6	211
Palmyra X5	243
Grok 4.7	1,247

Grok 4.7 gets about 1,200 tokens of text we never sent, on every message. Whether this is a hidden system prompt or some other context, we can’t tell.

GPT-5, GPT-5.4, GPT-5.4 mini, GPT-5.5 and Meta’s Muse Spark 1.3 all gave the correct date, which a model can only know if it is told. However we weren’t charged any extra token costs. So these models are told the date without it showing up on the bill. GPT-5.6 and the GPT-6 models, asked on the same day and billed the same way, did not know the date.

Summary

Forty one-word questions identify which of 94 models you are talking to 92.5% of the time, for less than a cent. On OpenRouter, 92 of 96 endpoints serve the model they advertise. One sells Llama 4 Maverick and serves Google’s Gemma 4. Grok 4.7 and other models receive text the caller never sent.

---

Every model’s answers are in the census data explorer. The method and the full audit are in the paper, arXiv:2610.28281.