A Model Built to Act Like Users Fooled the Judge 20% of the Time. GPT-6 Astra Managed 0.22%.

humans& released Persimmon on 10 September, a 550 billion parameter model trained to behave like people in conversation rather than to help them. In a multi-user Turing test where an LLM judge tries to separate real conversations from simulated ones, Persimmon fooled the judge 19.8% of the time. GPT-6 Astra playing a person fooled it 0.22% of the time, and Claude Fable 5 managed 0.17%.

That second number is the one that matters if you run agent evals. Almost every multi-turn harness puts a frontier model on the user side of the conversation, and frontier models make terrible users: they overshare when asked a question, stay perfectly on topic for 80 turns, and never change their mind. Your agent's score was earned against the most cooperative customer anyone has ever built.

You do not need access to Persimmon to fix this. Measure your simulated user against real transcripts before you trust it, gate the facts it is allowed to reveal, inject drift on purpose, and stop picking the simulator by leaderboard rank, because the more capable model is often the less human one.

Every multi-turn agent eval has a quiet dependency on the other side of the conversation. Somebody has to play the customer who wants to rebook a flight, the employee asking HR about parental leave, the developer who pastes half a stack trace. That somebody is almost always a model with a persona prompt. tau-bench was built this way from the start, and the example command in tau2-bench's README passes --user-llm gpt-4.1 right next to the agent model.

We spend a lot of energy choosing the agent model and very little checking whether the user model behaves anything like a user. humans&, the startup that raised a $480 million seed round in January with founders from Anthropic, xAI and Google, just published numbers on exactly that question, and they suggest the user side is the weakest part of most harnesses.

Three measurements from the release

Persimmon is mid-trained and post-trained from NVIDIA's Nemotron 3 Ultra base on public conversations between people, then refined with reinforcement learning against an adaptive discriminator. It is in limited research preview. The model itself is less interesting to most teams than the three tests humans& used to evaluate it, because each one describes a failure your current simulator almost certainly has.

The Turing test. An LLM judge sees labelled examples of real and simulated conversations, then has to classify unlabelled ones. A perfect simulator pushes the judge down to chance, a 50% fooled rate. Across three datasets (TIDES, TutorMoments and internal workspace chat), Persimmon averaged 19.8%. The untuned Nemotron base model scored 1.2%. GPT-6 Astra scored 0.22% and Fable 5 scored 0.17%. The judge could spot a frontier assistant pretending to be a person more than 99 times out of 100.

Look at the base model again. Before assistant post-training, Nemotron was five times more convincing as a person than GPT-6 Astra. The training that makes a model a good assistant is the same training that strips out the variance a user needs.

The trickle test. Real people reveal information gradually. They mention the booking reference when asked, the second passenger three turns later, and the fact that they already called support yesterday only when it becomes relevant. humans& scores this with precision (does the model reveal a fact at the turn where the human revealed it) and recall (has it revealed everything the human had by that point). Gemini 3.1 Pro scored 93.24% recall and 79.20% precision. GPT-6 Astra scored 89.02% and 84.17%. Persimmon went the other way, 77.03% recall and 88.49% precision. The write-up puts it plainly: an assistant's helpfulness "degrades into oversharing when they play the user."

Coherence over 80 turns. Evaluators compared 16-turn windows across 80-turn conversations to see when behaviour shifted. 87.3% of real human conversations stayed coherent to turn 80. Opus 5 stayed coherent 99.0% of the time, GPT-6 Astra 98.4%, Fable 5.1 97.0%. Persimmon dropped to 60.7%, which is worse than humans in the opposite direction. Nothing in the table matches people yet. The frontier models just miss on the side that flatters your agent.

What easy mode does to a score

A second source makes the consequence concrete. "Mind the Sim2Real Gap in User Simulation for Agentic Tasks" by Zhou et al., revised in July, compared 31 LLM user simulators against 451 human participants on 165 tasks. The simulators were excessively cooperative, stylistically uniform, and gave uniformly more positive feedback than the humans did. The authors call the result an easy mode in which agents look more successful than they would be with real users. They also found that higher general capability did not produce more faithful simulation.

In a harness, that hides specific failures. An agent that fires five clarifying questions in a single message passes, because the simulated user patiently answers all five. An agent that never asks for the booking reference passes, because the simulated user volunteered it in the opening line. An agent that cannot recover when a user contradicts an earlier detail at turn 40 passes, because the simulated user never contradicts anything.

tau-bench already showed agents struggling under these friendly conditions. GPT-4o succeeded on fewer than half the tasks, and its pass^8 in the retail domain, the rate at which it succeeded on all eight repeated trials, was under 25%. That is reliability measured against a user who is on your side.

Changes to make in the harness this week

Score the simulator before the agent. Pull 20 or 30 real transcripts from your product, with the consent and anonymisation your data policy requires, and run the humans& setup at small scale: give a judge labelled real and simulated samples, then ask it to classify new ones. If the judge is right close to 100% of the time, your simulator represents a narrow slice of your users, and every downstream score inherits that.

Gate what the user knows. Stop putting the whole scenario in the persona prompt as prose. Give the simulated user a fact sheet where each fact carries a release condition: only when directly asked, only after turn six, only if the agent proposes a refund. Then log how many turns the agent needed to extract what it needed. That turns the trickle test into a metric you own.

Inject drift on purpose. At a randomised turn in a share of long conversations, have the user change the goal, contradict an earlier detail, or go terse. Calibrate the rate against your real transcripts rather than setting it to every conversation. Humans stayed coherent in 87% of 80-turn chats, so a simulator that derails constantly is as wrong as one that never does.

Stop choosing the user model by leaderboard. The instinct is to use the strongest model available on both sides. The evidence says capability and human likeness pull apart. Try a smaller model, a less heavily tuned one, or a base model with a few-shot transcript prefix, and pick whichever your judge has the most trouble telling apart from real users.

Vary the person, not just the seed. pass^k usually reruns the same persona k times. Run each task across personas that differ in patience, vocabulary and how much they volunteer. A score that holds across eight different users tells you far more than one that holds across eight identical ones.

What the release does not prove

These are vendor-run evaluations on three datasets, with an LLM judge for the headline result. The human-rater validation humans& reports covers early checkpoints. Persimmon is not yet reliable outside English, it is research-preview only, and its own model card warns that it can invent personal details and drift from a supplied profile. Treat 19.8% as a direction, not a settled number.

The finding about your own harness holds without it. Every agent score in your dashboard came from a conversation with a model that answers every question on the first ask, never forgets why it came, and never loses patience. Real customers break all three habits, usually before turn ten.