Google Ran an Eval Neither Side Could See. Seventy-One Seconds of It Was Computation.

Google DeepMind, AVERI, OpenMined and MLCommons ran the first double-blind evaluation of a proprietary frontier model, published 27 August. The evaluator never saw Gemini's weights, Google never saw the test prompts, and the hardware attested to what ran before either asset went in.

The cryptography was the cheap part. The GPU enclave added under 5% overhead and the run itself took 1 minute 11 seconds. Getting the parties to agree on what would execute took 28 minutes and 3 seconds, with everyone on a synchronous call.

Contamination is the reason this exists. Roughly 29% of MMLU test items show signs of leakage, and substituting clean equivalents has knocked as much as 13 points off a model's GSM8K score.

What they actually built

The setup is simpler than the phrase "double-blind evaluation of a proprietary model" suggests.

AVERI encrypted a set of never-before-used prompts from AILuminate, the MLCommons safety benchmark family, covering CBRNE, cyberattacks, hate speech, self-harm and violent crime. Google DeepMind put Gemini 2.5 Flash-Lite in. Both assets went into an NVIDIA H100 secure enclave running OpenMined's PySyft, with Google Cloud Confidential Space underneath. The enclave ran the evaluation and released only the agreed outputs to the agreed recipient. AVERI decrypted the results and applied the AILuminate scoring criteria.

The load-bearing piece is the attestation. Before either party's asset enters, the hardware signs the integrity of every layer beneath the job, from root through BIOS and VM to the application, and a relying party verifies those signatures match what the stakeholders approved. In this pilot the relying parties were NVIDIA and Microsoft Azure's remote attestation services. OpenMined is direct about what that means: you do not have to trust those two companies specifically, anyone can run an attestation service, but you do have to trust whoever runs the one you use.

That is a real reduction in what you have to take on faith, not an elimination of it. The cloud provider and the cloud account holder drop out of the trust set entirely. The attestation provider does not.

The other detail worth stealing is how the evaluation code got written. Researchers built it against mock assets, fabricated stand-ins with identical interfaces to the real prompts and the real model. Every party reviewed and cryptographically signed the code before anything sensitive moved. Nobody debugged against live data, because nobody could.

Why anyone bothered

Benchmark scores have been drifting away from ability for a while, and the industry has mostly handled it by publishing more benchmarks.

The Johns Hopkins measurement is the one to keep in your head: about 29% of MMLU test items carry contamination signals. When researchers swap contaminated items for clean mirrors and re-run, scores fall, in one case by 13 points on GSM8K. Part of what the leaderboards were ranking was memorization.

NIST said the quiet version of this in February, noting that common benchmark practice often fails to quantify uncertainty or prevent overfitting to test sets. In March, its Center for AI Standards and Innovation signed a formal research agreement with OpenMined to use PySyft for secure evaluations, which is how a privacy-engineering library ended up in the middle of a national measurement program.

The failure mode is not usually fraud. It is that a test set stops being a test set the moment enough parties have seen it, and there is no clean way to prove how many have. A benchmark is a secret with a decay curve, and nobody publishes the curve.

The 28 minutes

Total wall clock for co-creating and approving the computation: 28 minutes and 3 seconds. Time spent transferring assets into the enclave, executing, and depositing results: 1 minute and 11 seconds.

Everything else was four organizations agreeing on what would run.

That ratio is the actual finding of the pilot, and it points the opposite way from where most people assume the difficulty lies. Confidential computing on H100s is a solved product with a sub-5% tax. The unsolved part is the sociotechnical layer, the review, the sign-off, the policy definition, the question of who is allowed to see which output. OpenMined names this directly and puts shortening the collaboration process in future work.

If you have ever tried to get a data-sharing agreement through two legal teams, 28 minutes on one call is an absurdly good result. It is only slow relative to 71 seconds of compute. That comparison is the trap. The governance was not overhead on the crypto, it was the work.

What it does not cover

The pilot is honest about its edges, which is more than most launch posts manage.

Model architecture was not protected. The pilot used publicly visible architectures, and hiding a proprietary one is listed as future work through NNsight integration. So the weights stayed private, the shape did not.

Side channels remain open at scale. Clustering multiple GPU enclaves introduces timing-based leakage that the current design does not address.

AVERI's own framing is that the security design handled many but not all tampering vectors, and that higher-stakes uses, the example given is verifying international agreements, would need additional assurances. Some of the pilot's code was proprietary and could not be fully inspected. Scores were delivered privately to Google DeepMind and were not published.

That last one is worth sitting with. The first double-blind evaluation of a frontier model produced a result the public cannot see. The mechanism proves an evaluation happened under agreed conditions. It does not, by itself, make the outcome public.

The part that transfers

You are not going to stand up an H100 enclave to test a vendor model. The framing is the part that travels.

Treat your held-out set as a depreciating asset with a visibility count, not a binary clean-or-dirty flag. The useful question is not whether your eval is contaminated. It is how many parties have seen it, through how many channels, and whether you could demonstrate that to someone who did not take your word for it.

Three things follow from that, and all of them are cheap.

Keep one holdout that never leaves your infrastructure and never goes into a prompt sent to a third party. Not your main eval, a small reserve you spend deliberately. The moment you send a test case to a vendor endpoint, you have added a party to the count, whatever the retention policy says. Policy is a promise. Attestation is a proof. Most of us are working with promises, so at least know which set you have only promises about.

Measure the delta, not the score. Run the public benchmark and your private clean set, and watch the gap between them over model versions. A widening gap is the signal, and it costs you one extra run to see it. The absolute scores get all the attention and tell you less.

Write down who approved what before you run anything that matters. The pilot spent 96% of its time on that and treated it as the interesting problem rather than the friction. In a normal team the equivalent is knowing which eval gates the release, who signed off on the threshold, and whether the person reading the number knows how old the test set is.

The enclave is the impressive artifact. The 28 minutes is the lesson.