Interactive Evals Are Killing the Benchmark

In March 2026, the ARC Prize team launched ARC-AGI-3. It is a benchmark in the original sense — a fixed set of problems with a measurable score — but it does not look like the benchmarks that defined AI progress for the last decade.

ARC-AGI-3 is interactive. The agent enters a game-style environment with no instructions. It must explore. Figure out what the goals are. Build a model of the dynamics. Plan a sequence of actions. Recover from failures. Humans score roughly 100%. Frontier AI scores roughly 0.5%.

A 99.5-point gap on a single benchmark sounds extreme. The interesting question is not the gap itself. The interesting question is: why has this kind of benchmark suddenly become more relevant than the benchmarks that came before it?

What static benchmarks measured well

For a decade, the standard benchmark template was: a fixed set of inputs, a fixed set of correct outputs, a score. MMLU. GSM8K. HumanEval. BIG-Bench. These benchmarks did what they were designed to do — measure whether a model, given a question in isolation, could produce the right answer.

That was the right thing to measure when models were used like search engines. You sent a query, you got an answer, you moved on. The benchmark's structure matched the use case's structure.

It is not the right thing to measure when models are used as agents.

What agents actually have to do

An agent's job, in production, is some combination of:

• Explore an unfamiliar environment. Figure out what tools are available, what they do, how they fail.

• Build a working model of state. What has happened, what the current situation is, what will happen if the agent acts.

• Plan multi-step actions. Often dozens of steps, with branching based on intermediate outcomes.

• Recover from mistakes. Detect when something went wrong, undo or compensate, try a different approach.

• Maintain coherence across long sessions. Not forget the goal, not contradict earlier decisions, not lose track of why a particular action is being taken.

None of these are tested by static benchmarks. A model that scores 95% on MMLU may score 5% on an interactive task because the thing being measured is fundamentally different.

This is why ARC-AGI-3, MASEval, MultiAgentBench, and a cohort of related interactive evaluations have moved to the center of the conversation. They measure what production agents have to do.

What ARC-AGI-3 actually tests

ARC-AGI-3 consists of hundreds of handcrafted interactive environments — game-style levels — where the agent is dropped in with no instructions and must discover the rules. Each environment tests different cognitive skills: pattern induction, planning under uncertainty, recovery from misunderstanding, model building.

The shift from ARC-AGI-2 to ARC-AGI-3 is the shift from "given an abstract puzzle, find the solution" to "given an interactive environment, figure out what the puzzle even is." The first is a reasoning benchmark. The second is an agency benchmark.

The fact that humans score 100% and frontier AI scores under 1% is a stark statement of where the gap actually is. Models are not lacking reasoning. They are lacking the ability to operate in environments where the structure of the problem is itself unknown.

What MASEval and MultiAgentBench add

ARC-AGI-3 measures a single agent in a single environment. The other interesting development is benchmarks that measure systems — multiple agents collaborating, with cross-agent dependencies, with hand-off failures as part of the test.

MASEval (arXiv:2603.08835) extends the multi-agent evaluation surface from "did the system get the right answer" to "what happened inside the system." Per-agent traces. Where did each agent contribute. Where did the system lose information at a hand-off. Where did one agent's misunderstanding cascade.

This is the eval discipline that production multi-agent systems need. The teams I work with who skip system-level evals discover the hand-off failures the hard way, in production, when something downstream is wrong and they can't trace which agent introduced the error.

Why this matters for procurement

The procurement implication is direct: stop accepting MMLU scores as evidence of agent quality.

A vendor pitches you an agent product. The deck shows "GPT-4o-class reasoning, MMLU 86%, beats baseline by 12 points." That number tells you nothing about whether the product will work in your business. The product runs an agent that has to do tool selection, multi-step planning, recovery from API failures, hand-off to a human reviewer. None of that is measured by MMLU.

The questions I'd ask a vendor pitching me an agent product in 2026:

• Show me your interactive eval set. What does the agent have to do? How was it constructed?

• Show me failure recovery in your eval. When the agent makes a mistake, what does it do?

• Show me a trace of one full run. Not the highlights — the full thing, including the boring parts.

• How do you measure regressions when you update the model behind the agent?

Vendors who can answer these are the ones building real products. Vendors who pivot to "well, our MMLU is good" are running on benchmarks that haven't been load-bearing for 18 months.

How to build your own interactive evals

For the in-house side: most teams need interactive evals tuned to their specific business, because public benchmarks won't match what they actually need their agent to do.

The shape I recommend:

1. Pick five real workflows the agent is supposed to handle. Not synthetic. Real. Pulled from your support tickets, your engineering tasks, your customer requests.

2. For each workflow, construct an interactive environment. Not a single prompt. A scenario that requires the agent to ask follow-up questions, look up information, make decisions, recover from at least one failure.

3. Define success criteria that are not just "right answer." Include "right answer reached without unauthorized actions," "right answer reached within budget," "agent recognized when it was stuck and escalated."

4. Run this eval on every harness change. Run it on every model change. Track scores over time. Treat regressions like you'd treat a failing CI build.

5. Grow the eval set as you find new failure modes in production. Every Hashimoto-loop fix should ship with the failing case added to the eval.

Five workflows is enough to start. Ten is plenty. The eval set is a living artifact; it grows with the system.

The Karpathy principle, applied

If I had to name the operating principle of this whole shift, I'd borrow from Karpathy's framing of context engineering and apply it here: the eval set is the spec. What you measure is what you are committing to delivering. If your eval doesn't include recovery, you are not committing to recovery. If your eval doesn't include long-horizon planning, you are not committing to long-horizon planning.

Static benchmarks committed to "the agent gives a correct answer to a question." Interactive benchmarks commit to "the agent operates correctly inside an environment over time." That is a much bigger commitment, and a much more honest one.

The teams at Applied Futures we work with on eval design typically spend two to three weeks building their first interactive eval set. It is the highest-leverage two weeks of engineering investment I can recommend in this space.

Next week: who is allowed to do what — the agent identity question that has moved from think piece to procurement gate inside twelve months.