Original research brief
A New Method Measures and Improves AI Agent Consistency
A new diagnostic tool measures the consistency gap in AI agents, where a ReAct agent using GPT-4.1 showed a 24.4-point difference between average and…
What this piece is grounded in
01
A ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions.
02
It succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap.
03
This post introduces consistency guidelines, a new guideline type in altk-evolve.
04
It is built on top of a diagnostic tool we call the Consistency Analyzer.
01 / FIELD NOTE
What Changed: A New Metric and Diagnostic Tool
A new research post introduces a method to measure and improve the consistency of AI agents. The source reports that standard agent evaluation reports Mean@k, which averages pass rates across multiple runs. The post defines a different metric, Pass^k, which is the fraction of tasks where the agent succeeds on all k runs. According to the source, a ReAct agent using GPT-4.1 posted a Mean@5 of 77.4% but a Pass^5 of only 53.0%, creating a 24.4-point consistency gap. The post introduces consistency guidelines, a new guideline type built on a diagnostic tool called the Consistency Analyzer, designed to target this gap directly. The supplied feed entry is a concise summary; consult the canonical article for full context.
02 / FIELD NOTE
Why Agents Make Different Decisions on Identical Tasks
The source explains that an agent's decision comes from a probability distribution over next tokens. A sharp distribution puts most mass on a single token, leading to repeatable choices. A flat distribution spreads comparable mass across several near-tied tokens, making the outcome vulnerable to small perturbations. The source states that because a trajectory chains dozens of decisions, a small per-step chance of flipping compounds into a large chance that some run goes differently. The post notes the ReAct agent runs at temperature 0.0, so the variance is not from ordinary sampling. This analysis turns the problem into identifying which steps have flat distributions.
03 / FIELD NOTE
Editorial Interpretation: Practical Implications for AI Work
This interpretation is based on source facts. The reported 24.4-point gap between Mean@5 and Pass^5 indicates that an agent's single-run success rate does not guarantee repeatable performance. For practical AI work, this means a workflow that succeeded once may fail on an identical subsequent request, which the source identifies as a reliability problem for mission-critical tasks. The introduction of a black-box Consistency Analyzer, which the source says replays each decision step through controlled resampling, provides a method to pinpoint at-risk decisions without requiring logits or model internals. The generation of targeted guidelines from flagged steps, as described, offers a path to mitigate inconsistency within an existing system architecture.
04 / FIELD NOTE
What a Reader Can Verify and What Remains Unclear
A reader can verify the definitions of Mean@k and Pass^k as presented in the source, and the specific numeric results: a ReAct agent backed by GPT-4.1 posts a Mean@5 of 77.4% and a Pass^5 of 53.0%. The source describes the two-stage pipeline: detection via the Consistency Analyzer and generation of targeted guidelines. The supplied feed entry is a concise summary; consult the canonical article for full context. For instance, the source provides an example of a generated guideline for an AppWorld task, but the full performance results and implementation specifics for the new consistency guidelines are detailed in the primary source.
Questions readers ask
What is the difference between Mean@k and Pass^k?
According to the source, Mean@k is the average pass rate across k runs of a benchmark. Pass^k is the fraction of tasks where the agent succeeds on all k runs. The source states that Pass^k is its pessimistic mirror image: every attempt must succeed, whereas Mean@k answers how good the agent is on average.
How does the Consistency Analyzer work?
The source states the Consistency Analyzer replays each decision step from a recorded trajectory through controlled resampling. It makes one additional model call per decision step offline, with the sampling parameter set to draw k completions at once, replayed against the already-recorded context. This yields a consistency score per decision step to pinpoint which decisions are at risk of flipping.