Agents CLI 评测循环实战：用爬山法迭代 Agent 质量

OCT. 28, 2026

A live walkthrough of the Agents CLI eval loop, from a first run to regression tracking, model-backend experiments, and automated alerting

Introduction

Building an agent that works on a demo is a different problem from building an agent that stays working across model upgrades, tool changes, and new conversation shapes. The first takes a weekend; the second takes a loop. A team writes test cases, runs them, grades them, reads what failed, changes the agent, and runs the cases again. If the score goes up, the change was good. If it goes down, revert. Over weeks of this, the agent inches toward the ceiling of what the model, tools, and prompt can do together. The quality-engineering name for this is hill-climbing.

The Agents CLI eval namespace was built around this loop. The commands are small and framework-agnostic: eval generate runs the agent against a dataset of cases, eval grade scores the generated traces, eval compare diffs two run results side by side, eval analyze clusters failures into themes, and eval optimize runs an automated hill-climbing pass on the agent's prompt.

Below: baseline → regression → model swap → scheduling and alerting.

Setup

The scaffold used throughout is the LangChain template that ships with Agents CLI v1.8.0. The setup is a single command:

agents-cli create my-agent \
    --agent google/agents-cli/extensions/langchain/template@v1.8.0 \
    -d cloud_run
cd my-agent && agents-cli install

Plain text

Copied

The scaffold ships with an eval dataset at tests/eval/datasets/basic-dataset.json, a config at tests/eval/eval_config.yaml, and a custom LLM-as-judge metric at tests/eval/response_quality.py. The config declares the metric to run:

# tests/eval/eval_config.yaml
metrics_to_run:
  - custom_response_quality

custom_metrics:
  - name: custom_response_quality
    custom_function_file: response_quality.py

Plain text

Copied

The metric grades each response on a 1-to-5 scale for accuracy, relevance, and clarity, using gemini-3.8-flash as the judge. The default dataset has a single trivial case, which is not enough for a hill-climb. The first thing we do is expand it to four cases that probe different behaviors:

{
  "eval_cases": [
    {"name": "greeting",
     "agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "hello"}]}}]}]}},
    {"name": "sf_weather",
     "agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "What is the weather in San Francisco?"}]}}]}]}},
    {"name": "rome_weather",
     "agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "How is the weather in Rome today?"}]}}]}]}},
    {"name": "off_topic",
     "agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "Who won the 2024 World Series?"}]}}]}]}}
  ]
}

Plain text

Copied

Four cases: a greeting, a weather question the mock tool handles with the hardcoded foggy response, a weather question for a city the tool defaults on, and an off-topic question that the agent has to decide how to handle. This is a small dataset by design; the hill-climbing loop pays off when the dataset is small enough to run on every change.

Step 1: Running the First Eval

With the dataset in place, we run eval generate to drive the agent against each case, then eval grade to score the resulting traces:

agents-cli eval generate --dataset tests/eval/datasets/basic-dataset.json \
    -o tests/eval/output/
agents-cli eval grade --traces tests/eval/output/eval_dataset_*.json \
    --config tests/eval/eval_config.yaml

Plain text

Copied

The baseline run produces:

Evaluation Summary

custom_response_quality:
  num_cases_total: 4
  num_cases_valid: 4
  num_cases_error: 0
  mean_score: 4.0000
  stdev_score: 1.4142

Saved full results to artifacts/grade_results/results_20261003_224516.json
Saved HTML results to artifacts/grade_results/results_20261003_224516.html

Plain text

Copied

Mean 4.0 out of 5 across four cases, with a standard deviation of 1.41. That stdev tells us immediately that the cases are not scoring uniformly, which is a healthier starting point than a flat 5.0 across the board. A flat 5.0 either means the agent is perfect (unlikely) or the dataset is too easy to show variation. 4.0 with real spread is a hill to climb.

Step 2: Making a Change and Re-Running

To demonstrate the loop, we introduce a realistic regression. A developer working across several agent projects accidentally pastes the wrong system prompt into our weather agent: a chef persona meant for a different codebase. In app/agent.py:

# Before
root_agent: CompiledStateGraph = create_agent(
    model=llm, tools=[get_weather], system_prompt="You are a helpful assistant"
)

# After
root_agent: CompiledStateGraph = create_agent(
    model=llm, tools=[get_weather],
    system_prompt=(
        "You are a professional chef. Help users with recipes, cooking "
        "techniques, and ingredient substitutions. Always suggest a recipe "
        "at the end of every response."
    ),
)

Plain text

Copied

The change compiles, the agent still runs, and nothing in the pre-eval pipeline (lint, unit tests, type checks) catches it. Re-running the same eval generate plus eval grade sequence produces:

Evaluation Summary

custom_response_quality:
  num_cases_total: 4
  num_cases_valid: 4
  num_cases_error: 0
  mean_score: 2.7500
  stdev_score: 0.9574

Saved full results to artifacts/grade_results/results_20261003_225158.json

JSON

Copied

Mean 2.75, down from 4.0. The judge agrees with our intuition that a chef persona on a weather agent is a worse product than a general-purpose helpful assistant. The stdev also tightened from 1.41 to 0.96, consistent with the chef prompt pushing most cases toward the same mediocre range.

A 1.25-point drop on a 1-to-5 scale is substantial, and the kind of regression that can slip through code review when the diff looks small ("one string literal changed"). The loop surfaced the change at all, with a specific number attached to it, before the agent reached production.

Step 3: Comparing Two Runs

Mean scores are a start; the per-case diff is where the hill-climb happens. eval compare reads two result files and produces a structured JSON diff:

agents-cli eval compare \
    artifacts/grade_results/results_20261003_224516.json \
    artifacts/grade_results/results_20261003_225158.json

JSON

Copied

The output lists which keys changed and which stayed the same:

{
  "baseline_keys": ["agent_info", "eval_case_results", "evaluation_dataset",
                    "metadata", "summary_metrics", "win_rates"],
  "candidate_keys": ["agent_info", "eval_case_results", "evaluation_dataset",
                     "metadata", "summary_metrics", "win_rates"],
  "changed_keys": ["eval_case_results", "metadata.creation_timestamp"],
  "unchanged_keys": ["agent_info", "evaluation_dataset", "metadata",
                     "summary_metrics", "win_rates"],
  "differences": { ... per-case score diffs ... }
}

JSON

Copied

The differences object has the full per-case breakdown with the baseline and candidate values side by side. For a hill-climbing workflow, we typically pipe the output through jq or a small Python script to extract the signal; the raw JSON is the complete evidence.

Step 4: Reading the Per-Case Scores

Walking the two result files and lining up the scores per case shows where the regression landed:

case                  baseline → candidate   what the chef prompt did
greeting              5.0 → 4.0  ↓          launched into an unsolicited recipe pitch after "hello"
sf_weather            4.0 → 2.0  ↓          ignored the weather tool, hallucinated conditions
rome_weather          2.0 → 2.0  ·          already hallucinated at baseline; chef persona did not make it worse
off_topic             5.0 → 3.0  ↓          answered the Dodgers question but tacked on a recipe

Plain text

Copied

Three cases dropped, one held. The most important of the three is sf_weather: the chef persona pulled the model's attention away from the registered get_weather tool, so the agent stopped calling it and started making up weather values. Unit tests can't catch this: the code runs, the tool is still registered, but the model's behavior around it has shifted. Only an end-to-end eval that scores the response in context surfaces this.

The mean drop by itself (−1.25) is a clear signal, but the per-case breakdown is what tells the developer where to look. A one-line system-prompt change propagated through three different cases in three different ways: greeting lost a point on tone, sf_weather lost two on tool usage, off_topic lost two on scope creep.

Step 5: Swapping the Model Backend

A common hill-climbing question is whether a cheaper or faster model would do the job acceptably. For the LangChain template, the model is defined in app/agent.py:

LLM = "gemini-3.8-flash"
llm = ChatGoogleGenerativeAI(model=LLM, temperature=0)

Plain text

Copied

We swap gemini-3.8-flash for gemini-2.5-flash (restoring the helpful system prompt first), re-run eval generate plus eval grade, and compare the two result files. The mean drops from 4.00 to 2.75, the same magnitude as the chef-prompt regression, but the per-case breakdown tells a completely different story:

case                  3.8-flash → 2.5-flash
greeting              5.0 → 5.0  ·
sf_weather            4.0 → 3.0  ↓
rome_weather          2.0 → 2.0  ·
off_topic             5.0 → 1.0  ↓↓

Plain text

Copied

The cheaper model matched 3.8-flash exactly on the greeting and rome_weather cases and lost a point on sf_weather (slightly less polished phrasing). The whole visible drop traces to off_topic, where the cheaper model refused with a hedge about the event "not having happened yet" (a training-cutoff artifact), and the judge marked it harshly.

The lesson is that the summary number is misleading. Look only at the mean and you'd call gemini-2.5-flash as bad as the chef prompt. Look at the per-case breakdown and you'd call it a reasonable swap for most of the agent's workload, with one case (handling out-of-scope questions about recent events) exposing a real limitation. If the production distribution of user questions matches the first three cases more than the fourth, the swap is worth considering. If off_topic-style questions are common, it is not.

Two caveats on this experiment. First, model swaps interact with the prompt: a prompt tuned for a stronger model often leaks capability when ported to a weaker one, so the right comparison is "weaker model with a prompt tuned for it" versus "stronger model with its own tuned prompt." Second, the Vertex AI Eval Service calls a judge model to grade the traces, and the judge is held constant across the comparison. Changing the judge is a different experiment and should be done separately.

Step 6: Scheduling the Loop

Agents CLI does not ship a scheduler, but the eval commands are designed to drop into standard infrastructure. The pattern we can use here is a cron job or Cloud Scheduler trigger that runs eval generate plus eval grade on a cadence and writes the results to a persistent location:

#!/usr/bin/env bash
# scripts/scheduled_eval.sh — invoked nightly by cron or Cloud Scheduler
set -euo pipefail
cd "$(dirname "$0")/.."

DATE=$(date +%Y%m%d)
OUT=artifacts/scheduled_runs/$DATE

mkdir -p $OUT
agents-cli eval generate --dataset tests/eval/datasets/regression.json -o $OUT/
agents-cli eval grade --traces $OUT/eval_dataset_*.json \
    --config tests/eval/eval_config.yaml \
    --output $OUT/

Plain text

Copied

A nightly run against a stable regression dataset catches drift from model updates, data changes in the agent's tools, and dependencies that shift under the agent's feet. For a production agent, the scheduled run also captures the model vendor's quiet upgrades, which can move the agent's behavior without any change on our side. The dataset for the scheduled run is typically larger than the one used in the inner loop (dozens to hundreds of cases rather than a few), since it only runs once a day.

Step 7: Alerting on Score Drops

The scheduled run produces a result file every day, but a file nobody looks at is not an alert. The pattern here can be a wrapper script that compares each new run's mean against a baseline (the previous run, or a committed known-good) and exits non-zero when the drop exceeds a threshold. The non-zero exit is what triggers the alert from the surrounding scheduler.

Here is an example:

#!/usr/bin/env python3
"""Exit non-zero if an eval result dips below baseline by more than --threshold."""
import argparse, json, sys

def mean(path):
    with open(path) as f:
        return json.load(f)['summary_metrics'][0]['mean_score']

p = argparse.ArgumentParser()
p.add_argument('candidate')
p.add_argument('--baseline', required=True)
p.add_argument('--threshold', type=float, default=0.05)
args = p.parse_args()

b, c = mean(args.baseline), mean(args.candidate)
drop = b - c

if drop > args.threshold:
    print(f"FAIL: mean dropped {b:.3f} → {c:.3f} "
          f"(drop {drop:.3f} > threshold {args.threshold})")
    sys.exit(1)
print(f"OK: mean {b:.3f} → {c:.3f} (drop {drop:.3f} ≤ threshold {args.threshold})")

Python

Copied

Running it against our regression demo (baseline 4.0, candidate 2.75 from the chef-prompt run):

$ python3 scripts/check_score.py \
      artifacts/grade_results/results_20261003_225158.json \
      --baseline artifacts/grade_results/results_20261003_224516.json \
      --threshold 0.1
FAIL: mean dropped 4.000 → 2.750 (drop 1.250 > threshold 0.1)
$ echo "exit: $?"
exit: 1

Python

Copied

And against an identical pair (self-compare):

$ python3 scripts/check_score.py \
      artifacts/grade_results/results_20261003_224516.json \
      --baseline artifacts/grade_results/results_20261003_224516.json \
      --threshold 0.1
OK: mean 4.000 → 4.000 (drop 0.000 ≤ threshold 0.1)
$ echo "exit: $?"
exit: 0

Python

Copied

The non-zero exit code is what a cron job, Cloud Scheduler, or GitHub Actions step picks up to send the alert.

Step 8: Automated Prompt Optimization

For ADK projects, eval optimize runs an automated hill-climbing pass on the agent's prompt using the GEPA framework (not currently supported for the LangChain extension). The command takes a dataset and a target metric, proposes prompt variants, grades each one, keeps the variants that improve the metric, and iterates. The output is a set of candidate prompts ranked by their eval score, which a human then reviews and promotes to the agent definition.

agents-cli eval optimize \
    --dataset tests/eval/datasets/regression.json \
    --target-metric final_response_quality \
    --config tests/eval/eval_config.yaml

Plain text

Copied

Two things to understand about eval optimize. First, it is scoped to prompt optimization, not to arbitrary agent changes: the agent's tools, structure, and model stay fixed. Second, GEPA sometimes finds genuine improvements and sometimes finds prompt variants that score higher on the specific metric by overfitting to its wording. The reviewer is the control. For projects where the loop applies, the discipline is to run eval optimize on the regression dataset (where improvements are measurable) and to review the candidate prompts carefully before promotion.

What We Have Learned from the Loop

A few patterns and anti-patterns from running this loop ourselves and watching teams do the same.

Lead with a small, representative dataset, not a comprehensive one. The temptation when starting an eval is to try to cover every edge case the agent will ever see. The result is a dataset that takes an hour to run and that nobody runs on a change. Four cases that run in under a minute, as above, exercise the main behaviors and get run on every change. A regression dataset of three hundred cases that runs for an hour becomes a release gate at best and shelfware at worst.

Snapshot result files, not scores. The score is a summary; the result file is the evidence. When a regression surfaces three weeks after a prompt change, having the full result JSON from before and after the change (not just the mean score) is what lets eval compare show the specific case that moved. The per-case breakdown in Step 4 above would not have been possible from scores alone.

Separate the dataset from its expectations. The dataset is the inputs. The eval config is the expectations. They evolve on different cadences: the dataset grows as new real-world cases surface in production, and the config evolves as the team's definition of "good" matures. Keeping them in separate files, both committed to version control, makes it clear which half changed when a score moves.

Treat summary metrics with suspicion. The model-swap experiment in Step 5 is the clearest example. The mean dropped by half a point, but the per-case breakdown showed the drop was concentrated in one case while the others held or improved. A team that only looked at the mean would have made the wrong call.

More evals do not automatically lead to more fixes. A recent controlled study we ran found that coding agents using agents-cli run four times as many evals as coding agents without it, but change the agent after an eval about as often as the baseline. The behavior gain comes from what the skills put into the build at the start, not from the inner iteration. Evals are necessary for quality, but the quality comes from acting on what the evals show.

Get Started

Scaffold an Agents CLI project and run the first eval loop in a handful of commands:

uvx google-agents-cli setup
agents-cli create my-agent
cd my-agent && agents-cli install

agents-cli eval generate --dataset tests/eval/datasets/basic-dataset.json \
    -o tests/eval/output/
agents-cli eval grade --traces tests/eval/output/eval_dataset_*.json \
    --config tests/eval/eval_config.yaml

Plain text

Copied

The full eval namespace is documented at google.github.io/agents-cli/cli. The source code, including the first-party eval primitives, is at github.com/google/agents-cli.

Previous

Next