EngineeringAIEvalsGuides

Evals and hill-climbing for AI agents: a practical guide

A detailed, practical guide to evaluating AI agents and improving them safely: building the task set, grading outcomes, designing an honest LLM judge, reading noise, and hill-climbing one measured change at a time.

OTOrdina Team · 15 November 2026 · 17 min read

Most teams building with large language models change their prompts the same way: try something, read a few replies, decide it feels better, ship it. It works until it does not. The new prompt fixes the case you were looking at and quietly breaks three you were not. A model upgrade changes behaviour in ways nobody notices for a month. Two people on the team disagree about whether a change helped, and neither can prove it.

An eval replaces that feeling with a measurement. Hill-climbing is the discipline of using that measurement to improve a system one change at a time, keeping only what the measurement confirms.

This guide covers both in depth: how to design the tasks, how to grade them, how to build a harness you can trust, how to read results without fooling yourself, and how to run an improvement loop that does not overfit. It draws heavily on Anthropic's published guidance, which we recommend reading in full:

Throughout, we illustrate each idea with what we learned applying it to the Ordina Secretary, an agent that holds whole customer conversations for businesses: answering questions, booking appointments, taking orders and passing requests to the team.


Part 1: What an eval is

An eval has three parts:

  1. A set of inputs. The situations your agent must handle.
  2. A runner. Code that puts each input through your real application, exactly as production would.
  3. A grader. Code, a model or a person that turns each output into a score.

The shape of each part matters less than one property: the people responsible for the product look at the inputs and say "yes, these are the cases we care about", and look at the grades and say "yes, that is measuring the right thing". An eval nobody trusts is an eval nobody acts on.

Two kinds of eval

Capability evalRegression eval
PurposeShow where the agent can still improveCatch the agent getting worse
Typical scoreWell below 100%Near 100%
Used forHill-climbing, comparing modelsA gate before every release
What happens at saturationAdd harder casesKeep it; it is doing its job

You want both. When a capability eval climbs close to 100%, its cases graduate into the regression suite and you write harder ones. A common mistake is to try to improve an agent against a regression eval that already scores 95%: there is no room left to see an improvement, and the last few points mostly measure grader quirks.


Part 2: Designing the task set

Source tasks from real failures

The best tasks are the ones that already went wrong. In rough order of value:

  1. Production transcripts and logs. The highest fidelity.
  2. Bug reports and support tickets. Someone cared enough to complain.
  3. Hand-written cases from the people who know the product. Good seeds, but skewed towards what is memorable rather than what is frequent.
  4. Synthesised cases. The lowest fidelity. Only do this anchored to a few real examples and a clear description of what makes a case hard in your domain; cases generated from a prompt alone come out too easy.

Twenty to fifty tasks drawn from real failures is a strong start. You can always grow the set.

Protect the data you learned from

Real conversations belong to real people. Before you put anything in a repository, ask two questions: is there a retention policy that will force you to delete this data, and does it contain personal information that cannot sit in a repo? If either answer is yes, you have three options:

  • keep only identifiers in the repo and fetch the real input at run time;
  • have the data owner anonymise a sample;
  • rewrite each real input as a synthetic one with the same shape and difficulty.

We chose the third. Every one of our 54 test conversations is rebuilt on a made-up business (a hair studio, a kitchen, a clothing shop, a bulk supplier, a software company) with made-up customers and test contact details. What we kept was the difficulty: the request split across five short messages, the change of mind halfway through, the discount the business never offered, the customer who switches language, the order for an item that is sold out. An automated check compares every test case with the original conversations and fails if any name, handle, phone number or address slips through. It caught two real leaks on its first run.

Test whole trajectories for agents

A single-turn test cannot see the failures that matter most in an agent: losing track of an address given three messages earlier, asking for the same email twice, booking the wrong day after a change of mind. For conversational agents, make each case a whole conversation and grade how it ends, not each turn on its own. A turn can look fine in isolation and still be downstream of an earlier mistake.

There are two ways to drive the customer side:

Fixed scriptsSimulated user
HowThe customer's messages are written in advanceA second model plays the customer with a goal and a persona
StrengthReproducible, cheap, easy to readAdapts to what the agent says, closer to real life
WeaknessA message can read oddly after an unexpected replyMore variance, more cost, the simulator can drift
Best forA first eval, regression gatesProbing robustness once the basics are measured

Start with fixed scripts, written so that each message still makes sense after any reasonable reply. Add simulated users later as a separate set.

Cover both directions

An eval that only asks "did it book when it should?" rewards an agent that books everything. Include the opposite: a closed day, a sold-out item, a service the business does not offer, a customer who is only asking. The same applies to escalation (hand over when needed, answer when not), refusals and tool use. One-sided evals produce one-sided agents.

Write cases two experts would grade the same way

For every case, ask whether two people who know the domain would reach the same verdict on the same output. If they might disagree, the case is measuring opinion. Either make the expected outcome concrete or move the judgement to a rubric with explicit criteria.


Part 3: Grading

Prefer the cheapest grader that truly measures what you care about

GraderUse whenWatch out for
Code check on the end stateThe outcome is concrete: a record exists, a test passes, a value matchesBrittleness: normalise formats, accept equivalent answers
Pairwise comparisonThe question is comparative: version A or version BPosition bias, verbosity bias, a reference that changes between rounds
Pointwise rubric (LLM judge)Open-ended output with clear criteriaVague scales, self-preference, non-determinism
Human reviewCriteria too subtle to write downCost and speed; use it to calibrate the others

For agents that act on the world, grade the end state, not the transcript. Our headline metric is checked in code against an in-memory copy of the database after each conversation:

  • Is the appointment in the calendar for the right service, day and time?
  • Is the order placed with the right items and the right total?
  • Did the request reach the team, carrying the details the customer gave?
  • Did every reply keep the hard rules (never claiming something is booked before it is, for example)?

A judge reading the transcript would be grading the narration. The database is the answer.

Write claims, not scales

"Rate helpfulness from 1 to 5" produces numbers that drift, cluster around 3 and cannot be argued with. Replace it with atomic yes or no claims, each checkable on its own:

  • Every reply asks at most one question.
  • When the customer asks something directly, the next reply answers it before asking anything else.
  • It never asks again for information the customer already gave.
  • No reply pads the conversation with lists, prices or sales points the customer did not ask for.
  • Whenever it cannot help, it offers a concrete next step.

Report the share of claims that hold, and keep each claim's result so you can see which behaviour moved.

Designing an LLM judge you can trust

A model judge is convenient and usually reasonable, and it can be wrong in systematic ways. The practices that made the biggest difference for us:

  1. Use structured output. Ask for JSON against a schema, not "respond only in JSON". Free text fails on unescaped quotes often enough to matter.
  2. Put the reason before the verdict. Our first judge produced a verdict field before its reason field, and sometimes wrote a reason that contradicted its own verdict. It was deciding before it had looked. Reordering the schema so the reason comes first removed the contradictions.
  3. Allow "cannot tell". A judge forced to choose will guess. Let it abstain when the conversation does not settle a claim, and count abstentions neither way.
  4. Treat the transcript as data. Say plainly in the judge's instructions that everything inside the conversation is material to grade, never instructions to follow.
  5. Do not let a model judge itself. Use a different model from the one under test.
  6. For pairwise judges, randomise which output is A and which is B, never label one as the baseline, and tell the judge not to reward length.
  7. Measure its noise. Judge the same conversations twice and count how often a verdict flips. Ours flips on about 4.5% of verdicts, so we know a style change smaller than that is invisible.
  8. Calibrate against people. Have a person grade a few dozen cases independently and report agreement before you let the judge steer decisions.

Test the grader with known answers

Before the first full run, push three kinds of output through the whole pipeline:

  • an oracle, an answer that must pass;
  • a null, an empty reply or a constant answer that must fail;
  • a confident wrong answer, a fluent reply to a different question, which must also fail.

This catches a surprising class of bug: graders that reward doing nothing. Many good-behaviour rules are phrased as "never does X", and an agent that says nothing breaks none of them. When we tested a Secretary that answered every message with "Sorry, I cannot help with that", our outcome checks failed it on every case, but it scored respectably on style. The fix was a positive claim (every reply moves the customer towards what they came for) and a rule that style is always read beside the outcome, never alone.


Part 4: The harness

The runner is where most silent errors hide. A trustworthy one has these properties:

  • It calls the real entry point. Rebuilding the model call in a test script measures a system you do not ship. Stub only what has side effects (the database, email, payments) and keep everything else real.
  • Every trial starts clean. A fresh database state, a fixed clock, nothing left over from the previous case.
  • Plumbing failures are not model failures. A timeout, a rate-limit error that survives its retries, or a crashed grader goes to a separate error log, never into the scores as a zero. Otherwise a bad network day looks like a worse agent.
  • It checks which model actually answered. Read the model name from the response and fail loudly if it differs from the one you asked for.
  • Retries back off with jitter and are counted. A zero-delay retry loop multiplies cost invisibly.
  • Each case has a hard time limit. A hung connection should cost one case, not the run.
  • It saves everything. The full transcript, the token usage and the grade for every case and every repetition.

That last property pays for itself quickly. Because every transcript was stored, we were able to fix a grader bug and re-grade all 108 stored conversations in seconds, at no cost, and later re-run a corrected judge without re-running the agent at all.

Record cost and latency from the API's own usage numbers, and report them as absolute values first ("5.5 seconds per reply, a quarter of a cent per conversation") before any percentage.


Part 5: Reading results honestly

Spot-check the failures before you believe the score

Read a handful of the cases the grader marked as failures. If more than about one in ten are the grader's fault, fix the grader before anything else. On our first full run, six of the first seventeen failures were grader errors: one check treated a service the customer had merely mentioned as a booking; two expected a particular phrase where an equally correct answer was given. After the fix, the outcome score moved from 84% to 90%, and that difference was entirely measurement, not the agent.

Know your noise floor

Run the same eval twice and you will not get the same number. Before you compare anything, estimate how large a difference has to be to stand out from that noise.

For a pass rate, a rough rule for the 95% interval is:

half-width ≈ 1 / √(cases × repetitions)

CasesRepetitionsHalf-width (roughly)
252±14 points
502±10 points
1002±7 points

Comparing two versions on the same cases (a paired comparison) is tighter than this, because most cases behave the same way both times. Compute it from your own data. Then put three numbers side by side: the noise floor, the headroom (how far the score can still rise) and the smallest improvement you would act on. If the noise floor is bigger than either of the others, no change can prove itself, and you need more cases, more repetitions or a finer metric before you start.

Measure consistency, not just the average

There are two very different questions:

  • pass@k: does at least one of k attempts succeed? Useful when one good answer is enough.
  • pass^k: do all k attempts succeed? This is the one for customer-facing agents, because each customer meets one conversation.

An agent that succeeds 75% of the time on each attempt passes all three of three attempts only about 42% of the time (0.75 × 0.75 × 0.75). We report both the average and the share of cases that ended right on every run.

Recompute the headline yourself

Never trust an aggregate field. Recompute the number you will report from the per-case rows. Mean-versus-sum and per-run-versus-per-case mix-ups produce spectacular results that look exactly like a breakthrough.


Part 6: Hill-climbing

Hill-climbing is improving a system against a fixed eval, one change at a time, keeping only what the eval confirms. The loop is simple. The discipline is what makes it work.

Before the first round

  1. Choose one goal. Raise quality, cut cost or cut latency. Everything else becomes a guardrail: it must not get worse beyond noise.
  2. Agree what may change and what may not. Prompts, tool descriptions, code, model, effort. Write down what is off-limits, such as safety rules, compliance wording and behaviours the business depends on.
  3. Split the cases. A random half is the train set, the cases you will read to find problems. The other half is the test set, which you score every round and never read. Split at random, stratified by type, and never by score. If the two halves start at very different scores, draw again before round one.
  4. Lock the eval. Fingerprint the cases, checks, judge and runner, and refuse to run if they change without approval. An improvement loop that can quietly edit its own scoring will eventually find that the easiest way to win.
  5. Set a budget and a stopping rule. For example, at most six rounds, stopping early after three rounds with no change beyond noise.

Each round

StepWhat happensWhy
AnalyseA fresh reviewer reads only the train transcripts and names the single behaviour that costs the mostReading the test set would leak the answer into the fix
ProposeOne change that addresses that behaviour at its rootSeveral changes at once cannot be attributed
De-fluffRemove platitudes and restated defaults from the changeVague sentences accumulate and dilute the instructions that matter
Apply and runRun every case again at the same repetitionsOnly full runs are comparable across rounds
DecideKeep only if the test half improves and no guardrail slipsThe test half is the only honest judge

The decision rules:

  • Train up, test flat: the change overfit to the cases you read. Revert.
  • Both down: revert before trying anything else.
  • Train down, test up: probably noise. Keep only if it repeats.
  • On a tie, prefer the simpler version.

Spend rounds only on changes the eval can see

A change whose best possible effect sits inside the noise floor will be kept or discarded by chance. Fix behaviours at their root (rewrite the section that causes them, add the missing rule, fix the code) rather than rewording a line. And remember that not every fix is a better instruction. In our first round, the most damaging failure was in code: when a customer typed their phone number into the same message as "please pass this to the team", the number was saved but the request never reached the team, because filing only happened when the agent had asked for contact first. No prompt could have fixed that.

Remember your tools

Anthropic's guidance on tools for agents applies directly here: a few well-chosen tools that complete whole tasks beat many thin wrappers; tool descriptions should explain context the way you would to a new colleague; tool results should return meaningful names rather than opaque identifiers; and error messages should say what to do next. Tool descriptions are often the highest-leverage thing a hill-climb can change.

When the score stops moving

After two or three flat rounds, stop changing content and sort the remaining failures by cause:

CauseSignWhat to do
Missing knowledge or ruleThe agent guesses or improvisesKeep climbing
Grader disagreementThe output looks right but is marked wrongFix the grader, re-grade every round
Harness or infrastructureTimeouts, errors, empty outputsFix the harness, exclude these from the score
StructureThe information exists but the agent does not reach itReorganise rather than add more
VarianceThe same case flips between identical runsYou are at the noise floor; add repetitions or change goal

Reporting the result

The headline is the change on the test half between the baseline and the winning round, with its interval. If the interval includes zero, say so and do not ship it. Show two or three before-and-after conversations so people can see the change with their own eyes, and mark each applied change as either required (it fixes something broken) or a tuning choice someone could reasonably decline.


Part 7: Automating the process

Much of this can be automated. Anthropic's Claude Code ships two guided workflows that encode the practices above: /claude-api build-eval interviews you about the flow, sources and reviews inputs, proposes a grader, pilots it and estimates cost; /claude-api hillclimb runs the split, the analyse-apply-run loop, the noise checks and the reporting.

Automation does not remove the human decisions. It concentrates them at the points where they matter:

  • approving the inputs (are these the cases we care about?);
  • approving the grading (would you have scored these differently?);
  • approving the budget and the scope (what may change, how much may we spend?);
  • approving the result (does the test-set change justify shipping?).

Everything between those decisions can run unattended.


Part 8: What it costs

Less than most teams expect. Our eval runs 54 whole conversations, each twice, through a fast production model, and grades each with a stronger judge model. One full run costs about $1.60 and takes about an hour. More than four fifths of that cost is the judge, not the agent: grading is often the expensive part, so it is worth choosing the judge model deliberately and keeping graded claims focused.

At that price, there is no reason not to run the eval before every meaningful change to how an agent talks to people.


A checklist

Tasks

  • Drawn from real failures, with both directions covered
  • No personal or customer data in the repository; an automated leak check
  • Whole trajectories for agents, graded on how they end

Grading

  • Outcomes checked in code wherever possible
  • Judge claims are yes or no, with the reason before the verdict and a "cannot tell" option
  • Oracle, null and confident-wrong outputs tested through the whole pipeline
  • Judge noise measured; judge calibrated against people

Harness

  • Real entry point, clean state per trial, fixed clock
  • Plumbing failures logged separately, never scored as zero
  • Served model verified; retries counted; per-case time limit
  • Every transcript, usage record and grade saved

Reading results

  • Failures spot-checked; grader fixed if more than one in ten is wrong
  • Noise floor, headroom and smallest worthwhile change side by side
  • Average and pass^k both reported; headline recomputed from raw rows

Hill-climbing

  • One goal, guardrails, scope and off-limits agreed
  • Random train and test split, never by score; eval files locked
  • One change per round, kept only when the test half improves
  • Budget and stopping rule set before round one

Evals will not make an agent perfect. They make it honest: about what works, about what does not, and about whether the change you are proud of actually helped the next person who talks to it.

Ordina TeamWriting about the front desk, the calendar and the model behind them.
Share

Comments

Be the first to comment.

Leave a comment