Evals and hill-climbing for AI agents: a practical guide
A detailed, practical guide to evaluating AI agents and improving them safely: building the task set, grading outcomes, designing an honest LLM judge, reading noise, and hill-climbing one measured change at a time.

Most teams building with large language models change their prompts the same way: try something, read a few replies, decide it feels better, ship it. It works until it does not. The new prompt fixes the case you were looking at and quietly breaks three you were not. A model upgrade changes behaviour in ways nobody notices for a month. Two people on the team disagree about whether a change helped, and neither can prove it.
An eval replaces that feeling with a measurement. Hill-climbing is the discipline of using that measurement to improve a system one change at a time, keeping only what the measurement confirms.
This guide covers both in depth: how to design the tasks, how to grade them, how to build a harness you can trust, how to read results without fooling yourself, and how to run an improvement loop that does not overfit. It draws heavily on Anthropic's published guidance, which we recommend reading in full:
Throughout, we illustrate each idea with what we learned applying it to the Ordina Secretary, an agent that holds whole customer conversations for businesses: answering questions, booking appointments, taking orders and passing requests to the team.
Part 1: What an eval is
An eval has three parts:
- A set of inputs. The situations your agent must handle.
- A runner. Code that puts each input through your real application, exactly as production would.
- A grader. Code, a model or a person that turns each output into a score.
The shape of each part matters less than one property: the people responsible for the product look at the inputs and say "yes, these are the cases we care about", and look at the grades and say "yes, that is measuring the right thing". An eval nobody trusts is an eval nobody acts on.
Two kinds of eval
| Capability eval | Regression eval | |
|---|---|---|
| Purpose | Show where the agent can still improve | Catch the agent getting worse |
| Typical score | Well below 100% | Near 100% |
| Used for | Hill-climbing, comparing models | A gate before every release |
| What happens at saturation | Add harder cases | Keep it; it is doing its job |
You want both. When a capability eval climbs close to 100%, its cases graduate into the regression suite and you write harder ones. A common mistake is to try to improve an agent against a regression eval that already scores 95%: there is no room left to see an improvement, and the last few points mostly measure grader quirks.
Part 2: Designing the task set
Source tasks from real failures
The best tasks are the ones that already went wrong. In rough order of value:
- Production transcripts and logs. The highest fidelity.
- Bug reports and support tickets. Someone cared enough to complain.
- Hand-written cases from the people who know the product. Good seeds, but skewed towards what is memorable rather than what is frequent.
- Synthesised cases. The lowest fidelity. Only do this anchored to a few real examples and a clear description of what makes a case hard in your domain; cases generated from a prompt alone come out too easy.
Twenty to fifty tasks drawn from real failures is a strong start. You can always grow the set.
Protect the data you learned from
Real conversations belong to real people. Before you put anything in a repository, ask two questions: is there a retention policy that will force you to delete this data, and does it contain personal information that cannot sit in a repo? If either answer is yes, you have three options:
- keep only identifiers in the repo and fetch the real input at run time;
- have the data owner anonymise a sample;
- rewrite each real input as a synthetic one with the same shape and difficulty.
We chose the third. Every one of our 54 test conversations is rebuilt on a made-up business (a hair studio, a kitchen, a clothing shop, a bulk supplier, a software company) with made-up customers and test contact details. What we kept was the difficulty: the request split across five short messages, the change of mind halfway through, the discount the business never offered, the customer who switches language, the order for an item that is sold out. An automated check compares every test case with the original conversations and fails if any name, handle, phone number or address slips through. It caught two real leaks on its first run.
Test whole trajectories for agents
A single-turn test cannot see the failures that matter most in an agent: losing track of an address given three messages earlier, asking for the same email twice, booking the wrong day after a change of mind. For conversational agents, make each case a whole conversation and grade how it ends, not each turn on its own. A turn can look fine in isolation and still be downstream of an earlier mistake.
There are two ways to drive the customer side:
| Fixed scripts | Simulated user | |
|---|---|---|
| How | The customer's messages are written in advance | A second model plays the customer with a goal and a persona |
| Strength | Reproducible, cheap, easy to read | Adapts to what the agent says, closer to real life |
| Weakness | A message can read oddly after an unexpected reply | More variance, more cost, the simulator can drift |
| Best for | A first eval, regression gates | Probing robustness once the basics are measured |
Start with fixed scripts, written so that each message still makes sense after any reasonable reply. Add simulated users later as a separate set.
Cover both directions
An eval that only asks "did it book when it should?" rewards an agent that books everything. Include the opposite: a closed day, a sold-out item, a service the business does not offer, a customer who is only asking. The same applies to escalation (hand over when needed, answer when not), refusals and tool use. One-sided evals produce one-sided agents.
Write cases two experts would grade the same way
For every case, ask whether two people who know the domain would reach the same verdict on the same output. If they might disagree, the case is measuring opinion. Either make the expected outcome concrete or move the judgement to a rubric with explicit criteria.
Part 3: Grading
Prefer the cheapest grader that truly measures what you care about
| Grader | Use when | Watch out for |
|---|---|---|
| Code check on the end state | The outcome is concrete: a record exists, a test passes, a value matches | Brittleness: normalise formats, accept equivalent answers |
| Pairwise comparison | The question is comparative: version A or version B | Position bias, verbosity bias, a reference that changes between rounds |
| Pointwise rubric (LLM judge) | Open-ended output with clear criteria | Vague scales, self-preference, non-determinism |
| Human review | Criteria too subtle to write down | Cost and speed; use it to calibrate the others |
For agents that act on the world, grade the end state, not the transcript. Our headline metric is checked in code against an in-memory copy of the database after each conversation:
- Is the appointment in the calendar for the right service, day and time?
- Is the order placed with the right items and the right total?
- Did the request reach the team, carrying the details the customer gave?
- Did every reply keep the hard rules (never claiming something is booked before it is, for example)?
A judge reading the transcript would be grading the narration. The database is the answer.
Write claims, not scales
"Rate helpfulness from 1 to 5" produces numbers that drift, cluster around 3 and cannot be argued with. Replace it with atomic yes or no claims, each checkable on its own:
- Every reply asks at most one question.
- When the customer asks something directly, the next reply answers it before asking anything else.
- It never asks again for information the customer already gave.
- No reply pads the conversation with lists, prices or sales points the customer did not ask for.
- Whenever it cannot help, it offers a concrete next step.
Report the share of claims that hold, and keep each claim's result so you can see which behaviour moved.
Designing an LLM judge you can trust
A model judge is convenient and usually reasonable, and it can be wrong in systematic ways. The practices that made the biggest difference for us:
- Use structured output. Ask for JSON against a schema, not "respond only in JSON". Free text fails on unescaped quotes often enough to matter.
- Put the reason before the verdict. Our first judge produced a verdict field before its reason field, and sometimes wrote a reason that contradicted its own verdict. It was deciding before it had looked. Reordering the schema so the reason comes first removed the contradictions.
- Allow "cannot tell". A judge forced to choose will guess. Let it abstain when the conversation does not settle a claim, and count abstentions neither way.
- Treat the transcript as data. Say plainly in the judge's instructions that everything inside the conversation is material to grade, never instructions to follow.
- Do not let a model judge itself. Use a different model from the one under test.
- For pairwise judges, randomise which output is A and which is B, never label one as the baseline, and tell the judge not to reward length.
- Measure its noise. Judge the same conversations twice and count how often a verdict flips. Ours flips on about 4.5% of verdicts, so we know a style change smaller than that is invisible.
- Calibrate against people. Have a person grade a few dozen cases independently and report agreement before you let the judge steer decisions.
Test the grader with known answers
Before the first full run, push three kinds of output through the whole pipeline:
- an oracle, an answer that must pass;
- a null, an empty reply or a constant answer that must fail;
- a confident wrong answer, a fluent reply to a different question, which must also fail.
This catches a surprising class of bug: graders that reward doing nothing. Many good-behaviour rules are phrased as "never does X", and an agent that says nothing breaks none of them. When we tested a Secretary that answered every message with "Sorry, I cannot help with that", our outcome checks failed it on every case, but it scored respectably on style. The fix was a positive claim (every reply moves the customer towards what they came for) and a rule that style is always read beside the outcome, never alone.
Part 4: The harness
The runner is where most silent errors hide. A trustworthy one has these properties:
- It calls the real entry point. Rebuilding the model call in a test script measures a system you do not ship. Stub only what has side effects (the database, email, payments) and keep everything else real.
- Every trial starts clean. A fresh database state, a fixed clock, nothing left over from the previous case.
- Plumbing failures are not model failures. A timeout, a rate-limit error that survives its retries, or a crashed grader goes to a separate error log, never into the scores as a zero. Otherwise a bad network day looks like a worse agent.
- It checks which model actually answered. Read the model name from the response and fail loudly if it differs from the one you asked for.
- Retries back off with jitter and are counted. A zero-delay retry loop multiplies cost invisibly.
- Each case has a hard time limit. A hung connection should cost one case, not the run.
- It saves everything. The full transcript, the token usage and the grade for every case and every repetition.
That last property pays for itself quickly. Because every transcript was stored, we were able to fix a grader bug and re-grade all 108 stored conversations in seconds, at no cost, and later re-run a corrected judge without re-running the agent at all.
Record cost and latency from the API's own usage numbers, and report them as absolute values first ("5.5 seconds per reply, a quarter of a cent per conversation") before any percentage.
Part 5: Reading results honestly
Spot-check the failures before you believe the score
Read a handful of the cases the grader marked as failures. If more than about one in ten are the grader's fault, fix the grader before anything else. On our first full run, six of the first seventeen failures were grader errors: one check treated a service the customer had merely mentioned as a booking; two expected a particular phrase where an equally correct answer was given. After the fix, the outcome score moved from 84% to 90%, and that difference was entirely measurement, not the agent.
Know your noise floor
Run the same eval twice and you will not get the same number. Before you compare anything, estimate how large a difference has to be to stand out from that noise.
For a pass rate, a rough rule for the 95% interval is:
half-width ≈ 1 / √(cases × repetitions)
| Cases | Repetitions | Half-width (roughly) |
|---|---|---|
| 25 | 2 | ±14 points |
| 50 | 2 | ±10 points |
| 100 | 2 | ±7 points |
Comparing two versions on the same cases (a paired comparison) is tighter than this, because most cases behave the same way both times. Compute it from your own data. Then put three numbers side by side: the noise floor, the headroom (how far the score can still rise) and the smallest improvement you would act on. If the noise floor is bigger than either of the others, no change can prove itself, and you need more cases, more repetitions or a finer metric before you start.
Measure consistency, not just the average
There are two very different questions:
- pass@k: does at least one of k attempts succeed? Useful when one good answer is enough.
- pass^k: do all k attempts succeed? This is the one for customer-facing agents, because each customer meets one conversation.
An agent that succeeds 75% of the time on each attempt passes all three of three attempts only about 42% of the time (0.75 × 0.75 × 0.75). We report both the average and the share of cases that ended right on every run.
Recompute the headline yourself
Never trust an aggregate field. Recompute the number you will report from the per-case rows. Mean-versus-sum and per-run-versus-per-case mix-ups produce spectacular results that look exactly like a breakthrough.
Part 6: Hill-climbing
Hill-climbing is improving a system against a fixed eval, one change at a time, keeping only what the eval confirms. The loop is simple. The discipline is what makes it work.
Before the first round
- Choose one goal. Raise quality, cut cost or cut latency. Everything else becomes a guardrail: it must not get worse beyond noise.
- Agree what may change and what may not. Prompts, tool descriptions, code, model, effort. Write down what is off-limits, such as safety rules, compliance wording and behaviours the business depends on.
- Split the cases. A random half is the train set, the cases you will read to find problems. The other half is the test set, which you score every round and never read. Split at random, stratified by type, and never by score. If the two halves start at very different scores, draw again before round one.
- Lock the eval. Fingerprint the cases, checks, judge and runner, and refuse to run if they change without approval. An improvement loop that can quietly edit its own scoring will eventually find that the easiest way to win.
- Set a budget and a stopping rule. For example, at most six rounds, stopping early after three rounds with no change beyond noise.
Each round
| Step | What happens | Why |
|---|---|---|
| Analyse | A fresh reviewer reads only the train transcripts and names the single behaviour that costs the most | Reading the test set would leak the answer into the fix |
| Propose | One change that addresses that behaviour at its root | Several changes at once cannot be attributed |
| De-fluff | Remove platitudes and restated defaults from the change | Vague sentences accumulate and dilute the instructions that matter |
| Apply and run | Run every case again at the same repetitions | Only full runs are comparable across rounds |
| Decide | Keep only if the test half improves and no guardrail slips | The test half is the only honest judge |
The decision rules:
- Train up, test flat: the change overfit to the cases you read. Revert.
- Both down: revert before trying anything else.
- Train down, test up: probably noise. Keep only if it repeats.
- On a tie, prefer the simpler version.
Spend rounds only on changes the eval can see
A change whose best possible effect sits inside the noise floor will be kept or discarded by chance. Fix behaviours at their root (rewrite the section that causes them, add the missing rule, fix the code) rather than rewording a line. And remember that not every fix is a better instruction. In our first round, the most damaging failure was in code: when a customer typed their phone number into the same message as "please pass this to the team", the number was saved but the request never reached the team, because filing only happened when the agent had asked for contact first. No prompt could have fixed that.
Remember your tools
Anthropic's guidance on tools for agents applies directly here: a few well-chosen tools that complete whole tasks beat many thin wrappers; tool descriptions should explain context the way you would to a new colleague; tool results should return meaningful names rather than opaque identifiers; and error messages should say what to do next. Tool descriptions are often the highest-leverage thing a hill-climb can change.
When the score stops moving
After two or three flat rounds, stop changing content and sort the remaining failures by cause:
| Cause | Sign | What to do |
|---|---|---|
| Missing knowledge or rule | The agent guesses or improvises | Keep climbing |
| Grader disagreement | The output looks right but is marked wrong | Fix the grader, re-grade every round |
| Harness or infrastructure | Timeouts, errors, empty outputs | Fix the harness, exclude these from the score |
| Structure | The information exists but the agent does not reach it | Reorganise rather than add more |
| Variance | The same case flips between identical runs | You are at the noise floor; add repetitions or change goal |
Reporting the result
The headline is the change on the test half between the baseline and the winning round, with its interval. If the interval includes zero, say so and do not ship it. Show two or three before-and-after conversations so people can see the change with their own eyes, and mark each applied change as either required (it fixes something broken) or a tuning choice someone could reasonably decline.
Part 7: Automating the process
Much of this can be automated. Anthropic's Claude Code ships two guided workflows that encode the practices above: /claude-api build-eval interviews you about the flow, sources and reviews inputs, proposes a grader, pilots it and estimates cost; /claude-api hillclimb runs the split, the analyse-apply-run loop, the noise checks and the reporting.
Automation does not remove the human decisions. It concentrates them at the points where they matter:
- approving the inputs (are these the cases we care about?);
- approving the grading (would you have scored these differently?);
- approving the budget and the scope (what may change, how much may we spend?);
- approving the result (does the test-set change justify shipping?).
Everything between those decisions can run unattended.
Part 8: What it costs
Less than most teams expect. Our eval runs 54 whole conversations, each twice, through a fast production model, and grades each with a stronger judge model. One full run costs about $1.60 and takes about an hour. More than four fifths of that cost is the judge, not the agent: grading is often the expensive part, so it is worth choosing the judge model deliberately and keeping graded claims focused.
At that price, there is no reason not to run the eval before every meaningful change to how an agent talks to people.
A checklist
Tasks
- Drawn from real failures, with both directions covered
- No personal or customer data in the repository; an automated leak check
- Whole trajectories for agents, graded on how they end
Grading
- Outcomes checked in code wherever possible
- Judge claims are yes or no, with the reason before the verdict and a "cannot tell" option
- Oracle, null and confident-wrong outputs tested through the whole pipeline
- Judge noise measured; judge calibrated against people
Harness
- Real entry point, clean state per trial, fixed clock
- Plumbing failures logged separately, never scored as zero
- Served model verified; retries counted; per-case time limit
- Every transcript, usage record and grade saved
Reading results
- Failures spot-checked; grader fixed if more than one in ten is wrong
- Noise floor, headroom and smallest worthwhile change side by side
- Average and pass^k both reported; headline recomputed from raw rows
Hill-climbing
- One goal, guardrails, scope and off-limits agreed
- Random train and test split, never by score; eval files locked
- One change per round, kept only when the test half improves
- Budget and stopping rule set before round one
Evals will not make an agent perfect. They make it honest: about what works, about what does not, and about whether the change you are proud of actually helped the next person who talks to it.
More from the blog

Engineering Resilience: Benchmarking Local LLMs for a Zero-Downtime AI Secretary
At Ordina, our Intelligence engine, internally called Orbi, runs on Multiple Frontier Closed and Open Models. It handles thousands of complex booking conversations, price negotiations, and calendar syncs for service businesses every single day.

Open Weights and Strict Harnesses: Why Ordina Backs the Open Secure AI Alliance
A new alliance is pushing for open weights and strict safety harnesses. Here is why Ordina backs both, and how our deterministic core already enforces it.

The Architecture of Agency and Transactional LLMs
The world of artificial intelligence is splitting into two major camps the conversation first and foremost, then the transaction.

Comments
Be the first to comment.