Lessons from building eval systems for agentic products, and the blueprint I would reach for today
There is no shortage of advice about evals right now. The big labs and the eval tooling companies have published playbooks, and much of that material is genuinely useful. Having built and run eval systems for agentic products, what I want to add is the stuff between the lines: the engineering culture around evals, not just the purely technical parts. This post is my attempt to write it down.
I have had the opportunity to build evals for three very different kinds of products. Conversations and action planning on a robot that could autonomously drive around the house and identify people, where the code is split between cloud and device. Code generation in a multi-agent system, where a requirements gathering agent, an architecture agent, backend and frontend agents, an integration agent, and a release agent all work together. And a two-way marketplace where clients hire freelancers to do work, and where we run long-running agents like a recruiter. Each has its unique challenges, of course, but what follows are the common things I have seen again and again.
When I compress everything I have learned, I keep hitting on four things: eval culture, developer experience, the flywheel, and versioning.
1. Eval Is a Team Sport
The most important thing I have learned about evals is not technical. It is about putting a culture in place where the team, both tech and non-tech, is focused on eval-based engineering. People are actively thinking about how to make evals easier to do. They are actively thinking about how to understand, read, and update evals, and how to update the golden datasets behind them, as a normal part of shipping.
You can tell when this culture is missing. Evals are owned by one engineer. Everyone else waits for a report that comes once a release, or once a month. The non-technical people on the team never look at the results because there is no way for them to. Quality gets discussed in terms of gut feel and anecdotes, and disagreements about whether the product got better or worse go unresolved because nobody can point at a number.
When the culture is working, it looks very different. Evals are woven into the product development lifecycle, not bolted on at the end. The PM defines what good looks like and writes the rubric criteria in plain language. Engineers build the pipeline. Subject matter experts label the golden data and review flagged production traces. Evals run continuously in CI/CD: a PR that changes a prompt runs the full suite and posts the results right on the PR as deltas against a baseline, a config change gets deterministic checks only, and a docs change skips evals entirely. And at launch review, the PM signs off on the eval results, not the code. When a launch review includes a sentence like "we pass 96 percent on factual accuracy, above our 95 percent threshold," you know the culture has taken hold.
The clearest sign is a PM reading through the failed cases and saying something like: cases two and three are real problems, case one is an acceptable edge case, and case four is a dataset issue, not a code issue. None of that requires an engineering background. It requires readable results and ownership of what good looks like.
One actionable practice here, which I believe in and also do: have PMs and other non-engineers write eval cases and submit them as PRs. An eval case is a small structured file, far easier to review than code, and the people closest to the customer are exactly the ones who should be encoding what good looks like.
2. The Developer Experience Needs to Be the Best
If you make evals hard to do, where people have to check out another package, or be on some kind of private VPN, or set up another machine, that is going to discourage people from running evals. If they are not getting tokens to run evals, or there is no visibility into what the eval results are, same outcome. All of these things need to be really top-notch. The developer experience needs to be the best: a one-liner to run an eval.
Results should land where people already are: a comment on the PR, a dashboard anyone on the team can open, a link posted where the team talks. Not a JSON file on someone's machine, not a notebook only one person knows how to run. And if a non-technical person cannot trigger an eval or read a result on their own, then the culture section above stays theory.
The token part is worth saying out loud: a fifty-case dataset judged on four criteria costs well under a dollar with a small judge model. Cost does become real at scale, once you are sampling production traffic for a few hundred thousand customers, but early on the scarcity is not money. It is friction and process. If someone has to file a ticket or ask permission to run an eval, they will run it less, and the regressions that would have been caught show up in production instead.
The structural way to get there is to build evaluation into the platform itself. If onboarding an agent onto your runtime automatically gives it eval infrastructure, so that the team only has to supply a dataset of inputs and expected outputs, you have removed the single biggest excuse for shipping without evals. The platform captures traces, replays inputs, runs judges, and reports results. The team brings the dataset.
Part of the developer experience is also a robust human-in-the-loop pipeline. Do you actually have subject matter experts who can review what the judge is saying, conclusively and predictably? If flagged traces sit in a queue nobody staffs, or the people reviewing them cannot say with authority whether the judge got it right, the loop is broken no matter how good the tooling is. Line up the experts, give them a simple review surface, and make their labels flow back into the datasets and the judge calibration.
And the same bar applies to the flywheel below: flagging a trace, promoting it into a dataset, or recording a judge disagreement should each be a one-liner too. If any of those takes more than a moment, the flywheel stops turning.
3. The Flywheel
Here is the honest part: it is very difficult to deterministically calibrate judges or to have perfect golden datasets. Accepting that is what led me to the most useful thing I have set up, which is a flywheel that continuously feeds reality back into the eval system. It has two halves.
The first half is a qualitative flywheel. Run very regular play sessions where you are releasing features as soon as possible, internally or with an external cohort, so people can give feedback. There should not be a lot of process around it: staging environments should automatically go to both technical and non-technical stakeholders who can test the product and react to it. This is how you understand how your agents are behaving when they are out in the world, and the feedback is real, actionable feedback in a way that dashboards are not. There is also a trick that makes play sessions do double duty: collect the conversation traces they generate and run your judges over them afterward. The humans give you qualitative reactions in the moment, the judges give you scored threads you can compare across sessions, and the low-scoring threads become candidates for your golden datasets.
The second half is quantitative: in addition to offline evals, set up online evals where you are running evals on sampled production traffic and traces. One framing that keeps the two modes straight: offline evals are reference-based, comparing against expected outputs, while online evals are necessarily reference-free, asking questions like "is this response helpful, grounded, and safe" without a golden answer to compare to. This is where the flywheel really turns, because every outcome feeds something:
- False positives, where the judge is telling us this should fail and we think it should not fail, become a reason for us to calibrate our judge.
- True positives, where the judge says it fails and we also think it fails, go back and become a use case for the golden dataset.
- True failures the evals were not able to catch, where it is a human who looks at the trace and realizes this is something we really should have caught, again become a reason to calibrate our judges and expand our datasets.
Two practical notes on making this half work. First, the low-scoring traces need somewhere to go: a review queue, on a regular cadence, feeding the same human-in-the-loop pipeline from the developer experience section. Second, production gives you quality signals for free if you collect them: thumbs up and down, and the user's next message. A user who immediately re-asks the same question or says "that's wrong" just labeled that trace for you, and those labels are exactly what you need to calibrate judges against.
4. Version the Moving Parts, See the Decision Graph
As you get into complex agentic systems, execution starts to look like a decision graph: every step branches on what came before, and you need very good visibility in terms of what led to what. When an eval fails on a multi-step trace, the question is rarely whether it failed but where along the path it went wrong. Good trace visibility is part of the eval system, not a separate observability project.
This is why I have come to think of evaluation at three altitudes. End-to-end: did the system produce a good final response? Trajectory: did the agent take a reasonable path, calling the right tools, in a sensible order, with the right arguments? And per-step: did each individual node do its job given its input? End-to-end tells you that something broke. Trajectory and per-step evals tell you what broke.
Versioning is what makes that visibility trustworthy. Make sure the different parts of your system are versioned: not just your agents, but the tools they use and the APIs underneath them. When you can view the decision graph and break this non-deterministic, awesome vehicle down into versioned, deterministic parts, you can actually understand what is happening behind the scenes. It puts structure on top of unstructured execution, and when an eval regresses, "what changed" has an honest answer.
Privacy cuts across all of this. A lot of production data is redacted or needs to stay private, and that constrains what you can sample for online evals and what you can promote into golden datasets. Thinking through these things early really helps.
The Blueprint
With culture, developer experience, the flywheel, and versioning in place, here is the technical shape I keep coming back to. Under each piece, the tactical details are the part that tends to get missed.
Product evals, not model evals. You are not benchmarking base models. You are evaluating the full system: prompt, retrieval, tools, guardrails, and everything in between, running through the same components that serve production. This also means evals are how you settle architecture debates. Should we fine-tune or prompt? Is this extra pipeline stage worth its latency? Without baseline comparisons, every one of those discussions stays an exchange of opinions. With them, complexity has to earn its place with data.
Both deterministic and non-deterministic judges. Deterministic checks for the things you can assert exactly, things like the right tool being called, valid output format, required keywords. LLM-as-judge for the things you cannot, like factual accuracy, grounding, and helpfulness. The tactical details for judges: use a judge from a different model family than your production model, because models favor outputs that look like their own. Prefer pass/fail criteria over 1-to-5 scales for most dimensions, since point scales fluctuate between runs. Have the judge reason step by step before giving its verdict, and return structured output you can parse. Know the biases: judges favor longer responses, and in pairwise comparisons verdicts can flip based purely on which answer was shown first, so swap the order and only accept consistent verdicts. For your most critical evals, use a small jury of diverse judges rather than one. And consider letting errors subtract value rather than just failing to add it: a response that is mostly right but hallucinates a source can be worse than no response at all, because now a human has to find and fix the error.
Golden datasets with both single-turn and multi-turn use cases. Agentic products fail in multi-turn conversations in ways single-turn evals will never surface: dropped context, forgotten filters, repeated information. The tactical details: start narrow, 25 to 100 cases per use case, covering the happy paths and edge cases; the adversarial and regression datasets below round out the coverage. Bootstrap with synthetic generation, but treat that output as silver, not gold; it only becomes golden after a subject matter expert has reviewed each case.
The ability to queue and prioritize requests. Eval runs compete for capacity with everything else, so you need to be able to queue eval requests and prioritize them when needed.
Sanity datasets, regression datasets, segmented datasets. Sanity datasets run fast and catch obvious breakage. Regression datasets grow every time something slips through: whenever something breaks in production, especially after a policy or content change your system should have known about, turn that incident into permanent regression cases. And segmented datasets, which I think are underrated: the ability to run evals for a specific segment, which might get ignored, which might be a minority segment, but could be dropping performance while we are only looking at the overall picture. Identifying your segments is part of the eval work, not an afterthought.
Adversarial and balanced datasets. Adversarial evals are very important: prompt injection attempts, requests the agent must refuse, questions that look answerable but deserve an honest "I don't know." And balance the dataset with negatives, not just positives. If you only test whether the agent searches when it should, you might end up with an agent that searches for almost everything. Test that it does not act when it should not, just as much as that it acts when it should.
Run multiple passes, and deep-dive your metrics again and again. The same model, the same agent, the same prompt can behave differently across different passes, and that gets missed when every eval is a single run. A single-run pass rate tells you capability; running the same cases several times tells you consistency, and consistency is what production actually demands. Keep a baseline from the last known-good run and report every result as a delta against it, because "88 percent" means nothing until you know yesterday was 92. And when a number moves, do not stop at the aggregate: dig into which cases, which segments, and which criteria moved it.
A good process for calibrating your judges, and a good process for calibrating your golden datasets. Before trusting a judge, run it against a calibration set of human-labeled cases and measure agreement; a judge is only calibrated when it agrees with humans well over 80 percent of the time, on both what should pass and what should fail. Then keep measuring on a schedule, because judges drift as your product's outputs change shape. The same goes for golden datasets: recurringly check that they are actually being reflective of the actual broad scenarios your product sees. A judge that was well calibrated three months ago is a rumor, not a measurement.
Thresholds with teeth, in tiers. Not every criterion deserves the same enforcement. Safety failures should block a launch at any rate above zero. Factual accuracy and hallucination should block below a high bar. Softer dimensions like tone and helpfulness can warn without blocking, and things like response length and cost per run should be tracked without gating at all. One rule I hold firmly: when an eval fails, iterate on the feature, not on the thresholds.
Predictions
Eval systems will get agentic. Based on the use case, evals will call tools to verify outcomes, and they will decide how many tokens they want to spend on evaluating the correctness of the other agent. Think of how someone really good at code review works: based on how important or critical the change is, they change how much effort they put into it. A one-line docs fix gets a glance, a payment-path change gets a deep read. Eval systems will get smarter the same way, spending their budget where the risk is.
Canonical simulated worlds. We will build canonical models that represent our product, with mock customers, mock artifacts, and mock workflows, and let our agents run in that simulated world. The big win is long-horizon metrics, which are very difficult to track when you are releasing features fast. If you release an agent on a robot that proactively engages the customer, does that change overall engagement and trust over months? Those things are hard to measure in the short term, and you have to measure them in the long term. If a simulation can capture the different vectors of how your product behaves with customers, along with seasonalities, that is going to be super cool. I think we need it.
Agents will be held accountable. This shows up most in enterprise and internal workflows. Say an agent writing a sales report makes an error. How do we hold it accountable? Right now the answer is: the eval fails, a human reads it, and that is it. I think there is room for a subsystem of the eval system that does a post-mortem on what the evals caught, when it is possible and sensible. What went wrong, which step, what should change. And accountability can have consequences, the way it does for people: an agent that erred gets more audit on its work, maybe suspended access to certain systems, and if it continues to fail, it gets downgraded or downleveled to less critical work.
FAQ
How do I put together my golden dataset? This is the question I get most, so here it is in one place. Work with product to define the common behaviors. Talk to your customers. Talk to your customer representatives, whether they are sales engineers, marketing people, sales support, or call center, these kinds of people. They hear the real questions and the real failures every day. And look into your production logs. Between those sources you will have more candidate cases than you can label, and the silver-to-gold review process above takes it from there.
Where do I start if I have nothing today? Do not start by picking a framework. Start by reading your traces: pull up twenty or thirty real conversations and note what good and bad look like. Pick one use case, write 20 to 30 cases from what you saw, add one deterministic check and one judge criterion, and wire it into CI so it runs on every prompt change. That is a working eval system. Everything else in this post, the datasets, the calibration, the flywheel, grows from there.
How do I keep testing the newer models that keep releasing? Treat the model as a moving part of your system, the same way the versioning section treats everything else. Set up a test where you can swap that one moving part while everything else stays the same: your prompts, your context, your tools. Run your golden datasets against it and see what moves. During that test you might realize you have to tweak a few other things, so tweak them and run it again. All of this becomes easy if your code makes the agent's loop or graph explicit, so that you can swap the model and play with it without touching much code. If trying a new model means a week of rewiring, the problem is not the model.
How do I evaluate my fine-tuned models? My first question back is always: what led you to fine-tune? Understanding that is crucial before you design any tests. Fine-tuning is great, it gives you control, privacy, and sometimes more deterministic results, but it also bounds your ability to play with the model, add more context, and update it for a dynamically moving product or business. The good news is that if you fine-tuned for a specific purpose, you probably already have very specific use cases, and those are exactly your eval cases. One smell to watch for: if you fine-tuned a model and you also have to supply a lot of context over RAG, you might be sitting on an anti-pattern. Fine-tune for behavior, let retrieval carry the knowledge, because knowledge baked into weights goes stale.
The Summary
If I had to compress all of this: the flywheel is what ties everything together. The judges, the datasets, the thresholds are all crucial, and the flywheel is what makes each of them better over time. Offline evals give you a floor, play sessions give you qualitative truth, online evals give you quantitative truth, and every disagreement between them makes the whole system better. Your job is to make that loop so easy to participate in that the entire team, not just the eval owners, keeps it spinning.