Most teams building AI agents know how to get them into production. Far fewer can answer a simple question: is it actually working? At our External Tech Workshop, Adam Horka - AI Engineer at ableneo - shared what happens when you set up a real evaluation framework on a project that had none.
Insights from External Tech Workshop
By Adam Horka, AI Engineer at ableneo · May 2026 · Last updated: June 2026
You built the agent. The demo went well. The client nodded. Now comes the question nobody asks: how do you actually know it works?
AI agent evaluation is the practice of systematically measuring whether an AI agent performs reliably across the full range of real-world inputs — not just the cases you hand-picked during development. It covers everything from choosing the right quality metrics, to building test datasets, to monitoring performance after deployment.
In most organizations rushing AI agents into production, systematic evaluation is missing. Teams have a handful of examples that looked good during development and a successful demo. What they do not have is measurement. At ableneo External Tech Workshop, Adam Horka walked through what it takes to build evaluation into an AI agent project from the ground up, what goes wrong when you skip it, and how to catch quality problems before your users catch them for you.
When Adam joined one client project, the team had already done substantial development work. The agent performed well in testing. The demo ran without issues. The client was satisfied with what they saw.
Adam asked one question: how do you know this works? Nobody could answer it with numbers. There were no systematic metrics, no comprehensive test set, no measurement framework. There were cherry-picked examples that the team felt represented the solution well.
When a proper evaluation framework was set up and run across a full, representative sample of real cases, the results shocked the team. The actual success rate was significantly lower than the testing samples had suggested. Not because the engineers had been careless. Because cherry-picked examples always lie. They are, by definition, the cases you already know work.
That project is not an outlier. The same pattern appears across different teams, different clients, different industries. The tooling for building agents has matured rapidly. The discipline of measuring whether those agents actually work in production has not kept pace.
The first problem is picking the wrong quality metrics, or picking none at all. Not every metric that sounds good is worth tracking. Some are easy to compute but tell you almost nothing about real-world usefulness. Others are harder to set up but map directly to whether users get correct answers.
The choice depends on what the agent is doing. A retrieval task, a classification task, and a generation task each call for different metrics. Precision, recall, BLEU scores, human preference ratings, task completion rates: each captures something different, and using the wrong one can give you a false sense of quality that collapses on contact with production traffic.
Equally important is defining KPIs before you build, not after. If you define what “good” means after you see the results, you will unconsciously calibrate to whatever the system already produces. That is not measurement. That is retroactive rationalization dressed up as evaluation.
Every evaluation framework eventually needs a golden dataset: a curated set of inputs with known-correct outputs that you can run your agent against and trust the results. In theory, this is straightforward. In practice, it is one of the most labor-intensive parts of the entire project.
A golden dataset needs to cover the real distribution of inputs your agent will face in production, including edge cases, ambiguous queries, and failure modes that developers did not anticipate during build. Creating that dataset requires domain expertise, time, and often significant disagreement between annotators about what “correct” even means for a given case.
Three people reviewing the same set of outputs with three different scoring rubrics can produce results that differ by 20 percentage points or more. That variance is not a sign of carelessness. It reflects genuine ambiguity in the task. Any evaluation framework that ignores that ambiguity is measuring false precision.
Golden datasets are worth building. But they need to be built with rigor, kept up to date as the task evolves, and treated as a living artifact, not a one-time exercise.
Using a large language model to evaluate the outputs of another large language model has become a standard technique, and for good reason. It scales. It is faster and cheaper than human annotation. For many evaluation tasks, it produces results that correlate well with human judgment.
But LLM-as-judge has failure modes that are easy to miss. LLM judges tend to be length-biased: they often favor longer, more detailed answers even when a shorter answer is more accurate. They can be swayed by confident-sounding language. And if the judge model has overlapping weaknesses with the agent being evaluated, it will not catch the errors that matter most.
LLM-as-judge works best as a first-pass filter and as a way to evaluate at scale. It works worst as a replacement for human review on high-stakes decisions or on task types where the judge has known blind spots. Use it as a layer in the evaluation stack, not as the entire stack.
At some point, automated evaluation is not enough. For tasks with legal, financial, or safety implications, human review of a sample of agent outputs is not optional. The question is how much review, and of what.
A practical frame: 20% human oversight of agent outputs, applied thoughtfully to the right slice of cases, catches the majority of consequential errors. 0% oversight, followed by a regulatory finding or a client escalation, costs orders of magnitude more. The math on human review is not actually complicated once you include the cost of the alternative.
The design question is where in the workflow human review sits and what triggers it. Reviewing everything is expensive and unnecessary. Reviewing nothing is a risk most organizations cannot afford. A tiered approach — where the agent handles high-confidence cases autonomously and flags lower-confidence cases for human review — is often the right architecture for regulated industries.
One of the most common misconceptions about AI agents in production is that deployment is the end of the evaluation process. It is the beginning of a new phase.
Production traffic is different from development traffic. Users phrase things differently than developers expect. Edge cases appear that never showed up in the golden dataset. Model behavior can shift subtly as underlying models are updated by providers. The distribution of inputs drifts over time as the user base grows or the product evolves.
Without regression testing, you will not know when quality drops. You will find out when users complain, or when a client calls. Running the same evaluation suite against production samples on a regular cadence is not overhead. It is the mechanism that keeps the gap between demo performance and production performance from becoming invisible.
The drop from 100% in controlled testing to 60% on day one in production is not a failure of the build. It is the natural consequence of moving from curated inputs to real ones. The question is whether you have the instrumentation to see it, understand it, and respond before it becomes a business problem.
Expecting an AI agent to achieve 100% accuracy is the wrong target. A skilled human performing the same repetitive task typically achieves around 95% accuracy — not 100%. Setting a higher bar for the agent than for the human it is assisting is not rigorous quality control. It is a standard designed to justify inaction.
The right benchmark depends on the task, the cost of errors, and the baseline performance of the current process, whether that process is human, automated, or a combination. In most cases, an agent that performs at 85-90% on a well-defined task, with human review covering the remainder, represents a meaningful improvement over the status quo.
Root Cause Analysis (RCA) is the tool that makes this conversation productive with clients. When a quality drop occurs, RCA lets you show exactly where in the pipeline the failure happened, what category of input caused it, and what the fix looks like. That shifts the discussion from “the AI is not working” to “here is the specific problem and here is how we address it.” That conversation is only possible if you have measurement in place to begin with.
What is the difference between AI agent testing and AI agent evaluation?
Testing typically refers to running a fixed set of cases before deployment to check for bugs and basic functionality. Evaluation is broader and ongoing: it covers how well the agent performs across the real distribution of inputs, including in production, using statistical metrics rather than pass/fail checks. Evaluation continues after deployment; testing usually stops at release.
How many examples do you need in a golden dataset for AI agent evaluation?
There is no universal number, but a dataset of fewer than 100 examples is rarely sufficient to measure performance reliably across the real distribution of inputs. For agents handling varied request types, 300-500 labeled examples is a more practical minimum. The distribution matters as much as the count: a dataset that overrepresents easy cases will give you an inflated success rate that does not hold in production.
When should you use LLM-as-judge versus human evaluation?
LLM-as-judge is effective for high-volume, low-stakes evaluation where speed and cost matter more than perfect precision — for example, tracking relative performance across development iterations. Human evaluation is necessary when the task has genuine ambiguity, when legal or regulatory accountability is involved, or when you need to calibrate your LLM judge itself. In practice, most robust evaluation pipelines use both: LLM-as-judge for scale, human review for calibration and edge cases.
What causes AI agent performance to drop after deployment?
The most common causes are: data drift (production inputs differ from training and test distributions), model updates by the underlying LLM provider that change baseline behavior, changes in user behavior or phrasing over time, and edge cases that never appeared in development testing. Regular regression testing against production samples is the primary mechanism for detecting these drops early.
Adam Horka is an AI Engineer at ableneo, where he focuses on agentic systems, evaluation frameworks, and AI deployment in enterprise environments. ableneo has delivered over 34 production AI projects across 4 countries.
External Tech Workshop is a monthly knowledge-sharing series run by ableneo, open to practitioners from the broader technology community.
