What does it mean for an AI agent to work?
It works when it completes its task measurably and repeatably on a question set fixed in advance. A demo proves nothing: five questions, all five chosen by the developer. Production receives thousands of inputs nobody prepared for. The gap shows up in a single number, the task completion rate across two hundred known cases.
This measurement layer is mostly missing from the market. We read the public material of twelve AI agencies in one European country and not one of them describes how it measures the quality of its own systems. Prices, yes. Technology lists, case studies, yes. No eval set, no hallucination rate, no regression test. I do not think the omission is accidental. It is comfortable: what you do not measure, you never have to defend.
95%
of enterprise generative AI pilots produced no measurable P&L impact
MIT Project NANDA, July 2025
40%+
of agentic AI projects will be cancelled by the end of 2027, on Gartner's forecast
Gartner, 2025-06-25
5.7% → 1.9%
retrieval failure rate with hybrid search and reranking in Anthropic's own measurement
Anthropic, 2024-09-19
The MIT Project NANDA study from summer 2025 worked from more than 300 announced deployments, 52 interviews and 153 executive responses. Weak models are not what sits behind the 95%. What sits behind it is that these systems do not learn from corrections, lose context between sessions, and live outside the place where the work actually happens. Every one of those three failures is measurable. The same research found that an internal expert paired with an external partner reached a 67% success rate, while projects built with internal IT alone reached 22%.
Gartner's forecast of 25 June 2025 names three reasons for the cancellations: escalating cost, unclear business value, inadequate risk control. A working measurement system surfaces all three weeks before the executive sponsor runs out of patience.
How do you build a golden dataset?
A golden dataset is a collection of cases where the correct answer is known. Each entry has three parts: the input, the expected output, and the acceptance criterion that says what makes an answer good. This set becomes the system's contract with itself. Every later measurement is relative to it.
Where the cases come from
Not from the developer's imagination. Real traffic is the best source: inbound emails for a support system, last year's invoices for a document pipeline, the questions colleagues actually type for an internal knowledge base. If nothing is live yet, a one-hour workshop with the domain team will produce eighty genuine questions, which in our experience is enough for version zero.
For the mix we use a rough rule of thumb. Half the set should be typical, everyday cases. A quarter should be the rare but important ones: malformed input, missing fields, ambiguous phrasing. A tenth should be cases where the correct answer is that the system does not know and hands over to a human. The remainder is deliberate attack: prompt injection, requests to reveal the system instructions, requests for data the user has no right to see.
How many, and who signs off
Start with 80 to 150 cases. After six months in production it will typically be somewhere between 300 and 500. Beyond that the runtime and the model bill start to hurt while the added information keeps shrinking. The expected answer is always approved by the domain specialist, never by the engineer. Three quarters of projects skip this step and then spend months arguing about whether the system got it wrong.
Hold back 20 to 30 cases that you never look at during development. That is your frozen holdout. Optimise only against the cases you can see and you will eventually be fitting to them; the holdout is what tells you how much of the improvement was real.
When to refresh it
Once a month, plus immediately after every incident. When a user finds a bad answer, that case should land in the set the same day, together with the expected output. The failure becomes a regression test, and the same bug cannot escape twice. It is the cheapest quality measure I know of, and almost nobody does it.
Which metrics show whether the agent is any good?
Seven numbers, read together. Four describe quality, three describe the cost of running the thing. None of them is sufficient alone: 99% task completion is useless if each request takes 40 seconds and costs 20 cents. The thresholds below are our defaults for mid-market projects, not an industry standard.
| Metric | How you compute it | Realistic threshold |
|---|---|---|
| Task completion rate | Successful runs divided by all runs on the golden dataset, judged by the acceptance criterion | above 85% for internal use, above 95% for customer-facing answers |
| Hallucination rate | Share of answers containing a claim not supported by any source | under 2% customer-facing, under 5% in internal workflows |
| Citation accuracy (RAG) | Share of answers citing the genuinely correct source, among those that cite anything | above 90%, with retrieval failure under 5% |
| Human override rate | Share of outputs edited by a person, among those sent for approval | under 15% by month three, and trending down |
| Latency (p95) | The time within which 95% of runs finish, not the mean | 4 seconds for chat, 60 seconds for background jobs |
| Cost per request | Monthly model bill divided by request count, broken down by step | know the p95 as well; the mean misleads here |
| Tool call failure rate | Share of failed or schema-violating tool calls out of all calls | under 2%, and zero schema violations |
Task completion rate
The most important number, and the easiest to get wrong. Everything hangs on what you write into the acceptance criterion. If it reads “the answer contains the correct amount”, a machine can check it. If it reads “the answer is helpful”, nothing can. Every case needs a concrete, checkable condition. For multi-step agents, break the score down by step too, because that is how you discover that half your runs die at step seven.
Hallucination rate
An answer hallucinates when it asserts something that appears neither in the provided sources nor in the expected output. RAG makes this reasonably measurable, since you have the context to check against. Open-ended generation is harder and needs human labelling. We count per sentence rather than per answer: a twenty-sentence summary with one invented date is as wrong as a completely bogus reply, and answer-level counting hides that.
Citation accuracy in RAG
Measure two separate things. Whether retrieval brought back the relevant passage at all, and whether the model then cited the right source in its answer. These come apart more often than people expect: the system finds the correct document and quotes a different one. In Anthropic's own measurement, the 5.7% failure rate of plain embedding search dropped to 3.7% with contextual embeddings, to 2.9% once contextual BM25 was added, and to 1.9% with reranking on top. If your system is far worse than that, the problem is probably in retrieval rather than in the model.
Human override rate
This is the one metric production hands you for free, as long as the workflow has an approval step. It counts how often a colleague edits the output. The trend matters more than the absolute value. If it has not fallen in three months, the system is not learning, and sooner or later the approver will start clicking through without reading. That last state is the dangerous one, because the human control that exists on paper has quietly stopped existing.
Latency and cost per request
Look at p95, not the mean. A two-second average happily hides a 25-second p95, and the 25 seconds is what users remember. On cost, the number of steps decides, not the choice of model: in Anthropic's measurement a Drive-to-Salesforce copy task fell from 150,000 tokens to 2,000 once a chain of tool calls became code execution. On the official Anthropic price list, Claude Sonnet 5 is 2 USD per million input tokens and 10 USD per million output, a cache hit costs a tenth of the input price, and the Batch API halves both sides. These items are visible in a trace-level cost breakdown. They are invisible on the monthly invoice.
Tool call failure rate
If the agent calls APIs, count badly parameterised calls, schema violations and provider errors separately. Schema violations have a tolerance of zero, because structured output and schema enforcement eliminate them entirely. Bad parameters are a model quality question, and they are a good signal that a tool description is ambiguous. In most cases a better tool description buys more than a more expensive model does.
What is an eval harness, and when do you run it?
An eval harness is a script that runs the golden dataset through the system, scores the answers, and compares the result against the previous release. It runs in CI before every prompt, model, retrieval or tool change, and it can fail the build. That is the difference between measured quality and hoped-for quality.
CASES = load_jsonl("golden/support_v7.jsonl") # 140 cases
for case in CASES:
for _ in range(3): # nondeterminism: 3 runs
out = agent.invoke(case["input"], model=PINNED_MODEL)
assert_hard(out.schema_valid) # hard gates: schema, PII, tools
assert_hard(out.no_pii_leak)
assert_hard(out.steps <= MAX_STEPS)
scores.add(
grounded = judge_grounded(out.text, case["context"]),
correct = rubric_or_exact(out.text, case["expected"]),
cost_usd = out.usage.cost,
latency = out.latency_ms,
)
report = aggregate(scores) # mean + p95 + stdev
gate(report, baseline="v6", tolerance=0.02) # >2 point drop = build failsHow this differs from a unit test
A classic unit test is deterministic: one input has exactly one correct output, and any deviation is a failure. With a language model the output is a distribution, not a value. Run the same question twice and you may get two different answers, both correct. So the logic of testing has to change with it.
| Aspect | Classic unit test | LLM regression test |
|---|---|---|
| Output | deterministic, one correct value | a distribution; the same input yields several good answers |
| Assertion | equality check | hard checks on form, a score on content |
| Runs per case | one is enough | 3 to 5 runs, and you read the aggregate |
| Failure condition | any deviation | the score drops below baseline by more than the tolerance |
| Runtime and cost | seconds, free | minutes, and you pay per model call |
| When it runs | on every commit | on every prompt and model change; full set overnight |
Nondeterminism is manageable if you split the checks into two layers. The first layer is hard and machine-checked, with no tolerance: the JSON schema validates, no personal data leaked, the step count stayed under the ceiling, the mandatory tool was called. All of that is decidable, and any breach fails immediately. The second layer is a score, where tolerance is the whole point: if the previous release scored 0.91 and this one scores 0.88, that is a failure, while 0.905 is acceptable noise. Derive the tolerance from the variance you measured across two consecutive runs of the same set.
When can you let another model do the grading?
When the task is a linguistic judgement, and only after you have measured how closely the judge agrees with human labels. Groundedness, tone, format compliance, instruction following: a strong model judges these well. Numeric correctness, regulatory compliance and domain accuracy: it does not, because it carries the same knowledge gaps as the system under test.
Where an LLM judge works
- Groundedness checks, where the judge receives the source text and has to decide whether every claim in the answer follows from it.
- Rubric scoring against a reference answer, where the judge looks at whether the response covers the reference's key elements.
- Format and register compliance, such as whether the answer stays formal, fits the length limit, or includes the mandatory legal sentence.
- Pairwise comparison of two outputs, which is more stable than asking for an absolute score.
Where it fails
- Arithmetic, date logic, units. Check these with deterministic code, not with a model.
- Grading its own output. Models tend to prefer their own phrasing, so the judge should never be the model that produced the answer.
- Ten-point scales. Use binary or three-way verdicts, because agreement with human labels degrades quickly on fine-grained scales.
- Nuance in a language the judge was barely trained on. There is no public evidence base here, so human validation carries more weight.
How to validate the judge itself
Take 50 to 100 cases from the golden dataset and have a human label them with the same rubric you would give the judge. Run the judge, compute the agreement rate, and look separately at the false positives, meaning the cases where the judge accepted a bad answer. That is the more dangerous error type. Below 80% agreement the judge is unusable for that task. Version the judge's prompt and model the way you version the system's, and rerun the validation whenever either changes.
The evaluator-optimizer pattern in Anthropic's agent design guidance applies the same logic at runtime: one model generates, another evaluates, and the output improves on the feedback. The same guidance also states that for most tasks a coded workflow beats an autonomous agent. Measurement backs that up. With fewer degrees of freedom the variance is smaller, so the system behaves more predictably.
How do you observe the agent in production?
Eval runs offline with known answers; observability runs on live traffic without them. There you lean on indirect signals: tool call errors, human overrides, thumbs-up feedback, latency spikes, cost jumps. Every run is stored as a trace, and the suspicious traces are where your next eval cases come from.
| Aspect | Langfuse | LangSmith | Braintrust |
|---|---|---|---|
| Free tier | 50,000 units / month | 5,000 base traces / month, 1 seat | 10 USD model credit, 1 GB, 10,000 scores |
| Entry paid plan | 29 USD / month (Core) | 39 USD / seat / month (Plus) | 249 USD / month (Pro) |
| Overage | 8 USD / 100,000 units | 2.50 USD / 1,000 base traces | 3 USD/GB and 1.50 USD / 1,000 scores |
| Self-hosting | MIT licence, no licence fee | Enterprise contract only | Enterprise only |
| EU data residency | EU region on the cloud plans | EU region available, rest on Enterprise | Enterprise tier |
| Retention | plan dependent | 14 days base, 400 days at extra cost | 14 days Starter, 30 days Pro |
The three price lists are not directly comparable, because a Langfuse “unit”, a LangSmith “trace” and a Braintrust “score” are different units of account and the conversion is not published. Any quote needs your own volume estimate. Our default for EU clients is self-hosted Langfuse, because the licence fee is zero and the data never leaves the client's own infrastructure. If the stack is already LangGraph, LangSmith is more convenient, though trace-based pricing gets expensive at volume. If comparative experimentation rather than tracing is the real pain, Braintrust is built around evals. We wrote the detailed head-to-head in our Langfuse and LangSmith comparison.
Logging also has a compliance dimension in the EU. The transparency obligations under Article 50 of the AI Act have applied since 2 August 2026: users have to be told they are talking to a language model. The logging obligation for high-risk systems moved to 2 December 2027 for standalone Annex III systems under Regulation (EU) 2026/1744. Anyone building trace storage now satisfies that deadline essentially for free.
What do you do when the vendor swaps the model underneath you?
You run the same eval set on the new version, compare it against the old result, and switch only if the score holds. This is the most tangible payoff of the whole measurement effort: a model migration becomes a twenty-minute decision instead of weeks of guesswork driven by user complaints.
Two distinct phenomena are worth separating. Model drift is when the provider updates weights or routing underneath you and behaviour changes. Prompt drift is when your own team adds fourteen corrective sentences to the system prompt over several months, each one answering a specific complaint, until nobody can say which sentence solves what. The second is more common and does more damage. The remedy is that every prompt change goes through the harness, so you can see what the fix broke elsewhere.
The migration sequence
- The new version runs the full golden dataset, three times per case, with the prompt unchanged.
- The report compares all seven metrics against the baseline, broken down by case category. A global average can hide the collapse of one narrow but important category.
- If the score holds, a canary starts: 5 to 10% of traffic goes to the new version for a week, watched on live metrics.
- Full cutover, with the previous version kept switchable for another two weeks.
Price changes carry the same class of risk. Google's price list states that the 0.75 and 3.75 USD Gemini Flash rates are promotional and double after 31 December 2026. Anthropic went the other way and made the Sonnet 5 introductory price permanent. If your cost per request metric is running, a change like that shows up as a number in next week's report rather than as a surprise at the quarterly close. TechCrunch reported in June 2026 that Uber had burned its entire annual AI coding budget by April, and that Priceline's Cursor renewal came back at four to five times the price. The model was not at fault in either case.
What belongs in an AI system SLA?
Two separate numbers. An availability figure for the infrastructure, and a quality figure re-measured monthly on the golden dataset. On its own, 99.9% only says the endpoint is silent for less than 43 minutes a month. It says nothing about correctness, and with AI systems the failure mode is rarely an outage. It is a confidently delivered mistake.
| What goes into the contract | Why in this form |
|---|---|
| 99.5% monthly availability, measured on the API endpoint | The model vendor's own SLA is the upper bound. 99.9% allows 43 minutes a month, 99.5% allows 3.6 hours. |
| Task completion rate on the golden dataset, re-measured monthly | This is the real quality promise. The set and the measurement method are an annex to the contract. |
| p95 response time, separately for the chat and the batch path | The mean hides the long tail, and the tail is what users feel. |
| Monthly token ceiling and the overrun procedure | A monthly budget alert tells you after the fact. You need a per-run, real-time ceiling next to it. |
| Mandatory steps of the fallback chain | What happens exactly during an outage, who decides, and within what time. |
| Incident definition and a 24-hour report | Write down when a bad answer becomes an incident: data leak, legal exposure, financial impact. |
| The golden dataset belongs to the client and is exportable | This is the strongest exit protection. Without the measurement set the next supplier starts from zero. |
| Model changes announced, with mandatory re-measurement | The supplier cannot switch models without showing the measurement. |
What fallback means for a language model
Four steps, in this order. First a retry with a short backoff, since some failures are transient. Second another provider or another model for the same task, which requires the system to be built model agnostic. Third a deterministic fallback: a cached earlier answer, a template, a rule-based path. Fourth a human queue, where the request goes to a person and the user is told roughly how long it will take. There is a fifth, bad option, which is the system inventing something so it has an answer to give.
The measurement layer costs less than most people expect. Our published price list puts a RAG knowledge base build at 3,000,000 to 8,000,000 Ft, roughly EUR 7,800 to 20,800 at the rounded 385 HUF/EUR rate we use for August 2026 quotes. Inside that, assembling the eval set and building the harness takes a few person-days, and running it costs a few dollars of model calls a month. The real line item is maintenance: refreshing the cases and revalidating the judge. On our side that sits inside the monthly operations fee of 30,000 to 150,000 Ft (EUR 78 to 390), because we do not consider operations without eval to be operations.
Summary and common questions
How many cases does a golden dataset need?
Eighty to a hundred and fifty is enough to start, provided you pull them from real traffic and cover both the common task types and the rare, awkward ones. After six months in production the set typically grows to three or four hundred. Size matters less than provenance: the questions have to be real, and a domain expert has to sign off on the expected answer, not the developer.
What hallucination rate is acceptable?
We hold customer-facing answers under 2% and internal, human-reviewed workflows under 5%. Those are our contractual thresholds, not an industry standard, and no public benchmark exists to compare them against outside English. What the hallucination touches matters more than the rate. A wrong date in a contract summary is worse than a clumsy pleasantry.
What is the difference between eval and monitoring?
Eval runs offline on known inputs with known expected outputs, usually in CI before a release. Monitoring runs on live traffic where no expected answer exists, so you watch indirect signals instead: tool call failures, human overrides, user feedback, latency, cost. They cover each other's blind spots and neither replaces the other.
Can I use another language model as the grader?
Yes, but only after you have measured how well its judgement agrees with human labels. Run it on fifty to a hundred hand-labelled cases, and if agreement sits below 80% the judge is useless for that task. It works well for groundedness, tone and format compliance. It fails on arithmetic and domain correctness.
Why is temperature zero not enough for a deterministic test?
Because batching, quantisation, hardware differences and routing on the provider side all shift the output, and the model version can change underneath you. In practice we run every case three to five times and compare the aggregate score against the previous release, rather than comparing one string to one string.
What does an eval harness cost to run?
Two line items. The observability tool: self-hosted Langfuse carries no licence fee, its Core cloud plan is 29 USD a month, LangSmith Plus is 39 USD per seat a month, Braintrust Pro is 249 USD a month. Then the model calls: a 150-case set run three times is 450 invocations, which at Claude Sonnet 5 list prices is a few dollars. Maintenance costs more than either.
What does 99.9% availability mean for an AI system?
It means 43 minutes of downtime a month, and it says nothing at all about whether the answers are correct. That is why two separate numbers belong in the contract: an availability figure for the infrastructure and a quality figure re-measured monthly on the golden dataset.
What happens when the vendor swaps the model underneath me?
If you pinned the model to a specific version, the first thing you get is a deprecation notice. You then run the same eval set on the new version, compare it against the old result, and switch only if the score holds. Without an eval set you find out from user complaints, typically weeks later.
If you are planning an agent right now, start the measurement in week one rather than week twelve. For the conceptual groundwork our introduction to AI agents gives the background, and the retrieval side is covered in the article on RAG systems. If the architecture is still open, the comparison of chatbots, n8n and custom agents is where to begin. What we build and operate is described on the process automation page, and live work sits in the AI portfolio, including the semantic search engine we built on the pop-culture archive of kultura.hu.
Sources
- MIT Project NANDA: The GenAI Divide, State of AI in Business 2025 (July 2025)
- Gartner: Over 40% of agentic AI projects will be canceled by end of 2027 (2025-06-25)
- Anthropic: Introducing Contextual Retrieval (2024-09-19)
- Anthropic: Building Effective Agents (2024-12-19)
- Anthropic: Code execution with MCP (2025-11-04)
- Anthropic model pricing (retrieved 2026-08-14)
- Google Gemini API pricing (retrieved 2026-08-14)
- Langfuse pricing (retrieved 2026-08-14)
- LangSmith pricing (retrieved 2026-08-14)
- Braintrust pricing (retrieved 2026-08-14)
- Péter Harang: How well do embeddings handle Hungarian? (2025-01-09)
- Hatvani Péter et al.: Training Embedding Models for Hungarian
- TechCrunch: The token bill comes due (2026-06-05)
- Regulation (EU) 2026/1744, Official Journal (2026-07-24)
- MNB exchange rates (HUF to EUR conversions rounded to 385)
