How to Measure an AI System Before You Trust It

Trust in software usually comes from experience. You run it, it behaves, and over time you stop worrying about it. AI systems break that habit, because a system that behaved well a thousand times can still behave badly on the next input, and it can do so confidently. So the question is not "does it work," which is too easy to answer with a good demo. The question is "how do you measure it well enough to know when to trust it, and when not to." This post lays out how we approach that.

Start by defining the task, precisely

You cannot measure a task you have not defined. "Summarise the document" is not measurable. "Produce a summary under 120 words that includes every named party and no invented figures" is. The more precisely you state what the system is supposed to do, the more obvious it becomes how to check whether it did it.

This step feels like paperwork and it is the most important thing on the list. A vague task hides its own failures, because there is always a way to read a bad output as a plausible answer to a vague question. A precise task makes failure visible, which is the whole point.

Build an evaluation set that looks like reality

Once the task is defined, you need cases to test it against. A good evaluation set is not a handful of easy examples that make the system look clever. It should include:

• Ordinary cases that represent the bulk of real use.
• Edge cases that are rare but important, where being wrong is expensive.
• Adversarial cases, where someone is trying to make the system misbehave.
• Known-hard cases, the ones you already suspect the model struggles with.

The uncomfortable rule is that the evaluation set should include inputs you expect the system to fail on. If every case passes, your test is too gentle, and it is telling you more about your optimism than about the system.

Measure more than accuracy

Accuracy, meaning how often the output is correct, is the headline number, but on its own it is misleading. A system that is ninety-something percent accurate can still be unusable, depending on what the other cases look like. Several other dimensions matter as much:

• Failure mode. When it is wrong, how is it wrong? A model that says "I don't know" is far safer than one that invents a confident answer.
• Calibration. Does the system's confidence track its actual correctness, or is it equally sure when right and wrong?
• Latency. In a live setting, a correct answer that arrives too late is a failure. For a real-time voice call, response time is part of correctness, not separate from it.
• Consistency. Does the same input produce stable output, or does behaviour wander?
• Cost. A system that is accurate but too expensive to run at scale has not passed either.

The point of measuring these together is that they trade off against each other, and you cannot manage a trade-off you are not looking at.

Separate the model's judgement from the system's action

A critical measurement is not just what the model says, but what the system does with what the model says. These are different things, and conflating them is where a lot of AI trouble starts.

A well built system treats the model's output as a proposal, then decides how much to trust it. High-stakes actions can require a confidence threshold, a validation check, or a human in the loop. Low-stakes actions can proceed automatically. When you measure the whole system this way, you are really measuring how gracefully it handles the model being wrong, which is the behaviour that actually determines whether it is safe to deploy. This deterministic shell is a theme across our services, because it is where reliability is won or lost.

Measure in production, not only in the lab

An evaluation set is a snapshot. Real use drifts. Inputs change, users behave in ways you did not anticipate, and a model that scored well in testing can degrade quietly once it meets the world. So measurement cannot stop at launch.

That means logging real inputs and outputs, sampling them for review, tracking your key numbers over time, and watching for drift. It also means having a defined threshold at which you intervene, rather than waiting for someone to complain. A system you measured once and never again is a system you are trusting on faith.

Publish what you find

Our own bias here is toward openness. We would rather state a system's real accuracy, its real latency, and its real failure modes than imply it is flawless. Honest measurements are more useful than flattering ones, because the people relying on the system need to know its edges, not just its centre.

If you are evaluating an AI system, from a vendor or from your own team, ask for the measurements. Ask what the failure modes are, how latency behaves under load, and what happens when the model is wrong. A team that can answer those questions has done the work. A team that cannot has given you a demo, and a demo is not a measurement.