Evaluating AI Systems in a Pilot: Evals, Guardrails, Red-Teaming

The 2021 version of this article was about testing machine learning models: training sets, validation sets, a facial recognition system that only recognized some faces. The systems being piloted in 2026 are different. They are LLM features and agents: non-deterministic, driven by natural language, able to call tools with side effects, and priced per token. They need a different evaluation discipline, and most pilots do not have one.

That is why most pilots end as demos. MIT’s NANDA initiative found 95% of enterprise GenAI pilots show no measurable P&L return, and the cause was a learning and workflow gap, not model quality. A demo shows that the system can work. A pilot has to show how often it works, on your data, at what cost, and what happens when it fails. This article covers the seven practices that turn a pilot into a decision: evaluation sets, LLM-as-judge, agent and tool testing, guardrails, red-teaming, monitoring, and cost and latency budgets.

What makes AI systems different to test

The five challenges from the original article still apply. What each one means has changed.

Variable output. The same input can produce different outputs. Exact-match assertions do not work. You evaluate whether the output falls within acceptable boundaries, which means writing down what acceptable means before the pilot starts.

Bias. A model trained on the internet carries the internet’s biases into invoice extraction and candidate screening alike. You test on a set that covers your real population of cases, including the rare ones, and you measure error rates per segment.

Changing environment. Providers upgrade and retire models on their own schedule. A prompt tuned for one version behaves differently on the next. Every change to the model, the prompt or a tool needs a regression run.

Data security. The system reads documents, emails and records. Some of that content will contain instructions aimed at the model. Prompt injection is the new failure class, and it is found with adversarial cases, not code review.

Ongoing training. Production models do not learn from your traffic, but your system does drift: new document formats, new products, new attack patterns. Monitoring replaces the old worry about a bot learning bad habits, and it is a permanent cost.

Build the evaluation set first

An evaluation set is a collection of real cases with known correct outcomes. It is the single most important artifact of the pilot, and it is built in week 1, before the first prompt.

Where cases come from. Your own records: last quarter’s invoices, tickets, contracts, shipments. Three groups:

• Golden cases: typical inputs with unambiguous correct outputs.
• Edge cases: unusual formats, missing fields, multiple languages, very long inputs.
• Failure cases: inputs the system should refuse, escalate or flag, including adversarial ones.

How many. Between 50 and 200 cases is enough to start a pilot. The set grows weekly from production samples once the system is live. Stratify it so rare-but-costly cases are represented, not just frequent ones.

Labels. Each case gets the expected outcome and the reason. Labels are produced or checked by the people who do the job today. Disagreement between labelers is itself a finding: if two experts disagree on 15% of cases, the system cannot score above 85% agreement, and the threshold has to reflect that.

Metrics by task type.

Task	Primary metrics
Extraction (invoices, forms, bills of lading)	Field-level accuracy; straight-through rate (cases needing no human touch); error rate on money and date fields
Classification and routing	Precision and recall per class; cost-weighted error
Generation (summaries, drafts, answers)	Rubric score; groundedness against sources; refusal on out-of-scope inputs
Agents	Task success rate; tool-call correctness; steps per task; escalation precision

The baseline. Measure the current process on the same cases: the manual team, the rules engine, the vendor tool. A pilot that reports 91% accuracy without a baseline has reported nothing. The threshold for go is set against the baseline, in writing, before build starts.

The same set becomes the regression suite in production. Our invoice automation system reached ~95% of invoices digitized without human intervention because straight-through rate was the metric from day one and the evaluation set was built from the companies’ own invoices.

LLM-as-judge, and when to trust it

Some outputs cannot be checked by string comparison: a summary, a drafted reply, an explanation. The practical answer is to have a second model grade the first against a rubric. This is now standard practice. In-production LLM-as-judge evaluation, tracing and cost monitoring are described as table stakes for enterprise AI by Modern Data 101.

Rules that keep a judge honest:

• Rubric, not impressions. The judge scores named criteria: factual consistency with the source, completeness against required fields, tone, refusal where required. Each criterion is scored separately, with the evidence quoted.
• Calibrate against humans. Run the judge on 100 cases that experts have graded. Report the agreement rate. A judge that agrees with experts 90% of the time is a useful screening tool. One at 70% is noise.
• Different model, different prompt. A model grading its own output shares its blind spots. Use another model family for the judge, or at minimum a separate prompt with no access to the generator’s reasoning.
• Pairwise where possible. “Which of these two outputs is better, and why” is more reliable than an absolute score.
• Deterministic first. Anything checkable by code is checked by code: schema validity, required fields, numbers that must sum, dates that must be in range. The judge handles what is left.

The judge is a screening tool. It decides which cases a person reviews, not whether the system ships.

Testing agents: tools, state and side effects

An agent is a program that calls other programs. Testing it means testing the program, not only the model’s answers.

• Unit tests for tools. Every tool the agent can call is tested like any API integration: valid inputs, invalid inputs, timeouts, partial failures, permission errors. The tool layer is deterministic and should have ordinary test coverage.
• Sandboxed systems of record. The agent runs against a staging ERP, a mock carrier API, a copy of the ticket queue. No pilot agent writes to production during evaluation.
• Trace replay. Every run is recorded as a trace: prompt, tool calls, tool results, decisions. Failed runs are replayed as new evaluation cases. Assertions are written against the trace: the agent called the budget check before creating the order; it never called a write tool after a validation failure; it escalated when confidence was below the threshold.
• Idempotency and recovery. Rerunning a failed step does not create a duplicate order. A run paused for approval resumes from the checkpoint. Timeouts and step budgets end runaway loops.
• Regression on every change. New model version, new prompt, new tool: the full evaluation set runs, and the results are compared with the previous run before the change is promoted.

Guardrails

Guardrails are constraints enforced outside the model. They are tested like security controls: you try to get past them.

• Input validation. Size limits, format checks, language detection, and detection of content that looks like instructions rather than data.
• Output validation. Structured outputs are validated against a schema. Free text is checked for leaked identifiers, unsupported claims and out-of-scope content before it reaches a user.
• Allowlisted tools and least privilege. The agent can call the tools in its list, with the scopes assigned, and nothing else. Write tools are separate from read tools.
• Deterministic verification before release. For a law firm’s contract generation system, a verification controller checks every LLM-generated clause against rules before a document is released. The system produces contracts 10x faster and saves 28,800 hours and €3.5M a year. The controller is what made lawyers willing to use it.
• Confidence gates. Outputs below a threshold route to a person. The threshold is chosen on the evaluation set to hit a target error rate on what passes through unattended.
• PII and PHI controls. Masking before data reaches the model, or a model deployed inside your VPC. Retention limits. A data-flow diagram that a security lead has signed.

Red-teaming

Red-teaming is structured adversarial testing: people and scripts trying to make the system do what it must not. It belongs in the pilot, because the findings change the architecture, and architecture is cheaper to change in week 3 than in production.

What to attack.

• Direct prompt injection: instructions in the user input that override the system’s rules.
• Indirect prompt injection: instructions hidden in documents, emails, web pages or tool results the system reads. The most important class for agents, because the attacker never talks to the model directly.
• Data exfiltration: getting the system to reveal other users’ data, system prompts, credentials or retrieved documents it should not surface.
• Over-permissioned tools: using a legitimate tool for an unintended effect, such as a search tool that can reach internal hosts.
• Denial of wallet: inputs that trigger long loops or huge outputs and run up the bill.
• Policy evasion: getting refused content or actions through rephrasing, role-play, encoding or multi-turn setup.

How to run it.

1. Write a threat model: what the system can access, what it can do, who would want to abuse it, what the worst outcome is.
2. Build an attack library: public injection and jailbreak corpora, plus cases specific to your domain and tools.
3. Automate the library as an adversarial evaluation set and run it on every change.
4. Add manual sessions by engineers who know the tools, because automated suites miss the creative attacks.
5. Fix at the architecture level (permissions, validation, isolation), not with a prompt patch, and re-run.

The output is a list of findings with severity, a fix for each, and the adversarial cases added to the regression suite.

Monitoring in production

The pilot ends. The evaluation does not. What runs in production:

• Tracing on every run. Inputs, outputs, tool calls, tokens, latency and outcome, with the identity that triggered it. This is the audit log and the debugging tool.
• Sampled human review. A fixed share of runs, plus every escalation and every low-confidence case, reviewed weekly by the people who own the outcome. Their corrections become new evaluation cases.
• LLM-as-judge in production. The calibrated judge scores a sample of outputs continuously, so quality drift is visible before users report it.
• Drift detection. Input distribution (new document types, new languages, new products) and output distribution (rising escalation rate, falling straight-through rate) tracked against the pilot baseline.
• Alerts with thresholds. Error rate, escalation rate, latency and cost per transaction each have a threshold that pages someone. Silent degradation is the common failure.
• Dashboards in your stack. CloudWatch, Datadog, Grafana: whatever your operations team already watches. A separate AI dashboard nobody opens is not monitoring.

Cost and latency budgets

Cost is a test criterion, not an afterthought, because it decides whether the system pays for itself at production volume.

• Cost per transaction. Measured at pilot volume, projected at production volume, compared with the cost of the current process. Reported in the go/no-go.
• Model routing. A small model handles the easy majority of cases; the frontier model handles the rest. The evaluation set tells you where the line is.
• Caching. Repeated prompts, retrieved documents and generated artifacts are cached. Two identity management agents we built cut LLM API spend by 90% by caching the scripts they generate. Account setup fell from 15 minutes to under 1 minute.
• Latency budgets per step. Each step in an agent has a time limit; the whole run has one too. Users tolerate seconds for a draft and nothing for a lookup.
• Hard stops. Spend limits per run, per user and per day, enforced by the platform. An agent in a loop should hit a budget, not a finance review.

What a pilot should end with

A pilot is a 4-week project whose deliverable is a decision. Ours ends with four things:

1. Data governance. A data-flow diagram of what the system reads, writes and stores, with PII and PHI classification, access scopes and the residency of every model call, reviewed with your security lead.
2. Evaluation results. The evaluation set and its results on your records: accuracy, straight-through rate, error rate and failure categories, against the baseline and the threshold set in week 1. Test coverage of tools and integrations. Red-team findings and fixes.
3. Observability and cost. Tracing from every pilot run, a dashboard of volume, latency, failures and token spend, and a cost per transaction at pilot volume with a projection at production volume.
4. Integration and handover. A working integration into at least one system of record, and a production plan with architecture, remaining governance items, team and timeline.

If any of the four cannot be produced, that is itself the finding. A no-go with a clear reason is a good outcome. It cost 4 weeks instead of 4 quarters, which is the point of the Erbis AI Pilot in 4 Weeks: fixed fee, fixed timeline, your data, and a go/no-go decision at the end. Most pilots end as demos. Ours end as decisions.

FAQ

How is testing an LLM feature different from testing software?

Outputs vary, so you test against acceptable boundaries and rubrics instead of exact values. Inputs are natural language, so the test set is drawn from real records rather than written by hand. The model changes underneath you, so every change needs a regression run on the full evaluation set. The tool and integration layer around the model is tested like ordinary software.

How large does an evaluation set need to be?

Between 50 and 200 labeled cases to start a pilot, stratified so that rare, costly cases are represented. It grows weekly from production samples and human corrections. Size matters less than provenance: cases must come from your own operation and be labeled by the people who do the job.

Can an LLM reliably judge another LLM’s output?

As a screening tool, yes, when it grades against a written rubric, uses a different model family, and has been calibrated against expert grades on at least 100 cases with the agreement rate reported. As the final authority on whether a system ships, no. Deterministic checks and human review remain in the loop.

What does red-teaming an AI agent involve?

A threat model of what the agent can access and do; an attack library covering direct and indirect prompt injection, data exfiltration, tool misuse and cost attacks; automated runs of that library on every change; and manual sessions by engineers who know the tools. Findings are fixed at the architecture level and added to the regression suite.

Erbis has run production AI since 2017 and holds ~95% AI accuracy in production across systems that have saved clients $110M+. If your pilot is producing a demo rather than a number, book a discovery call. We will tell you what the evaluation should look like, whether or not we build it.