Guardrails and Evals: How to Test an AI Agent Before You Trust It
Guardrails limit what an agent can do; evaluations tell you whether it does the right thing. You need both, and you need them before launch, not after the first incident.
By RFsoftLab Team, AI Engineering · · Updated · 3 min read
Software teams are used to unit tests that pass or fail. Language-model behaviour is probabilistic, so the testing discipline has to change shape rather than disappear. Two practices carry most of the weight: guardrails at runtime and evaluations before and after every change.
Guardrails: limit the blast radius
A guardrail is a control that sits around the model. It does not make the model smarter; it makes the system safer when the model is wrong. Useful layers include:
- Input checks. Detect out-of-scope requests, obvious prompt-injection patterns, and sensitive data that should be redacted before it reaches a model.
- Tool-level constraints. Validate arguments, enforce allow-lists, cap amounts and rates, and require approval for consequential actions. These are the most dependable guardrails because they are ordinary code.
- Output checks. Verify structure, check that cited sources actually exist in the retrieved context and block content that violates policy.
- Budgets and limits. Maximum steps, maximum tool calls, maximum spend and a timeout per run.
- Escalation paths. When confidence is low or a rule fires, hand the case to a person with the context attached.
Be realistic about what each layer can promise. Pattern-based and model-based filters both miss things. That is why the strongest guardrails are the ones that do not rely on the model behaving: permissions, validation and approvals.
Evaluations: measure what matters
An evaluation set is a collection of representative cases with a definition of success. Build it from real work:
- Collect cases from historical tickets, documents or transactions, anonymised as required.
- Label expected outcomes. For some tasks this is an exact answer; for others it is a checklist ("cites the right clause", "asks for the missing field", "does not promise a refund").
- Include hard cases. Ambiguous requests, missing information, contradictory documents and attempts to push the agent out of scope.
- Include "should refuse or escalate" cases. An agent that always answers is not being helpful; it is being unsafe.
- Score consistently. Combine exact checks, rule-based checks and, where useful, a model-based grader calibrated against human judgments.
Run the suite on every change to the prompt, tools, retrieval settings or model version, and store the results so you can compare runs over time.
Test the trajectory, not only the answer
Agents act over several steps. A correct final answer reached by calling the wrong tool, or by reading data it should not have seen, is still a defect. Record the sequence of tool calls and assert on it: which tools were called, in what order, with what arguments, and whether any forbidden call occurred.
Red-team deliberately
Set aside time to attack your own agent. Plant instructions inside documents the agent will retrieve, ask it to reveal its configuration, try to get it to call tools outside its remit and test odd inputs such as very long text or mixed languages. Add every successful attack to the evaluation set so it cannot return silently.
Monitor in production
Offline tests will not cover everything. Sample live runs for human review, track the rate of escalations and corrections, and watch for drift after provider-side model updates. A rising correction rate is an early warning that is far cheaper to act on than a complaint.
Make results decision-ready
Agree thresholds with the business owner before launch: what pass rate is needed on which categories, and which categories must be perfect (for example "never sends external email without approval"). Hard rules like that should be enforced by code, not measured by evaluation.
Keep it proportionate
A low-risk internal drafting assistant needs a lighter regime than an agent that touches financial records. Match the effort to the consequence of failure, but do not skip the evaluation set. Even a modest one gives you something that most early agent projects lack: evidence.
Frequently asked questions
- What is the difference between a guardrail and an evaluation?
- A guardrail is a runtime control that blocks, rewrites or escalates unsafe behaviour. An evaluation is an offline or sampled test that measures how well the agent performs against expected outcomes. Guardrails reduce harm; evaluations reveal quality and regressions.
- Can an LLM be used to grade another LLM?
- Yes, as one signal. Model-based grading scales well but can be biased or wrong, so calibrate it against human-labelled examples and keep humans reviewing a sample.