The output
Evidence is the product. Testing is how we get it.
What you get is not a dashboard. It is a signed, self-contained pack that an engineer, a release manager and a risk reviewer can each read and reach the same conclusion from, months later, offline.
What is in a pack
- The tested artifact: model, prompts, tools, policies, versions, hashes
- Scenario and control versions
- Execution lineage, including handoffs between agents
- Conversation and redacted trace references
- Tool names, arguments, results, retries and latency
- System state before and after
- Deterministic control results
- Statistical results and confidence
- Flake classification
- Cost and runtime per run
- How each result maps to your own internal controls
- A signed release recommendation
Structure
A pack on disk
Every run directory describes itself and can be checked on its own. Signature checking and offline re-evaluation are separate commands, so you do not need our service running, or even reachable, to audit a decision it made.
runs/<SCENARIO_ID>-<framework>-<variant>-<timestamp>/ ├── manifest.yaml artifact under test, versions, hashes, signature ├── pack.json control results, gate decision, statistics ├── events/ canonical normalized events (trustrail.events.v0.1) ├── spans/ source OTel spans, kept for traceability ├── annotations/ judge outputs and reviewer notes └── report.html the human-readable gate report $ make verify RUN_DIR=runs/<pack> # signature check $ make replay RUN_DIR=runs/<pack> # offline re-evaluation
Controls
Eight controls that apply to any acting agent
These do not depend on the domain. Moving from a refund to a cancellation to an access grant changes the entity and the system of record. It does not change the question being asked.
| Control | What it checks |
|---|---|
| Identity and ownership | The agent acted for the right user, on something that user actually holds |
| Authorization sufficiency | Auth state was good enough at the moment of the action, not just earlier in the session |
| Explicit confirmation | An irreversible action was confirmed before it happened, checked mechanically rather than inferred from tone |
| Correct entity selection | The thing acted on is the one the user chose, out of the set the agent actually retrieved |
| Sensitive data minimisation | The agent did not surface more than the journey needs |
| Idempotency and retry safety | A retried tool call did not do the same thing twice |
| Result and state consistency | What the agent told the user matches what the system of record actually did |
| Escalation | The agent handed off when it was not allowed to automate |
Two journeys in two different domains already run against this same control set, with only the domain pack changed. Concrete control IDs and a worked example.
Probabilistic systems
One run proves nothing
An agent that gets confirmation right once has not shown that it gets confirmation right. So scenarios run repeatedly, and the gate separates three outcomes: a deterministic pass, a statistically stable pass with a stated confidence, and flaky.
Flaky gets reported as flaky. We do not round it up to a pass because most of the runs were fine. On an irreversible action, an intermittent failure is the finding.
The gate stays advisory at first
TrustRail will not hard-block your pipeline until a design partner has months of real gate statistics behind the thresholds. A false block at a first customer destroys the reference, and the reference is worth more than the feature.
Run this against one of your agents
Six to eight weeks, fixed scope, fixed price, against your existing test environment.