Agent Assurance · Example
Agent assurance testing
Cadre name · Invigilator
Replays a supported pack's evaluations and inspects the recorded steps, returning findings and limitations.
Operating dossier
The job, permitted decisions, and owner
Mandate
Replays a supported pack's own evaluations and inspects observed execution paths, packaging findings and limitations.
- Case arrives with
- Target agent ID · Sample size
- Allowed outcomes
- No Material Automated Findings · Findings · Unable To Assess
- Where it fits
- Agent Assurance
Operating logic
- 01Supports an institution's independent assurance or validation process with an automated review of another Cadre blueprint; it supplies evidence for the reviewers who own the conclusion.
- 02Re-runs a defined sample of the target's evaluation cases and inspects the available execution evidence; it names the sample and what remains outside scope.
- 03Challenges how the target worked: missing commitments, unsupported conclusions, or control-relevant path anomalies become specific findings.
- 04Scans sampled execution records for declared prohibited-basis field names. This narrow exposure check is not fair-lending testing or a fairness conclusion; unobservable controls remain limitations.
- 05Reports findings, no material automated findings, or unable to assess (not a validation pass). The target owns its evidence, independence limits stay explicit, and accountable reviewers retain the full-scope conclusion.
Decision logic
- 01Supports an institution's independent assurance or validation process with an automated review of another Cadre blueprint; it supplies evidence for the reviewers who own the conclusion.
- 02Re-runs a defined sample of the target's evaluation cases and inspects the available execution evidence; it names the sample and what remains outside scope.
- 03Challenges how the target worked: missing commitments, unsupported conclusions, or control-relevant path anomalies become specific findings.
- 04Scans sampled execution records for declared prohibited-basis field names. This narrow exposure check is not fair-lending testing or a fairness conclusion; unobservable controls remain limitations.
- 05Reports findings, no material automated findings, or unable to assess (not a validation pass). The target owns its evidence, independence limits stay explicit, and accountable reviewers retain the full-scope conclusion.
- Must not
- Invent missing evidence or act outside the permitted outcomes.
- Human owner
- Independent review and remediation acceptance
- Evidence retained
- Structured outcome, observed path, and the source evidence used.
Case and record
One case, from supplied evidence to recorded decision
Supplied case
- Target agent ID
- fincrime.sanctions.disposition
- Sample size
- 3
View the case data
Case key validate-cerberus-pass
{
"targetAgentId": "fincrime.sanctions.disposition",
"sampleSize": 3
}Evaluation and limits
Evaluation coverage and stopping points
Published evaluation map
Expected decisions, boundaries, and costly failure modes.
These cases show what the pack is asked to decide, when it should stop, and which plausible errors the evaluation is meant to expose. They are not a reliability score or independent validation.
Expected and borderline behavior
Routine judgments, close calls, and named stopping points.
Replay Cerberus target-owned evals and produce a scoped challenge report
Clean checks may report no material automated findings, but never a validation pass.
Case key
validate-cerberus-passReplay Spark target-owned evals, run the declared-basis name-exposure check, and state that it is not fair-lending validation.
Case key
validate-spark-fairness
Boundary and misuse cases
Plausible inputs that could produce a costly or overconfident decision.
Unsupported target
do not manufacture evidence or approval; return unable-to-assess with independent-review follow-up.
Case key
adv-unknown-target
Operating fit
Workflow placement and validation
Operating pattern
Independent review
Separate the work from the party that tests it, then require findings to point back to evidence and a named standard.
Decision sequence and workflow placement
- 01Receive completed work
- 02Reconstruct the standard
- 03Run independent checks
- 04State findings and limits
- 05Assign accountable review
Wider workflows
Institution-specific validation
These questions shape how the example fits an institution.
Validation questions
- Policy and legal interpretation
- Source coverage, quality, and freshness
- Thresholds and exception calibration
- Integration and degraded-mode behavior
- Human approval and escalation design
- Performance on the institution’s own case mix
Inputs, actions, and outcomes
- Target agent ID
- Sample size
- Read manifestread_manifest
- Run eval samplerun_eval_sample
- Analyze trajectoryanalyze_trajectory
- Check guardrail exposurecheck_guardrail_exposure
- Commit challenge reportcommitcommit_challenge_report
- No Material Automated Findings
- Findings
- Unable To Assess
Related examples