Challenge-Set Design Agent
Builds representative, boundary and adversarial cases before results are known.
Derives cases from user journeys, loss events, complaints, near misses and plausible misuse, then balances them across normal, rare and high-cost conditions. It records provenance and expected invariants without leaking answer keys into the system under test.
Authority
Prepare
Team role
Provides specialist analysis
Handoffs
Named collaborators
The role
What it owns and where its authority ends
Desk
Independent Validation Design & Opinion
Desk workflow
Validation scope, then challenge-set design, then quantitative and control tests, then the independent opinion, then committee disposition.
Collaboration
Works within a defined desk workflow
Decision boundary
Assembles the work product; approval remains elsewhere.
Systems and capabilities involved
Sanitized production-pattern sampler
Synthetic data sandbox
Incident and complaint corpus
Evaluation registry
Handoffs
What this role gives and receives
Capabilities offered
Design an independent challenge set
Create traceable baseline, boundary, subgroup and adversarial cases.
- Receives:
- Validation claims, failure modes, use context and sampling constraints
- Returns:
- Versioned cases, invariants, provenance and coverage map
Handoff to
Handoff to
Receives from
Receives from
External handoff
Domain SME
External handoff
Conduct risk
External handoff
Data privacy
Context
What the role needs to do the work
- Current work
- Claims, failure modes, candidate cases and coverage gaps.
- Prior interactions
- Prior failures, production incidents and challenge-set blind spots.
- Policies and reference
- Domain journeys, typologies, protected cohorts and abuse patterns.
- Working method
- Case independence, provenance and leakage-prevention rules.
Illustrative workflow
How the work moves
Starting point
Validation needs cases for an agent that triages customer complaints.
- 01
Map ordinary, vulnerable-customer, deadline and adversarial failure modes.
- 02
Sample sanitized structures and synthesize boundary cases with independent expected invariants.
- 03
Deduplicate semantic variants and publish a coverage matrix.
Result
A sealed corpus spanning normal work, rare deadlines and adversarial instructions.
Checks and boundaries
What must be tested or reviewed
- 01Includes legitimate hard negatives alongside prompt attacks so refusal cost is measured alongside attack blocking.
- 02Keeps protected-class labels available to the evaluator while preventing them from entering the decisioning prompt.
- 03Detects duplicated paraphrases that would overstate scenario coverage.
Human authority
- Domain SME signs coverage rationale
- Privacy approves production-derived sampling
Keep exploring