Validation Design Orchestrator
Builds the independent validation plan and commissions the right specialist challenge.
Translates the risk tier, intended use, failure costs, architecture and change history into testable claims, acceptance thresholds and independent work packages. It routes GenAI, quantitative, fairness, security and agentic-control tests, and sets the limits of challenge itself rather than inheriting the development team's test plan.
Authority
Prepare
Team role
Coordinates the work
Handoffs
Named collaborators
The role
What it owns and where its authority ends
Desk
Independent Validation Design & Opinion
Desk workflow
Validation scope, then challenge-set design, then quantitative and control tests, then the independent opinion, then committee disposition.
Collaboration
Works within a defined desk workflow
Decision boundary
Assembles the work product; approval remains elsewhere.
Systems and capabilities involved
Governance inventory and card
Validation pattern library
Test-agent directory
Work-paper manager
Handoffs
What this role gives and receives
Capabilities offered
Design an independent validation
Create claims, failure modes, tests, thresholds, samples and specialist assignments.
- Receives:
- Review-ready evidence pack, risk tier and material-change record
- Returns:
- Signed validation plan with work packages and acceptance criteria
Delegates
Create independent baseline and adversarial cases from the stated use and failure costs. Trigger: Every validation plan Returns: Versioned challenge set with coverage rationale.
Delegates
Measure performance, robustness and subgroup behavior against locked thresholds. Trigger: System makes predictive, ranking or extraction claims Returns: Reproducible metrics, uncertainty and threshold results.
Delegates
Commission retrieval-grounding, prompt-injection and factuality testing. Trigger: System generates content, uses retrieval or accepts untrusted text Returns: Consolidated GenAI challenge report and critical findings.
Handoff to
Receives from
Receives from
External handoff
Independent validation lead
External handoff
Fair-lending
Context
What the role needs to do the work
- Current work
- System claim set, risk tier, validation scope, work packages and open evidence.
- Prior interactions
- Prior validation plans, missed failure modes and committee feedback.
- Policies and reference
- Validation standards, system archetypes and regulatory expectations.
- Working method
- Independence, sampling, threshold and change-materiality rules.
Illustrative workflow
How the work moves
Starting point
A retrieval-grounded lending agent moves from drafting to committing adverse-action notices.
- 01
Reframe the material change around decision impact, factual grounding and tool authority.
- 02
Define acceptance thresholds and commission challenge, GenAI and agent-control work packages.
- 03
Route the completed evidence to the opinion judge without seeing the owner's preferred conclusion.
Result
A tier-proportionate plan with locked thresholds, named tests and independent owners.
Checks and boundaries
What must be tested or reviewed
- 01Adds authority and rollback tests when a summarization assistant gains a write tool.
- 02Commissions fairness tests only where a decision affects a person, and leaves a back-office parser out of scope.
- 03Locks thresholds before results arrive and preserves failed work packages in the record.
Human authority
- Validation lead approves scope and thresholds
- Material scope changes require documented approval
Keep exploring