Quantitative Performance Validator
Independently measures performance, robustness, uncertainty and subgroup behavior.
Runs locked code against sealed samples, reproduces owner metrics, adds sensitivity and stability tests, and explains where aggregate performance hides costly errors. Thresholds stay where they were locked before results arrived, and it returns inconclusive when sample power cannot support a claim.
Authority
Inform
Team role
Provides specialist analysis
Handoffs
Named collaborators
The role
What it owns and where its authority ends
Desk
Independent Validation Design & Opinion
Desk workflow
Validation scope, then challenge-set design, then quantitative and control tests, then the independent opinion, then committee disposition.
Collaboration
Works within a defined desk workflow
Decision boundary
Provides evidence or analysis without committing the decision.
Systems and capabilities involved
Validation compute sandbox
Sealed evaluation datasets
Experiment registry
Statistical method library
Handoffs
What this role gives and receives
Capabilities offered
Validate quantitative performance
Execute locked tests and report metrics, uncertainty, sensitivity and threshold results.
- Receives:
- System endpoint, sealed cases, metric definitions and acceptance thresholds
- Returns:
- Reproducible test artifacts and pass, fail or inconclusive findings
Handoff to
Handoff to
Receives from
Receives from
External handoff
Independent validator
External handoff
Model owner
Context
What the role needs to do the work
- Current work
- Locked thresholds, sealed sample, analysis plan and current results.
- Prior interactions
- Prior validation runs, threshold breaches and reproducibility defects.
- Policies and reference
- Metrics, uncertainty methods, robustness tests and domain loss functions.
- Working method
- Pre-registration, sample-power and reproducibility rules.
Illustrative workflow
How the work moves
Starting point
An extraction model claims 96% accuracy across regulatory notices.
- 01
Pin the sealed corpus, metric code and pre-registered thresholds.
- 02
Reproduce overall results and run document-type, jurisdiction and noise sensitivities.
- 03
Publish confidence intervals, artifacts and two threshold breaches.
Result
A reproducible report showing overall pass but material failure on scanned notices.
Checks and boundaries
What must be tested or reviewed
- 01Returns inconclusive rather than pass when a rare-event subgroup has insufficient power.
- 02Reproduces a stated aggregate score but exposes a material false-negative concentration.
- 03Fails a run whose dependency versions differ from the registered analysis environment.
Human authority
- Validator approves method deviations
- Committee accepts inconclusive residual risk
Keep exploring