# Sovereign EvalOps example contract

Revision: 2026-09-26. Contract schema: sovaihub-evalops-reference/v2.

This is a reference data contract, not a configuration accepted directly by
Promptfoo, Ragas, or an existing SovAIHub CLI. Build and test an adapter to your
selected runner and CI system. No runner is bundled with this download.

## Implement the contract

1. Replace every REQUIRED value with an approved immutable revision, digest,
   endpoint, owner, or evidence reference. Validate the contract before execution.
2. Load the companion JSONL using the RAG protocol. Map fixture principals and
   ACLs to isolated test identities and use sandboxed tool stubs.
3. Run each required case for every configured repeat. Record pass, fail,
   error, or skipped explicitly; absence is never success.
4. Evaluate critical authorization/injection checks separately from quality
   rates. A critical violation in any repeat blocks the release.
5. Compute case success rates per slice and repeat. Compare with the approved
   baseline using the same cases and scoring definitions. Apply the floors and
   regression limits to every required slice and repeat, not just the average.
6. Emit the specified JSON evidence contract and human review summary. Missing
   outputs, invalid metrics, or unresolved errors block the promotion step.
7. Exercise a passing candidate, an intentional critical failure, a missing
   case, and a missing baseline. Demonstrate that the latter three block release.

## Metric semantics

`task_success_rate` = cases passing all required behavioral checks / required cases
in that slice in that repeat. A case with missing evidence is not a pass; it also
triggers the incomplete-evidence blocker. Report numerator and denominator.
These cases have no numeric weighted average that can override a critical gate.

`max_regression_percentage_points: 2` means candidate rate minus baseline rate
must be at least -0.02 on the 0-to-1 scale. It does NOT mean a 2% relative change.
Security and abstention floors in the example are 1.0. With tiny sample counts,
rates have little precision; replace the illustrative thresholds and coverage
with workload-specific requirements before production use.

Eight synthetic cases (four answerable, one abstention, three security) demonstrate
the schema. Repeating them does not create independent coverage of more tasks.
Track per-repeat variability and use representative held-out cases for meaningful
confidence analysis. Do not claim zero hallucination or universal security.

## Evidence JSON requirements

- schema_version, run_id, started_at_utc, completed_at_utc
- contract_digest, dataset_digest, baseline_run_id, all artifact revisions
- per_case: case_id, slice, repeat, status, critical_failures, evidence references
- per_slice_per_repeat: numerator, denominator, rate, baseline_rate, delta_pp
- decision: pass or block, blocker reasons, reviewer, approval timestamp
- outputs: restricted raw-trace references and a redacted summary

Approved evidence storage and actual signing, if required, need configured
credentials, access control, retention, key management, and signature verification.
Never place secrets or unrestricted production traces into the manifest.

Guide: https://sovaihub.com/solutions/sovereign-evalops