Reference architecture / Evaluation & release engineering
Turn AI changes into evidence-backed release decisions. Design repeatable tests and enforceable gates for models, RAG pipelines, prompts, and agents, with evaluation data and judges kept inside the approved boundary.
Reviewed · Vendor-neutral guidance
01 / Decision framework
A model upgrade is only one release trigger. Prompt edits, source updates, chunking, embeddings, rerankers, tool policies, and serving configuration can all introduce regressions.
| Evaluation family | What the suite checks | Gate evidence |
|---|---|---|
| RAG and knowledge | Answer correctness, authorized retrieval, evidence support, citations, freshness, and abstention. | Case-level results, corpus snapshot, identities, evidence IDs, and reviewed failures. |
| Model and prompt | Task success, schema validity, language slices, quantization changes, and long-input behavior. | Pinned artifacts, sampling settings, repeated runs, and comparable baseline results. |
| Agents and tools | Authorized task completion, approval requirements, bounded loops, recovery, and tool argument validation. | Sandboxed action traces and denied actions; no destructive production tools in the test environment. |
| Security and operations | Injection, cross-tenant access, data exposure, timeouts, service targets, and accepted-outcome cost. | Critical-failure counts, missing-result checks, load conditions, and telemetry boundaries. |
02 / Decision framework
The architecture separates test inputs, execution, scoring, policy, and evidence so that each part can be changed without hiding how a release was approved.
Version the contract: test IDs, dataset and corpus digests, permissions, model/runtime revisions, prompts, judge rubric, thresholds, and baseline reference.
Execute in the approved environment: isolate tenants, restrict network egress, use secret references, freeze tool fixtures, and record complete per-case outcomes.
Score transparently: use deterministic checks where possible and human-calibrated judges where necessary. Record judge versions, disagreements, run variation, and slice-level sample counts.
Apply the gate: fail on critical violations, required cases not run, evaluator errors, insufficient coverage, or defined regressions. Produce machine-readable results and a reviewer-readable decision.
Promote and observe: require the named approver, retain the evidence, canary the change, and keep an approved previous artifact available for rollback. Feed incidents back into the suite.
Offline evaluation includes the judge, embeddings, dependencies, telemetry, and artifact storage. A local application calling a cloud judge is not an offline evaluation pipeline.
03 / Decision framework
Agree thresholds with the workload owner and security reviewer. The downloadable contract supplies illustrative values and explicit units; it requires an adapter to your CI system or evaluation runner.
Treat any observed unauthorized disclosure or tool action as a blocker, regardless of average quality. Distinguish a confirmed violation from an inconclusive evaluation, but block both until required evidence is resolved.
Check absolute minimums and permitted changes versus an approved baseline per required slice. A two-percentage-point change in a rate differs from a two-percent relative change. Record denominators, repeat counts, and confidence where meaningful.
An empty dataset, skipped critical case, timeout, invalid judge result, absent baseline, or missing slice must not become a green build. Fail the gate and preserve the error reason for triage.
Keep artifact digests, timestamps, decisions, owners, and redacted traces with access and retention policies. If signing is required, agree the key management and verification workflow; an unsigned JSON report is not signed evidence.
04 / Decision framework
Implement an evaluation workflow that fits the existing engineering stack before expanding it into a platform.
One application, one approved environment, a versioned holdout set, a baseline run, critical checks, and a CI decision. The handover should include a deliberate failing change that proves the gate blocks promotion.
Add more languages, tenants, agent tools, cost gates, load tests, and production sampling after the first pipeline is stable. Define who maintains labels, approves changed thresholds, and responds to drift.
The materials on this page are a reference design, not an installed SovAIHub runner or hosted monitoring product. Integration, report formats, and ongoing operation are agreed in the implementation scope.
Working materials
Download the editable materials, adapt them to your environment, and keep the assumptions alongside the results.
Work with SovAIHub
Scope an EvalOps pilot that produces a working release decision for one priority AI application.
Bring the application interface, current release process, representative test data, approved runtime boundary, CI access constraints, and the people authorized to accept quality and security risk.
Scope, acceptance criteria, delivery schedule, and fees are agreed before implementation.
Scope an EvalOps pilotConnect the decision to the next engineering or operating step.
Official documentation checked for this revision. Confirm the exact model, runtime, and license versions when implementing.