Evaluation and release gates
Turn acceptance criteria into repeatable promotion decisions with recorded evidence and rollback conditions.
Before you start
Module prerequisites: SAI-100. Check the exercise-specific tools and environment before running commands.
Module outcomes: Design a private ingestion and retrieval flow; Implement permission-aware retrieval; Evaluate grounding, citations, and no-answer behavior. This lesson contributes to that sequence.
A release gate is a decision system
An AI release gate turns requirements into a repeatable allow, reject, hold, or exception decision. It applies to models, prompts, retrieval pipelines, policies, runtimes, tools, infrastructure, and complete workload releases.
A gate is more than a test suite. It defines the candidate, acceptance criteria, evidence, decision authority, exception route, promotion destination, and rollback conditions.
Build the gate contract
For each gate, specify:
- Candidate identity: the immutable versions and configuration being evaluated.
- Approved purpose: workload, users, data classes, deployment boundary, and prohibited uses.
- Evaluation set: representative, boundary, adversarial, permission, failure, and regression cases.
- Measures: quality, safety, security, latency, capacity, reliability, cost, and policy outcomes that matter.
- Thresholds: pass, fail, warning, and manual-review criteria with justified tolerances.
- Decision authority: who can approve, reject, or accept an exception.
- Evidence: inputs, environment, results, logs, reviewer notes, decision, and integrity information.
- Aftercare: monitoring signals, rollback trigger, expiry, and required reassessment events.
Avoid metric theatre
A high aggregate score can hide an unacceptable failure for a specific user group, permission case, language, content type, or high-impact task. Keep disaggregated results and inspect failure severity as well as frequency.
Do not reuse a pass outside its evaluated context. A new prompt, knowledge corpus, quantization, runtime, policy, tool, hardware profile, or deployment boundary can change behavior and may require partial or complete reassessment.
Exceptions
An exception should identify the failed criterion, rationale, compensating controls, owner, affected scope, expiry date, monitoring, and closure evidence. Exceptions must not silently redefine the acceptance threshold.
Release evidence
The final record should allow an independent reviewer to reproduce the decision logic: what was tested, under which conditions, what failed, who decided, what was promoted, and how it can be rolled back.
Module worked example
Two users ask the same finance question, but only one may read the source document.
- Apply the control
- Filter active documents by the caller's permissions before retrieval context is assembled.
- Expected result
- The permitted actor gets a supported citation; the denied actor receives no finance context.
- Troubleshooting
- Inspect retrieved document IDs and cache identity. Removing a citation after generation cannot undo disclosure.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…