Evaluation and release gates
Turn acceptance criteria into repeatable promotion decisions with recorded evidence and rollback conditions.
Before you start
Module prerequisites: SAI-220, SAI-240, SAI-250. Check the exercise-specific tools and environment before running commands.
Module outcomes: Define an allowlisted tool registry; Design human approval and confinement boundaries; Produce verifiable run receipts for agent actions. This lesson contributes to that sequence.
A release gate is a decision system
An AI release gate turns requirements into a repeatable allow, reject, hold, or exception decision. It applies to models, prompts, retrieval pipelines, policies, runtimes, tools, infrastructure, and complete workload releases.
A gate is more than a test suite. It defines the candidate, acceptance criteria, evidence, decision authority, exception route, promotion destination, and rollback conditions.
Build the gate contract
For each gate, specify:
- Candidate identity: the immutable versions and configuration being evaluated.
- Approved purpose: workload, users, data classes, deployment boundary, and prohibited uses.
- Evaluation set: representative, boundary, adversarial, permission, failure, and regression cases.
- Measures: quality, safety, security, latency, capacity, reliability, cost, and policy outcomes that matter.
- Thresholds: pass, fail, warning, and manual-review criteria with justified tolerances.
- Decision authority: who can approve, reject, or accept an exception.
- Evidence: inputs, environment, results, logs, reviewer notes, decision, and integrity information.
- Aftercare: monitoring signals, rollback trigger, expiry, and required reassessment events.
Avoid metric theatre
A high aggregate score can hide an unacceptable failure for a specific user group, permission case, language, content type, or high-impact task. Keep disaggregated results and inspect failure severity as well as frequency.
Do not reuse a pass outside its evaluated context. A new prompt, knowledge corpus, quantization, runtime, policy, tool, hardware profile, or deployment boundary can change behavior and may require partial or complete reassessment.
Exceptions
An exception should identify the failed criterion, rationale, compensating controls, owner, affected scope, expiry date, monitoring, and closure evidence. Exceptions must not silently redefine the acceptance threshold.
Release evidence
The final record should allow an independent reviewer to reproduce the decision logic: what was tested, under which conditions, what failed, who decided, what was promoted, and how it can be rolled back.
Module worked example
A reviewer approves a proposal, then another process changes a selected source file.
- Apply the control
- Bind approval to the displayed diff and original source hashes, then recheck all targets before writing.
- Expected result
- The stale proposal is rejected without applying an earlier file in the same change set.
- Troubleshooting
- Distinguish rejection, application failure and uncertain partial state. Reconcile side effects before retrying.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…