Evaluation and release gates
Turn acceptance criteria into repeatable promotion decisions with recorded evidence and rollback conditions.
Before you start
Module prerequisites: SAI-100. Check the exercise-specific tools and environment before running commands.
Module outcomes: Create model acceptance criteria; Record licensing and provenance evidence; Define model update and retirement controls. This lesson contributes to that sequence.
A release gate is a decision system
An AI release gate turns requirements into a repeatable allow, reject, hold, or exception decision. It applies to models, prompts, retrieval pipelines, policies, runtimes, tools, infrastructure, and complete workload releases.
A gate is more than a test suite. It defines the candidate, acceptance criteria, evidence, decision authority, exception route, promotion destination, and rollback conditions.
Build the gate contract
For each gate, specify:
- Candidate identity: the immutable versions and configuration being evaluated.
- Approved purpose: workload, users, data classes, deployment boundary, and prohibited uses.
- Evaluation set: representative, boundary, adversarial, permission, failure, and regression cases.
- Measures: quality, safety, security, latency, capacity, reliability, cost, and policy outcomes that matter.
- Thresholds: pass, fail, warning, and manual-review criteria with justified tolerances.
- Decision authority: who can approve, reject, or accept an exception.
- Evidence: inputs, environment, results, logs, reviewer notes, decision, and integrity information.
- Aftercare: monitoring signals, rollback trigger, expiry, and required reassessment events.
Avoid metric theatre
A high aggregate score can hide an unacceptable failure for a specific user group, permission case, language, content type, or high-impact task. Keep disaggregated results and inspect failure severity as well as frequency.
Do not reuse a pass outside its evaluated context. A new prompt, knowledge corpus, quantization, runtime, policy, tool, hardware profile, or deployment boundary can change behavior and may require partial or complete reassessment.
Exceptions
An exception should identify the failed criterion, rationale, compensating controls, owner, affected scope, expiry date, monitoring, and closure evidence. Exceptions must not silently redefine the acceptance threshold.
Release evidence
The final record should allow an independent reviewer to reproduce the decision logic: what was tested, under which conditions, what failed, who decided, what was promoted, and how it can be rolled back.
Module worked example
A newer model advertises higher coding scores but changes its license and chat template.
- Apply the control
- Review the exact revision and terms, then rerun representative acceptance cases with the intended runtime.
- Expected result
- The decision records quality, operating constraints, rejected alternatives and a fallback.
- Troubleshooting
- Check tokenizer, template, quantization and parser versions before attributing a regression to model quality.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…