Model lifecycle workshop
Evaluate an open-weight model candidate and produce a controlled lifecycle decision package.
Before you start
Module prerequisites: SAI-100. Check the exercise-specific tools and environment before running commands.
Module outcomes: Create model acceptance criteria; Record licensing and provenance evidence; Define model update and retirement controls. This lesson contributes to that sequence.
Scenario
Evaluate two candidate open-weight models for a synthetic internal summarization workload. No candidate is assumed acceptable.
Tasks
- Define intended purpose, prohibited uses, workload distribution, quality, safety, privacy, latency, capacity, and portability criteria.
- Create representative, boundary, adversarial, and failure cases before reviewing results.
- Record candidate provenance, license evidence, model and tokenizer identities, runtime, hardware, and test conditions.
- Compare results and document severe failures separately from averages.
- Select, reject, or hold each candidate with rationale and limitations.
- Build the approved package and registry record.
- Design the promotion gate, monitoring signals, rollback triggers, update comparison, and retirement process.
Deliverables
Submit the requirements, evaluation plan and results, license/provenance record, model decision record, package manifest, registry entry, approval, and lifecycle plan.
The review should be reproducible: another reviewer must be able to identify exactly what was evaluated and why the decision applied only to the stated context.
Module worked example
A newer model advertises higher coding scores but changes its license and chat template.
- Apply the control
- Review the exact revision and terms, then rerun representative acceptance cases with the intended runtime.
- Expected result
- The decision records quality, operating constraints, rejected alternatives and a fallback.
- Troubleshooting
- Check tokenizer, template, quantization and parser versions before attributing a regression to model quality.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…