Model monitoring, updates, and retirement
Detect material change, reassess fitness, control updates, preserve rollback, and retire safely.
Before you start
Module prerequisites: SAI-100. Check the exercise-specific tools and environment before running commands.
Module outcomes: Create model acceptance criteria; Record licensing and provenance evidence; Define model update and retirement controls. This lesson contributes to that sequence.
Monitor fitness, not personality
Monitor the properties connected to the approved purpose: workload success, supportedness, refusal and escalation, policy outcomes, latency and capacity, security findings, license or supplier changes, and user-reported harm. Protect sensitive prompts and responses while retaining enough structured evidence to investigate.
Update triggers
Reassess after new weights, tokenizer, quantization, prompt, runtime, hardware, retrieval, policy, deployment boundary, user population, data class, security finding, incident, or material drift. Use the controlled-change model to determine evaluation depth.
Compare the candidate with the currently approved version using the same baseline plus new regression cases. Record gains, regressions, migration needs, exception decisions, and rollback conditions.
Retirement
Define an end-of-use date, replacement or shutdown plan, owner communication, deployment discovery, approval revocation, credential changes, artifact retention, evidence retention, and deletion obligations. Verify that routing aliases, caches, batch jobs, and hidden endpoints cannot still invoke the retired model.
A model may remain retained for investigation or reproducibility while being prohibited from active use. Make those states distinct.
Module worked example
A newer model advertises higher coding scores but changes its license and chat template.
- Apply the control
- Review the exact revision and terms, then rerun representative acceptance cases with the intended runtime.
- Expected result
- The decision records quality, operating constraints, rejected alternatives and a fallback.
- Troubleshooting
- Check tokenizer, template, quantization and parser versions before attributing a regression to model quality.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…