Model evaluation, selection, and licensing
Compare model fitness, risk, licensing, operational constraints, and sovereignty consequences.
Before you start
Module prerequisites: SAI-100. Check the exercise-specific tools and environment before running commands.
Module outcomes: Create model acceptance criteria; Record licensing and provenance evidence; Define model update and retirement controls. This lesson contributes to that sequence.
Define acceptance before comparison
State the tasks, languages, context distribution, quality and safety expectations, privacy constraints, latency, capacity, hardware, operating boundary, portability, and evidence needs. Include prohibited uses and failure cases.
Evaluate candidates on the same representative set. Record model and tokenizer identity, configuration, prompt or harness, runtime, hardware, dataset version, results, variance, and limitations. Aggregate scores should not hide high-impact or subgroup failures.
License and origin
Record the license text and version obtained with the artifact, source location, publisher claims, usage and redistribution conditions, required notices, restrictions, dependencies, and internal legal decision. Do not infer permission from the ability to download weights.
Selection record
Compare at least:
- Workload fitness and known failure patterns.
- Security, safety, privacy, and misuse exposure.
- Provenance and supplier transparency.
- Runtime and hardware feasibility.
- License and policy compatibility.
- Operational support, update path, portability, and exit.
- Evaluation and monitoring effort.
Record why the selected candidate is acceptable, where it is not, compensating controls, review expiry, and triggers for reconsideration.
Refresh candidates without changing the acceptance bar
Use the model registry for source-linked candidates, including Qwen3-8B, Qwen3-Coder-30B-A3B-Instruct and the hybrid Qwen3.5-9B. Compare exact artifacts on the same tasks and hardware; family names and public benchmark scores cannot establish enterprise readiness.
Record model-card URL, revision, license, tokenizer, chat template, tool-call parser, runtime build, quantization, native context and any extension settings. Separate total MoE parameters used for weight memory from active parameters used per token. Hybrid and multimodal models need architecture-specific state and encoder budgets.
The basic coding lab retains qwen2.5-coder:7b as its instructional baseline. A replacement must pass the same preview, approval, file-safety and before/after tests; do not silently update the lab when the registry changes. Use the hardware sizing guide to plan a measured trial.
Module worked example
A newer model advertises higher coding scores but changes its license and chat template.
- Apply the control
- Review the exact revision and terms, then rerun representative acceptance cases with the intended runtime.
- Expected result
- The decision records quality, operating constraints, rejected alternatives and a fallback.
- Troubleshooting
- Check tokenizer, template, quantization and parser versions before attributing a regression to model quality.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…