Capacity, reliability, and recovery
Define capacity envelopes, degradation behavior, service objectives, recovery targets, and tested restoration evidence.
Before you start
Module prerequisites: SAI-210. Check the exercise-specific tools and environment before running commands.
Module outcomes: Match runtimes and hardware to workloads; Design governed model endpoints and routing; Plan capacity, reliability, and failure behavior. This lesson contributes to that sequence.
Define the service envelope
AI capacity is shaped by model size, precision, context length, request mix, concurrency, batching, hardware, memory, retrieval, and tool dependencies. A single throughput number cannot describe every workload.
Create a measured envelope that states the model and runtime versions, hardware profile, request distribution, concurrency, latency percentiles, error behavior, power or cost basis, and test limitations. Re-measure after material change.
Service objectives
Choose indicators that reflect user outcomes and control behavior. Examples include successful supported answers, permitted retrieval success, time to first useful output, completion latency, tool-action success, policy-decision availability, and evidence completeness.
For each objective, define the population, measurement window, exclusions, target, owner, and response when the error budget is consumed. Do not label unsafe or unauthorized output as successful simply because the endpoint returned a response.
Design degradation
Plan behavior for accelerator exhaustion, model unavailability, retrieval failure, identity or policy outage, evidence-store failure, excessive context, and dependent-tool failure. Options include queueing, reduced concurrency, an approved smaller model, retrieval-only results, cached approved content, no-answer, or service suspension.
Every fallback changes risk. It requires its own approval and evaluation rather than automatic routing to any available system.
Recovery
Define recovery time and recovery point targets for configuration, model and artifact stores, knowledge indexes, source data, evidence, secrets, and operational state. Distinguish rebuildable derived data from authoritative records.
Test restore and rebuild procedures. A backup job marked successful is not proof that a compatible, authorized system can be restored.
Recovery evidence
Retain the scenario, starting state, versions, backup identity, restore steps, duration, validation results, data loss, exceptions, and approval. Feed failures into the controlled-change backlog and repeat the exercise after material architecture changes.
Module worked example
A model loads successfully but requests fail when several users submit long prompts.
- Apply the control
- Measure peak cache demand, active sequences, queueing and tail latency on the declared hardware.
- Expected result
- The serving decision limits concurrency and context to an observed workload envelope.
- Troubleshooting
- Loading weights is not a capacity test. Separate weight memory, cache, runtime overhead and reserve.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…