Routing, capacity, and hardware
Design routing and hardware profiles from measured workload envelopes, isolation needs, and failure behavior.
Before you start
Module prerequisites: SAI-210. Check the exercise-specific tools and environment before running commands.
Module outcomes: Match runtimes and hardware to workloads; Design governed model endpoints and routing; Plan capacity, reliability, and failure behavior. This lesson contributes to that sequence.
Routing is a policy decision
A router may select a destination by approved workload, model capability, data classification, latency, region, tenant, health, capacity, or cost, and by fallback state. Record which inputs are authoritative for that decision, when a route is denied outright, and which destinations are allowed for each context — a router without an explicit deny path will eventually route somewhere nobody approved.
Do not silently route to an external or less-controlled endpoint when local capacity is exhausted. A fallback destination changes the request's boundary and needs independent approval and evaluation, not an automatic failover rule buried in configuration.
Capacity profile
Measure with a declared model, tokenizer, precision, runtime, hardware, context distribution, input and output lengths, concurrency, batching, and quality settings — a throughput number without these conditions cannot be reused for planning. Capture throughput, time to first output, completion latency percentiles, queue time, error rate, memory headroom, saturation, and stability over a sustained period, not just a short burst.
Hardware decision
Consider accelerator availability, memory, interconnect, CPU and memory support, storage load time, power and cooling, isolation, firmware and driver lifecycle, supply constraints, and recovery. Higher peak throughput may not improve the workload if queueing, context size, or memory pressure dominate actual behavior — buying faster hardware does not fix a design that is bottlenecked elsewhere.
Plan headroom for failures, maintenance, model loading, and demand variance, and link capacity assumptions directly to monitoring and admission control so a violated assumption is visible rather than discovered during an incident. See Capacity, reliability, and recovery for the shared reliability pattern this connects to.
Record every capacity assumption in the runtime architecture decision record so a future hardware or model change is evaluated against the same declared baseline rather than against an untested new set of assumptions.
Module worked example
A model loads successfully but requests fail when several users submit long prompts.
- Apply the control
- Measure peak cache demand, active sequences, queueing and tail latency on the declared hardware.
- Expected result
- The serving decision limits concurrency and context to an observed workload envelope.
- Troubleshooting
- Loading weights is not a capacity test. Separate weight memory, cache, runtime overhead and reserve.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…