Interactive guide / Capacity planning
Plan memory, throughput, and resilience before buying hardware. Estimate an explicit attention-cache budget, then validate the exact model and serving configuration against your workload and service targets.
Reviewed · Vendor-neutral guidance
Defaults are an illustrative 8B configuration, not a named model profile. Replace every assumption with your model and serving configuration. Inputs stay in this page.
01 / Decision framework
This calculator models conventional decoder-only full-attention MHA/GQA inference. It uses GiB (2³⁰ bytes); vendor labels and other tools may report decimal GB.
Estimate weights from total resident parameters and effective bytes per parameter, including quantization metadata. For mixture-of-experts models, active parameters per token are not the total resident weight footprint. Use measured loaded weights where available.
For uniform full-attention layers: KV bytes = 2 × layers × KV heads × head dimension × cache bytes per element × cached tokens × active sequences. The factor of two accounts for keys and values. Count prompt and generated tokens; registered users are not simultaneous sequences.
Activations, CUDA graphs, temporary buffers, allocator behavior, adapters, and neighboring services consume memory. The illustrative reserve is configurable. Replace assumptions with peak measurements from the exact runtime and workload.
Hybrid/state-space models, latent attention, mixed sliding-window layers, multimodal encoders, training, and fine-tuning require different accounting. Prefix sharing and cache offload also change the budget. Do not apply this formula blindly to models such as Qwen3.5.
02 / Decision framework
Memory fit answers whether an experiment is plausible. Throughput, latency, resilience, and total operating cost decide whether it is useful.
| Deployment | What to validate | Common planning error |
|---|---|---|
| CPU or unified memory | Available system memory, bandwidth, thermals, and acceptable token latency. | Assuming a model that loads will deliver interactive performance. |
| One accelerator | Usable device memory, supported kernels, peak cache demand, and p95 latency. | Choosing a GPU from parameter count alone. |
| Several accelerators | Per-device allocation, tensor/pipeline parallel support, interconnect, and communication overhead. | Adding advertised VRAM as though all devices form one uniform pool. |
| Replicated service | Queue behavior, load balancing, failover capacity, and one replica unavailable. | Spending all capacity on normal peak load with no maintenance reserve. |
| Disconnected deployment | Offline artifacts, drivers, licenses, package mirrors, monitoring, and recovery media. | Budgeting GPUs while omitting the surrounding operating platform. |
03 / Decision framework
Use the worksheet to record assumptions, observed results, and the person who accepts the capacity decision.
Pin model revision, tokenizer, quantization, runtime image, drivers, and hardware. Record cache dtype separately from weight quantization.
Replay representative input/output lengths and arrival patterns. Include long requests, burst demand, mixed tenants, and worst credible cache occupancy.
Report warm and cold behavior, p50/p95 first-token and end-to-end latency, output tokens/second, errors, queue time, and peak memory. Goodput counts only work meeting the agreed quality and latency targets.
Run a sustained thermal test and a failure scenario. Include CPU, host RAM, storage, networking, power, spares, support, and growth in the bill of materials.
The calculator estimates an aggregate device-memory budget. It does not recommend a GPU SKU, certify a model/runtime combination, or predict throughput.
Working materials
Download the editable materials, adapt them to your environment, and keep the assumptions alongside the results.
Work with SovAIHub
Scope a sizing review before a procurement decision or a production expansion.
Bring candidate model revisions, input/output length distributions, simultaneous demand, latency targets, availability needs, and access to representative hardware.
Scope, acceptance criteria, delivery schedule, and fees are agreed before implementation.
Request a hardware sizing reviewConnect the decision to the next engineering or operating step.
Official documentation checked for this revision. Confirm the exact model, runtime, and license versions when implementing.