How it works
Inference needs the weights, the KV cache, and some working memory:
weights = parameters × bytes per parameter
KV cache = 2 × layers × KV heads × head dim × tokens × sequences × bytes per value
total = (weights + KV cache) × (1 + overhead)
The KV cache stores one key and one value vector per layer and per KV head for every token in the context. Models with grouped-query attention (most recent ones) have far fewer KV heads than attention heads. For Llama 3.1 8B that is 128 KiB per token at 16-bit, so an 8k context costs 1 GiB and 128k costs 16 GiB.
Full fine-tuning with mixed-precision Adam needs about 16 bytes per parameter: 16-bit weights and gradients (2 + 2), a 32-bit master copy of the weights (4) and two 32-bit Adam moments (4 + 4).
LoRA keeps the base weights frozen (at any precision; 4-bit is QLoRA) and trains small adapters, typically 0.1–2 % of the parameters, at 16 bytes each.
What the estimate leaves out
- Training activations, which grow with batch × sequence length and are often the largest term; gradient checkpointing trades them for compute.
- Framework-specific buffers. The inference overhead % stands in for activations, the CUDA context (~0.3–1 GB) and fragmentation; 10–20 % is a reasonable start.
- Paged or offloaded KV caches (vLLM, llama.cpp) can use less than the full-context worst case.
Presets
Layer counts, KV heads and head dims come from each model’s published config.json. Treat results as a planning estimate; measure on the real runtime before buying hardware.