Where the Memory Actually Goes
The weights dominate: parameters times bytes-per-parameter, which is why quantization matters so much — the same 70B model is 141 GB at FP16 but about 42 GB at Q4_K_M. The KV cache comes second and is the part people forget: the model stores attention keys and values for every token in context, so memory grows with every message in a long chat. Its size depends on the model’s layer count and attention design — models with grouped-query attention (Llama 3.1, Qwen2.5) keep it small, which this calculator accounts for using each model’s real numbers. The rest is runtime overhead: CUDA buffers, compute scratch space, and your desktop’s own claim on the card.
Choosing a Quantization Honestly
Quantization trades precision for size. Q8 is effectively indistinguishable from full precision; Q5_K_M and Q4_K_M lose a little sharpness on hard reasoning but remain the community default because they halve — or better — the footprint. Below Q4 the degradation gets noticeable quickly. The classic dilemma is a bigger model at Q4 versus a smaller one at Q8: in most benchmarks and most people’s experience, the bigger model at Q4 wins, which is why the fit table is worth checking before settling for the smaller download.
If It Doesn’t Fit
You have real options short of a new GPU. Drop the context length — most tasks don’t need 32k, and halving context halves the KV cache. Step down one quantization level. Offload some layers to CPU RAM (llama.cpp and Ollama do this automatically), accepting slower generation for the layers that spill. Or run a KV cache in 8-bit, which many runtimes now support. The breakdown above tells you which lever moves the most for your configuration.
