Reader intent
Estimate GPU memory before running an LLM workload
The model file is not the full VRAM bill
A common mistake is to look at parameter count, multiply by quantization size, and assume that number is the GPU requirement. That gets you the approximate weight memory, but inference also needs KV cache, runtime buffers, attention workspace, tokenizer/runtime overhead, and sometimes extra memory for batching.
That is why a model can load successfully at short context and then fail when users send longer prompts or when concurrency increases.
Start with model weights
Weight memory is the easy part. A 7B parameter model at FP16 is roughly 14 GB before overhead. At 8-bit it is roughly 7 GB. At 4-bit it is roughly 3.5 GB. Real files can differ because of architecture, metadata, tensor packing, and quantization format.
- FP16 or BF16: about 2 bytes per parameter
- 8-bit quantization: about 1 byte per parameter
- 4-bit quantization: about 0.5 bytes per parameter
Then add KV cache
KV cache stores attention keys and values for tokens already seen by the model. It grows with context length and concurrent sequences. Long context can turn a comfortable 24 GB plan into a 48 GB or 80 GB plan.
For transformer-style models, the rough direction is simple: more layers, bigger hidden size, more context tokens, and more simultaneous sequences all increase KV memory.
Keep runtime overhead visible
Inference engines need memory for CUDA kernels, graphs, temporary buffers, fragmentation, and framework-level overhead. A 15% to 30% planning buffer is usually saner than renting a GPU that is exactly equal to the spreadsheet estimate.
When hosted APIs are the better path
If the workload is mostly chat, coding, reasoning, or product prototyping, a hosted model API can be cheaper and simpler than renting a full GPU. Dedicated GPU rental makes more sense when you need custom weights, full machine access, batch jobs, notebooks, ComfyUI, or a specific serving stack.