Why Your GPU Utilization is 30% and You Are Paying 100%
You paid $10/hour for an A100. Your GPU utilization says 32%. Where did the other 68% go? It did not go anywhere — you are paying for it every single second
GPU Cost | 10 min read | 2026-04-24
You paid $10/hour for an A100. Your GPU utilization says 32%. Where did the other 68% go? It did not go anywhere — you are paying for it every single second while the GPU sits idle, waiting for data that never arrives. This is the most expensive waste in GPU rental, and almost nobody notices it until the bill comes.
The Lie That nvidia-smi Tells You
Most developers check GPU utilization with nvidia-smi and see a number like 85%. They think everything is fine. The problem is that nvidia-smi reports memory utilization and compute utilization as separate metrics, and the "GPU-Util" column only shows how busy the GPU is at that exact moment. A 2-second spike to 90% followed by 58 seconds at 5% still looks like "high utilization" if you only glance at it.
The real utilization number comes from averaging compute activity over the entire rental period. When teams do this properly — using nvprof, Nsight, or even a simple nvidia-smi dmon loop — the honest number is usually between 25% and 40%. The rest of the time, the GPU is waiting. Waiting for data to load from disk. Waiting for CPU preprocessing to finish. Waiting for the next batch because the batch size was too small to keep all SMs busy.
And the billing does not care. Whether your GPU is at 5% or 95%, the meter runs at the same rate. This is why a "cheap" GPU with poor utilization often costs more than a "expensive" GPU with good utilization. You are not paying for compute. You are paying for time.
The 4 Silent Utilization Killers
1. Data loading bottleneck
Your GPU can process 10,000 tokens per second. Your DataLoader can feed it 3,000. The GPU spends 70% of its time waiting for the next batch. This is the single most common utilization killer, and it happens because most training scripts use the default DataLoader settings — num_workers=0, no prefetching, no pinned memory.
The fix is not complicated. Set num_workers to at least 4 (or 2x your CPU core count, whichever is lower). Enable pin_memory=True for faster CPU-to-GPU transfers. Add prefetch_factor=2 to keep batches ready before the GPU asks for them. These three changes alone can push utilization from 30% to 60% on most training workloads.
Typical waste: 40-60% of GPU time spent idle waiting for data
Cost impact: On a $10/hour A100, that is $4-6/hour burned on nothing. Over a 24-hour training run, that is $96-144 wasted.
2. Batch size too small
GPUs are designed for massive parallelism. An A100 has 6,912 CUDA cores. If your batch size is 8, you are using maybe 200 of them. The rest sit idle. This is not a bug — it is a fundamental property of how GPUs work. Small batches mean small parallelism, which means low utilization regardless of how fast your data loader is.
The right batch size depends on your model and GPU memory. A good starting point: fill 70-80% of available VRAM with your batch. If your model uses 20GB of VRAM and you have an 80GB A100, you have 60GB free for batches. Experiment with batch sizes until you hit that 70-80% VRAM utilization mark. Then measure compute utilization — it should be above 60%.
Rule of thumb: If VRAM usage is below 50%, your batch size is too small and you are wasting money.
Why it hurts: Small batches not only waste GPU time — they also hurt training quality. Larger batches give more stable gradients and faster convergence.
3. CPU preprocessing bottleneck
Image augmentations, tokenization, text cleaning, feature engineering — all of this runs on the CPU. If your preprocessing pipeline takes 200ms per sample and your GPU processes a sample in 5ms, the GPU waits 195ms for every 5ms of actual work. That is 97% idle time, and no amount of GPU power will fix it.
The solution is to move preprocessing off the critical path. Use background workers to preprocess data ahead of time. Cache preprocessed samples to disk or memory. For image workloads, consider doing augmentations on the GPU itself (using libraries like Kornia or Albumentations with GPU backend). For text workloads, pre-tokenize your dataset and store it in an efficient format like Arrow or Parquet.
Diagnosis: Run your training script with num_workers=8 and watch GPU utilization. If it jumps from 30% to 70%, your CPU was the bottleneck all along.
Quick win: Pre-tokenize your dataset before training starts. This alone can cut preprocessing time by 80% for NLP workloads.
4. Architecture mismatch
Some models are simply not designed to use GPUs efficiently. Mixture-of-Experts (MoE) models activate only a fraction of their parameters per forward pass. A MoE model with 100B parameters might only use 10B per token, leaving 90% of the GPU's compute capacity idle. This is not a configuration problem — it is an architectural property of the model.
Similarly, models with heavy attention overhead (long context windows, many attention heads) spend more time on memory-bound operations than compute-bound ones. The GPU's tensor cores sit idle while memory bandwidth becomes the bottleneck. This is why a model that "fits" in VRAM can still have terrible utilization.
The hard truth: No amount of optimization will make an inherently memory-bound model use a compute-bound GPU efficiently. The solution is to choose the right GPU for the model, not the biggest GPU you can afford.
Example: An MoE model on an A100 might get 25% utilization. The same model on an RTX 4090 might get 45% — and cost 60% less per hour.
The Real Math of Wasted Utilization
Here is what happens when you compare the advertised GPU price against the actual cost per useful compute second. These numbers use real-world utilization measurements from production training workloads.
| GPU | Hourly rate | Actual utilization | Effective cost per useful hour | Wasted per 24h |
|---|---|---|---|---|
| A100 80GB (well-tuned) | $10.00 | 75% | $13.33 | $60/day |
| A100 80GB (default settings) | $10.00 | 32% | $31.25 | $163/day |
| RTX 4090 24GB (well-tuned) | $3.50 | 65% | $5.38 | $30/day |
| RTX 4090 24GB (default settings) | $3.50 | 28% | $12.50 | $72/day |
| H100 80GB (MoE model) | $25.00 | 22% | $113.64 | $468/day |
The H100 row is the most painful. At 22% utilization, you are paying $113.64 for every hour of actual compute work. An A100 at 75% utilization does the same work for $13.33 per useful hour. The H100 is 8.5x more expensive for the same actual compute, despite only being 2.5x more expensive on paper.
How to Measure Your Real Utilization
Before you can fix utilization, you need to measure it correctly. nvidia-smi gives you a snapshot. You need an average.
Quick utilization measurement script
# Run this alongside your training script
nvidia-smi dmon -d 1 -s u -o T > util.log &
# After training finishes, calculate average:
awk '{sum+=$4; n++} END {print "Avg GPU util:", sum/n "%"}' util.log
# Also check memory utilization:
awk '{sum+=$3; n++} END {print "Avg mem util:", sum/n "%"}' util.log
If average GPU utilization is below 50%, you have a utilization problem. If memory utilization is above 80% but GPU utilization is below 40%, you are memory-bound — your model needs a GPU with more memory bandwidth, not more compute.
5 Fixes That Actually Work
1. Optimize your DataLoader
This is the single highest-impact change for most training workloads. Set num_workers to at least 4, enable pin_memory=True, and add prefetch_factor=2. If your dataset is stored on a slow disk, copy it to a local SSD or tmpfs before training starts. For cloud storage (S3, GCS), use a caching layer like WebDataset or streaming DataLoader that prefetches data in the background.
Expected improvement: 30% → 55-65% utilization
Time to implement: 15 minutes
2. Use gradient accumulation for small batches
If your model is too large to fit a good batch size in VRAM, use gradient accumulation. Instead of batch_size=64, use batch_size=8 with accumulation_steps=8. This gives you the same effective batch size while keeping VRAM usage manageable. The GPU still processes small batches, but the gradient updates happen less frequently, reducing overhead and improving overall throughput.
Expected improvement: Better training stability + 10-15% utilization gain
Time to implement: 30 minutes
3. Enable mixed precision training
FP16 or BF16 training uses half the memory and often runs 1.5-2x faster on modern GPUs (A100, H100, RTX 4090). This means you can double your batch size without running out of VRAM, which directly improves utilization. PyTorch's autocast makes this a one-line change: wrap your forward pass in torch.autocast(device_type='cuda', dtype=torch.bfloat16).
Note: BF16 is preferred over FP16 for training stability. A100 and newer GPUs support it natively. If your GPU does not support BF16, use FP16 with a loss scaler.
Expected improvement: 40% → 70% utilization + 50% VRAM savings
Time to implement: 10 minutes (one-line change)
4. Profile before you guess
Most utilization problems are not what you think they are. You might blame the DataLoader when the real bottleneck is a slow tokenizer. You might increase batch size when the real issue is CPU-bound image augmentation. Use PyTorch Profiler or Nsight Systems to find the actual bottleneck before you spend hours optimizing the wrong thing.
Quick start: Add torch.profiler.profile() to your training loop, run for 100 steps, and open the trace in Chrome's about://tracing.
What to look for: Long gaps between GPU kernel launches = CPU bottleneck. Short kernels with low occupancy = batch size too small.
5. Right-size your GPU
The most effective utilization fix is choosing the right GPU for your workload. If your model uses 8GB of VRAM and you rent an 80GB A100, you will never get good utilization — the GPU is 10x larger than you need. An RTX 4090 (24GB) or even an RTX 4080 (16GB) will give you better utilization at a fraction of the cost.
Conversely, if your model needs 70GB of VRAM and you rent a 24GB RTX 4090, you will spend hours on workarounds (gradient checkpointing, offloading, sharding) that destroy utilization. Rent the GPU that matches your model size, not the one with the best headline specs.
Decision rule: Your model + batch should use 60-80% of the GPU's VRAM. Below 40% = downsize. Above 90% = upsize or optimize.
Cost impact: Right-sizing can cut your hourly cost by 40-70% while improving utilization by 20-40 percentage points.
When to Downsize vs When to Optimize
Not every utilization problem is worth fixing. Here is a practical decision tree:
Utilization decision framework
- Utilization below 30%: Your GPU is massively oversized. Downsize immediately. The cost savings will dwarf any optimization effort.
- Utilization 30-50%: You have a real problem. Start with DataLoader optimization and mixed precision. These two fixes take less than an hour and typically push you to 60-70%.
- Utilization 50-70%: This is acceptable for most workloads. Further optimization is possible but has diminishing returns. Focus on reducing total training time instead.
- Utilization above 70%: You are doing well. Do not waste time chasing the last 10%. Focus on model quality and convergence instead.
- Memory-bound (high mem util, low GPU util): Your model needs more memory bandwidth, not more compute. Consider GPUs with HBM (A100, H100) or reduce model size.
The Takeaway
GPU utilization is the most important metric you are probably not measuring. The hourly rate is a lie — the real cost is hourly rate divided by utilization. A $3.50/hour GPU at 65% utilization is cheaper than a $10/hour GPU at 32% utilization. Always measure before you rent, always profile before you optimize, and always right-size before you scale up.
The best GPU is not the biggest one. It is the one that keeps its SMs busy, its memory full, and its utilization above 60%. Everything else is just burning money while the GPU waits.
Stop guessing which GPU fits your workload.
Compare live GPUs with real utilization data, right-sized for your model. Pay per second, not per idle hour.
Find the right GPU