LLM Inference Cost Leaks: The GPU Is Not Always the Expensive Part

Cost leaks rarely announce themselves as one big bill

LLM inference spend usually leaks in small places: long answers, retries, regenerate clicks, oversized context, idle GPU minutes, cold model loads, and traffic routed to expensive hardware when a cheaper model would have worked.

A dashboard that only shows total spend hides the real decision: whether the workload belongs on a hosted API, a rented GPU, or a hybrid path.

The five leaks to measure first

Do not start with provider comparisons. Start with workload shape. A cheap GPU can be expensive if utilization is low. A cheap token price can be expensive if outputs are long or retries are common.

• Idle GPU time after a notebook, job, or demo stops being active.
• Output tokens growing faster than input tokens.
• Frontend retries, backend retries, and user regenerate clicks multiplying requests.
• Long context sent repeatedly instead of summarized or cached.
• Every request using the same model instead of routing by difficulty.

The hybrid rule

Use hosted APIs while demand is uncertain and traffic is bursty. Use rented GPUs when the workload is sustained, custom, or needs machine control. The mistake is forcing every product flow into one infrastructure path too early.

What to build into the product

Add request-level usage logging, per-feature spend tracking, output length caps, retry accounting, and a fallback route. These are not finance features; they are survival features for AI products.

References

• Workload-Router-Pool inference optimization paper
• PagedAttention and vLLM paper