Prompt caching for repeated LLM workloads
Reduce repeated-prefix inference cost with cached input token accounting on Lumino's OpenAI-compatible model APIs.
Who this is for: AI tutoring apps, legal AI tools, RAG and document platforms, grant-writing workflows, support bots, and agent builders with repeated context.
The problem: repeated context gets billed again and again
AI tutoring apps, legal document workflows, RAG systems, grant document tools, agents, and support bots often send the same large prefix on every request. Prompt caching helps when stable context is reused and the provider reports cached input tokens.
The benchmark: first call vs second call
In a repeated-prefix benchmark, 3,066 prompt tokens became 3,040 cached tokens on the repeat call and request cost dropped by about 76%.
| Run | Token accounting | Result |
|---|---|---|
| First call | 3,066 prompt tokens | Cold prefix billed as normal input |
| Repeat call | 3,040 cached tokens | Stable prefix reused as cached input |
Best-fit workloads
- AI tutoring with repeated rubrics, lesson context, and examples
- Legal document review with contracts, statutes, and clause-level questions
- RAG ingestion with repeated retrieved context and citation rules
- Grant documents with repeated guidelines and applicant context
- Agents with stable tool rules, repo summaries, and task history
- Support bots with repeated product docs and escalation policy
How to structure prompts
Put stable prefix content first: system instructions, documents, schemas, policies, rubrics, and agent rules. Append dynamic user input later. Avoid changing timestamps, random IDs, or reordered context near the beginning of the prompt.
Pricing visibility
Lumino usage accounting is designed to separate uncached input, cached input, and output tokens when provider usage includes cached-token reads.
See cached vs uncached tokens in usage, then optimize the prompt instead of guessing where the bill came from.
{
"usage": {
"prompt_tokens": 3066,
"completion_tokens": 184,
"prompt_tokens_details": {
"cached_tokens": 3040
}
}
}