Spec: F95 — Provider-Aware LLM Prompt Caching & Cost Telemetry

Status: partial — provider cache context, native usage normalization, and assistant trace aggregation exist; backend runner, live-provider, and boundary-reconciliation evidence remain open

1. Problem & CEO Job

F95 serves trust the signal at the system level. Without provider-aware cache metrics, repeated prompt prefixes raise cost and latency, savings are unprovable, and the team may build a false in-house cache instead of using supported provider mechanisms.

2. User Stories & Acceptance Criteria

• As a platform engineer, I want provider-aware cache routing and usage accounting, so that Lumi can reduce repeated LLM cost without inventing a misleading cache layer.

  • Given an OpenRouter-backed assistant turn, when a stable org/thread cache context is available, then the provider request includes a stable session_id suitable for sticky routing.
  • Given a DeepSeek response with prompt_cache_hit_tokens / prompt_cache_miss_tokens, when Lumi parses the response, then those values are captured as normalized cache metrics.
  • Given an OpenRouter response with usage.prompt_tokens_details.cached_tokens / cache_write_tokens, when Lumi parses the response, then those values are captured as normalized cache metrics.
  • Given a Gemini response with usageMetadata.cachedContentTokenCount, when Lumi parses the response, then that value is captured as normalized cache metrics.
  • Given an assistant turn that makes multiple provider round-trips, when the turn completes, then ChatMessage.agent_trace_json records aggregate cache metrics for that turn.

• As a maintainer, I want prompt-cache work to stay honest, so that we do not overclaim a “cache system” that does not exist.

  • Given the first slice ships, when reviewing the implementation, then there is no Redis/global prompt cache or speculative cache invalidation framework; provider-native Gemini cache objects remain bounded to the explicit lifecycle adapter.
  • Given a provider returns no cache-related fields, when Lumi processes the response, then the request still succeeds and cache metrics remain zero/empty instead of erroring.

3. Scope Boundaries

• In:

  • provider-layer support for an optional stable cache context
  • OpenRouter session_id routing for cache-friendly sticky provider selection
  • normalized cache-usage extraction from provider responses
  • Gemini cachedContent create/reuse lifecycle through the provider adapter
  • assistant-turn cache telemetry persisted in agent_trace_json
  • prompt-prefix discipline as a written contract for Lumi prompts

• Out:

  • Redis / database prompt caching owned by Lumi
  • generic cache invalidation infrastructure
  • Anthropic-specific cache_control implementation (not currently a configured Lumi provider)
  • local inference-server caching
  • user-facing UI for cache statistics

4. Data Model

No new tables.

Existing model touched:

• backend/apps/chat/models.py → ChatMessage.agent_trace_json

New trace payload section:

• cache_metrics.cached_input_tokens
• cache_metrics.cache_write_input_tokens
• cache_metrics.cache_miss_input_tokens
• cache_metrics.session_id

This is additive JSON only. No migration.

5. API Contract

No external Lumi API changes in the first slice.

Internal provider contract additions:

Tenant scoping:

• cache context keys derive from tenant-scoped ids already in memory (organization_id, optional thread_id, optional member_id)
• no cross-tenant shared cache key is introduced

AI evidence/confidence:

• unchanged; this feature does not change extraction or answer confidence bands

6. UI / UX

• No new route or page in the first slice.
• No new user-visible states.
• i18n: n/a
• Assistant path (F91 doctrine): n/a with reason — this is provider/infra work with no direct intake surface. Its assistant-facing value is indirect: cheaper and more stable repeated assistant turns.

7. Test Plan

8. Current Behavior

The repository contains provider cache context/session routing, DeepSeek/OpenRouter/Gemini usage normalization, assistant-trace cache aggregation, and a bounded Gemini cachedContents create/reuse adapter. F95 does not claim live provider effectiveness or a Lumi-owned global cache.

9. Error States

• Provider returns no cache-related usage fields:

  • User-visible message: none
  • Internal handling: treat metrics as zero
  • Logging: none required
  • Retry: n/a

• Cache context missing:

  • User-visible message: none
  • Internal handling: request proceeds without session routing
  • Logging: none required
  • Retry: n/a

• Provider rejects request for unrelated reasons (429 / 5xx / malformed payload):

  • unchanged existing provider error handling applies

10. Diagrams & Docs Touched

• docs/MVP_SCOPE.md
• this spec
• if the assistant trace payload shape is treated as a contract surface, update F91 changelog when behavior ships

11. Implementation Audit (2026-07-07)

Before this feature, the codebase already had one thing right and three things wrong:

• Right: Lumi prompt builders already place stable system/product rules ahead of the dynamic question, which is the prerequisite for any provider-side prefix cache to help.
• Wrong: provider calls had no stable routing/session concept, so OpenRouter could not be nudged toward cache-friendly sticky routing.
• Wrong: provider usage accounting was mostly discarded, so the team could not measure cache benefit.
• Wrong: the repo had no explicit product contract preventing an overbuilt fake cache layer.

First-slice decision:

• ship routing + telemetry plus the bounded Gemini provider adapter
• keep broader cache corpus selection and live effectiveness measurement out of repository proof
• do not build any Lumi-owned prompt cache in v1

12. Open Questions / Decisions

1. Should explicit Gemini cachedContent objects be limited to one or two high-reuse corpora (for example, company policy or large imported dossiers), or remain deferred until a real workload proves the need?
2. If Anthropic becomes a first-class Lumi provider, should cache_control support live in the shared OpenAI-compatible base or a provider-specific subclass?
3. Do we want cache metrics only in agent_trace_json, or also in structured logs / HQ observability?

13. Bug / Change Log

2026-07-07 — Scope change approved from feedback issue #42

Type: Feature

Summary:

• Registered provider-aware prompt caching / cost telemetry as a first-class Lumi feature instead of leaving it as an untracked feedback idea.
• Chosen first slice: provider cache context, OpenRouter sticky session routing, provider cache-usage parsing, and assistant trace telemetry.
• Explicitly rejected a fake global prompt cache for v1.

Reason:

• The repo had no source-of-truth feature entry for this work, so implementation would have violated the scope contract.

Impact:

• Backend/provider-layer work only in the first slice.

Tests:

• Provider tests + assistant trace test added in the implementing PR.

Notes:

• “Prompt caching” in Lumi means using provider-native mechanisms and measuring them, not inventing a home-grown cache system unless the provider capabilities prove insufficient.