Gemini context caching: the bug was a timestamp

the invoice didn't match the dashboard

In the last week of April, Joishi billed 41.6 million input tokens to Gemini. My own dashboard said the context cache was fine. Every cachedContents.create returning 200, TTLs refreshing on schedule, zero errors across seven days.

Both of those were accurate. The cache worked. Nobody was using it.

Hit rate: 3.1%.

We were paying to store context we then re-sent in full, every single request. That's the worst of both bills. It took me two days to find the cause and about forty minutes to fix, which is the usual ratio for this class of bug.

what's actually in the dossier

Joishi answers questions about a person's chart. Before the model sees a question, it needs the person: birth data, the computed panchanga for the day, planetary longitudes and house cusps, the dasha table, yoga flags, and whatever family members are linked to the account. We call that blob the dossier. It averages about 28,000 tokens once rendered, and it does not change between a user's first question and their fifth.

That's the textbook case for explicit caching. You hand Gemini the stable prefix once, get back a handle, and every subsequent request references the handle instead of resending 28,000 tokens. Cached input bills at a fraction of normal input. In exchange you pay storage per token-hour for as long as the handle lives.

The Redis side is a two-line idea:

import { Injectable } from '@nestjs/common';
import { GoogleGenAI } from '@google/genai';
import { Redis } from 'ioredis';

@Injectable()
export class ContextCacheService {
  constructor(
    private readonly ai: GoogleGenAI,
    private readonly redis: Redis,
  ) {}

  async acquire(userId: string, dossier: Dossier): Promise<CacheHandle | null> {
    const key = cacheKey(userId, dossier);
    const existing = await this.redis.get(key);
    if (existing) return { name: existing, source: 'hit' };

    const cached = await this.ai.caches.create({
      model: MODEL,
      config: {
        contents: renderDossier(dossier),
        ttl: `${TTL_SECONDS}s`,
        displayName: key,
      },
    });

    await this.redis.set(key, cached.name, 'EX', TTL_SECONDS - 30);
    return { name: cached.name, source: 'created' };
  }
}

The - 30 is deliberate. Redis should forget the handle slightly before Gemini does, so we never hand the model a name it has already dropped.

the numbers before

	before
cache hit rate	3.1%
mean billed input tokens / request	30,700
p95 time to first token	3.4s
model spend / 1,000 requests	$9.42
cache storage / 1,000 requests	$0.71
all-in / 1,000 requests	$10.13

Look at the storage line. Seventy-one cents per thousand requests to keep caches warm that were read roughly never. We were paying rent on apartments nobody moved into.

how i found it

I'd been assuming the miss was on the Gemini side — TTL expiry, regional eviction, something out of my control. It wasn't. The clue was that cacheKey() was producing a different key for the same user thirty seconds apart, with no writes to their data in between.

So I added a debug route behind an admin guard that dumped the exact bytes we were hashing, called it twice, and diffed the two dumps.

curl -s "$HOST/internal/ai/dossier-bytes?userId=$UID" > /tmp/a.json
sleep 30
curl -s "$HOST/internal/ai/dossier-bytes?userId=$UID" > /tmp/b.json
diff /tmp/a.json /tmp/b.json

Out of 214 KB of serialised dossier, twenty-four bytes differed.

- "generatedAt": "2026-04-28T09:14:02.117Z",
+ "generatedAt": "2026-04-28T09:14:32.406Z",

A timestamp. Somebody — me, eight months earlier — had stamped the dossier with its build time for debugging. It was never read by anything. It rode along in the payload we hashed and the payload we sent, and it guaranteed that no two requests ever produced the same cache key. The cache was healthy the whole time. It was a cache of exactly one-use entries.

The same diff surfaced a second offender once I looked properly: the set of active yogas was serialised with [...activeYogas], and that set was populated by rule evaluators running under Promise.all. Same members, different insertion order depending on which evaluator resolved first. Roughly one request in nine got a different ordering.

And a third, quieter one. Planetary longitudes came back from the ephemeris as raw doubles. Two of our workers run on different base images, and the twelfth decimal place of an arcsecond calculation didn't always agree between them. Same chart, different bytes, different key.

Three separate causes, one symptom. That's why it took two days.

the fix: one serialiser, one instant

Nothing goes into a cache key or a cached payload unless it passes through one function. That function sorts keys, fixes float precision, and refuses to serialise anything that reads a clock.

import { createHash } from 'node:crypto';

const FLOAT_PRECISION = 6;

/**
 * Two structurally equal dossiers must produce byte-identical output on any
 * machine, any Node version, any evaluation order. Everything here exists
 * because something violated that.
 */
export const canonicalize = (value: unknown): string => {
  if (value === null || value === undefined) return 'null';

  if (typeof value === 'number') {
    if (!Number.isFinite(value)) {
      throw new TypeError('non-finite number in cacheable payload');
    }
    return Number.isInteger(value) ? String(value) : value.toFixed(FLOAT_PRECISION);
  }

  if (typeof value === 'string' || typeof value === 'boolean') {
    return JSON.stringify(value);
  }

  if (value instanceof Date) {
    throw new TypeError('Date in cacheable payload — pass AnalysisInstant explicitly');
  }

  if (value instanceof Set) {
    return canonicalize([...value].map(canonicalize).sort());
  }

  if (Array.isArray(value)) {
    return `[${value.map(canonicalize).join(',')}]`;
  }

  const entries = Object.entries(value as Record<string, unknown>)
    .filter(([, v]) => v !== undefined)
    .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));

  return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${canonicalize(v)}`).join(',')}}`;
};

export const cacheKey = (userId: string, dossier: Dossier): string =>
  `ai:ctx:v3:${userId}:${createHash('sha256').update(canonicalize(dossier)).digest('hex').slice(0, 32)}`;

The Date branch is the important one. It doesn't sanitise the timestamp, it throws. If a future version of the dossier grows a clock read, the request fails in staging with a message that names the problem, instead of quietly costing us money in production for eight months. I'd rather have the outage.

The other half is the instant. Astrology is a function of time, so the dossier genuinely does depend on "now" — you can't just delete the clock. What you can do is stop reading it more than once.

export type AnalysisInstant = {
  readonly epochMs: number;
  readonly zone: string;
};

export const freezeInstant = (zone: string): AnalysisInstant => ({
  epochMs: Math.floor(Date.now() / INSTANT_GRANULARITY_MS) * INSTANT_GRANULARITY_MS,
  zone,
});

Resolved once, at the controller, and threaded down as an explicit parameter. No service below that line is allowed to call Date.now(). INSTANT_GRANULARITY_MS is fifteen minutes, which means the instant is stable across a conversation and everything derived from it hashes the same. Transits don't move enough in fifteen minutes for anyone to notice, and I checked that with an astrologer before picking the number rather than after.

the numbers after

	before	after
cache hit rate	3.1%	78.4%
mean billed input tokens / request	30,700	14,900
p95 time to first token	3.4s	1.9s
model spend / 1,000 requests	$9.42	$4.57
cache storage / 1,000 requests	$0.71	$0.34
all-in / 1,000 requests	$10.13	$4.91

Fifty-one percent off the all-in number. Not the 75% the pricing page implies, and I want to be clear about why, because I've seen people quote the headline discount internally and then get asked awkward questions at the end of the quarter.

The discount only applies to the cached prefix. Our requests carry roughly 3,400 tokens of question and conversation history that will never be cacheable, and they're billed at full rate. Then there's storage, which went down but didn't vanish. And 21.6% of requests still miss, by design.

why "inline" is not an error

The metric is ai_context_cache_total, labelled mode="hit" or mode="inline".

Inline means we sent the dossier in the request body instead of referencing a handle. When I first wired this up, inline incremented an error counter, because in my head a miss was a failure. That was wrong, and it cost me a Saturday morning of chasing a page that was reporting correct behaviour.

Inline is the right answer in at least four situations. The first request of a session, before any cache exists. Dossiers under the model's minimum cacheable size, which for a new user with no linked family members is genuinely common. TTL expiry in the middle of a long conversation. And the case where caches.create fails and we'd rather answer the question at full price than return a 500 to someone who asked what their week looks like.

So inline is a first-class path with its own code, its own tests, and its own label. The alert watches the ratio, not the absence:

- alert: AiContextCacheHitRateLow
  expr: |
    sum(rate(ai_context_cache_total{mode="hit"}[30m]))
      /
    sum(rate(ai_context_cache_total[30m]))
      < 0.5
  for: 45m
  labels:
    severity: ticket
  annotations:
    summary: "context cache hit rate under 50% for 45 minutes"
    runbook: "diff the canonical dossier bytes for one user before touching TTLs"

That runbook line is there so the next person doesn't spend two days where I spent two days.

the ttl mistake

While I was in there I also changed the TTL policy, and this is the part I'd do differently from the start.

Original design: 60-minute TTL, extended on every hit. It felt generous. What it actually did was pay an hour of storage for every user who asked one question and closed the app, which is most users. The extension logic made it worse — an engaged user's cache would live for hours across a session that had long since moved on.

Now it's 20 minutes, and we only extend on the second hit. One-question users cost us a third of what they used to. Sessions that are genuinely alive still get extended. The median session on Joishi is 2.6 questions, and the TTL should be shaped around the median, not the enthusiast.

I should have measured the session-length distribution before picking a TTL. I picked 60 because it sounded safe.

what i still can't measure

Storage cost per surface. Billing is per token-hour against the cache handle, and a handle is shared across every surface a user touches in a session. If someone opens panchanga, then chart, then milan, all three ride the same cached dossier. I can attribute model spend per surface cleanly. Storage I can only attribute per session, and I've stopped pretending otherwise in the dashboards.

Whether a hit and an inline produce the same answer. Same prefix, same model, same sampling config — they should. I have no mechanism that proves it. We spot-check about thirty pairs a month by hand, which is not a control.

Early eviction. Gemini can drop a cache before its TTL, and when it does, it looks identical to normal expiry from our side. That means my 20-minute number is tuned inside a band of uncertainty I can't narrow. It's probably fine. "Probably fine" is where the TTL work has been sitting since May.

And the counterfactual. The only way I know to measure what we'd be paying without caching is to turn caching off, which I've done exactly once, for an hour on a Sunday, and won't do again with real traffic. Everything above is measured against a week of production before the fix and a week after. That's the honest version of the comparison, and it's weaker than an A/B, because the traffic mix moved slightly between those weeks and I can't fully subtract it.