Redis limits that survive a crash

09:14 — "I paid for this"

The ticket said: I'm on the paid plan and it tells me I have too many chats open. I have one tab open.

Joishi lets a paid account run three AI streams at once. Free accounts get one. The cap exists because a stream holds a model connection for as long as it takes to answer, and one enthusiastic user with a script can eat the capacity of forty. It's a boring limit. It had been running for months.

By 09:40 there were nine tickets. By 10:30 we could count 61 accounts that couldn't start a single stream, all of them telling the client the same thing: stream_limit_reached.

This is what actually happened, in the order I understood it, including the fifty-five minutes I spent looking in the wrong database.

08:47 — the deploy

A release went out at 08:47. Nothing dramatic — three containers rolled on EC2 behind nginx, the usual docker stop, ten-second grace, new image up. The changed code was in credit settlement: how many credits a stream costs, and when the reservation gets converted into a charge.

Remember that. It's the whole reason the next hour went badly.

09:41 — the wrong hypothesis

Credits and the stream cap look adjacent if you squint. Both are per-user. Both are checked before a stream starts. Both say no. And a deploy that touched credit settlement had gone out fifty minutes before the first ticket.

So my first theory was that reservations weren't being released — that we were reserving credits at stream start, failing to settle at stream end, and the user's available balance was pinned at zero. That story fit everything I knew at 09:41.

I went into Postgres and started counting. Reservations without a matching settlement row. Settlements with a null completion. Anything orphaned in the ledger since 08:47.

There was nothing. Every reservation had its settlement. The ledger balanced to the paisa. I checked it three ways because I didn't believe it, and by 10:35 I'd proved the thing I was sure about was fine.

The tell was in the error string the whole time. stream_limit_reached is not the credits error. The credits error is insufficient_credits. I'd read past it because I'd decided what the bug was before I read the code.

10:38 — the actual cause

The concurrency cap was a counter. Two lines, honestly:

// what it used to be
const used = await redis.incr(`joishi:streams:${userId}`);
if (used > limit) {
  await redis.decr(`joishi:streams:${userId}`);
  throw new StreamLimitError(userId, limit);
}

try {
  await runStream(req, res, userId);
} finally {
  await redis.decr(`joishi:streams:${userId}`);
}

Read it and it looks right. It passed review. I wrote it.

Then I ran this against production Redis:

redis-cli --scan --pattern 'joishi:streams:*' | \
  while read k; do echo "$(redis-cli get "$k") $k"; done | sort -rn | head

Forty-three keys sitting at 3. Nine at 2. A handful above 3, which shouldn't be reachable at all.

The finally block only runs if the process lives long enough to run it. docker stop sends SIGTERM and then SIGKILL ten seconds later. Our streams routinely run thirty to ninety seconds. Three containers went down at 08:47 with roughly nineteen streams in flight each, and every one of those decrements simply never happened.

A counter that only goes down when you're polite is a counter that goes up forever. Nothing in the system ever reconciles it against reality, because there is no reality to reconcile against — Redis doesn't know what a stream is, it knows what an integer is. The leaked count is indistinguishable from a real one.

Client aborts leaked too, just slower. A phone that goes into a tunnel mid-stream drops the socket; if the abort path doesn't reach the finally, that's another permanent +1. That drip had been running the whole time. The deploy just did in one second what backgrounded phones had been doing over weeks, which is why nobody had noticed it before.

10:52 — the mitigation

I deleted the keys by hand.

redis-cli --scan --pattern 'joishi:streams:*' | xargs -r redis-cli del

That's a one-liner typed into a production shell from a bastion by a person whose hands were not entirely steady, and it worked, and it should never have been the tool. A user whose stream was genuinely running at that moment got their slot released early. On that morning I did not care.

Tickets stopped within a minute or two of the delete. Then I went and rewrote the thing properly, and shipped it two days later.

the fix: expiry is a property of the slot

The counter's problem isn't the counting. It's that a count has no provenance. You can't ask an integer which streams it represents, so you can't ask whether they still exist.

So don't store a count. Store the slots, and give each one an expiry.

The shape that works is a sorted set per user, where the member is the stream id and the score is the wall-clock millisecond at which that slot stops being believable:

ZADD joishi:streams:{userId} <expiresAtMs> <streamId>

Acquiring a slot is then: drop everything whose expiry is in the past, count what's left, and add yourself if there's room.

-- acquire.lua
-- KEYS[1] = joishi:streams:{userId}
-- ARGV[1] = now (ms)   ARGV[2] = streamId
-- ARGV[3] = slot ttl (ms)   ARGV[4] = limit
redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', ARGV[1])
local used = redis.call('ZCARD', KEYS[1])
if used >= tonumber(ARGV[4]) then
  return { 0, used }
end
redis.call('ZADD', KEYS[1], ARGV[1] + ARGV[3], ARGV[2])
redis.call('PEXPIRE', KEYS[1], ARGV[3] * 2)
return { 1, used + 1 }

Slot TTL is 120 seconds. While a stream is alive it heartbeats every 20 seconds and pushes its own score forward:

await redis.zadd(key, 'XX', 'GT', String(Date.now() + SLOT_TTL_MS), streamId);

XX and GT are both load-bearing. XX means "only update a member that already exists" — without it, a heartbeat that races the release call resurrects a slot that was just given back, and you've reinvented the leak with extra steps. GT means "only move the score forward", which makes a late heartbeat arriving out of order harmless instead of a small time machine.

Release is ZREM. If release never happens — SIGKILL, tunnel, panic, power — the slot expires 120 seconds later and the next acquire prunes it. Worst case a user waits two minutes. On the morning of the incident, the worst case was forever.

why there is no reaper

The obvious alternative is a background job that walks Redis and cleans up abandoned slots. I've written that job before, in other systems, and I don't want it here.

A reaper is a second process with its own deploy, its own health check, its own alert, and its own ability to be the thing that's broken. When it dies quietly at 3am, the symptom is identical to the bug it was written to prevent, and now you have two suspects. It also needs a definition of "abandoned" that agrees with the one used on the read path, which means the same rule is written twice in two languages.

The TTL version puts cleanup on the path that already cares. Nobody has to notice the garbage for it to disappear, because the only code that could ever be confused by a stale slot is the code that removes it, one line earlier, in the same atomic call. Crash safety stops being a job somebody runs and becomes a property of the data.

Redis key expiry alone won't get you there, incidentally — you can't expire individual members of a sorted set, only the whole key. The score is the per-member TTL, and ZREMRANGEBYSCORE is what enforces it. The PEXPIRE on the key is just so an idle user's set eventually evaporates instead of sitting in memory for a year.

Lua, not MULTI

Check-then-act needs to be atomic or the limit isn't a limit. Three tabs opened at the same instant will all read used = 2 and all decide there's room.

Redis transactions don't solve this. MULTI/EXEC is a batch, not a transaction in the sense you want — you can't read ZCARD inside it and branch on the result, because nothing returns until EXEC. The workaround is WATCH plus an optimistic retry loop, and that loop is where the interesting bugs live: it's the code that only runs under contention, which is exactly the code you never exercise in dev.

Lua is one round trip, atomic by construction, eleven lines, and the branch reads like ordinary code. It runs on Redis's main thread, so it has to be short — ours is O(log N) with N capped at 3, which is nothing. On a clustered setup the script touches exactly one key derived from the user id, so it never straddles slots.

Load it once with SCRIPT LOAD, call it with EVALSHA, and keep a NOSCRIPT fallback for the case where Redis restarted and forgot your script. In ioredis, defineCommand handles that for you.

the limit is not the billing system

The strongest thing I took out of this: the concurrency cap and the credits ledger must never share a store, a code path, or an incident.

Credits are money. They live in Postgres, they're append-only, every change is a row somebody can point at six months later during a billing dispute. They must survive a Redis flush, a region failover, and me.

The concurrency cap is an abuse control. It's approximate, it's ephemeral, and its correct failure mode is to forget. Deleting the entire key space costs a few users an early slot release and costs the business nothing. Deleting the ledger ends the company.

They also want different answers when Redis is unreachable. The cap should fail open — if we can't count streams, let the stream run, because refusing paying users is worse than briefly not enforcing a soft cap. Credits must fail closed. Same outage, opposite decision. You cannot encode both in one system without one of them being wrong.

And there's the human cost, which is the one I actually felt: because the two were adjacent in my head, a stream-limit error read as a billing error, and I spent fifty-five minutes in the ledger while users sat locked out of something they'd paid for.

what I'd do differently

The acquire script returns used. That number should have been a gauge from the first day — per-user maximum, per-minute. A leaked slot is a floor that never returns to zero, which on a chart is the most obvious shape in the world. I'd have seen it at 08:50 instead of reading about it at 09:14.

The mitigation should be a button. "Release all slots for this account" belongs in the admin panel, scoped to one user, audited, available to support. Not a redis-cli pipeline typed under pressure.

And I'd write the TTL version first. The counter took an afternoon; the sorted-set version took a day. I saved most of a day in March and spent four hours of it in May, plus whatever the tickets cost us in goodwill.

The part I still don't have a good answer for is the heartbeat. Every live stream writes to Redis every 20 seconds, which at today's peak is about 45 writes a second of pure bookkeeping. That's free. I don't know what it costs at fifty times the traffic, and I don't like that the honest answer is "we'll find out."