Six idempotency rules for a notification service

the 1,140 emails

A purchase order got approved once. The approver got told about it 1,140 times.

This was a notification service I built on Node and Express for a construction and property development group in South Asia — one service, sitting beside a large internal ERP, responsible for every email, SMS, and in-app alert the ERP wanted to send. The ERP itself is somebody else's codebase. I owned the notifier: its architecture, its data model, its queue, and its pager.

Everything below is a rule I hold now. None of them are principles I read somewhere. Each one is the thing I wrote down after a specific bad afternoon, and I've kept them in roughly the order the afternoons happened.

1. the idempotency key belongs to the event, not to the sender

The 1,140 emails came from four characters of code:

const idempotencyKey = randomUUID();

That line lived inside the function that talks to SES. It looked defensive. It was decorative. A fresh UUID per attempt means every retry is a brand-new request as far as anything downstream can tell, so the dedupe layer underneath had nothing to compare against. SES was rate-limiting us that morning and BullMQ was doing exactly what I'd told it to.

The key has to be derived from the thing that happened, upstream of anything that can retry:

export const notificationKey = (e: DomainEvent, channel: Channel, recipientId: string) =>
  createHash('sha256')
    .update([e.type, e.entityId, e.occurredAt, channel, recipientId].join('|'))
    .digest('hex')
    .slice(0, 32);

occurredAt is the emitting system's timestamp, not Date.now(). That matters more than it looks. If our worker retries the same message it has to produce the same key; if the ERP genuinely re-emits "PO-4482 approved" an hour later, that's a different event and should get a different one. Only the sender can tell those apart.

I now treat a randomUUID() anywhere near a send path as a defect on sight.

2. every consumer gets handed the same message twice, and the dedupe window has to outlive the retry chain

At-least-once delivery is not a queue limitation you can configure away. It's the only honest guarantee a distributed queue can make, and it means duplicates are a normal operating condition, not an incident.

So we dedupe at the edge of the send. One Redis key, set-if-absent, before the provider call:

const key = `notify:sent:${notificationKey(event, channel, recipientId)}`;
const fresh = await redis.set(key, '1', 'EX', DEDUPE_TTL_S, 'NX');
if (fresh === null) {
  logger.info({ key, channel }, 'duplicate suppressed');
  return { status: 'duplicate' };
}

Then I got this wrong in a way that took a month to show up. DEDUPE_TTL_S was 300. Five minutes felt generous.

Our retry policy is six attempts with exponential backoff from 30 seconds, so the last attempt fires about sixteen minutes after the first. For a message that failed five times and succeeded on the sixth, the dedupe key had expired eleven minutes before the send that mattered — and any later duplicate of that event walked straight through as new. It only bit on the rare messages, which is why it took a month to surface and why I didn't believe the report.

The window has to outlive the whole retry chain, with room for a backed-up queue. Compute it, don't feel it:

const ATTEMPTS = 6;
const BASE_S = 30;
// gaps between attempts: 30 + 60 + 120 + 240 + 480
const RETRY_HORIZON_S = BASE_S * (2 ** (ATTEMPTS - 1) - 1); // 930s ≈ 16 min
const DEDUPE_TTL_S = 24 * 60 * 60; // horizon is minutes; a stuck queue is hours

Redis holds a few hundred thousand 32-byte keys for a day. Nobody has ever noticed the memory.

The other half of this rule: the Redis key is a fast path, not the record. The durable record is a Postgres row with a unique index on the key, written when the provider accepts. A Redis flush costs us a few duplicates. A flush of the only copy would erase six months of delivery history, which is the thing you produce when someone asks whether a contractor was ever told about a variation order.

3. retries need jitter, and the reason isn't politeness

SES throttled us one Sunday evening during a month-end batch. About 4,000 queued notifications failed inside the same three-second window.

Exponential backoff then scheduled all 4,000 to retry at the same instant. Thirty seconds later they arrived together, got throttled together, and rescheduled themselves for sixty seconds later. Together. Five times, in formation, and the graph looked like a heartbeat monitor.

Backoff spreads retries out in time. It does nothing to spread them apart from each other, and a burst of failures is precisely the case where they all share a schedule.

// full jitter — pick uniformly from [0, exponential window]
const backoff = (attempt: number) => {
  const window = Math.min(BASE_MS * 2 ** attempt, MAX_MS);
  return Math.floor(Math.random() * window);
};

await queue.add('deliver', payload, {
  attempts: 6,
  backoff: { type: 'custom' },
  removeOnComplete: 5_000,
  removeOnFail: false,
});

Full jitter, not "backoff plus a bit of noise" — a small random addition to a shared schedule still gives you a peak, just with soft edges. Our worst-case burst went from 4,000 requests in one second to about 40 a second across a hundred-second spread, which SES doesn't notice at all.

4. don't promise ordering, promise state

A user got "your approval was granted" before "your approval is requested". Both were correct, both were delivered, and they arrived in the order the two workers happened to finish — which the user reasonably read as the system being broken.

The request that came back was "make notifications arrive in order." Per-recipient serialization means a queue or a lock per recipient, so one slow send blocks everything else for that person. And then providers reorder anyway, mail clients sort by their own received-at, and the SMS aggregator has a queue I can't see. I can guarantee order up to the moment the message leaves the building, which is about 5% of the distance to the user's eyes.

So we don't sell it. Every notification carries the entity's version at the time it was generated, and the in-app inbox renders by version rather than arrival:

type Notification = {
  entityType: 'purchase_order';
  entityId: string;
  entityVersion: number; // monotonic per entity
  renderedAt: string;
};

Anything whose entityVersion is lower than one already shown collapses into the newer card instead of stacking under it. Email and SMS got their subject lines rewritten to be independently true — each message states the current state and the entity reference, so reading them out of order is confusing for a second rather than wrong. "PO-4482 is now approved" survives arriving first in a way "your request has been processed" does not.

A copywriting fix to a distributed systems problem, and it worked better than the distributed systems fix would have.

5. a dead letter nobody reads is a deleted message with a storage bill

I found 12,000 messages in the dead-letter queue. The oldest was 23 days old. I'd built the DLQ, configured it, tested it, and then never once opened it, because nothing ever asked me to. Every one of those was a notification somebody was supposed to receive and didn't.

Two changes. First, a dead letter has to be understandable by someone who isn't holding the codebase in their head — the original event, the rendered body as it would have gone out, every attempt with its provider response, and a plain-language reason:

await dlq.add('dead', {
  key,
  channel,
  recipient: { id, email, phone },
  event,
  rendered: { subject, body },
  attempts: attemptLog, // [{ at, error, providerStatus }]
  reason: classify(lastError), // 'hard_bounce' | 'template_error' | 'provider_5xx' | …
});

Second, it has to arrive where a human already looks. A cron at 08:30 posts a digest into the ops channel — total dead since yesterday, grouped by reason, top three examples, a replay link per group. It posts even when the count is zero, because a silent channel is indistinguishable from a broken cron. That digest caught fourteen hard bounces one morning: a directory sync had mangled one department's addresses.

6. templates fail at send time, so render before you commit to sending

Two ways this bit us, and the second one is my favourite bug of the whole project.

The dull one: a template referenced {{ approver.name }}, the approver had been deactivated, and the render threw — inside the send function, after the dedupe key had been set. The job failed, the retry hit the key, saw a duplicate, reported success. A notification that was never sent got recorded as delivered. Rendering now happens before the SET.

The good one: our SMS templates are bilingual. English fits 160 characters per segment in GSM-7. One Devanagari character anywhere in the template flips the whole message to UCS-2, where a segment is 70. A properly translated template quietly turned every alert into three segments — truncated on some handsets, triple-priced on all of them. The aggregator reported 100% delivery throughout, because from its side nothing had failed.

There's now a test that renders every template against a fixture recipient in every supported locale and asserts the segment count:

it.each(TEMPLATES)('%s fits one segment in every locale', (id) => {
  for (const locale of LOCALES) {
    const body = render(id, fixture, locale);
    expect(segments(body)).toBe(1);
  }
});

Localisation is not a presentation concern in a notification system. It changes the encoding, the length, the cost, and sometimes whether the message arrives at all.

the one I'm still arguing with myself about

All six assume the interesting question is "did we deliver it?". I think the better one is "did we decide not to?" — quiet hours, channel preferences, digest batching, every place where the correct behaviour is to send nothing.

We don't record those. When somebody says they never got an alert I can prove we sent it, or prove we tried. I can't prove we deliberately held it until morning because that's what they'd asked for, so every suppression looks like a bug to the person it happened to. The fix is obviously to write a row for every decision, sent or suppressed. I haven't worked out how to do that without the decision log outgrowing the delivery log tenfold, and I don't have a good answer yet.