Making Notification Delivery Recoverable

01What Was Failing

I found the problem while changing the production notification flow behind a customer-facing app. Discounts, app milestones, stock changes and order updates could produce an in-app inbox entry, a push notification, or both.

The path looked reasonable when you read it top to bottom: a scheduled fetcher discovered eligible notifications, rendered their content, saved the inbox rows, called Firebase, then wrote the delivery logs. The problem was hiding between two of those steps, saving the inbox entry and recording Firebase's result.

Discovery treated the inbox row as proof that a notification was already handled. So if the process died after saving the row but before Firebase answered, the next run would just skip it. The entry existed, but nothing in Postgres said whether Firebase still needed to be called. Nothing durable to retry.

“Put it on a queue” was the obvious answer, and only half an answer. BullMQ could move the work into a worker and retry it. But a rerun hit the same flow: discovery saw the inbox row and skipped the notification. The crash window hadn't closed, it had just moved into a worker.

02The Durable Handoff

Fetchers now turn notification triggers into PENDING Outgoing rows: rendered content, channel, delivery and retry state. I persist the rendered payload so old work stays sendable even if someone deletes the template in the meantime.

Postgres becomes the durable handoff between discovery and delivery. BullMQ is the alarm clock; Outgoing is the note on the desk explaining what still needs doing. If the alarm is late or a worker disappears, the note is still there.

Each sender records its verdict on the same Outgoing row. In the push path the inbox still comes before Firebase, but its unique outgoingId stops a reclaimed row from creating a second inbox entry. That closes one duplicate path. It doesn't make Firebase transactional; a unique constraint is not a distributed transaction.

03The Fixed Lease

Several workers can poll the same backlog. They claim rows in one Postgres statement: FOR UPDATE SKIP LOCKED keeps concurrent claimers out of each other’s batches, while claimedUntil gives each claim an expiry. The row remains PENDING; it becomes claimable again when the lease expires.

WITH batch AS (
  SELECT id FROM "Outgoing"
  WHERE status = 'PENDING'
    AND channel = ANY($1::"DeliveryChannel"[])
    AND ("claimedUntil" IS NULL OR "claimedUntil" < NOW())
  ORDER BY "createdAt"
  LIMIT $2
  FOR UPDATE SKIP LOCKED
)
UPDATE "Outgoing"
SET "claimedUntil" = NOW() + make_interval(mins => $3),
    "updatedAt" = NOW()
WHERE id IN (SELECT id FROM batch)
RETURNING *;

My first version had a PROCESSING status, a watchdog for stuck rows, and heartbeats so the watchdog wouldn't kill batches that were merely slow. At some point I was checking the thing that checks the thing that checks the workers, and had to admit the design was telling me something. So I sat down and thought: there has to be an easier way to do all this. I threw it all out for a fixed lease: the claim predicate itself is the recovery. Lease expired, row claimable. That's it.

Simpler, but not magical. A lease is a parking meter, not proof that the previous car has left: too short and a slow worker overlaps its replacement, too long and real crash recovery waits. I didn't add renewal or fencing tokens, so that trade-off stays out in the open.

04Retry Is a Verdict

Once claimed, each sender does its channel-specific delivery and writes the result back to Outgoing. The retry loop is the boring part. The push path is where it gets interesting: when Firebase fails, how much does that failure actually tell you?

A missing verdict is not a failed delivery. Marking a Firebase outage as a pile of confident-looking DEAD rows would be tidy and wrong. The reverse is just as bad: call a permanent rejection an infrastructure failure and the row orbits PENDING forever. Treating “no viable push target” as SENT is a product decision. The user already has the inbox copy.

05How Far It Scales

The next question was whether more NestJS instances actually bought throughput. So I ran two local tests: one shaped like a slow provider, one designed to pressure the database path.

With provider latency dominating, scaling came out close to linear and Postgres stayed almost idle. That's the shape you want in production: workers hide the provider's latency without the database becoming the bottleneck.

That test mostly measures how well the workers hide a slow provider. My first pressure run seemed to peak at eight instances and slow down at twelve. That was a real result and the wrong conclusion. Twelve watch-mode NestJS containers plus Postgres were fighting inside the same ten-core Docker allocation. I had found the edge of the MacBook, not the edge of the mechanism.

That's the knee on this machine. The claim-and-lease mechanism still held, but doubling the workers bought 7.6% more throughput while the shared database absorbed the extra load.

06So, Did It Work?

Mostly, yes. It fixed the stupid failure mode that started this whole thing: a notification could disappear between the database and Firebase, with no durable state left to retry. Now the work exists before a sender touches it. If BullMQ is late, a worker dies or Firebase falls over, the row is still there.

The tests showed that workers can share a backlog without grabbing each other's rows, and that adding instances actually moves more work. They say nothing about production capacity. An M2 on my desk is not a datacenter. But they did show the next limit: once the provider stops being slow, the cost moves to Postgres and the gains flatten out.

It still doesn't give us exactly-once delivery. Firebase can accept a push and the process can die before SENT is written, so an expired lease may trigger another provider call. The gap is real, but the user sees less of it than the raw call count suggests: the unique outgoingId keeps inbox creation idempotent, and Firebase collapse identifiers coalesce pending repeats. A device can still get both pushes if the first was already delivered. In the usual retry path, though, the duplicate stays mostly invisible. The safeguards narrow the gap. They don't turn Postgres and Firebase into one transaction.

That's the whole deal. BullMQ wakes the sender up; Postgres remembers what needs to happen, who claimed it, and what happened last time. SENT means we recorded a final result, not that Firebase delivered exactly once. I'll take that over a missing notification and a shrug.