Idempotency is not optional when money moves

Every webhook provider in the billing chain retries. Stripe retries failed deliveries on a backoff schedule. HubSpot retries. QuickBooks' event notifications retry. This is correct behaviour — at-least-once delivery is the only guarantee a distributed system can cheaply make — and it means your integration will receive the same event more than once.

If the handler creates something, you now have two of them. When the thing being created is an invoice, you've billed a customer twice.

This post is about what it actually takes to make that impossible, because "use an idempotency key" is advice that's true and insufficient.

Three layers, and most people only do one

Layer 1 — the request key. Stripe accepts an Idempotency-Key header; send the same key with the same request and Stripe returns the original result instead of doing the work twice. Keys are scoped to a time window (24 hours at time of writing — check the current docs), which matters: a retry that arrives after the window expires is a fresh request.

This layer protects you against your own retries of the same HTTP call. It does not protect you against your handler running twice with two different generated keys, which is what happens when the webhook itself is redelivered and your code generates a UUID per invocation.

The fix is to make the key derived, not random. Not uuid() — something like hash(event_id + operation), or deal_41208:create_subscription. Two invocations of the same logical operation must produce the same key, every time, including after a restart.

Layer 2 — event deduplication. Store every event id you've processed and check before handling. Stripe events have stable ids; so do HubSpot's. A table with a unique constraint on (source, event_id) and an insert-before-process pattern kills the majority of duplicate work at the door.

The subtlety: dedupe on the event, not the object. Two genuinely different events about the same subscription must both be processed. Deduping too aggressively is its own bug, and a more annoying one to find, because it looks like missing data rather than duplicated data.

Layer 3 — a natural key in the destination. This is the layer people skip, and it's the one that actually saves you.

Every write should be keyed on something the destination system can enforce or that you can query before writing:

• Stripe subscription: metadata.hubspot_deal_id — query for an existing subscription with that deal id before creating.
• QuickBooks invoice: DocNumber set to the Stripe invoice id — query by DocNumber first. QuickBooks won't enforce uniqueness on it for you, so the check is yours, but the field gives you something stable to check.
• QuickBooks customer: DisplayName is uniqueness-enforced by QuickBooks itself, which is why create-before-search fails loudly there rather than silently.

Why this layer matters more than the other two: layers 1 and 2 depend on your own state being intact. If your database is restored from a backup, if you re-run a backfill, if you migrate environments, your event table and your key derivation both lose. The natural key is in the destination, and it survives everything on your side.

Ordering is a separate problem, and it's worse

Idempotency stops you doing the same thing twice. It does nothing about doing two things in the wrong order.

A deal is created and amended forty seconds apart. Both events fire. If the amendment is processed first and the creation second, the subscription ends up reflecting the original amount and the correction is lost — and no error is raised anywhere, because both operations succeeded.

The mitigations, roughly in order of cost:

• Serialise per entity. All events touching deal 41208 go through a single queue partitioned on that id. Parallel across customers, serial within one.
• Version-check on write. Carry the source record's updatedAt or version, and refuse a write whose source version is older than the last one you applied.
• Re-read before write. Instead of trusting the event payload, treat the event as a signal and fetch the current state of the deal before acting. Slower, more API calls, dramatically more robust. For money-moving flows it's usually worth it.

The third option has a pleasant side effect: it makes replay safe. If you can rebuild the correct state from a fetch at any time, then reprocessing a backlog after an outage is just... running the flow again.

Replay, and why you need it before you think you do

Something will break for three days — an expired token, a vendor incident, a bug you shipped. When it's fixed, you have a backlog. Without replay you either process it by hand or you don't process it at all.

Replay only works if every layer above is in place. Reprocessing three days of events through a handler that isn't idempotent turns an outage into a billing incident. Which is why replay is the honest test of whether you got the rest right: if you're afraid to press the button, you didn't.

What to build, minimally

If you're doing this yourself, the shape that works:

1. Webhook endpoint acknowledges immediately (200) and enqueues. Never process inline — slow processing causes the retries that cause the duplicates.
2. Dedupe table with a unique constraint on the event id; insert first, process after.
3. Queue partitioned by entity id so events for one deal are serial.
4. Handler fetches current state from the source rather than trusting the payload.
5. Every write carries a derived idempotency key and checks the destination's natural key first.
6. Anything that fails after retries goes to a dead-letter queue with a human notification — never silently dropped, never retried forever.

That's six things, and none of them are exotic. They're just all required at once, which is why a Zap that does the happy path in two steps feels like it works right up until the week it doesn't.

TruelineHQ's writes are idempotent at all three layers, and every flow can be replayed. See how it works.