Your webhook endpoint works in testing. Then a provider retries one delivery, your handler runs twice, and a customer is billed or provisioned twice. The first action is a two-minute check: open your handler and find the line that records the event ID. If it runs after the business write, or doesn't exist, that's the fix this post walks through. It's part of the SaaS Backend Infrastructure guide, and every webhook handler that touches money needs the same three layers: signature verification, idempotency keyed to the event ID, and a dead-letter queue with a replay path.
What breaks without webhook idempotency?
Without idempotency, the first retry produces a second charge, a second subscription or a second user account, and providers do retry.
Stripe's webhook documentation (checked 30 September 2026) says it attempts delivery for up to three days with exponential backoff in live mode, and retries sandbox events three times over a few hours. That gap is why the bug hides in development and shows up in production. The same page says webhook endpoints might occasionally receive the same event more than once, and that Stripe doesn't guarantee delivery in generation order: creating a subscription can produce customer.subscription.created, invoice.created, invoice.paid and charge.created in any sequence. Stripe's advice is to log the event IDs you've processed and skip repeats. It adds that in some cases two separate Event objects are generated, and to spot those you compare the data.object ID together with the event.type.
Why does check-then-act fail?
Consider a handler that SELECTs for an existing event_id, finds nothing, provisions the subscription, then INSERTs the event as processed. Two retries arriving within milliseconds of each other can both SELECT before either INSERTs. Both proceed, and two subscriptions get created. That's a hypothetical, but it follows directly from the ordering of the steps.
The fix is to make the claim atomic. A processed_webhooks table with a unique constraint on event_id lets the database arbitrate:
INSERT INTO processed_webhooks (event_id, received_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING
RETURNING id;
The PostgreSQL INSERT documentation (version 18, checked 30 September 2026) says ON CONFLICT DO NOTHING avoids inserting the row instead of raising an error, and that only rows actually inserted are returned. So if RETURNING yields no rows, another delivery already claimed the event and you stop. This is the webhook idempotency key pattern in its simplest form: the event ID is the key and the database constraint is the lock.
Does signature verification actually matter?
Yes, because without it anyone who knows your webhook URL can trigger your billing logic with fabricated payloads.
Stripe's documentation states the risk directly: without verification, an attacker could send fake events to fulfil orders, grant access or modify records. It recommends two protections together, IP allowlisting against its published address list and signature verification. The Stripe-Signature header carries a timestamp (t=) and one or more signatures, and the only valid live scheme is v1, an HMAC with SHA-256 over the timestamp, a dot and the raw JSON body, keyed with your endpoint secret. Stripe's libraries apply a default tolerance of five minutes between the timestamp and the current time, which blocks replayed payloads, and the docs warn that a tolerance of 0 disables the recency check entirely. Use the library helper (construct_event in Stripe's Ruby example) rather than hand-rolling it.
One subtlety costs people a day. Stripe requires the raw body for verification, and any manipulation of it makes verification fail. If your framework parses JSON globally before your handler runs, scope raw-body handling to the webhook route only. You can test locally with stripe listen --forward-to localhost:4242/webhook, and a body-parsing mismatch shows up on the first event.
How do you build webhook idempotency that doesn't race?
Claim the event ID atomically before touching business state, then run the business write and the claim in one database transaction so they commit together or not at all.
Ordering matters more than almost any other detail. Hookdeck's idempotency guide puts it plainly: mark events as processed before executing side effects, because if you send an email first and mark second, a crash between the two steps means the retry sends the email again. The sequence I'd use:
- Verify the signature on the raw body. Return 400 on failure.
- Enqueue the raw event to a durable queue and return 200 immediately.
- In the worker, claim the event with
INSERT ... ON CONFLICT DO NOTHING RETURNING. - If no row comes back, the event is already handled. Acknowledge the message and stop.
- In the same transaction, run the business operation.
- Commit.
Step 2 follows Stripe's guidance to return a 2xx quickly before complex logic and to process from an asynchronous queue, since a spike such as all subscriptions renewing at the start of a month can overwhelm synchronous handlers. Steps 5 and 6 avoid the second bug: committing the business change and the idempotency record separately. A crash between two commits leaves the business operation done and the event unrecorded, so the next retry repeats it.
How long should you keep processed event IDs?
Longer than any path by which an old event can come back. Stripe retries automatically for up to three days, and its docs say a manual resend from the Dashboard works for up to 15 days after the event was created, and from the CLI for up to 30 days. My recommendation is to keep IDs for at least 30 days plus a margin, and to delete older rows with a scheduled cleanup job. On Postgres, including the Supabase and Stripe setup, a created_at index on the table keeps that cleanup cheap and avoids adding Redis.
What about events that arrive out of order?
Don't derive state from the expected sequence. Stripe says not to depend on receiving events in a specific order, and not to use the event's created value to decide order or whether you've processed an event, because it's recorded in seconds and distinct events can share a timestamp. Instead, use the event as a trigger and fetch the current object from the API. For example, if invoice.paid arrives first, retrieve the invoice, charge and subscription with the IDs it carries. The Stripe integration glossary entry covers the overall billing pattern, and the Stripe and Supabase subscription pattern shows it in a Postgres schema.
What does a dead-letter queue need, and how should you retry?
A dead-letter queue needs four things: the original payload, the error with its stack trace, a retry count, and a replay path your on-call person can actually reach.
A DLQ is where an event goes after the worker exhausts its retry budget. The alternatives are silently dropping it or blocking the worker forever, and silently dropping a Stripe webhook for invoice.payment_succeeded isn't a decision you want made by accident.
| Property | Main queue | Dead-letter queue |
|---|---|---|
| Retention (example) | 4 days | 14 days |
| Auto-retry | Yes, with backoff | No, manual replay only |
| Alert trigger | Repeated failure | First entry |
| Stored context | Event payload | Payload, error, retry count, timestamps |
Those retention numbers aren't arbitrary. The Amazon SQS developer guide (AWS documentation, checked 2026) lists a default message retention of 4 days and a maximum of 1,209,600 seconds, which is 14 days. Its dead-letter queue page adds that a message's expiry is based on its original enqueue timestamp, so a message that spent a day in the source queue has one less day in a 4-day DLQ, and that you should set the DLQ retention longer than the source queue's. Hookdeck's dead-letter queue guide gives the same 14 versus 4 days as an example. Treat them as starting points for your own queue, not a standard.
What's a sensible webhook retry policy?
Backoff with jitter, a hard cap on attempts, then the DLQ. Backoff stops you hammering a recovering dependency, jitter stops every worker retrying on the same tick, and the cap stops a permanently broken event from looping. On SQS the cap is the maxReceiveCount in the redrive policy, and AWS notes that setting it too low, such as 1, moves a message to the DLQ after one failure. The provider side is separate: Stripe already retries deliveries to you, so your own retries apply to the work inside your worker.
How should replay and alerting work?
Replay in batches with pauses between them. Releasing 500 backed-up events at once after an outage creates a thundering herd against your own database, so watch DLQ depth fall instead of queue depth spike. For alerts, I'd page on the first DLQ entry for financial event types such as invoice.payment_succeeded and use a threshold for the rest. That's a recommendation, not a documented rule.
What should you monitor per provider?
Index your processed_webhooks table by event_id and event_type, then track failure rate by event type over 24 hours, DLQ inflow per hour, the age of the oldest unprocessed DLQ event (your real SLA clock) and the share of events processed within 30 seconds of receipt. Set separate thresholds for financial and data events, since a failed invoice.payment_succeeded isn't the same severity as a failed customer.updated. Callidus, the React, Firebase and Stripe Connect clinic SaaS I built, handles twenty Stripe webhook events end to end, including customer.subscription.trial_will_end and invoice.payment_action_required with 3D Secure escalation. With that many event types, knowing which one is failing matters more than knowing something failed. BookBed has its own Stripe webhook handlers and trial lifecycle wired, though subscriptions aren't yet activated for paid traffic. The telemetry side is covered in observability for SaaS.
Open your handler now and answer one question: what does it write first, the event ID or the side effect? If it's the side effect, move the claim in front of it and put both in one transaction.