nestonexStart a project Why do webhooks arrive more than once, and what does a handler have to do about it?
Because senders retry when they do not get a fast success, and a response lost in transit looks identical to one that never arrived. So delivery is at-least-once, order is not guaranteed, and a handler has to verify the signature, record the event, respond immediately, and process it separately and idempotently.
Less than people assume. A webhook is an HTTP request another system sends when something happens, instead of you asking repeatedly. That is the whole contract, and it is a good one — it removes polling latency and polling cost.
What it does not guarantee: exactly-once delivery, ordering, or timeliness. Senders retry on any non-success and on timeouts, and a success response lost on the way back is indistinguishable from a request that never arrived, so the sender retries a delivery you already processed.
Order is worse than most designs assume. A retried 'created' event can land after the 'updated' event that followed it, so a handler applying events in arrival order will overwrite newer state with older. Events usually carry a timestamp or sequence; use it rather than trusting arrival.
Verify first. Check the signature against the shared secret using a constant-time comparison, on the raw body before any parsing, and reject anything that fails. An unauthenticated webhook endpoint is a public API for creating events in your system.
Then record and respond. Write the event to a table or a queue, return 200 immediately, and do the work separately. Processing inline means a slow downstream dependency turns into a timeout, which turns into a retry, which turns into the same work running twice concurrently.
Then process idempotently. Store the sender's event id with a unique constraint and let the database reject the duplicate. That single constraint is the difference between a retry being harmless and a customer being charged twice.
And keep the raw payload. When something goes wrong three weeks later, the ability to replay exactly what arrived is the difference between diagnosing it and guessing.
The endpoint being down. Senders retry with backoff for a while and then give up, and how long varies by provider. Anything critical needs a reconciliation path — a periodic fetch that catches what was missed — because the webhook is an optimisation, not a source of truth.
Events for objects you have not seen. A webhook can reference a record created by a request whose response has not finished processing on your side. Handlers should tolerate an unknown reference by retrying rather than erroring permanently.
Secret rotation. Accepting both the old and new secret during a window is the only way to rotate without dropping deliveries, and it needs to be a deliberate procedure rather than an outage.
And volume spikes. A bulk operation upstream produces a burst downstream. An intake that queues absorbs it; an intake that processes inline falls over exactly when the data matters most.
If you are the sender, the obligations reverse. Sign every payload, include a stable event id and a timestamp, retry with exponential backoff, and document how long you retry for.
Send the event, not the state. A payload saying what happened, with an id the receiver can use to fetch current state, ages better than one embedding a full snapshot that may already be stale by the time it arrives.
And publish from the same transaction that made the change, or you will eventually announce something that was rolled back — which is the problem the transactional outbox exists to solve.
The three-sided food-delivery backend leans on this pattern. Diners, restaurants and riders operate on the same orders through one API described by a single OpenAPI spec, with rider payouts running through Stripe Connect and transfers gated on onboarding eligibility. Payment webhooks drive state that money depends on, which is exactly where at-least-once delivery stops being academic.
AgencyOS uses a transactional outbox with message queues for the same reason from the other side: the record and the event announcing it commit together, so nothing downstream ever hears about a change that did not happen.
Compute an HMAC of the raw request body with the shared secret and compare it to the header using a constant-time comparison. Use the raw bytes, before any JSON parsing — re-serialising changes whitespace and key order and will produce a different digest.
200 as soon as the event is safely recorded, without waiting for processing. Return a non-2xx only when you want a retry. Returning an error after the work has already succeeded causes the sender to redeliver something you have done.
Do not rely on receiving them in order. Use the timestamp or sequence number in the payload to decide whether an event is newer than your current state, and ignore it if it is not. Ordering guarantees, where a provider offers them, usually only hold per object rather than globally.
For latency and cost, yes. For correctness, no. Anything important needs a reconciliation job that periodically fetches state and repairs what was missed during an outage or a retry exhaustion.