Payments, retries and webhooks: where certainty ends
A payment system often has to make decisions with incomplete information.
A request can time out after the provider has already charged the customer. A webhook can arrive after another event that happened later. The difficult part is not making these failures impossible. It is designing the system so that uncertainty, duplication and reordering do not corrupt its state.
01 / A timeout is not a result
The backend knows that it did not receive a response. It does not know that the payment failed. The charge may not have happened, may still be processing, or may have succeeded exactly as requested while only the response disappeared.
TIMEOUT ≠ FAILURE
That distinction is operational, not philosophical. If the backend treats the timeout as proof of failure and sends a new payment request, the provider may see two independent instructions and create two charges. An ambiguous payment outcome has become a duplicate payment.
Payment idempotency gives the operation an identity that can survive multiple request attempts. A retry carrying the same idempotency key can ask for the result of the same logical operation instead of asking the provider to perform the side effect again.
It is not magic, and it is not a universal “exactly once” guarantee. The useful guarantee depends on where idempotency is enforced, how the key is scoped, how long it is remembered, and whether the retried payload actually represents the same operation. Reusing a key for a changed amount is not a safe retry; it is a conflict that should be detected.
INTERACTIVE EXPERIMENT 01 How to charge someone twice ↗︎02 / Delivery order is not event order
The same uncertainty returns in asynchronous form. A payment webhook is an observation delivered through a network, not a privileged view of the provider’s clock.
PROCESSING → SUCCEEDED → REFUNDED
REFUNDED → PROCESSING → SUCCEEDED
Webhooks tell us that an event was delivered now. They do not necessarily tell us that the event happened now. Network delays, webhook retries, queues and independent processing paths can make delivery order differ from causal or event order unless the provider explicitly guarantees otherwise.
EVENT ≠ DELIVERY
DELIVERY TIME ≠ EVENT TIME
A handler that blindly assigns every incoming event as the newest payment state can move its own record backwards. The provider may already consider a payment REFUNDED; a delayed payment.succeeded webhook can still leave the local row as SUCCEEDED. Every message was valid. The projection is wrong.
03 / Events are not state
An event says: “payment.succeeded happened.” State asks: “what is true about this payment now?”
Those are related questions, not interchangeable ones. A historical event can remain completely true while being stale as a representation of current payment state. Receiving a true fact therefore does not automatically tell a consumer what value should be stored right now.
EVENT ≠ CURRENT STATE
This is the bridge between retries and webhooks: the system is always deciding how much a partial observation proves. A missing response does not prove failure. A newly delivered event does not prove recency.
04 / The provider is usually the authority
Our backend does not directly observe provider reality. It observes messages about it. Those observations may be delayed, duplicated, reordered or temporarily missing, so local state can diverge even when every individual component is behaving as designed.
LOCAL STATE ≠ PROVIDER REALITY
Reconciliation is how we acknowledge that gap. When event history is ambiguous, processing has failed, or a local projection may be stale, fetching the current payment from the authoritative provider can be safer than reconstructing reality from arrival order alone.
That does not mean every webhook should trigger an API read. Real systems choose between version checks, transition validation, serialized processing, scheduled reconciliation and targeted fetches according to provider semantics, consistency requirements, rate limits, cost and the importance of the operation. The point is to identify an authority and design an explicit path back to it.
05 / Real systems are messier
The experiments on this site deliberately isolate one failure mode at a time. Production payment systems rarely have that luxury.
Duplicate webhook deliveries interact with webhook retries. Multiple workers race to process the same resource. Database transactions commit while synchronous API responses and asynchronous payment webhooks cross in flight. Reads can be stale. Authorization and capture may be separate operations. Refunds can be partial. Provider-specific state machines branch in ways a tidy linear diagram does not. Reconciliation jobs have their own retries, blind spots and alerting thresholds.
None of these complications changes the core model. They create more places where an observation can be incomplete and more boundaries where an unintended side effect can escape. The engineering work is to make identity, ordering, authority and state transitions explicit at those boundaries.
06 / What actually matters
Reliable payment systems are not built by assuming that requests arrive once, responses always return, and events stay ordered.
They are built by deciding what remains true when none of those assumptions hold.
TIMEOUT≠FAILURE
DELIVERY≠EVENT TIME
EVENT≠CURRENT STATE
RETRY≠NEW OPERATION
LOCAL STATE≠PROVIDER REALITY