Idempotency and retries: why payments get charged twice

The lost reply

A shopper's app asks your payment service to charge ₹500. Your service asks the bank to debit the account, the bank agrees, and your service sends the success reply back. Then the mobile network drops the connection and the reply never arrives.

From the app's side, a reply that never came looks exactly like a request that never arrived. It cannot tell "the payment failed" from "the payment worked and I did not hear about it". The natural reaction is to try again, and if your service treats the retry as a new payment, the shopper is charged twice.

Timeouts are not rare events in payments. Mobile connections drop, load balancers restart, and banks are sometimes slow to answer. At millions of payments a day, even a tiny fraction becomes a steady stream of double charges and angry calls.

What idempotent means

An operation is idempotent if doing it twice has the same effect as doing it once. Setting an address is idempotent: set it twice and it is still the same address. Charging a card is not: charge it twice and the customer pays twice.

You cannot make the network reliable, so you make the operation idempotent instead. The standard tool is an idempotency key: a unique identifier the client attaches to a payment, and sends again unchanged on every retry of that same payment. Your service remembers each key it has processed. When a key arrives that it has seen before, it does not charge again; it returns the result it recorded the first time.

One key per payment, not per attempt

The key must identify the shopper's intention to pay, so it is created once, before the first attempt, and reused for every retry. If the app generates a fresh key for each attempt, the retries look like new payments and the key protects nothing.

In practice the key is often a random UUID created when the shopper presses Pay, or a natural identifier such as an order number. The app must keep it somewhere that survives a crash or a restart, otherwise a retry after the app reopens arrives with a different key.

Getting the details right

The idea is simple. Most production bugs come from these details:

• Check and record in one step. "Look up the key, and if it is new, charge" has a gap: two copies of the same request arriving together can both see the key as new. Let the database enforce it with a unique constraint on the key, so exactly one insert succeeds.
• Handle the request that is still running. A retry can arrive while the first attempt is still waiting on the bank. Return a clear "in progress" answer and let the client ask again shortly, rather than starting a second charge.
• Reject a reused key with different contents. If key K1 first arrived for ₹500 and now arrives for ₹5,000, something is wrong. Store a fingerprint of the request with the key and refuse a mismatch instead of returning the old result.
• Decide which outcomes are final. A decline from the bank is a real answer and should be returned again on retry. An internal error before anything was sent to the bank is not; the retry should be allowed to try again.
• Keep keys long enough. Keys can expire, but only after every reasonable retry is over, including retries from an app that was closed and reopened the next day. Twenty-four hours is a common minimum.

A minimal table for the keys looks like this:

CREATE TABLE idempotency_keys (
  client_id    BIGINT      NOT NULL,
  idem_key     VARCHAR(64) NOT NULL,
  request_hash CHAR(64)    NOT NULL, -- request fingerprint
  status       VARCHAR(16) NOT NULL, -- in_progress or completed
  response     TEXT        NULL,     -- returned on retry
  created_at   TIMESTAMP   NOT NULL,
  PRIMARY KEY (client_id, idem_key)
);

The flow is then: insert the key as in_progress; if the insert fails because the key exists, return the stored result, an "in progress" answer, or a mismatch error. If it succeeds, make the charge and record the outcome against the key in the same transaction that records the payment.

The same problem, one step further

Your own service is safe now, but its call to the bank or card network has exactly the same problem. If that call times out, your service does not know whether the debit happened either.

The answer is the same idea, applied downstream. Send a stable reference with every request to the bank, and when a reply goes missing, ask about that reference before sending anything new. Payment networks are built around this: card messages carry unique references, an uncertain authorisation can be followed by a reversal, and UPI apps check the status of a transaction rather than starting it again.

Retry carefully

Idempotency makes retries safe. It does not make every retry a good idea.

• Retry only what can succeed next time: timeouts and "temporarily unavailable" errors. A declined card or an invalid request will fail the same way again.
• Back off. Wait longer after each attempt, with some randomness, so thousands of clients do not retry at the same instant and knock over a service that is just recovering.
• Retry in one layer only. If the app, the gateway and the payment service each retry three times, one slow bank call can become 27 requests. Pick the layer that owns retries and keep the others simple.
• Stop eventually. After a few attempts, tell the shopper the payment status is being confirmed, rather than looping forever.

Reconciliation is the backstop

Even with every safeguard, some payments will end in doubt: a debit the bank recorded that your system never heard about. Daily reconciliation compares your records with the bank's and the network's, then reverses or completes the mismatches. Idempotency keeps those cases rare; reconciliation makes sure none of them is permanent.

In an interview

"How do you stop a customer being charged twice?" is one of the most common payment system design questions. A strong answer covers the lost reply, the idempotency key and where it is created, the race between concurrent retries, the downstream call to the bank, and reconciliation as the final safety net.