blog/backend-engineering/idempotency-keys-safe-retries
Backend Engineering

Idempotency: Why POST Should Sometimes Act Like PUT

The client's network dropped right after they hit 'Pay.' Did the charge go through? They retry. Now you might charge them twice. Idempotency keys are how you let clients retry safely without double-processing — and they're more subtle than they look.

·8 min read

The Double-Charge Problem

A customer taps "Pay." Their phone sends the request, your server charges the card, and then — right before the response comes back — their connection drops. The client has no idea whether the charge succeeded. So it does the reasonable thing: it retries.

Your server sees a second, identical request. Nothing tells it this is a retry. It charges the card again.

This isn't a rare edge case. Mobile networks flake, load balancers time out, and clients are built to retry on failure. Any endpoint that changes state and can be retried has this problem. The fix is a property called idempotency, and getting it right is one of those things that separates an API that survives production from one that generates angry support tickets.


What Idempotency Actually Means

An operation is idempotent if running it multiple times has the same effect as running it once. Not the same response necessarily — the same effect on the system.

📌 Idempotent vs Safe

A safe operation doesn't change state at all — GET is safe. An idempotent operation may change state, but doing it twice lands you in the same place as doing it once. PUT user.email = "x" is idempotent: set it once or five times, the email is "x" either way. POST /charges is naturally not idempotent — each call is meant to create a new charge.

The HTTP spec already assigns these properties:

MethodSafeIdempotentWhy
GETYesYesReads only, never mutates
PUTNoYesReplaces the resource with a full new value
DELETENoYesDeleting twice = still deleted
POSTNoNoEach call creates a new resource by default
PATCHNoNot guaranteedDepends: 'set x=5' is, 'increment x' isn't

Notice PATCH is the tricky one. PATCH balance: {set: 100} is idempotent. PATCH balance: {increment: 10} is not — run it twice and you've added 20. The method doesn't decide idempotency; the operation does.

The real challenge is POST. Creating a charge, placing an order, sending a message — these are inherently "do it once" operations, but the network can make them "do it twice." We need a way to make a non-idempotent operation behave idempotently when retried. That's the idempotency key.


The Idempotency Key

The client generates a unique token — a UUID — for each logical operation and sends it with the request, usually in a header:

POST /v1/charges
Idempotency-Key: 3f9a1c7e-8b2d-4e6f-a1b0-9c8d7e6f5a4b
Content-Type: application/json
 
{ "amount": 4999, "currency": "usd", "source": "card_xyz" }

The key stays the same across retries of the same logical action. The server uses it to recognize "I've seen this exact operation before" and return the original result instead of doing the work again.

Retry With an Idempotency Key
Client
sends key
POST + key
API
checks store
Key Store
seen it?
Charge
once only

On the first request, the key isn't in the store, so the server processes the charge and saves the result under that key. On a retry with the same key, the server finds the stored result and returns it — no second charge. The client can retry ten times; the card is charged once.


The Part Everyone Gets Wrong: The Race

The naive implementation looks like this:

// BROKEN under concurrency
const existing = await store.get(idempotencyKey);
if (existing) return existing.response;   // already processed
 
const result = await chargeCard(request);  // <-- gap!
await store.save(idempotencyKey, result);
return result;

Now imagine the client's retry arrives while the first request is still charging the card. Both requests check the store, both find nothing, both proceed to charge. The idempotency key did nothing — you double-charged anyway.

⚠️ Idempotency without locking is a false sense of safety

The whole point is to handle retries, and retries are exactly when two identical requests are most likely to be in flight at the same time. If your check-then-act isn't atomic, concurrent retries slip right through the gap between the check and the save.

The fix is to claim the key atomically before doing the work. Insert the key into the store in a "processing" state as the very first step, using a uniqueness constraint so only one request can win:

1

Claim the key

Atomically insert the key with status processing. The database's unique constraint guarantees exactly one request succeeds. If the insert fails because the key already exists, this is a duplicate.

2

Losers wait or replay

The request that lost the race either waits for the winner to finish, or — if the winner already stored a final response — returns that stored response immediately.

3

Winner does the work

Only the winning request charges the card. It then updates the key's record with the final response and status completed.

4

Later retries replay

Any future request with the same key finds status completed and returns the saved response. Fast, and never re-charges.

This turns "check then act" into "atomically claim, then act" — the race disappears because the store itself is the arbiter.


The Details That Bite You

Getting the happy path right isn't enough. Three things routinely go wrong.

Same key, different body

A client reuses an idempotency key but sends a different request body — maybe a bug, maybe a malicious replay. If you blindly return the cached response, you've silently ignored what they actually asked for. The right move: fingerprint the request body, store it with the key, and if a later request presents the same key with a different fingerprint, reject it with a 422.

🔴 A key binds to one exact request

An idempotency key doesn't mean 'skip if you've seen this key.' It means 'this key represents this exact operation.' Same key + different payload is a client error, not a cache hit. Fail loudly.

How long do you keep keys?

You can't store idempotency records forever — the table would grow without bound. But expire them too soon and a legitimately slow retry (a client that backed off for an hour) gets treated as a brand-new request. Most APIs keep keys for 24 hours to a few days. Pick a window longer than your worst-case retry, and communicate it.

What about failures?

If the first attempt failed — say the card was declined — should the retry replay the failure or try again? Generally: cache deterministic outcomes (a declined card will decline again) but allow retries past transient errors (a timeout talking to the payment processor). The nuance is whether the failure is a property of the request or a property of that one attempt.

Does This Endpoint Need an Idempotency Key?

Does the request change server state?


Where This Fits in the Bigger Picture

Idempotency isn't just an API nicety — it's load-bearing infrastructure for reliability. Every retry mechanism above your endpoint assumes it:

  • Message queues deliver at-least-once. Your consumers will see duplicate messages. Idempotent processing is the only thing that makes that safe.
  • Sagas and workflow engines retry steps on failure. Each step has to be idempotent or the retries corrupt state.
  • Client SDKs (Stripe, AWS, and most serious APIs) retry automatically with backoff. They can only do that safely because the server honors idempotency keys.

Idempotency is what makes "just retry it" a safe default instead of a way to double-bill your customers.


Key Takeaways

  • Idempotent means running an operation N times leaves the same effect as running it once — not necessarily the same response.
  • GET, PUT, and DELETE are idempotent by design; POST is not, and PATCH depends on the operation (set vs increment).
  • Make non-idempotent creates safe with an idempotency key: a client-generated token, constant across retries, that lets the server return the original result instead of redoing the work.
  • The hard part is concurrency: claim the key atomically first, then do the work. A plain check-then-act races and double-processes exactly when retries pile up.
  • Bind each key to a request fingerprint, expire keys on a sensible window, and decide deliberately whether to replay failures.

References