API Idempotency Keys: Prevent Duplicate Requests Safely

API idempotency keys prevent duplicate requests from creating duplicate side effects when clients retry uncertain writes. They matter most on endpoints like POST /payments, POST /orders, POST /subscriptions, and any API call where a timeout can leave the client unsure whether the server already committed the operation.

The dangerous case is ordinary: the server creates the order, charges the card, or schedules the job, but the response is lost before the client receives it. Without an idempotency key, a retry can look like a new request. With a correct idempotency design, the retry returns the original result instead of repeating the write.

The key word is correct. A thin implementation that only stores "we saw this key" can still race, replay the wrong response, hide ambiguous failures, or let duplicates through after a short TTL.

What An API Idempotency Key Solves

An idempotency key is a client-generated value that identifies one logical write operation across retries.

The client is saying:

I may send this request more than once, but I mean one operation.

The server uses that key to decide whether an incoming request is new work, a retry of completed work, a retry while the first request is still running, or an invalid reuse of the same key with a different payload.

That is different from making every operation naturally idempotent. RFC 9110 defines HTTP idempotent methods in terms of the intended server effect of repeating the same request. PUT and DELETE can be idempotent by method semantics. POST usually is not. An idempotency key adds an application-level contract for write endpoints where the method itself does not make retries safe.

For API writes, the target guarantee is:

• one logical operation creates one side effect
• retries with the same key and same payload do not create another side effect
• completed retries replay the authoritative original result
• concurrent retries do not both execute the side effect
• reused keys with different payloads are rejected

This is why idempotency belongs near the API boundary, not as a best-effort cleanup job after the fact.

Why Duplicate API Requests Happen

Duplicate requests are not rare client bugs. They are normal distributed-system behavior.

Common causes include:

• a mobile client loses connectivity after the server commits
• the client times out before reading the response
• a load balancer retries after an ambiguous upstream failure
• a user double-clicks a submit button
• a background worker retries a failed API call
• a payment or order flow retries after a 502, 503, or connection reset

The server cannot infer intent from the second request alone. It needs a stable identifier that survives the retry.

Retries improve availability. Idempotency preserves correctness.

If you add retries without idempotency, temporary network failures can become duplicate orders, duplicate payments, duplicate subscriptions, or repeated downstream jobs.

The Request Lifecycle You Want

A safe idempotency flow for POST /payments usually looks like this:

1. Client sends Idempotency-Key.
2. Server validates the request shape.
3. Server calculates a canonical request fingerprint.
4. Server atomically reserves (scope, key).
5. First request processes the write.
6. Server stores the final status code and response body.
7. Completed retries replay the stored response.
8. Concurrent retries get a clear conflict or retry response.
9. Reused keys with a different fingerprint are rejected.
10. Old records expire only after the real retry window is over.

The atomic reservation is the correctness boundary.

If two requests can both pass a "does this key exist?" check before either one writes the key, the design is still unsafe.

The Race That Breaks Naive Implementations

This is the race to test before trusting the implementation:

Both requests did a read before either one reserved ownership. The idempotency table existed, but it did not protect the side effect.

The fix is not a faster lookup. The fix is a single atomic step where exactly one request wins the right to process the operation.

In SQL, that is usually a unique constraint plus INSERT ... ON CONFLICT DO NOTHING. In Redis, it is often SET NX with careful state transitions and expiration handling. The storage choice matters less than the invariant: only one worker may own a new (scope, key).

A Practical Idempotency Table

A useful SQL table stores more than the key:

CREATE TABLE api_idempotency_keys (
  id BIGSERIAL PRIMARY KEY,
  scope TEXT NOT NULL,
  idempotency_key TEXT NOT NULL,
  request_fingerprint TEXT NOT NULL,
  status TEXT NOT NULL,
  response_status INT,
  response_body JSONB,
  resource_type TEXT,
  resource_id TEXT,
  error_code TEXT,
  locked_until TIMESTAMPTZ,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at TIMESTAMPTZ NOT NULL,
  UNIQUE (scope, idempotency_key)
);

The important fields are:

For most APIs, scope should include the account, tenant, merchant, user, or integration boundary. A globally unique key is useful, but the real uniqueness contract is usually "unique for this actor and this endpoint family."

Atomic Reservation With SQL

The reservation should be one write guarded by a unique constraint:

INSERT INTO api_idempotency_keys (
  scope,
  idempotency_key,
  request_fingerprint,
  status,
  locked_until,
  expires_at
) VALUES (
  $1,
  $2,
  $3,
  'in_progress',
  now() + interval '30 seconds',
  now() + interval '24 hours'
)
ON CONFLICT (scope, idempotency_key) DO NOTHING
RETURNING id;

If the insert returns a row, this request owns the operation.

If it returns no row, fetch the existing record and decide what to do based on its fingerprint and status.

That follow-up read is safe because the ownership decision has already happened.

Handler Shape

A simplified payment handler might look like this:

async function createPayment(req, res) {
  const key = req.headers['idempotency-key']

  if (!key) {
    return problem(res, 400, 'Idempotency-Key is required for this endpoint')
  }

  const scope = `account:${req.auth.accountId}:payments`
  const fingerprint = fingerprintPaymentRequest(req.body)

  const reservation = await idempotency.tryReserve({
    scope,
    key,
    fingerprint,
    ttlHours: 24,
  })

  if (reservation.kind === 'payload_mismatch') {
    return problem(res, 409, 'Idempotency key reused with a different payload')
  }

  if (reservation.kind === 'completed') {
    res.statusCode = reservation.responseStatus
    return res.json(reservation.responseBody)
  }

  if (reservation.kind === 'in_progress') {
    res.setHeader('Retry-After', '1')
    return problem(res, 409, 'A request with this idempotency key is still running')
  }

  try {
    const payment = await payments.charge(req.body)
    const body = { id: payment.id, status: payment.status }

    await idempotency.markCompleted({
      scope,
      key,
      responseStatus: 201,
      responseBody: body,
      resourceType: 'payment',
      resourceId: payment.id,
    })

    res.statusCode = 201
    return res.json(body)
  } catch (error) {
    await idempotency.markAmbiguousOrFailed({
      scope,
      key,
      error,
    })

    throw error
  }
}

The exact status codes are a product/API decision. The important part is that every branch is explicit.

Do not let the second request "fall through" to normal business logic just because the first request has not finished yet.

Request Fingerprints Prevent Dangerous Reuse

An idempotency key should represent the same logical request. If a client reuses the same key for a different amount, different customer, or different target resource, replaying the old result is dangerous.

That is why the server needs a request fingerprint.

Do not hash the raw bytes blindly unless your API contract truly requires byte-for-byte equality. Many clients can send semantically equivalent JSON with different field order, omitted defaults, harmless metadata, or formatting differences.

Prefer a canonical fingerprint based on business-relevant fields:

function fingerprintPaymentRequest(body: PaymentRequest) {
  return stableHash({
    orderId: body.orderId,
    amount: body.amount,
    currency: body.currency,
    paymentMethodId: body.paymentMethodId,
  })
}

The fingerprint should be strict enough to catch accidental key reuse and stable enough that legitimate retries do not conflict.

Stripe's idempotent request documentation describes the same practical idea: save the first result for a key, compare later parameters, and reject mismatches instead of replaying an unrelated response.

What To Return For Each State

Idempotency needs a response policy, not just storage.

Payment-like systems should be conservative. If the side effect might already exist, do not delete the idempotency record and allow a fresh charge just because the first attempt ended in a timeout.

Preserve uncertainty explicitly.

Ambiguous Failures Are The Hard Part

Clean success is easy. Clean validation failure is usually easy. Ambiguous failure is where duplicate side effects slip in.

Imagine this sequence:

1. Your API reserves the idempotency key.
2. It calls a payment provider.
3. The provider accepts the charge.
4. Your service times out before receiving the provider response.
5. The client retries with the same key.

At that moment, the safe answer is not "try the charge again." The system does not yet know whether the first side effect happened.

Better options include:

• store an ambiguous state
• return a pending response to the client
• reconcile with the provider by provider request ID or metadata
• complete the idempotency record after reconciliation
• replay the final result on later retries

This is also where idempotency interacts with webhook retries, background jobs, and the transactional outbox pattern. The synchronous API boundary is only one part of the duplicate-processing story.

TTL Should Match Real Retry Behavior

A TTL is both a storage policy and an API contract.

If the TTL is too short, duplicates can leak through after the record expires. If it is too long, the table grows and accidental key reuse becomes more likely.

Useful starting points:

Stripe documents that keys can be removed after they are at least 24 hours old, and that a reused key after pruning is treated as a new request. Your API can choose a different window, but clients need to know it.

Publish the replay window. Do not make clients guess when a retry is still protected.

Idempotency Does Not Replace Transactions Or Locks

Idempotency protects a retry boundary. It does not automatically solve every race inside the operation.

If two different idempotency keys attempt to reserve the last inventory item, create conflicting state transitions, or update the same aggregate, you still need database constraints, transactions, locks, or version checks.

The relationship is:

• idempotency handles duplicate attempts of the same logical request
• transactions keep related database changes atomic
• uniqueness constraints prevent impossible duplicates
• optimistic or pessimistic locking coordinates competing operations
• outbox or queue patterns preserve downstream side effects

If the same operation touches shared rows under contention, pair this design with Optimistic vs Pessimistic Locking in SQL and the broader patterns in How to Prevent Race Conditions in Backend Systems.

Tests That Catch Real Bugs

Idempotency code often looks correct in review and fails under timing.

High-value tests include:

This is a good fit for API integration tests because the correctness boundary crosses HTTP handling, storage, and business logic. Unit tests can verify helpers, but the duplicate-request race only becomes meaningful when the reservation path uses the real storage mechanism. I covered that testing boundary in How to Write API Integration Tests.

Production Signals To Watch

Once idempotency is live, make it observable.

Track:

• missing key rate on endpoints that require one
• replay rate
• payload mismatch rate
• in_progress conflict rate
• age of unresolved in_progress records
• ambiguous outcome count
• expired-key retry count
• number of completed records without a resource ID

These metrics show whether clients are retrying more often, whether a new integration is reusing keys incorrectly, whether timeouts changed, or whether reconciliation is falling behind.

Idempotency should not be invisible glue. It is part of the API's correctness model.

Practical Checklist

Before enabling idempotency on a production write endpoint:

1. Require a key only where duplicate side effects matter.
2. Scope the key by account, tenant, user, or endpoint family.
3. Use an atomic reservation guarded by a unique constraint.
4. Store a canonical request fingerprint.
5. Store the authoritative response status and body.
6. Model in_progress, completed, and ambiguous outcomes explicitly.
7. Define the response policy for each state.
8. Choose a TTL that matches real retry behavior.
9. Publish the TTL and mismatch behavior in the API contract.
10. Add concurrency and ambiguous-failure integration tests.
11. Instrument replays, mismatches, conflicts, and unresolved records.

The Short Version

API idempotency keys make retries safe only when the server treats them as a correctness boundary.

The safe design is not "store a key somewhere." It is atomic reservation, scoped uniqueness, payload fingerprinting, explicit in-progress behavior, deterministic response replay, a realistic TTL, and a policy for ambiguous failures.

That combination turns duplicate API requests from a production incident into a predictable part of the contract clients can rely on.