Payment Retries Review

Retry failed payment calls so fewer checkouts fail

PR #482  ·  billing-app  ·  +214 −31  ·  payment-retries
BILL-214 (example ticket)

A customer is charged twice, because a retry repeats a charge that already went through. Or one slow provider makes every checkout wait half a minute.

What could go wrong. The reviewer writes this line before reading the logic. It is the ranking key for every card below. Without it, cards get ranked by what looks scary, not by what costs most.

I skipped style, naming and test coverage. I am not saying whether to merge. The questions below are yours to answer.

What this touches outside its own code

Never capped, never collapsed. Every point where the change reaches past its own code gets one row: an outside API, a setting, a function that other code calls. The "Shared code" row counts the callers, so the reach of the change is visible at a glance. Highlighted rows were not in the ticket, or rest on docs only.

KindWhatNew or changedWhereIn the approved design?Confirmed how?
Shared codehttpClient.post() now retries by default. 11 callers outside the diff: emails 4, outgoing webhooks 3, analytics 2, tax lookup 1, shipping quotes 1changedsrc/lib/httpClient.ts:22no. The ticket asks for payment calls onlyread the code. A repo-wide search for httpClient.post(, each hit read
External APIPayment provider POST /v1/charges, now up to 3 attempts per checkoutchangedsrc/payments/chargeClient.ts:15yes. BILL-214docs only. Not confirmed with a real call
Config and secretsHTTP_MAX_ATTEMPTS. The code uses 3 when it is unset, and it is unset in productionnewsrc/config.ts:12noread the code

4 of 12 files need you

Triage. Most of a pull request is routine. The skipped files are still listed by name, so the human can catch a wrong skip.

FileWhy you need to read it
src/lib/httpClient.tsIt adds retries for every caller, not just payments.
src/payments/chargeClient.tsIt decides which payment errors are tried again.
src/checkout/checkoutController.tsIt decides how long a customer waits for an answer.
src/config.tsIt picks the number of attempts when nobody sets one.
The other 8 files, and why I skipped them

What happens, step by step

Before and after. The real path, traced twice: a flowchart for the shape of the change, then pseudocode for the real lines. Here the customer used to decide whether to pay again. Now the code decides.

Now on main

flowchart TD
  classDef added stroke:#c08a2a,stroke-width:3px
  classDef removed stroke:#c0503f,stroke-width:2px,stroke-dasharray:5 4
  A["Customer pays"] --> B["Send the charge"]
  B --> C["On a timeout, ask the customer to try again"]:::removed

With this change

flowchart TD
  classDef added stroke:#c08a2a,stroke-width:3px
  classDef removed stroke:#c0503f,stroke-width:2px,stroke-dasharray:5 4
  A["Customer pays"] --> B["Send the charge"]
  B --> C["On a timeout, wait and send it again, up to 3 times"]:::added
  C --> D["Show the result"]

Thick outline: a hop the change adds. Dashed outline: a hop it removes.

Pseudocode of the real paths. Shaded lines are new.

1. A customer pays at checkout

Now on main

POST /checkout
  chargeClient.create(order)
    httpClient.post("/v1/charges")
                      // 10 s timeout
  on timeout: "Payment failed,
               please try again"

With this change

POST /checkout
  chargeClient.create(order)
    httpClient.post("/v1/charges")
                      // 10 s timeout
    on timeout or 5xx:
      wait 200 ms, then 400 ms
      send again, 3 attempts in all
  on final failure: "Payment failed"

2. Any other code sends a request

Now on main

httpClient.post(url, body)
  send once

With this change

httpClient.post(url, body,
  { attempts = 3 })   // the default
  send, and retry like checkout
// 11 callers outside the diff

What changes for people

Who wins and who loses. When the change alters what a signal means, a small before/after table comes first.

QuestionBeforeAfter
What happens after a timeout?The customer decides whether to pay again.The code sends the same charge again.
What stops a second charge?The customer, who sees the error.Nothing in our code. Only the provider, if it rejects repeats.
Better offA customer who hits a brief provider error pays on the first try, without seeing an error.
Worse offA customer whose first charge went through but timed out. The retry may charge them again.
Now fails quietlyEmails and webhooks that failed once now go out up to three times, with no sign to anyone.
Now depends onThe provider rejecting a repeated charge. Only its docs say it does.

What you need to decide

Every decision, ranked. Each card has the code pair, one yes/no question, how the reviewer knows, and when the choice is wrong. The reviewer does not rule. The weakest evidence ranks first, not the scariest code. The blue tag names the kind of problem the card shows.

Edge case · retry

01A charge that timed out may have gone through, and the retry charges again

// BEFORE
await httpClient.post("/v1/charges", body);

// AFTER — no idempotency key is sent
await httpClient.post("/v1/charges", body, { attempts: 3 });

The question

A timeout does not mean the charge failed. Does the provider reject a repeated charge that carries no idempotency key?

How I know

not checkedOnly the provider's docs were read. Nobody sent a repeated charge on staging.

This is wrong if

A customer on a slow connection is charged twice for one order.

The other option

Send an idempotency key built from the order id, so a retry cannot charge twice.

Further reading. Stripe's post "Designing robust and predictable APIs with idempotency" explains why a retried payment needs a key.

Shared code · 11 other callers

02Every caller of the HTTP client now retries, not just payments

// BEFORE — httpClient.ts
export function post(url, body) {

// AFTER — the default applies to all 11 callers
export function post(url, body, { attempts = 3 } = {}) {

The question

Four of the new retrying callers send emails. Is a customer getting the same receipt up to three times acceptable?

How I know

read the codeA repo-wide search for httpClient.post(, each of the 11 hits read.

This is wrong if

A slow mail provider makes customers receive duplicate receipts and password emails.

The other option

Default to one attempt, and let the payment client ask for three.

Numbers · waiting time

03During a provider outage, checkout waits about 31 seconds before an error

// BEFORE
{ timeoutMs: 10000 }                // worst case 10 s

// AFTER
{ timeoutMs: 10000, attempts: 3 }   // 3 × 10 s + 0.6 s of waits

The question

The 10-second timeout now applies to each attempt, not to the whole payment. Is a 31-second wait acceptable at checkout?

How I know

read the codeThree attempts of 10 s, plus waits of 200 ms and 400 ms.

This is wrong if

Customers give up, reload the page and pay again while the first payment is still being retried.

The other option

Give the whole payment one 10-second budget, shared by all attempts.

Performance · outage

04In an outage, retries triple the traffic, and all at the same moments

// BEFORE — did not exist

// AFTER — httpClient.ts:38, the same wait for every caller
await sleep(200 * 2 ** attempt);

The question

With no random spread in the waits, every failed checkout retries at the same instant. Can a provider that is already failing take three times the calls?

How I know

not checkedNobody knows the provider's rate limit during an incident.

This is wrong if

A short provider hiccup becomes a long outage, because our retries keep it overloaded.

The other option

Add random jitter to each wait, and stop retrying after a run of failures.

Further reading. The AWS Builders' Library article "Timeouts, retries, and backoff with jitter" explains how retries can overload a failing service.

Second pass: security and performance

A second agent hunts. The cards above only ask. A fresh agent that never saw them reads the diff for security and performance defects, which often leave no before/after pair to show. Its findings appear as returned. "Also card 0N" marks one that repeats a card.

Security

S1The retry log line writes the whole request, card token included

src/lib/httpClient.ts:41

log.warn("retrying request", { url, body, attempt });

What happens

Each retry copies the card token and the customer's email into the logs, for all 12 callers.

How I know

read the codebody is the full request; the charge body holds cardToken.

Performance

P1A failing provider holds each checkout request open for 31 seconds

Also card 03

src/checkout/checkoutController.ts:29

const charge = await chargeClient.create(order);

What happens

During an outage, open checkout requests pile up, so the rest of the store slows down too.

Depends on

Checkouts per minute at peak — not measured. Read it from the checkout dashboard.

How I know

read the codeThe handler awaits every attempt inside the request.

The ticket against the code

Silence is the finding. Where the ticket did not specify something and the code chose anyway, that choice is where a discussion would have happened if there had been time for one.

The ticket asked. Retry payment calls that fail with a timeout or a server error, so brief provider errors stop failing checkouts.

The code does. It retries payment calls, by adding retries to the shared HTTP client that 11 other callers also use.

The ticket did not say

Smaller things I noticed