100s of payments ended up in a broken state. 💀
And we had no idea what happened in production.
The flow looked simple:
Payment gateway sends us webhooks → we read the row → update it → save.
Then this happened.
Two webhooks landed at the exact same millisecond — a SUCCESS and a PENDING status for the same payment.
- Both read the same row
- Both wrote back
- Last write wins… and one update just vanished 👻
Classic concurrent write race. Razorpay retried the webhook, we processed both, and the DB picked a winner at random.
How we fixed it
1. Optimistic locking
Added @Version to the payment entity. Every update now carries a version number. Version mismatch on save? The update gets rejected — not silently overwritten.
@Version
private Long version;2. Retry loop
On a conflict, the handler refetches the latest row and tries again — so no update is silently dropped.
3. Terminal state guard
After refetch, if the row is already SUCCESS, we stop. A late-arriving PENDING should never overwrite a finished payment.
if (payment.getStatus() == SUCCESS) return; // already done, ignoreVersioning gives safety. Retry gives resilience. Together they keep payments consistent — no table locks, happy path stays fast.
Payment gateways will redeliver, retry, and reorder webhooks. Our services need to be ready for that.
Distributed systems are fun… until you see “same row is being updated by another transaction” 😄