We used Redis locks to prevent duplicate cron jobs.
Users started getting duplicate notifications. 💀
The setup made sense: multiple instances of the service all had the same cron running. So we added a Redis distributed lock — only one instance acquires the lock, runs the job, sends notifications.
Then this happened in production:
- Job execution time exceeded the lock’s TTL
- Lock expired mid-execution
- Another instance acquired the lock
- Same job started again on a different server
- Both instances were now sending notifications simultaneously 💥
Redis locks decide who starts. Not who finishes.
The fix: idempotency check
We added a version key to the job. Before processing any batch, the instance checks:
“Is this still the latest execution?”
- If yes → proceed ✅
- If stale → skip 🗑️
String currentVersion = redis.get("cron:notification:version");
if (!currentVersion.equals(myVersion)) {
// another instance already took over — bail out
return;
}
// safe to process
sendNotifications();When the lock expires and Instance B takes over, it writes a new version. Instance A picks up the version mismatch on its next batch and stops. No duplicate sends.
Locks reduce the chance of duplication. Idempotency guarantees correctness.
Distributed systems don’t fail because things break — they fail because things run twice. 😄