One Redis Key Expired. 100 Requests Hit Our DB.

Cache stampede: one expired key = everyone hits DB at once

A single Redis key expired. API latency jumped from 200ms → 2s.

No deploy. No traffic surge. Nothing changed.

Turns out a config key in Redis had expired. And then everything hit the database at once.

What is a cache stampede?

flowchart TD A[Redis key expires] --> B[Request 1 misses cache] A --> C[Request 2 misses cache] A --> D[Request 100 misses cache] B --> E[DB query] C --> E D --> E E --> F[DB overwhelmed — latency spikes] style F fill:#C4583A,color:#F7F5EE style A fill:#D69E2E,color:#1A1A17

When a cached value expires, every in-flight request simultaneously falls through to the database. All of them fire the same query. All at once.

At small traffic? Milliseconds of pain, nobody notices.

At production scale? Hundreds of identical queries hit your DB in the same instant. Latency spikes. Everything slows down.

This is a cache stampede — also called a thundering herd.

What happened in our case

Our service cached a config object in Redis with a fixed TTL. When it expired:

Every incoming request missed the cache
All of them fell through to the DB
Every single one fired the same query
DB got hammered with hundreds of identical reads
The cache wasn’t protecting the database anymore

The config object took ~200ms to compute from DB. Multiply that by 100+ concurrent requests = chaos.

Fix 1: Mutex lock with SETNX

Only let one request rebuild the cache. Everyone else waits for that result.

public Config getConfig(String key) {
    // Try cache first
    String cached = redis.get(key);
    if (cached != null) return deserialize(cached);

    // Try to acquire rebuild lock
    String lockKey = "lock:" + key;
    boolean acquired = redis.set(lockKey, "1", SetArgs.Builder.nx().ex(10));

    if (acquired) {
        try {
            Config config = db.fetchConfig(key);
            redis.setex(key, 300, serialize(config));
            return config;
        } finally {
            redis.del(lockKey);
        }
    } else {
        // Another thread is rebuilding — wait briefly and retry
        Thread.sleep(50);
        return getConfig(key);
    }
}

SET key value NX EX 10 is atomic. Even if 100 requests arrive simultaneously, only one wins the lock. The rest retry after 50ms and hit the freshly warmed cache.

flowchart TD A[Cache miss] --> B{SETNX lock?} B -->|acquired| C[Fetch from DB] C --> D[Write to Redis] D --> E[Release lock] B -->|not acquired| F[Wait 50ms] F --> G[Retry — cache is warm now] style C fill:#0F4C3A,color:#F7F5EE

Fix 2: TTL jitter

If you cache multiple keys with the same TTL, they all expire simultaneously. One stampede per config key, all at once.

Add randomness to the TTL:

int baseTtl = 300; // 5 minutes
int jitter = ThreadLocalRandom.current().nextInt(0, 60); // 0–60s random
redis.setex(key, baseTtl + jitter, serialize(value));

Keys now expire at different times. No synchronized expiry. No synchronized stampede.

The mental model shift

Most engineers think about cache hits and misses. The question is “is the data in cache?”

But the real question under high traffic is: what happens the moment it’s not?

The cache wasn’t the bottleneck. The cache expiry was.

Distributed systems are fun… until every request misses the cache at once.