Your Redis Cache Is Serving Yesterday's Data: Fixing Invalidation Properly

The bug report says "the price on the product page is wrong". You open the admin panel and the price is right. You open the product page in a private window and it is wrong. You run the query straight against Postgres and it is right. Somewhere between a correct database and a wrong page there is a Redis key nobody is looking at, and it has been serving the old number since about 04:00 this morning.

Cache invalidation gets treated as a joke, one of the two hard problems, and that joke has done real damage. It frames the problem as unsolvable and therefore not worth engineering. It is not unsolvable. What is true is that almost nobody treats the cache as part of the write path, so the invalidation call gets written last, by hand, inside whichever service happened to do the update, using a key string typed from memory.

The position this article is built on: a TTL is not invalidation. A TTL is a bound on how long you have agreed to be wrong. If being wrong for an hour is unacceptable, then you need an explicit invalidation on the write path, and it has to be impossible to get the key wrong.

The failure has no error message, which is exactly why it survives

There is no stack trace here. The closest thing to an error is this, and it is not reported as one:

127.0.0.1:6379> DEL product:42
(integer) 0

DEL returns the number of keys that were actually removed. Zero means "there was nothing at that key". It does not mean the call failed, so no client library raises, no logger fires, and no alert exists. Every Python, Node and Go Redis client will happily return 0 into a variable nobody reads. Meanwhile the real key is still sitting there:

127.0.0.1:6379> EXISTS products:v3:t7:42
(integer) 1
127.0.0.1:6379> TTL products:v3:t7:42
(integer) 3187

That is the whole incident. A writer deleting product:42, a reader populating products:v3:t7:42, and 53 minutes of remaining TTL during which every customer sees the old price.

Why this happens

The cache is not actually in the write path

Cache-aside is drawn as a neat pair: read misses go to the database and populate the cache, writes go to the database and delete the cache. The drawing assumes exactly one writer. In a system that has been alive two years there are five: the API handler, an admin action, a nightly import job, a background task that recalculates something, and whatever a developer ran in a psql session during the last incident. Four of them do not know the cache exists. This is the single most common root cause, and it is not really a coding mistake. It is an architecture that requires every future writer to remember an invisible obligation.

Key namespaces drift and nothing enforces agreement

Read path and write path build the key independently. Someone adds a tenant ID to the read key for multi-tenancy, or bumps a v2 to v3 when the serialised shape changes, and updates the reader only. Both code paths are still correct in isolation. They just no longer refer to the same key, and Redis has no schema to tell them so.

Invalidation fires at the wrong moment

If you delete the key inside the transaction and it later rolls back, you have evicted a correct entry and the next reader repopulates it correctly. Wasteful, harmless. The dangerous variant: the delete happens before the commit, a concurrent reader misses, reads the pre-commit value from the database, and writes it back into the cache. You have now cached a value that is about to be wrong, with a fresh full TTL. That is the classic cache-aside race, and it makes the stale entry outlive the one you deleted.

The TTL was silently doing all the work

Plenty of systems have a broken explicit invalidation path that nobody discovers, because a 60 second TTL was masking it. The incident arrives on the day someone raises the TTL to an hour to cut database load. Nothing about the invalidation code changed. The bug was always there.

There is no degradation, so one bad key becomes an outage

If your read path does value = json.loads(redis.get(key)) with no branch for None and no exception handling around the connection, then a Redis restart, a failover, or an OOM command not allowed when used memory > 'maxmemory' does not degrade the page. It 500s the page. A cache is an optimisation. The moment your code cannot run without it, it is a database, and you did not design it like one.

The fix

Step 1: Make the key impossible to type by hand

One module. Both sides import it. No string literal for a cache key exists anywhere else in the codebase.

# cache_keys.py - the only place a cache key is constructed
PRODUCT_SCHEMA = 3  # bump when the cached payload shape changes

def product_key(tenant_id: int, product_id: int) -> str:
    return f"products:v{PRODUCT_SCHEMA}:t{tenant_id}:{product_id}"

def product_version_key(tenant_id: int) -> str:
    return f"products:ver:t{tenant_id}"

Then add a CI check that greps the source for a quoted string being passed directly to a cache call, excluding this module, and fails the build if it finds one. It is crude, and it has caught more real bugs for me than any amount of code review.

Step 2: Invalidate after commit, never inside the transaction

In SQLAlchemy this is a session event, not something you sprinkle into handlers:

from sqlalchemy import event
from sqlalchemy.orm import Session

@event.listens_for(Session, "after_commit")
def _flush_pending_invalidations(session):
    for key in getattr(session, "_cache_invalidations", ()):
        redis.unlink(key)          # UNLINK frees memory off the main thread
    session._cache_invalidations = set()

Django's equivalent is transaction.on_commit(lambda: redis.unlink(key)). Node with Prisma or Knex needs the same discipline: collect keys during the unit of work, fire them once the commit returns. The point is that the invalidation is attached to the transaction, not to a request handler that a future writer might never go through.

Step 3: Version the namespace so a missed delete cannot persist

Deleting the exact key is precise and brittle. Bumping a version counter is coarse and almost impossible to get wrong. You keep an integer per logical group, and it forms part of every key in that group:

def read_product(tenant_id, product_id):
    ver = redis.get(product_version_key(tenant_id)) or b"0"
    key = f"{product_key(tenant_id, product_id)}:g{ver.decode()}"
    hit = redis.get(key)
    if hit is not None:
        return json.loads(hit)
    row = db.fetch_product(tenant_id, product_id)
    redis.set(key, json.dumps(row), ex=3600)
    return row

def bump_products(tenant_id):
    redis.incr(product_version_key(tenant_id))   # every old key is now unreachable

One INCR makes every key in the group unreachable at once. The orphaned entries expire on their own TTL and are never read again. You trade one extra round trip on reads, plus a burst of misses after each bump, for the guarantee that a key-name mismatch cannot serve stale data indefinitely. On a read-heavy path where the group is bumped every few seconds that trade goes bad, and you should scope the group tighter.

Step 4: Add a backstop that does not depend on application code

Application-level invalidation misses whatever bypasses the application. Redis keyspace notifications let you watch what is actually happening, which is a diagnostic aid rather than a fix:

redis-cli CONFIG SET notify-keyspace-events KEA
redis-cli --csv PSUBSCRIBE '__keyevent@0__:del' '__keyevent@0__:expired'

The real backstop is change data capture: a logical replication slot on Postgres, or the MySQL and MariaDB binlog, feeding a small consumer that bumps the version counter for any table it sees change. It catches the manual UPDATE in psql, the migration, and the third-party job, because all of them must write to the database.

Step 5: Make the read path survive Redis being gone

try:
    hit = redis.get(key)
except RedisError:
    CACHE_ERRORS.inc()
    hit = None            # fall through to the database, do not raise

Set real timeouts on the client (socket_timeout and socket_connect_timeout, a few hundred milliseconds) so a hung Redis cannot hold your worker threads. Without that, a cache that is merely slow will exhaust your connection pool and take down the application, which is the same failure class as running out of Postgres connections.

Verification

Three checks, in increasing order of confidence.

1. Watch the write happen. In one terminal run redis-cli MONITOR, in another perform the update through the real UI. You should see the UNLINK or INCR land, and the key it names must be byte-identical to the one the reader builds:

1721800000.123456 [0 10.0.0.5:41022] "INCR" "products:ver:t7"
1721800000.401200 [0 10.0.0.5:41030] "GET" "products:ver:t7"
1721800000.402911 [0 10.0.0.5:41030] "GET" "products:v3:t7:42:g18"

MONITOR costs real throughput, because every command is echoed to your client. Use it on staging or briefly against one production node, and never leave it attached.

2. Assert the invalidation in a test, not the delete call. A test that mocks Redis and asserts delete was called proves nothing about the key name. Run a real Redis in the test suite and assert on behaviour:

def test_price_update_is_visible_immediately(client, db):
    assert client.get("/api/products/42").json()["price"] == 1200
    db.execute("UPDATE products SET price = 1450 WHERE id = 42")
    db.commit()
    assert client.get("/api/products/42").json()["price"] == 1450

That test fails on a key mismatch, on a missing commit hook, and on a new writer that skipped the cache layer. A mock-based test fails on none of them.

3. Run a staleness canary in production. A job every few minutes that reads one known row through the cached path and directly from the database, compares them, and emits a metric. Alert on any mismatch that survives two consecutive runs, so you do not page on the normal sub-second window. This is the only check that catches the writer you did not know existed, and it is about thirty lines of code.

What people get wrong

Common advice	Why it fails
"Just lower the TTL"	Shrinks the window, does not fix correctness, pushes load back onto the database, and hides the real bug for another year.
KEYS 'product:*' then delete	KEYS walks the entire keyspace and blocks the single-threaded server while it does. On a large instance that is a self-inflicted outage. Use SCAN, or version the namespace so you never need a sweep.
FLUSHALL when in doubt	Turns a stale-key bug into a thundering herd against the database, usually at the worst moment. It also wipes sessions, rate-limit counters and queues sharing the instance.
"Add a circuit breaker"	Good for availability, irrelevant to correctness. It protects you when Redis is down. It does nothing when Redis is up and confidently wrong, which is the case that costs money.
"Write the new value in instead of deleting"	Write-through looks tidier but makes concurrent writers race on ordering, and the loser's value sits there for a full TTL. Deleting is idempotent and order-insensitive.

One more deserves naming: treating the postmortem as the deliverable. Nearly every public write-up of a cache incident ends at "we added monitoring and improved our testing". No tool detects cache-invalidation bugs for you, and none is coming, because the correct value is only knowable from your own domain. The compensating control has to be structural.

When it is still broken

• Look for a second cache you forgot about. An in-process LRU inside each worker, a functools.lru_cache, a module-level dict, is invisible to Redis and immune to every fix above. It also disagrees between workers, so the page alternates between correct and stale on refresh. That alternation is the tell.
• Check the HTTP layer. If responses carry Cache-Control: public, max-age=... then Cloudflare, an nginx proxy_cache, or the browser is serving the old body and Redis is innocent. Curl the origin directly and compare it against what the edge returns. My nginx reverse proxy notes cover where those headers get set and overridden.
• Check whether you repopulate from a replica. If the write goes to the primary and the cache is filled from a read replica a second behind, you cache the pre-write value the instant you invalidate it. Repopulate from the primary.
• Check the database number. Redis exposes 16 logical databases by default, and one service on db=0 while another uses db=1 produces exactly this symptom. redis-cli -n 1 --scan --pattern 'products:*' against each settles it in ten seconds.

None of this is exotic. It is the same discipline as any other correctness problem: reduce the number of places a mistake can be made, then measure the thing you care about rather than the thing that is easy to measure. A cache hit rate dashboard will never tell you the cache is lying. A canary will.