How Notion Decreased Latency by 20% with Caching

The Global Context

At hyperscale, the challenges of data retrieval transition from simple database query optimization to complex multi-tenant resource orchestration. When platforms scale to accommodate tens of millions of active users interacting with deeply nested, collaborative document graphs, the underlying database layer faces severe resource contention. Multi-tenant isolation becomes a primary concern, as a single highly active workspace can generate a disproportionate volume of read requests, leading to a classic noisy-neighbor scenario. Without a robust caching layer, these localized spikes in traffic bypass application-level boundaries and directly saturate the database's I/O operations per second (IOPS), CPU, and memory bandwidth, threatening the availability of the entire platform.

To mitigate this, infrastructure teams must design caching topologies that act as protective shields, isolating database clusters from unpredictable access patterns. This requires moving beyond naive, single-node cache deployments toward distributed, multi-tiered caching architectures. At this scale, the blast radius of a cache invalidation failure or a misconfigured eviction policy can be catastrophic, causing cascading degradations across downstream microservices. Therefore, caching is no longer treated as an optional performance optimization, but as a critical tier of the core infrastructure, requiring strict rate-limiting, tenant-aware resource allocation, and deterministic data routing protocols to maintain system-wide stability.

Deep-Dive Challenge: Distributed Failure Modes

Operating a distributed cache at scale introduces severe failure modes that can quickly bring down an entire infrastructure if not proactively engineered against. One of the most critical failure states is the cache stampede, also known as the thundering herd problem. This occurs when a highly cached, high-concurrency data key expires or is invalidated. In a matter of milliseconds, thousands of concurrent application threads detect the cache miss and simultaneously query the upstream database. This sudden, massive spike in read demand can exhaust database connection pools, spike CPU utilization to 100%, and cause query latencies to degrade exponentially, ultimately leading to a complete service outage.

Another insidious failure mode is replication lag within the caching tier itself, particularly in globally distributed deployments. When a write operation updates a primary cache node, that change must be asynchronously propagated to read replicas across multiple geographic regions. During this propagation window, read requests routed to lagging replicas will return stale data, violating eventual consistency guarantees and causing application-level anomalies. If the application attempts to resolve this by immediately writing back the stale data to the primary database, it can trigger write-write conflicts or corrupt the source of truth, highlighting the delicate balance between cache consistency and system availability.

Furthermore, cascading failures often emerge when a caching node fails and its keyspace is rehashed across the remaining active nodes. If using a naive hashing algorithm, a single node failure can cause a massive percentage of cache keys to map to different nodes, resulting in a systemic cache miss storm across the entire cluster. Even with consistent hashing, the sudden redistribution of load can overwhelm the neighboring nodes, causing them to fail in a domino effect. To prevent these catastrophic scenarios, engineers must implement sophisticated mitigation strategies, such as probabilistic early expiration, request coalescing via single-flight patterns, and resilient consistent hashing rings with virtual nodes.

The Solution Architecture

1. Local and Edge Data Flow Dynamics

To achieve a 20% reduction in end-to-end latency, the architecture employs a multi-tiered caching topology consisting of an in-memory L1 cache local to each application process, coupled with a distributed L2 cache cluster powered by Redis. When an application node receives a read request, it first queries its local L1 cache (implemented using a high-performance, lock-free LRU cache in Go or Rust). If an L1 miss occurs, the request falls back to the L2 Redis cluster, which is partitioned using consistent hashing with virtual nodes to ensure uniform key distribution. Only when both L1 and L2 tiers miss does the request proceed to the primary PostgreSQL database. To prevent cache stampedes at the L2 layer, we implement a probabilistic early expiration algorithm (XFetch), which recalculates the cache validity dynamically based on the read frequency and the time taken to compute the value.

2. Control Plane vs. Data Plane State Management

The separation of concerns between the control plane and the data plane is vital for maintaining state consistency without sacrificing throughput. The data plane handles the high-velocity routing of read and write payloads, utilizing optimized serialization formats like Protocol Buffers (Protobuf) to minimize CPU overhead during serialization and deserialization. The control plane, on the other hand, manages cluster topology, node discovery, and cache invalidation broadcasts. Invalidation events are published to a high-throughput Apache Kafka topic. A dedicated invalidation daemon consumes these events and dispatches invalidation signals to the L1 caches of all active application instances via gRPC streams, ensuring that stale L1 entries are purged within milliseconds of a database write.

3. Observability, Telemetry, and Dead-Letter Queue (DLQ) Strategy

Comprehensive observability is integrated directly into the caching client libraries to track cache hit/miss ratios, serialization latency, and network round-trip times (RTT). Telemetry data is exported via OpenTelemetry gRPC exporters to a centralized Prometheus and Grafana stack. When an invalidation event fails to process due to network partitions or consumer lag, the event is routed to a Dead-Letter Queue (DLQ) hosted on Kafka. A specialized reconciliation worker processes the DLQ, executing out-of-band database consistency checks and force-invalidating the affected keys across both L1 and L2 tiers. This ensures that transient network anomalies do not lead to permanent data drift between the cache and the persistent storage layer.

-- Redis Lua Script for Probabilistic Early Expiration (XFetch)
-- KEYS[1]: cache key
-- ARGV[1]: delta (computation time in ms)
-- ARGV[2]: beta (constant > 0, higher means more aggressive early refresh)
-- ARGV[3]: ttl (time to live in seconds)

local res = redis.call("HMGET", KEYS[1], "value", "ttl", "delta")
local value = res[1]
local ttl = tonumber(res[2])
local delta = tonumber(res[3])

if not value then
    return nil -- Cache miss
end

local rand = math.random()
local beta = tonumber(ARGV[2])
local threshold = -beta * delta * math.log(rand)

if threshold > ttl then
    return {value, "REFRESH_TRIGGERED"}
end

return {value, "HIT"}

Implementation & Trade-offs

Evaluating this multi-tiered caching architecture through the lens of the PACELC theorem reveals deliberate engineering trade-offs. In the event of a network partition (P), the system prioritizes availability (A) over consistency (C), allowing read operations to continue serving potentially stale data from the L2 cache rather than failing the request. When the system is operating normally (E), we choose to trade consistency (C) for latency (L). The asynchronous nature of our L1 invalidation pipeline means there is a sub-millisecond window where different application nodes may serve slightly different versions of a document. This eventual consistency model is highly acceptable for collaborative document editing, where real-time operational transformation (OT) or Conflict-free Replicated Data Types (CRDTs) resolve concurrent edits downstream, but would be unacceptable for transactional financial systems.

Furthermore, we must account for the CPU and memory overhead introduced by the caching client libraries. While L1 caching dramatically reduces network RTT to zero for hot keys, it increases the memory footprint of the application process and introduces garbage collection (GC) pressure. To mitigate GC pauses in our Go-based application services, we utilize off-heap memory allocation or highly optimized byte arrays for storing serialized Protobuf payloads. The serialization and deserialization overhead itself is minimized by reusing Protobuf message pools, preventing frequent allocations on the heap. This optimization ensures that the sub-millisecond latency gains achieved by bypassing the database are not clawed back by runtime execution overheads.

Senior Perspective: Infrastructure Maturity

Achieving a sustainable 20% latency reduction at scale is as much an organizational and operational challenge as it is a technical one. It requires establishing a rigorous culture of Chaos Engineering, where failure modes are actively simulated in production environments. By utilizing tools like Chaos Mesh or Gremlin, we inject artificial network latency, terminate Redis sentinel nodes, and simulate Kafka partition splits to verify that our fallback mechanisms, single-flight coalescing, and DLQ reconciliation pipelines behave deterministically under extreme stress. This continuous validation ensures that the caching layer remains a resilient shield rather than becoming a single point of failure.

Ultimately, the maturity of an infrastructure team is reflected in their ability to balance compute costs with system performance. By offloading up to 90% of read traffic from the primary database clusters to the caching tier, we significantly reduce the provisioning requirements for expensive, high-IOPS database instances. This cost-of-compute optimization frees up capital and engineering resources to focus on product innovation rather than constant database scaling bottlenecks. Maintaining this level of discipline requires continuous monitoring of cache efficiency metrics, automated load testing of the caching topology during CI/CD pipelines, and strict architectural reviews for any new data access patterns introduced to the codebase.