Inside Atlassian’s Forge Billing Architecture for Distributed Usage Tracking at Scale

The Global Context

At hyperscale, tracking distributed usage for billing purposes ceases to be a simple database transaction problem and becomes a complex distributed systems challenge. When operating a multi-tenant platform like Atlassian Forge, which hosts thousands of third-party applications executing untrusted code across a global footprint, resource isolation and accurate telemetry collection are paramount. Traditional monolithic billing systems rely on synchronous database writes, which quickly fail under the weight of millions of concurrent API calls and event emissions. In a multi-tenant environment, a single tenant experiencing a sudden traffic spike can easily saturate shared database connections, leading to severe resource contention and starving adjacent tenants of compute resources. This noisy-neighbor effect can rapidly degrade the performance of the entire platform if not mitigated by a robust, isolated ingestion layer.

To prevent localized traffic anomalies from escalating into platform-wide outages, modern SaaS architectures must decouple usage tracking from the core billing engine. This separation of concerns requires a highly scalable, asynchronous ingestion pipeline capable of buffering massive bursts of telemetry data while maintaining strict multi-tenant isolation. At this scale, the system must enforce rate limits and quotas at the edge, ensuring that no single tenant can monopolize the ingestion infrastructure. Furthermore, the architecture must guarantee that every single usage event is captured, validated, and processed with high fidelity, as any lost telemetry directly translates to lost revenue or degraded customer trust. Achieving this level of reliability requires a deep understanding of distributed queueing, backpressure mechanisms, and partition strategies designed to isolate tenant workloads at the physical layer.

Deep-Dive Challenge: Distributed Failure Modes

Designing a distributed usage tracking system introduces several critical failure modes that can compromise both system stability and financial accuracy. One of the most pervasive challenges is the cascading failure, which occurs when a downstream component, such as a database or a processing worker, becomes overloaded and begins to fail. As the downstream service slows down, upstream services attempt to retry failed requests, creating a thundering herd effect that further exacerbates the overload. Without proper backpressure and circuit-breaking mechanisms, this feedback loop can quickly propagate backward through the entire ingestion pipeline, eventually causing the edge API gateways to reject incoming telemetry events and leading to data loss.

Another significant challenge is the cache stampede, which occurs when a highly active cache key (such as a tenant's current usage balance) expires or is invalidated. If thousands of concurrent requests attempt to read and update this expired key simultaneously, they will all bypass the cache and hit the underlying database, causing a sudden spike in read/write latency and potentially crashing the database instance. To mitigate this, systems must employ strategies like probabilistic early expiration or mutex locking to ensure only a single worker updates the cache. Additionally, in a globally distributed deployment, replication lag between database replicas can lead to split-brain scenarios or inconsistent state reads. If a tenant's usage is updated in one region but read from another before replication completes, the system may make incorrect decisions regarding quota enforcement, leading to double-spending or unauthorized resource consumption.

Finally, network partitions are an inevitable reality in large-scale cloud environments. When a partition occurs, the system must decide whether to prioritize consistency or availability, as dictated by the CAP theorem. In a billing context, choosing availability means allowing usage events to be recorded even if the central ledger cannot be reached, which can lead to temporary over-allocation of resources. Conversely, choosing consistency means rejecting usage events when the ledger is unavailable, which directly impacts the availability of third-party applications running on the platform. Resolving these conflicting requirements requires a carefully designed event-driven architecture that leverages eventual consistency, idempotent processing, and robust reconciliation loops to ensure that the system eventually converges on a single, accurate state once the partition is resolved.

The Solution Architecture

Edge Ingestion and Local Data Flow Dynamics

At the edge, incoming telemetry events are received by a high-throughput API gateway layer, which performs initial validation, authentication, and rate limiting. To minimize latency overhead, the gateway does not perform synchronous database lookups; instead, it relies on a local, highly optimized cache (such as Redis) to verify tenant credentials and active quotas. Once validated, the event is immediately serialized using Protocol Buffers (Protobuf) to minimize payload size and serialization overhead. The serialized event is then appended to a local, high-performance write-ahead log (WAL) or a local Kafka topic. This asynchronous write pattern ensures that the edge gateway can respond to the client with a 202 Accepted status within single-digit milliseconds, decoupling the client from the downstream processing pipeline.

Control Plane vs. Data Plane State Management

The architecture strictly segregates the control plane from the data plane to ensure high availability and fault tolerance. The data plane is responsible for the high-throughput ingestion, serialization, and streaming of usage events, utilizing Apache Kafka as the central event backbone. Kafka topics are partitioned by tenant ID to guarantee that events for a specific tenant are processed in order and can be scaled horizontally across multiple consumer groups. The control plane, on the other hand, manages tenant metadata, billing configurations, and quota definitions. It stores this authoritative state in a highly consistent relational database (such as Amazon Aurora PostgreSQL) and propagates updates to the data plane asynchronously via Change Data Capture (CDC) pipelines. This separation ensures that even if the control plane database experiences an outage, the data plane can continue to ingest and process usage events using cached configuration state.

Observability, Telemetry, and Dead-Letter Queue (DLQ) Strategy

To maintain operational visibility into the distributed pipeline, the architecture incorporates a comprehensive observability stack. Every event is tagged with a unique correlation ID at the edge, allowing it to be tracked as it flows through the various processing stages. Metrics such as ingestion latency, consumer lag, and processing error rates are exported to Prometheus and visualized via Grafana dashboards. When an event fails to process due to schema mismatches, corrupted payloads, or transient database errors, it is not discarded. Instead, the consumer routes the failed event to a tiered Dead-Letter Queue (DLQ) system. The first tier handles transient errors with exponential backoff and jittered retries. If the event continues to fail, it is escalated to a secondary DLQ for manual inspection and replay, ensuring zero data loss for critical billing telemetry.

syntax = "proto3";

package billing.v1;

message UsageEvent {
  string event_id = 1;
  string tenant_id = 2;
  string app_id = 3;
  string metric_name = 4;
  int64 quantity = 5;
  int64 timestamp_ms = 6;
  map<string, string> metadata = 7;
}

// Example of an idempotent consumer processing loop in pseudocode:
//
// function processUsageEvent(UsageEvent event) {
//   if (cache.exists(event.event_id)) {
//     log.warn("Duplicate event detected, skipping: " + event.event_id);
//     return; // Idempotency check passed
//   }
//   
//   db.beginTransaction();
//   try {
//     db.execute("INSERT INTO usage_ledger (event_id, tenant_id, quantity, timestamp) VALUES (?, ?, ?, ?)", 
//                event.event_id, event.tenant_id, event.quantity, event.timestamp_ms);
//     db.execute("UPDATE tenant_balances SET balance = balance + ? WHERE tenant_id = ?", 
//                event.quantity, event.tenant_id);
//     db.commit();
//     cache.set(event.event_id, "PROCESSED", "EXPIRE", 86400); // Cache for 24 hours
//   } catch (Exception e) {
//     db.rollback();
//     throw e; // Trigger consumer retry
//   }
// }

Implementation & Trade-offs

When evaluating this architecture through the lens of the PACELC theorem, we explicitly choose availability and eventual consistency over strict consistency during network partitions (PA/EL). In the event of a network partition between the edge ingestion nodes and the central database, the system continues to accept and queue usage events locally. While this means that a tenant's balance may temporarily lag behind reality, it ensures that the platform remains fully operational and third-party applications can continue to execute. Once the partition is resolved, the queued events are processed in order, and the system eventually converges on the correct balance. To mitigate the risk of temporary over-allocation during a partition, we enforce conservative local quotas at the edge, which are periodically synchronized with the central ledger.

This trade-off introduces a minor latency overhead at the edge, primarily due to the local cache lookups and serialization steps. However, by utilizing highly optimized Redis clusters and Protobuf serialization, this overhead is kept under 5 milliseconds, which is well within the acceptable threshold for high-performance API gateways. Furthermore, by applying Amdahl's Law to our processing pipeline, we identify that the serial portion of our code—specifically, the database writes—is the primary bottleneck. To maximize parallel execution, we partition our Kafka topics extensively, allowing us to scale our consumer instances horizontally. This parallelization ensures that we can process millions of events per second with minimal queuing delay, even during peak traffic periods.

Senior Perspective: Infrastructure Maturity

Operating a distributed billing and usage tracking system at scale requires a high degree of operational discipline and a mature infrastructure engineering culture. It is not enough to design a robust architecture; the system must be continuously tested under simulated failure conditions to ensure its resilience. By incorporating Chaos Engineering practices—such as injecting network latency, terminating Kafka brokers, and simulating database failovers—we can verify that our circuit breakers, retry mechanisms, and DLQ routing behave as expected. These automated chaos experiments help us identify hidden failure modes and race conditions before they can impact production workloads, ensuring that our billing pipeline remains highly resilient.

Additionally, cost-of-compute optimization is a critical consideration when running high-throughput pipelines. By optimizing our serialization formats, leveraging efficient connection pooling, and utilizing auto-scaling consumer groups, we can significantly reduce the infrastructure footprint required to process our telemetry data. This focus on efficiency not only lowers our operational costs but also improves the overall sustainability of our cloud infrastructure. Ultimately, maintaining a system of this complexity requires a continuous commitment to engineering excellence, comprehensive documentation, and a blameless post-mortem culture that views every production incident as an opportunity to refine and strengthen our architecture.