26 Aug 2026 8 min read Backend

Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)

The KV-Cache Bottleneck: Memory Overhead and Fragmentation in LLM Inference

In high-throughput Large Language Model (LLM) serving, performance is fundamentally bounded by memory bandwidth rather than floating-point compute capability (FLOPS). During the autoregressive generation phase, the Transformer architecture predicts tokens sequentially. Each newly generated token requires compute pass-through for all preceding tokens in the sequence. To avoid recomputing Key and Value state vectors across all previous layers at every iteration step, systems cache these intermediate representations in GPU VRAM as the Key-Value (KV) Cache.

The memory footprint of the KV-cache for a given sequence scales linearly with sequence length $S$, batch size $B$, number of Transformer layers $L$, number of attention heads $H$, and head dimension $D$. Quantified in bytes for standard 16-bit floating-point precision (FP16 or BF16, where precision factor $P = 2$ bytes per element):

$$\text{Memory}_{\text{KV}} = 2 \times B \times S \times L \times H \times D \times P$$

For a standard 70-billion parameter model (such as Llama-3-70B with $L=80$, $H=8$ Grouped-Query Attention KV heads, and $D=128$), a single context window of 8,192 tokens consumes approximately 1.34 GB of VRAM solely for its KV-cache. Multiplying this across high concurrency levels (e.g., $B = 64$ or $128$) results in tens to hundreds of gigabytes of memory consumption—frequently exceeding total available accelerator VRAM.

Traditional inference engines (e.g., standard PyTorch implementations or naive FasterTransformer setups) managed this memory by allocating contiguous memory buffers up front based on the maximum context length (e.g., reserving contiguous memory blocks for 2,048 or 8,192 tokens upfront). This strategy introduces severe architectural pathologies:

Internal Fragmentation: Memory reserved for the maximum sequence length remains unallocated and idle when requests terminate early or process short prompts. Up to 60-80% of reserved VRAM sits completely unutilized.
External Fragmentation: Variable-length request lifetimes create fragmented memory holes across contiguous allocation pools, blocking new requests even when aggregate unallocated VRAM is technically sufficient.
Over-allocation Virtual Limits: Because VRAM must be pre-allocated contiguously per request slot, batch sizes are severely artificially constrained, driving GPU Tensor Core utilization down to single-digit percentages during the decode phase.

PagedAttention: Virtual Memory Abstraction for Tensor Block Management

To solve the memory fragmentation crisis, vLLM introduces PagedAttention, an algorithm inspired by classic operating system dynamic virtual memory allocation (paging). Rather than requiring KV-caches to reside in contiguous physical VRAM, PagedAttention breaks the KV-cache of every request down into fixed-size logical Key-Value Blocks.

Each block contains the key and value vectors for a fixed number of tokens (typically $B_{\text{size}} = 16$ or $32$). Logical blocks map to non-contiguous physical blocks in a globally managed VRAM physical memory pool via a dynamic Block Table managed by the vLLM engine runtime.

Physical Block Mapping and Kernel Invocation

During execution, as an incoming request generates new tokens, the runtime engine dynamically allocates physical blocks from the pool on demand. When a logical block fills up, a new physical block is assigned from the free-block array. The core PagedAttention CUDA kernel modifies the standard multi-head attention operation by accepting an extra array of physical block addresses per request:

$$\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{Q K^{\text{T}}}{\sqrt{d_k}}\right) V$$

In the PagedAttention kernel execution, key vector $K_i$ and value vector $V_i$ for position $i$ are fetched dynamically by resolving the physical block index and offset:

$$\text{Block Index} = \lfloor i / B_{\text{size}} \rfloor, \quad \text{Block Offset} = i \pmod{B_{\text{size}}}$$

This abstraction reduces internal memory fragmentation to strictly less than a single block per request (the final partially filled block), bringing memory utilization efficiency up above 96% in typical production loads.

Zero-Copy Branching and Parallel Sampling

PagedAttention's virtual block mapping allows multiple logical sequences to reference the exact same physical memory blocks via reference counting. This capability optimizes advanced decoding strategies:

Parallel Sampling & Beam Search: Multiple candidate outputs share the identical prompt KV physical blocks. New physical blocks are allocated only when individual worker sequences diverge and generate distinct tokens, implementing a high-performance Copy-on-Write (CoW) memory mechanism.
Prefix Caching: Common system prompts, long-context system instruction sets, or multi-turn conversational history blocks are cached across separate API requests. Sub-sequence hashes map directly to pre-allocated physical VRAM blocks, bypassing initial Prefill calculation phases entirely.

Continuous Batching and Iteration-Level Request Scheduling

Traditional batching models in deep learning operate on static request batches: a batch of $N$ inputs enters the GPU, runs through execution, and holds the compute context until the longest sequence completes generation. This leads to massive idle tail latency (the "straggler problem").

vLLM bypasses static batch processing via Iteration-Level Continuous Batching (pioneered conceptually by Orca). The inference engine runs a persistent scheduling loop that operates at the granularity of individual token generation iterations rather than full end-to-end request lifetimes.

Prefill vs. Decode Phase Disaggregation

LLM processing consists of two distinct execution regimes:

Prefill Phase (Prompt Processing): Compute-bound operation. Evaluates all incoming prompt tokens in parallel, saturating GPU Tensor Cores via large Matrix-Matrix multiplications (GEMM).
Decode Phase (Token Generation): Memory-bandwidth-bound operation. Generates one token per sequence per iteration step using Matrix-Vector multiplications (GEMV), operating under heavy memory access overhead.

vLLM's scheduler dynamically merges newly arrived Prefill requests with ongoing active Decode iterations in single batch execution passes (Chunked Prefills). If new requests arrive, their prompts are chunked and scheduled alongside existing running decode passes, preventing compute stall while retaining high throughput across Tensor Cores.

Preemption Mechanics: Swapping vs. Recomputation

Under extreme request spikes where physical VRAM blocks become fully exhausted, vLLM prevents Out-Of-Memory (OOM) fatal crashes using structural preemption strategies managed by its scheduler:

1. Swapping: The engine allocates a dedicated memory pool in host CPU RAM (pinned memory accessible via high-speed PCIe transfer). When VRAM allocation reaches critical thresholds, lower-priority requests have their physical KV blocks evicted to CPU RAM via asynchronous cudaMemcpyAsync transfers over PCIe lines. Once VRAM clears, blocks are swapped back into active GPU memory.

2. Recomputation: For workloads with high PCIe bottlenecks or long sequences, the scheduler drops physical KV blocks entirely and preempts the request. When VRAM capacity frees up, the request is re-queued into the system and undergoes a fast Prefill phase to recompute its KV cache.

Multi-GPU Distributed Serving: Tensor Parallelism vs. Pipeline Parallelism

For models exceeding single GPU VRAM limits (e.g., Llama-3-70B requiring >140 GB FP16 capacity), vLLM integrates distributed execution primitives across multi-node, multi-GPU clusters using High-Performance Computing communication frameworks (NCCL).

Tensor Parallelism (Megatron-LM Style)

Tensor Parallelism (TP) splits weight matrices within individual Transformer layers across multiple GPUs operating within the same NVLink domain. Matrix multiplications are partitioned column-wise for weight projections ($W_q, W_k, W_v$) and row-wise for output projections ($W_o$):

Column-Parallel (Q, K, V Projections):
[ X ] x [ W_1 | W_2 ] = [ X*W_1 | X*W_2 ]

Row-Parallel (Output Projection):
[ Y_1 | Y_2 ] x [ W_1_out ] = [ Y_1*W_1_out + Y_2*W_2_out ] (Requires All-Reduce)
                [ W_2_out ]

Each TP worker GPU holds a distinct slice of the model's total attention heads. Consequently, the KV-cache physical blocks on each GPU store only the KV head slices assigned to that worker rank. PagedAttention physical block allocations are synchronized across TP ranks using deterministic block management, ensuring all worker GPUs allocate physical block IDs in identical sequence alignments without needing runtime cross-GPU manager coordination communication.

Pipeline Parallelism and Async Execution Layers

When scaling across nodes where inter-GPU connections are limited to PCIe or standard InfiniBand Ethernet networks, Pipeline Parallelism (PP) groups sequential blocks of layers onto distinct GPU nodes. Inter-node layer transitions require point-to-point (P2P) transfers via NCCL primitives (ncclSend / ncclRecv).

vLLM combines TP and PP using asynchronous Python host engine drivers backed by Rust/C++ execution layers, executing Ray worker processes or native PyTorch distributed process groups to overlap GPU kernel execution streams with IPC metadata synchronization.

Code Architecture: Custom Block Manager and Scheduler Implementation

The following structural Python implementation demonstrates the algorithmic core of vLLM's Block Manager, illustrating how logical token sequences translate to physical memory allocation, reference counting, and Copy-on-Write branching logic.

import math
from typing import Dict, List, Optional, Set

class PhysicalTokenBlock:
    def __init__(self, block_number: int, block_size: int):
        self.block_number: int = block_number
        self.block_size: int = block_size
        self.ref_count: int = 0
        self.is_allocated: bool = False

class BlockAllocator:
    """Manages physical VRAM GPU blocks pool."""
    def __init__(self, num_gpu_blocks: int, block_size: int):
        self.block_size = block_size
        self.free_blocks: List[PhysicalTokenBlock] = [
            PhysicalTokenBlock(i, block_size) for i in range(num_gpu_blocks)
        ]
        self.used_blocks: Set[int] = set()

    def allocate(self) -> PhysicalTokenBlock:
        if not self.free_blocks:
            raise MemoryError("VRAM Out-Of-Memory: No physical GPU blocks available")
        block = self.free_blocks.pop(0)
        block.ref_count = 1
        block.is_allocated = True
        self.used_blocks.add(block.block_number)
        return block

    def free(self, block: PhysicalTokenBlock) -> None:
        block.ref_count -= 1
        if block.ref_count == 0:
            block.is_allocated = False
            self.used_blocks.remove(block.block_number)
            self.free_blocks.append(block)

class LogicalSequence:
    def __init__(self, seq_id: int, block_size: int):
        self.seq_id: int = seq_id
        self.block_size: int = block_size
        self.logical_token_count: int = 0
        self.block_table: List[PhysicalTokenBlock] = []

class PagedAttentionBlockManager:
    """Handles Sequence-to-Block translation and Copy-On-Write logic."""
    def __init__(self, num_gpu_blocks: int, block_size: int):
        self.block_size = block_size
        self.allocator = BlockAllocator(num_gpu_blocks, block_size)

    def allocate_token_slot(self, seq: LogicalSequence) -> None:
        # Check if new physical block is required
        if seq.logical_token_count % self.block_size == 0:
            new_block = self.allocator.allocate()
            seq.block_table.append(new_block)
        else:
            # Handle Copy-on-Write (CoW) if block is shared
            last_block = seq.block_table[-1]
            if last_block.ref_count > 1:
                # Branch/Fork detected: duplicate shared block before write
                new_block = self.allocator.allocate()
                last_block.ref_count -= 1
                seq.block_table[-1] = new_block
                
        seq.logical_token_count += 1

    def fork_sequence(self, parent_seq: LogicalSequence, child_seq: LogicalSequence) -> None:
        """Zero-copy fork for Beam Search or Parallel Sampling."""
        child_seq.block_table = list(parent_seq.block_table)
        child_seq.logical_token_count = parent_seq.logical_token_count
        for block in child_seq.block_table:
            block.ref_count += 1

    def free_sequence(self, seq: LogicalSequence) -> None:
        for block in seq.block_table:
            self.allocator.free(block)
        seq.block_table.clear()

Performance Benchmarks, Trade-offs, and Infrastructure Realities

While vLLM radically changes GPU utilization profiles for inference infrastructure, deploying it effectively in high-concurrency production environments requires understanding structural trade-offs.

Block Size Optimization Trade-offs

Choosing the physical block size ($B_{\text{size}}$) presents a trade-off between memory overhead and CUDA execution efficiency:

Small Block Size ($B_{\text{size}} = 8$): Reduces internal fragmentation to trivial levels. However, it significantly increases the total length of the Block Table vector passed to custom CUDA kernels, increasing CPU-side overhead and GPU shared-memory translation latencies.
Large Block Size ($B_{\text{size}} = 32$ or $64$): Optimizes memory bus alignment and enables wider memory vector fetches in custom CUDA kernels. However, it increases internal memory fragmentation on short sequence completion rates. In general benchmark suites, $16$ or $32$ serves as the optimal sweet spot for modern Ampere, Hopper, and Blackwell GPU architectures.

Production Optimization Matrix

The following deployment parameter configuration matrix summarizes optimal production runtime tunings based on target SLA characteristics:

1. Maximum Throughput Priority (Batch Processing / Offline Pipelines)

--gpu-memory-utilization 0.95: Maximize allocation footprint for physical blocks.
--max-num-seqs 256 / 512: Allows massive concurrent decoding pools.
--enable-chunked-prefill true: Prevents large prefill steps from starving active decode threads.

2. Low-Latency Target Priority (Interactive Chat / Real-time APIs)

--max-num-seqs 32 / 64: Limits queue size to protect Time-To-First-Token (TTFT) and Inter-Token Latency (ITL).
--swap-space 16: Keep CPU swapping limits constrained to reduce host-device communication latency spikes.
--kv-cache-dtype fp8: Reduces KV-cache bandwidth requirements by half, effectively doubling context capacity at a minor loss in precision.

By transforming raw VRAM from contiguous static buffers into dynamic virtual memory pages, vLLM converts idle memory capacity directly into compute density. This shift elevates modern LLM inference platforms from bottlenecked, single-tenant scripts into high-density, multi-tenant distributed systems.