The economics of large language model serving have collided squarely with the physical boundaries of on-board GPU memory. While high-bandwidth memory (HBM) delivers the terabytes-per-second throughput necessary for compute intensive transformer layers, its physical capacity remains severely constrained by silicon packaging limits and fabrication yields. For workloads characterized by long document reasoning, deep agentic memory, or multi-turn conversational histories, the storage required for key-value (KV) caches rapidly outgrows available HBM, forcing operators into aggressive batch eviction or expensive distributed sharding.

The Beluga architecture addresses this fundamental imbalance by introducing Compute Express Link (CXL) switched fabrics to manage disaggregated KV-caches at rack scale. Rather than relying on traditional RDMA network protocols, which impose operating system overhead, buffer copies, and packet serialization latencies, CXL delivers byte-addressable, cache-coherent load and store semantics. GPUs can read and write to pooled memory modules through hardware switches with latencies measured in hundreds of nanoseconds rather than microseconds.

In a disaggregated CXL topology, active token generation retains immediate attention layers within local GPU memory, while historical KV-cache blocks are transparently offloaded to pooled CXL expansion sleds. When an autonomous agent references earlier context blocks or executes a multi-step backtracking maneuver, the inference engine streams the relevant cache segments back across the CXL fabric without stalling the execution pipeline.

Production evaluations show that this architecture slashes Time-To-First-Token (TTFT) metrics and significantly expands batch capacity across shared server clusters. By unlinking context retention from fixed accelerator memory ceilings, datacenter operators can maximize GPU utilization, preventing costly computational units from sitting idle due to localized memory exhaustion. The migration toward CXL pooled memory marks the beginning of true disaggregated infrastructure for enterprise artificial intelligence.