Many AI teams discover that their GPUs look “busy” in dashboards but are not actually generating much useful output. The main culprit is the Key-Value (KV) cache used during large language model (LLM) inference.

As context windows grow and multi-turn conversations become the norm, the KV-Cache quickly fills GPU memory (HBM). When that happens, the system starts evicting cache entries for less-active sessions. When those users return, the model must re-run the entire prefill phase to rebuild the KV-Cache, instead of focusing on generating new tokens. This re-computation tax directly reduces effective GPU utilization.

HPE Alletra Storage MP X10000 addresses this by acting as a high-performance, shared KV-Cache tier that sits behind GPU and system memory:

• Offloads KV-Cache from limited GPU/system RAM to scalable object storage.
• Uses RDMA and GPU Direct Storage to move KV data directly between storage and GPU memory, bypassing the host CPU and OS.
• Provides a global, shared KV-Cache that any inference node can access, supporting session mobility across GPUs.

In testing with Signal65 and Kamiwaza on real-world, agentic workloads, this approach delivered:

• Up to 21.5x reduction in Time to First Token (TTFT) versus no KV-Cache.
• Up to 19.4x higher output token generation rates versus no KV-Cache.
• 5.6x lower TTFT and 5.9x higher token throughput versus memory-only KV offload.

The net effect is that GPUs spend far more time generating output tokens and far less time re-processing prompts, improving both user experience and cost efficiency.

HPE Alletra Storage MP X10000 is designed as a shared, high-performance object storage layer that underpins multiple parts of an AI stack, with a particular focus on inference and KV-Cache.

In a typical modern LLM deployment, you might see:

• Prefill nodes using compute-optimized GPUs to process prompts in parallel and build the initial KV-Cache.
• Decode nodes optimized for memory bandwidth and low-latency token generation.
• HPE Alletra X10000 acting as a secondary KV-Cache tier and AI data platform, accessible from both prefill and decode nodes.

Key architectural roles of the X10000 include:

• KV-Cache offload: Stores large KV-Cache objects when GPU and system memory are full, effectively moving the memory limit from tens of GBs to petabyte-scale capacity.
• Linear scaling: A scale-out, disaggregated design where each controller can access all NVMe SSDs. As you add controller nodes, bandwidth and capacity scale linearly, supporting growth from hundreds to thousands of GPUs.
• High-speed connectivity: Supports 100 Gbps Ethernet with RDMA and delivers up to 20 GB/s read throughput per node for large RDMA-enabled reads.
• GPU Direct Storage (GDS): Uses NVIDIA’s cuObject library and RDMA to move data directly between X10000 and GPU memory, coordinated by NVIDIA BlueField-3 DPUs.

Beyond KV-Cache, the X10000 also supports:

• RAG pipelines: Optional data intelligence nodes with NVIDIA L40S GPUs can generate vector embeddings on ingest, turning the system into an AI data platform for semantic search.
• Lifecycle management: Object storage semantics allow you to set policies (for example, a 7-day TTL on KV-Cache buckets) so transient data is automatically expired.
• Global namespace: A unified object store that any inference node can access, enabling session mobility and reuse of KV-Cache across the cluster.

In short, the X10000 sits between your GPUs and your long-term data, reshaping how state (KV-Cache and embeddings) is stored and shared so that inference remains compute-bound rather than memory-bound.

Using HPE Alletra Storage MP X10000 as a KV-Cache tier is primarily about getting more useful work out of the GPUs you already own, rather than solving scaling challenges by simply buying more hardware.

Based on Signal65 and Kamiwaza’s testing with real-world, agentic workloads (up to 96 simultaneous users on an HPE inference server with 8 x NVIDIA H200 GPUs), the measured benefits include:

Performance and user experience

• Time to First Token (TTFT)
  • Up to 21.5x faster TTFT versus a system with no KV-Cache.
  • 5.6x faster TTFT versus memory-only KV-Cache offload.
• Output token generation rate
  • Up to 19.4x higher throughput versus no KV-Cache.
  • 5.9x higher throughput versus memory-only offload.
• Stability under load: Under heavy concurrency, TTFT tail latency stayed stable with X10000 offload, while baseline systems saw TTFT spikes as re-computation increased.

Infrastructure efficiency and TCO

• Higher effective GPU utilization: GPUs spend more time generating output tokens and less time re-running prefill for previously seen prompts.
• Better cost-per-token: A single X10000 cluster can support KV-Cache needs for hundreds of GPUs, often providing a more attractive cost profile than adding more high-end GPUs to compensate for memory limits.
• Reduced CPU overhead: HPE reports up to an 80% throughput improvement using RDMA for S3-compatible storage versus S3 over TCP, thanks to GPU Direct Storage and offload to NVIDIA BlueField-3 DPUs.

Operational advantages

• Automatic KV-Cache lifecycle management via object storage policies (for example, TTL-based cleanup).
• Global, shared KV-Cache across inference nodes, enabling session mobility and reuse.
• Security and compatibility: BlueField-3 DPUs provide line-rate encryption and can emulate block/file storage for legacy frameworks while still leveraging object performance.

For most teams, the business case centers on reimagining inference efficiency: instead of scaling GPUs to cover memory constraints, you use X10000 to keep the system compute-bound, increase concurrency per GPU, and improve the economics of each token generated.