Supermicro, IBM, and NVIDIA Cut Inference Latency 56x With KV Cache on Shared Storage

Analyst(s): Brendan Burke
Publication Date: November 6, 2026

Supermicro, IBM, and NVIDIA launched a jointly validated KV cache storage solution that persists inference context on shared flash instead of recomputing it on GPUs. Futurum examines what the underlying white paper proves about shared storage as a production memory tier for large-scale AI inference.

What Is Covered in This Article:

• Joint Supermicro, IBM, and NVIDIA KV cache storage solution launched November 4, 2026
• 56x faster time-to-first-token at 130K-token prompts versus GPU recompute
• 22x throughput gain from 0.19 to 4.26 requests per second across 200 concurrent requests
• White paper codification of NVIDIA Dynamo’s five-tier KV cache memory hierarchy, including the CMX flash appliance
• A noisy-neighbor stress test sustaining 3.6 requests per second against 200 GB/s of competing network traffic
• Competitive field spanning WEKA, VAST Data, DDN, Lightbits, and NVIDIA’s own pooled flash tier

The News: Supermicro…