By Tobias Mann
Publication Date: 2026-05-03 13:05:00
AI adoption is reaching an inflection point as the focus shifts from training new models to serving them. For the AI startups vying for a slice of Nvidia’s pie, it’s now or never.
Compared to training, inference is a much more diverse workload, which presents an opportunity for chip startups to carve out a niche for themselves. Large batch inference requires a different mix of compute, memory, and bandwidth than an AI assistant or code agent.
Because of this, inference has become increasingly heterogeneous, certain aspects of which may be better suited to GPUs and other more specialized hardware.
Nvidia’s $20 billion acquihire of Groq back in December is a prime example. The startup’s SRAM-heavy chip architecture meant that, with enough of them, Groq’s LPUs could churn out tokens faster than any GPU. However, their limited compute capacity and aging chip tech meant they couldn’t scale all that efficiently.
Nvidia side stepped this problem by moving the compute heavy prefill bit of the inference pipeline to its GPUs while it kept the bandwidth-constrained decode operations on its shiny new LPUs.
This combination isn’t unique to Nvidia. The week after…