By Sean Michael Kerner
Publication Date: 2026-10-08 18:44:00
“I would say that’s the right thing,” Kar argued.
Kar noted that users pay based on tokens today. He named time to first token, tokens per second, tokens per dollar, and tokens per watt as the measures. Optimizing those measures requires networking, especially in clusters that scale to hundreds of thousands of accelerators.
There is, however, a bit of a gap with many networks today that makes it difficult to optimize for token operations. According to Kar, the main components of the gap are latency, bandwidth, and scale-up features in the silicon. On the software side, he said predictive analytics and telemetry keep machines running.