OpenAI says its own models optimised Astra Ultrafast’s inference on Nvidia GPUs

OpenAI used its own models to tune the inference software that serves GPT-6 Astra Ultrafast on Nvidia hardware, two of its executives say on the record in a post Nvidia published on October 1. “We used our internal models to optimize inference on NVIDIA GPUs, and NVIDIA’s programmability helped us deliver the acceleration behind Astra Ultrafast,” says Uday Ruddarraju, chief technology officer of compute at OpenAI.

Ultrafast is served from Nvidia Blackwell GPUs, according to Nvidia.

Ultrafast is the fast service tier for GPT-6 Astra. Nvidia says it “offers up to 8x faster token generation than the Astra Standard mode”, runs on Nvidia Blackwell GPUs, and is “available now in the OpenAI API and to eligible ChatGPT Work and Codex users”.

The 8x is a vendor figure for one thing only

That number is Nvidia’s, published on Nvidia’s own blog by Dion Harris, and it covers token generation rather than the time a request takes from end to end. No benchmark, prompt, model size or batch configuration is given with it, Nvidia does not say which Blackwell part the comparison ran on, and the “up to” is doing the usual work. Nvidia’s artwork for the post shows a Standard run at 17.0 seconds against 6.4 seconds…