Nvidia Ships the Foundation Model Physical AI Has Been Waiting For | PYMNTS.com

Nvidia launched Cosmos 3, an open world foundation model for physical AI trained on 20 trillion tokens of multimodal data — including nearly a billion images, 400 million real and synthetic videos, audio and action data from humans and robots.

The model targets machines that need to understand the physical world before acting in it. At the launch, Nvidia founder and CEO Jensen Huang said that “the big bang of physical AI is just around the corner thanks to breakthroughs in multimodal reasoning language, vision and world models.”

The distinction matters for anyone building or deploying physical AI. An LLM learns from text. A world foundation model learns from physical environments: how objects move, collide, fall and interact over time. What makes Cosmos different from a video generator, Axios noted, is the action data: Cosmos 3 doesn’t just generate realistic scenes, it predicts what a robot or vehicle should do next within them.

The Data Problem Physical AI Has to Solve

Training a chatbot on internet-scale text is expensive but tractable. Training a robot or autonomous vehicle on real-world physical experience is neither. A robot learning to handle objects needs millions of interaction examples. An autonomous vehicle needs exposure to rare and dangerous scenarios,…