NVIDIA Releases NemotronLabs VoiceChat 11B: An Open Full-Duplex Speech-to-Speech Model with ~450 ms Turn-Taking and Live Tool Calling

NVIDIA has released NemotronLabs VoiceChat 11B, an open 11B end-to-end speech-to-speech model for real-time, full-duplex conversation. Instead of chaining ASR, an LLM, and TTS, it performs streaming speech understanding and speech generation in one unified network. That removes the multi-model orchestration and API handoffs a cascaded stack requires, and cuts end-to-end latency: measured smooth turn-taking latency is 448 ms on Full-Duplex-Bench 1.0. The model listens while it speaks, so a user can barge in mid-turn and the agent yields, with a take-over rate of 1.00 at 480 ms. It is also first open full-duplex model to support tool calling while conversation keeps flowing, using a separate output channel for scripts along with operator-defined “on-hold” lines that fill the gap while an API runs.

Is it deployable?

PARTIAL — deployable today for pilots, not for production. Weights and container are both public, and the license is permissive. But NVIDIA team states the checkpoint is ‘ready for research purposes only,’ and the repo documents real failure modes: a two-minute audio context ceiling, degradation into non-recoverable gibberish after several turns, runaway self-talk after a turn ends, and dropped words in user transcription.

• Which companies: any team that can allocate one GPU with at least 80 GB of VRAM — A100, H100, RTX 6000 Pro, or B200 on x86_64 Linux. That covers AI-native startups, funded scaleups, enterprise R&D and…