How Does AI Voice Tech Work in Realtime?

Knowing when to speak is actually a lot tougher for voice AI than you'd think - I mean, humans can hand off to each other in a split second, no problem. But previous voice AI systems just couldn't keep up with that rhythm, mainly because they relied on a turn-based architecture that used tiny models called turn detectors to decide when to jump in. It's a tough job, really - guess too soon and you cut the user off, guess too late and your response feels sluggish.

GPT-Live, OpenAI's latest voice system, does things differently - it lets the voice model listen and speak at the same time, which is a pretty big deal. This full-duplex capability means it doesn't need a separate detector, and conversations feel way more immediate and natural. Plus, when it needs to do some deeper thinking or use a tool, GPT-Live can consult other models without breaking the flow of the conversation.

The key to making all this work is a system architecture that's optimized for low latency - it's what lets the experience scale. Instead of the typical back-and-forth inference, the system just streams audio in and speech back out, handling any handoffs on a separate path. This setup is what powers a lot of the new features in ChatGPT Voice, like controlling your computer or coordinating with other agents in the desktop app.

To tap into this tech, devs can check out the OpenAI platform and tools like the Realtime API to build their own voice AI apps. By using GPT-Live and its low-latency architecture, they can create way more natural and responsive voice interactions - the kind that let users have really seamless conversations with AI systems.

Source: OpenAI