OpenAI Details GPT-Live Architecture for Continuous Stateful Voice

OpenAI designed GPT-Live by isolating media processing and model inference from tools, delegation, and persistence. [2] The live path exclusively runs the media pipeline and inference loop, while application logic operates behind an asynchronous remote procedure call (RPC) boundary. [1] This separation addresses the need for conversations to remain responsive even when external services or other operations introduce variable latency. [1]

By keeping application logic outside the latency-sensitive path, a tool call can wait on a database or API without blocking audio delivery. [2] Justin Uberti, Head of Realtime AI at OpenAI, stated that achieving target latency required consistent media delivery, leading to the architectural choice to restrict the live path. [1]

GPT-Live utilizes dedicated, stateful inference for each session on an assigned instance. [1] While sessions reserve capacity on their assigned instance, OpenAI can move an existing session's context to another instance when draining capacity or when a conversation reaches its context limit. [1][2]

For its media foundation, OpenAI retained WebRTC and introduced WebRTC Abridged Roundtrip Protocol (WARP) improvements and Instant Connect to reduce startup latency. [1] Before launching the system, OpenAI conducted a "silent" test that mirrored production voice traffic while discarding the output. [1][2] This testing method exposed load-related behavior and failures that synthetic tests had failed to identify. [1][2]

OpenAI's architectural breakdown of GPT-Live illustrates the technical requirements for delivering fluid, real-time voice AI at scale. By strictly decoupling the media and inference loops from application logic like database lookups or tool use, OpenAI addresses the core latency bottleneck that has historically plagued voice agents. The use of stateful session management with movable context further demonstrates a sophisticated approach to dynamic resource allocation. What the sources don't address: How much overhead the context migration process adds to overall system latency during peak traffic periods.