Benchmarking Voice Agents: Why Time-To-First-Token Isn't Enough

Time to first token (TTFT) is the metric teams use to pick an inference API for voice. [1]

It is also the metric that misleads them. [1]

TTFT marks when generation starts; a text-to-speech model cannot speak until a full clause arrives. [1]

Between those two points sits the difference between an agent that feels conversational and one that gets interrupted. [1]

A voice agent is a latency budget with a language model inside it. [1]

Every stage spends milliseconds the user can hear. [1] Tokens per second dictate how quickly the first sentence completes, making it a critical factor in perceived speed. [1] Consequently, experts suggest that time-to-first-sentence provides a more accurate measure of the actual user experience. [1]

LiveKit puts the practical end-to-end target at 700ms to 1.2s. [1]

The emphasis on TTFT is a holdover from text-based chatbots, where receiving the first word immediately improves perceived speed. However, voice AI requires a fundamentally different latency architecture. Because text-to-speech engines need syntactic context—at least a full clause—to generate natural prosody, the speed at which subsequent tokens arrive (tokens per second) is just as critical as the initial delay. Developers building realtime agents must prioritize time-to-first-sentence (TTFS) and optimize the entire stack, from speech-to-text to WebRTC transport, to stay within the 700ms to 1.2s budget. What the sources don't address: How new end-to-end native speech models, which bypass text generation entirely, might alter this latency math.