Time-to-first-token: why voice agents live or die on latency
A latency budget for real-time voice agents: the 200ms human turn-taking rule, where cloud pipelines spend 600ms–1.7s, and how on-device buys it back.
Ask anyone who has demoed a voice-enabled robot: the feature nobody writes on the spec sheet is the one that kills you. Not word error rate, not voice quality — the pause. The half-second-too-long gap between when a person stops talking and when the robot starts. Users don't say "the latency is high." They say "it's broken," and they stop talking to it.
This post is a latency budget for voice agent latency, built the way you'd actually engineer one: start from the number humans enforce, subtract what each stage costs, and see what's left. Spoiler: if there's a network round-trip in your critical path, there usually isn't anything left.
The 200ms rule humans enforce
Human turn-taking is brutally fast and remarkably universal. In a cross-linguistic study of 10 languages — from Japanese to Danish to Tzeltal — Stivers et al. (PNAS, 2009) found the mean gap between conversational turns is about 200 milliseconds, with every language's mean within 250ms of that. Two hundred milliseconds is roughly the minimum human reaction time to anything — sprinters responding to a starting pistol live in the same range.
That's the bar your users' brains are calibrated to. They will forgive a machine some slack, but not much: industry guidance converges on sub-second response feeling fluid and delays past one second reading as disruptive, with many teams treating ~500ms as the practical target and one second as the ceiling before users start bailing out of the interaction.
For a robot the bar is arguably higher, not lower. A voice assistant on a phone is an app; a humanoid or AMR standing in front of you is a social actor. A long pause from an embodied agent doesn't read as "loading" — it reads as ignoring you.
TTFT, TTFA, and what to actually measure
Two metrics get conflated constantly:
- Time-to-first-token (TTFT) — how long the language model takes to emit its first token after receiving the prompt. This is the LLM-serving metric, and it's the one model providers quote.
- Time-to-first-audio (TTFA) — how long from the user finishing speaking to the first audio frame leaving the speaker. This is the metric your user experiences.
TTFT is one term inside TTFA. The full accounting is:
TTFA = endpointing delay (deciding the user is done)
+ STT finalization (last partial → final transcript)
+ LLM TTFT (prompt in → first token out)
+ TTS first-frame latency (first sentence → first audio)
+ every network hop in between
Benchmark TTFA mic-stop to speaker-out, on the target hardware, over dozens of utterances — and look at p95, not the median. A voice agent that's snappy on average and occasionally hangs for three seconds is experienced as unreliable, which is worse than uniformly mediocre.
Where a cloud pipeline spends your budget
The standard cascaded cloud stack — streaming STT API, hosted LLM, TTS API — spends the budget like this, per published breakdowns from voice-infrastructure vendors (Twilio, Telnyx, Chanl):
| Stage | Typical cost |
|---|---|
| Speech-to-text (hosted) | 100–300ms |
| LLM inference (hosted, TTFT) | 350–1,000ms+ |
| Text-to-speech (hosted, first audio) | 90–200ms |
| Network round-trips between vendors | 50–200ms |
| Total TTFA | ~600ms–1.7s |
Three things make the real number worse than the table:
- The stages are serialized. Each one can't start until the previous one delivers, so the costs add — and so do their tails. Three stages each hitting their p95 simultaneously is rare; one of them hitting its p95 on any given turn is routine.
- The table assumes a good network. Those 50–200ms hop costs are datacenter-grade assumptions. Your robot is on warehouse Wi-Fi behind racks of steel shelving, or on LTE in a field. Radio conditions turn a 100ms hop into a 500ms hop without warning, and no amount of model optimization gets it back.
- You don't control any of it. Provider-side queuing, rate limits, regional routing — your worst-case latency is set by someone else's infrastructure. We wrote up exactly how this failed us in production before we moved on-device.
Do the arithmetic against the human bar and the conclusion falls out: a serialized cloud pipeline starts near the one-second ceiling on a good day. There is no budget left to spend.
How on-device buys the budget back
Moving the whole loop onto the robot — local STT, a small language model, local TTS — doesn't make any single stage free. What it does is delete the terms you can't engineer away and make the rest overlappable:
- The network terms go to zero. No hops between vendors, no radio variance, no provider queues. Your p95 becomes a property of your silicon and your scheduler.
- Stages can overlap aggressively. On one device with shared memory, streaming partial transcripts into the model and streaming first tokens into TTS is an architecture decision, not a cross-vendor integration project. The agent starts speaking while it's still thinking.
- Latency becomes deterministic. The same utterance costs roughly the same milliseconds every time. For an embodied agent, consistency is half of what makes the interaction feel alive.
The trade is that you now own the compute budget. On Jetson-class hardware that's a real engineering exercise — model sizes, quantization, GPU scheduling across STT and the SLM — and it's exactly the exercise we walk through, with numbers, in our offline voice assistant on Jetson Orin guide.
Engineering the last milliseconds
Once you're on-device, TTFA optimization is a short list of high-leverage moves:
- Tune endpointing first. The delay between the user's last word and your pipeline deciding they're done is pure dead time, and it's often 300ms+ of default VAD hangover. Aggressive endpointing plus graceful barge-in handling beats a faster model.
- Right-size the model. TTFT scales with model size; a well-prompted 3B SLM that answers in 300ms beats a 70B that answers in 3 seconds, for the short conversational turns a robot actually handles.
- Stream everything. Feed TTS the first clause, not the full response. First-sentence chunking routinely cuts perceived latency by hundreds of milliseconds.
- Measure on the target. Desktop-GPU numbers do not transfer to a 25W embedded module sharing memory bandwidth with a perception stack.
This integration layer — streaming orchestration, endpointing, barge-in, all in C++ with no glue-code seams — is the part of the stack that EdgeAI's voice AI platform exists to own, so you don't rebuild it around three separate open-source projects.
FAQ
What is a good latency for a voice agent?
Under ~500ms time-to-first-audio feels responsive; under one second is acceptable; beyond one second users perceive the interaction as broken. Human conversational turn gaps average around 200ms, which is the bar users are implicitly calibrated to.
What's the difference between TTFT and TTFA?
TTFT (time-to-first-token) measures only the language model — prompt in to first token out. TTFA (time-to-first-audio) measures what the user experiences: from end of speech to first audio out, including endpointing, STT, TTFT, TTS, and every network hop. Optimize and report TTFA.
Why is cloud voice latency so variable?
Because a cascaded cloud pipeline serializes three hosted services plus network hops, and each contributes independent tail latency. Radio conditions (Wi-Fi, LTE) add variance you cannot engineer away. On-device pipelines remove the network terms entirely, making latency deterministic.
Does on-device inference actually beat cloud TTFT?
For the model stage in isolation, a hosted frontier LLM on datacenter GPUs can produce a faster raw TTFT than a 3B model on an embedded module. But TTFA is what matters, and the cloud's network, queuing, and serialization costs typically dwarf the model-stage difference — while on-device stages can overlap in shared memory.
The takeaway
Voice agents live or die on a number humans set 200 milliseconds at a time, and a serialized cloud pipeline spends that budget before your model produces a single token. Put the loop on the device, overlap the stages, and tune endpointing — that's the whole game.
If you'd rather start from a stack where that work is already done: try EdgeAI free, skim the getting-started docs, or talk to us and bring your p95s.