In a voice agent, everything competes for the same budget: the moment between someone finishing their sentence and hearing a reply. Past roughly a second and a half, people start talking over the system. That constraint shapes every other decision.
The latency budget, component by component
01Voice activity detection ~210msDeciding the caller has stopped speaking.
02Speech to text ~310msTranscription of the turn.
03Intent classification ~180msA dedicated classifier, not the language model.
04Response generation ~390msThe only step most people think about.
05Speech synthesis first byte ~410msStream from the first token, not the last.
Generation is under a third of the budget. Optimising the model while ignoring the other four components is the most common way these projects stall.
Four things that are harder than they look
01Barge-inPeople interrupt. The agent must stop mid-sentence, discard queued audio, and treat what it heard as the new turn.
02Knowing when to stopSilence does not mean finished. Tuning end-of-turn detection tends to improve perceived quality more than any prompt change.
03Numbers and namesPolicy numbers, dates and amounts must be read back and confirmed. A misheard digit is worse than no answer.
04The handoffBelow a confidence threshold, or on any distress signal, transfer with full context attached — never a cold restart.
With those in place, resolving around 88% of calls without human intervention is a realistic target. Roughly 15% of the effort goes into the model; the rest is plumbing and latency.
