Nine States for One Phone Call

01Context

Narusec builds a Greek voice agent for call centers, and the first job we gave it is the least glamorous one in telephony: answer the phone and book an appointment. Managed voice platforms treat Greek as an afterthought, so we assembled the pipeline from parts. Soniox for the ears (the best Greek word error rate we measured: 7.4%), Gemini 3.1 Flash-Lite for the brain, Google's Gemini TTS for the voice, and a WebSocket in front that speaks the same media-stream shape VoIP providers use, so a browser mic and a phone line look identical to the server.

The test business is a fictional Athens barbershop. The agent greets you as ο ψηφιακός βοηθός της Ρούλας, checks a calendar, offers slots, books, and hangs up. The calendar behind it is deterministic and fake on purpose: every run costs real API money, so the only part allowed to be unpredictable is the part we're actually testing.

02Choosing Parts With a Stopwatch

Before the agent existed we built the thing that would pick its parts: a live eval harness where every candidate model answers the same scripted Greek booking scenarios behind the same mocked tools. Every run is a paid measurement against the real APIs, so the shortlist stayed deliberately narrow. Latency came first.

Raw speed nearly picked the wrong winner, and raw speed alone hides the shape of an answer. MiniMax reached its first token in a reasonable second, then kept talking for six: its answers averaged 467 characters where everyone else stayed near 100 or 200, and a phone call has no scrollbar. The grounded round, same scenarios but with the calendar tools mocked live, is where the real sorting happened: GPT-5.4 mini stayed fastest and made exactly zero tool calls across every run, answering availability questions without ever checking availability. DeepSeek hit its first token in 0.8s but needed 4.3s to finish a tool loop. Gemini 3.1 Flash-Lite took 1.2s to first token, called the calendar every single time, passed all four booking scenarios, and its share of a full scenario came to about a tenth of a cent.

03Architecture

The sentence buffer is why the agent starts talking before the model has finished thinking: the first complete sentence goes to TTS while the rest of the answer is still streaming in. That one decision buys most of the perceived speed.

04The State Machine

Our first version was an imperative loop: a 400ms setTimeout debounce, a pile of booleans, and a generation counter for throwing away stale work. Every new bug was a timing bug, and every fix was another flag. We threw it out and rebuilt the session as an XState machine with nine named states: idle, greeting, debouncing, generating, speaking, interrupting, tooling, ending, cleanup. The old loop is still in the tree, unused. The machine version has survived contact with real conversations; the loop didn't.

The playback estimate deserves a second look, because it's the load-bearing hack: the client never reports that audio finished playing. The server just adds up chunk durations into playback.until, and every question that matters (is the bot audible, is the turn over, is this caller barging in) becomes one comparison against the clock.

05Getting Interrupted

Not every sound is an interruption. A barge-in has to be longer than three characters, above 0.7 STT confidence, and it has to get past a regex whose entire job is to refuse μμμ and εεε as speech, because Greek callers backchannel constantly and the bot shouldn't stop mid-sentence for an agreeable hum. One moment is deliberately rude in the other direction: while the agent is reading back a booking confirmation, barge-in is disabled, so the one sentence that commits a slot is heard whole.

/** Greek filler/backchannel sounds that should never
    trigger barge-in or generation. */
export const FILLER_SOUND_RE = /^[μαεωχ]+$/i;

/** The model claiming a booking it never made. */
export const BOOKING_HALLUCINATION_RE =
  /(?:έκλεισε|σε έβαλα|σε βάζω|επιβεβαιώθηκε|κλείστηκε|τα λέμε.*τότε)/i;

When a real interruption lands, the machine doesn't just cut the audio. It goes back into the chat history and rewrites its own last turn down to the sentences that actually made it out of the speaker, then appends a note telling the model where it was cut off. The next answer starts from what the caller heard, not from what the model said.