Our voice agent worked perfectly. On some phones. On some networks. Some of the time.

At Vaya, a client talks to an AI intake agent before booking their first therapy session, so the therapist walks in with a real brief instead of just a name.

Our first version streamed audio over WebSockets. In testing it was fine. With real users it was a coin toss: one device worked, another didn’t. Home Wi-Fi was fine, mobile data wasn’t. The same person could have a smooth call one day and a broken one the next.

When we dug in, the pattern was clear. The WebSocket connection to our server kept dropping and stalling on real-world networks. A WebSocket runs over TCP, which delivers every byte in order. On a lossy mobile link, one lost packet holds up everything behind it until it’s resent. That’s the right behaviour for a file download and the wrong one for a live voice, where late audio is as bad as missing audio.

Even when the connection held, the design had deeper problems.

• ~2 seconds of dead air after every answer. Each stage (transcribe → think → speak) waited for the previous one to fully finish. Two seconds sounds small until you’re on the other end of it, wondering if anyone heard you.
• You couldn’t interrupt. We had to mute the mic while the agent spoke, or it would transcribe its own voice coming back through the speaker. So the client had to wait politely for the agent to finish every sentence.
• Pauses ended your turn. We ended a turn on silence. Someone stopping to find the right word got cut off.

In a mental health intake, people pause because the question is hard. We were punishing hesitation.

We moved to Pipecat (open source) with WebRTC, the technology video calls run on. Four things changed.

• Streaming end to end. Transcription, the LLM and the voice all stream, so the stages overlap instead of queueing. The agent starts speaking its first sentence while the rest is still being written.
• Real interruption. Clients can cut in, and the agent stops and listens.
• Echo cancellation, jitter buffering and packet-loss recovery, built in. WebRTC was designed for exactly this kind of network. On mobile data, that’s the difference between usable and not. Echo cancellation is also what let us un-mute the mic while the agent talks.
• A turn-detection model that listens to how a sentence trails off, not just the silence after it. Pauses no longer end your turn.

A WebSocket only dials out, which every network allows. WebRTC tries to build a direct two-way path between the two peers, and our hosting platform doesn’t accept inbound UDP at all. So no client could ever connect directly.

The fix is a TURN relay: a small server both sides dial out to, which forwards audio between them. Every single call goes through it.

We could have rented one. We self-host it instead, because it carries clinical audio, and we didn’t want another third party hearing someone describe their mental health.

Trains, tunnels, flaky networks. WebRTC recovers well from packet loss, but not from losing the connection completely. So if the connection drops, the client reconnects to the same interview. The conversation is saved turn by turn, and the agent picks up where it left off instead of asking their name again.

The old agent ended interviews by outputting the word DONE, which we stripped before speaking. That worked because we had the whole reply in hand before any of it was spoken.

With streaming, words go to the voice as they’re generated. There’s no longer a moment where the full reply exists and can be cleaned up. The agent literally said “done” out loud to clients.

Now it ends with a tool call that never reaches the speaker. The general lesson: once output is streamed, anything the model emits as text will be heard. Control signals belong in structured calls, not in the prose.

An LLM writes faster than a voice speaks. When a client interrupts, the agent has often generated a full question the client only heard half of, and the transcript counts it as asked. For an intake whose whole job is to give the therapist a complete brief, that’s a quiet gap.

So we’re adding recovery:

• Interrupted questions get flagged instead of being counted as asked.
• The agent handles what the client said first, then returns to the question at a natural moment.
• It re-asks at most once.
• If someone avoids a question twice, we don’t push. The therapist just sees it wasn’t discussed.

Voice AI is mostly not about the model. It’s the network, the timing, and the half-second moments where a conversation quietly goes wrong.