Agent in the middle

How we made an AI speech pipeline feel like one continuous thing

Partway through building voice mode, OpenAI released GPT-Live, a native speech-to-speech model. Audio in, audio out, one model, full duplex. It listens and answers the way a person does, one continuous thing, and we explored whether we could build on it.

A speech-to-speech model makes the voice model the agent. It can call a function, but everything our assistant is: tools, memory, skills, approval flows, would have to be rebuilt inside the provider’s runtime. And the model that speaks is the model that thinks: there’s no routing a hard question to a stronger brain. Maybe that changes someday. Today, for an assistant whose whole value is the agent loop, that trade kills the product before it ships.

So we kept the classic cascade, the one that’s always felt robotic: transcribe the speech, run the full agent loop, synthesize the reply. Three stages bolted together, and every joint between them is a seam you can hear as dead air. We spent the engineering on hiding those seams. The goal, written down early and returned to often, was to make voice mode feel natural.

How it started

Voice began as a feature of the existing chat surface. Press a button, talk instead of type, watch the same transcript fill in. It worked on the first try, and it felt wrong in a way that took a few days to name: watching both sides of the conversation render as text while you talk turns the conversation into a deposition. You stop talking to something and start reading a court record of yourself. The transcript was breaking it.

That observation ended voice as a feature of our chat interface. We built it a room instead. Start a conversation and the assistant grows to fill the screen: the walls take its color, its two cartoon eyes look back at you, a “Listening” label sits beneath them, and that’s the whole interface. The surface came together in days. It looked alive.

But it didn’t sound alive. Underneath the eyes, the interaction model was a walkie-talkie: you talk, it pauses, it thinks, eventually it talks back. Every silence reminded you there was machinery back there. The gap between how it looked and how it felt lived entirely in the turn-taking.

We could have shipped it that way. It worked, and the launch date was close. We decided to spend another week building, because voice gets one first impression. Someone who tries it once and feels the walkie-talkie closes the window, and no changelog entry brings them back to check whether it got better.

The pipeline

A single WebSocket carries the whole conversation between the browser and the server.

Each turn routes through the same agent loop that text chat uses, so voice inherits tools, memory, and skills for free. That’s the naive version, and the naive version feels mechanical. A handful of mechanisms make it feel alive.

Speculative launch

The longest pause in a cascade sits between the user’s last word and the model’s first token. We start early to hide it: the moment the voice detector hears trailing silence, the agent turn dispatches, before we’ve confirmed the person is actually done talking. There is no separate confirmation step to wait for: the same model call that begins the reply judges, from the wording, whether the person was actually finished. If it turns out they were only pausing mid-thought, the whole turn rolls back, persisted message included, and the listening window quietly extends. You pause, it waits. You finish, it’s already answering. This overlap is the single biggest latency win in the system.

The mouth and the brain

Every turn goes first to a fast, cheap “front” model, and the first tokens of its reply are a verdict.

Three verdicts are possible.

Simple turn: the front model just answers. Its output streams straight to text-to-speech, and first audio arrives fast because a small model is doing the talking.

Hard turn: the front model escalates. It says one short phrase out loud, usually naming what it’s about to do, like “let me check your calendar.” It then hands the turn to the strong model, which runs it in the background while that phrase plays. The mouth covers for the brain, so the caller never hears the big model’s think time as silence.

Not a turn at all: if the wording says the person isn’t finished, a trailing “and...” or a thought left hanging, the front model stays quiet and we keep listening.

The front model has no tools, but it knows what the strong model can do, so it hands off hard requests instead of refusing them or making things up.

Floor-holders

Even with the front model in place, there are stretches where nothing is being said: usually a turn running long on tool calls. The assistant fills those gaps out loud. Some of it is a simple acknowledgment, “one sec, let me look.” But on long turns it gets better than filler: the main agent bubbles up what it’s actually doing, and the front model speaks it. “Searched the web, reading through it now” is a live progress report on the work happening behind the voice.

The narration is audio-only. It never appears in the transcript and never persists to the conversation, so the record stays clean. And it’s timed off silence the user can actually hear: the server tracks when the last audio finishes playing on their end, not when it finished generating, so the quiet clock starts when the room goes quiet. Narration fires only when someone is genuinely sitting in silence, and never talks over the assistant’s own voice.

Gapless synthesis

The opening clause of a reply is synthesized the instant it arrives instead of waiting for a full sentence. A prefetch pump keeps the next segment’s synthesis hidden behind the current segment’s playback, and the client schedules each audio buffer against a running playhead cursor rather than waiting for the previous one to end. Chunks stitch together with no audible gaps.

Interrupting

Anyone who has used a voice assistant has tried to talk over it and been ignored. The assistant keeps talking, your words land on deaf code, and the only fix is to wait it out. Real conversation doesn’t work that way. You interrupt, the other person stops.

Ours stops. Cut in anywhere, over a word, mid-sentence, even during the thinking before any audio has played, and the assistant yields immediately and listens to the new thing you’re saying. Interrupting doesn’t feel like a feature you invoke. It feels like talking to someone with ears.

Getting there meant fighting the browser. Echo cancellation is half-duplex: while the assistant speaks, the microphone ducks your voice, so real interruptions arrive choppy while the assistant’s own audio bleeds through and registers as speech. Our first detector kept hearing itself and stopping mid-sentence. The guard we shipped is deliberately simple: wait for a quarter second of sustained speech before yielding, while tolerating the brief gaps the canceller chops into a real voice. Long enough to ignore its own echo, short enough that you never feel ignored.

But stopping is only half of it. What happens to the task it was doing?

Most systems throw it away. Interrupt a flight search to ask a side question, and the search dies with the sentence you cut off.

We hand it off instead. The unfinished work moves to a background agent that keeps running with the same tools and context while the assistant turns its attention to what you just said. So you get your side question answered, and the flight search never stopped. When it finishes, the answer folds back in: woven into the next reply if you’re mid-conversation, or spoken on its own once there’s a moment of silence.

The live conversation always has priority. Background work waits for its moment to speak. And if you tell the assistant to drop it, it drops.

Fail-open everywhere

Everything we added in the name of latency fails open. If the verdict doesn’t arrive in time, the turn releases anyway. If the escalation phrase comes back broken, a canned one plays instead. If an acknowledgment or progress update can’t be generated, silence wins, unless the silence itself has gone on too long, in which case a canned phrase is the lesser evil. None of it can break turn-taking, and none of it can wedge a call. The cleverness lives in the happy path, and the floor stays solid underneath it.

Putting it together

No single mechanism here is the trick. Speculative launch hides the endpoint decision, the front model hides the strong model’s think time, progress narration hides the tool calls, the prefetch pump hides the synthesis, and the handoff hides the cost of changing your mind. Each one covers a different seam, and the seams were the whole difference between a pipeline and a presence.

The cascade is all still in there, transcribing, looping, synthesizing. If we did our job, you’ll never hear it.