A practical prompt architecture for production voice agents.

Everyone has a prompt. Almost nobody has an architecture. The difference shows up by the third call. A clever prompt demos beautifully, survives call one, wobbles on call two, and dies on call three, the moment a real caller does something nobody scripted.

I learned this at volume. At iExcel I run a mass-tort reply engine, an AI voice system that calls back people who asked a law firm for help with an injury claim. It has logged 50,000 calls. I already published the results from those 50,000 calls. This article is the how: the structure that holds when the volume arrives.

A prompt is words. An architecture is words plus memory plus facts plus fallbacks plus guardrails, each in its own layer, each with exactly one job. Build the layers and the agent holds up under load. Skip them and you own a demo.

Voice gives you about one second

Voice is the hardest surface in AI. Harder than chat, harder than email, harder than anything with a screen. The reason is simple. A human on the phone gives you about one second of silence before something feels wrong. After that, trust starts leaking, and on an outbound call trust is the entire product.

Look at what has to happen inside that second. The caller’s words become text. The model reads and decides. The reply becomes a voice. The audio travels back across the phone network. Four steps, one second, every single turn of the conversation.

Here is the part most builders miss. Every token the model writes is silence the caller hears. A token is a chunk of a word, and models produce them one at a time. On a screen nobody notices generation time, because the text just appears. On a phone, the caller sits inside every one of those tokens, waiting, deciding whether to hang up.

That is your latency budget, the amount of waiting you can spend before the caller decides the call is broken. The budget drives every design choice that follows. Small prompts, because the model reads faster when there is less to read. Short answers, because long ones take longer to speak. Facts fetched only at the moment they are needed. None of this is style. It is arithmetic.

Layer one: the system frame

The bottom layer is a tight system frame, the agent’s job description. Who it is. Who it works for. What it is allowed to do. What it may never do, stated in plain language, near the top.

Mine runs short, and short is the discipline. Every extra instruction is text the model must reread on every turn, and reading time is silence. Worse, models follow ten clear rules far better than a hundred vague ones. When I audit a broken agent, I usually find a system prompt the size of an employee handbook. Nobody follows a handbook under pressure. Not humans. Not models.

The hard limits are the part I write first. Never quote a settlement number. Never promise an outcome. Never keep talking past a request to stop. Each ban gets one short sentence, because a ban the model has to interpret is a ban it will eventually break.

The frame on my reply engine fits on a page. Identity, purpose, tone, hard limits, nothing else, because everything else belongs in a better layer. If your agent’s whole identity does not fit on a page, it does not have an identity. It has clutter.

Layer two: state, so nothing gets asked twice

State is everything the system already knows about this caller. Their name. The form they filled out. Which firm they contacted, when, and about what. What they said two turns ago, and what they said when we called last week.

This layer decides whether the call feels human. The people my engine calls already told a law firm what happened to them. If the agent opens with “Can I get your name?”, the caller hears something brutal: nobody read my file. Trust dies in the first sentence, and no model is charming enough to win it back.

So the state layer loads what we know before the model speaks a word, and the opening proves the homework got done. “Hi Maria, you reached out on Tuesday about your claim. Is now still a good time?” One sentence in, the caller knows this call is about them.

State lives outside the model, in a database, and gets written back after every turn. The model never has to remember anything. It only has to read. That sounds like a small distinction. At call 50,000 it is the difference between a system and a coin flip.

Layer three: retrieval, facts on demand

Retrieval means pulling facts from a knowledge base at the exact moment the conversation needs them. The opposite is the mega-prompt: paste the FAQ, the compliance manual, and every product detail into one giant prompt and hope.

The mega-prompt fails three ways. It is slow, because the model rereads all of it on every turn and the caller hears that delay as dead air. It is mushy, because the one rule that matters drowns in a pile of rules that might. And it goes stale, because the day a deadline or a phone number changes, somebody has to find it inside a wall of text and hope nothing else breaks.

The rule I build by: only facts, only when needed. A caller asks about a filing deadline, the system fetches that one answer, hands the model a clean paragraph, and the model speaks it. The prompt stays small and fast. The facts stay current, because you update a database instead of performing surgery on a prompt.

Retrieval also keeps the agent honest. When the answer is not in the knowledge base, the agent says so and offers the next step, instead of inventing something that sounds confident.

Layer four: response shaping

The last layer edits the model before the voice speaks, because voices are not screens. A bullet list read aloud sounds like a robot reading a receipt. A paragraph that looks tidy in a chat window becomes a seven-second monologue on the phone, and the budget from earlier never stopped applying.

Response shaping enforces hard rules. Short sentences. One question at a time, because callers answer the last thing they heard. No lists, no headings, no symbols. Numbers spoken the way people say them, so the voice says “five thousand dollars” instead of reading characters. And the agent yields: the moment the caller starts talking, it stops.

The shaping layer also writes for the ear in smaller ways. It front-loads the point, because callers remember openings. It confirms in fragments, the way people do. “Got it. Tuesday at ten.” Not a paragraph restating the whole appointment. The voice sounds human because the text was built to be spoken, not displayed.

These rules do not live in the model’s personality, because personality drifts from call to call. They live in a layer that checks the output before it becomes sound. Personality drifts. Layers hold.

Fallback ladders

Models stumble. Speech-to-text mishears a name. An API times out mid-call. A caller switches to Spanish, coughs through a sentence, or asks something nobody scripted. The question is not whether your agent will stumble at volume. It is what the caller experiences in the seconds after it does.

I build fallback ladders, and every rung is decided before launch. Rung one: ask again, with a scripted, human line, never an error message. “Sorry, you cut out for a second. Could you say that once more?” Rung two: stop improvising. The agent drops to a scripted recovery path that still moves the call forward, like confirming the callback number. Rung three: a warm handoff to a human, with the state layer passing along everything the caller already said, so nobody starts over.

The model is allowed to fail. The call is not.

One rule governs the whole ladder: the caller never hits a dead end. No loops. No third “I didn’t catch that.” No silence that just ends. Every confused path lands somewhere useful: a person, a scheduled callback, a clear next step. Callers never notice the ladder exists. That is exactly the point.

Guardrails that cannot be talked out of

The system frame says what the agent should do. Guardrails are the lines it cannot cross, even when a caller pushes, and even when the model talks itself into something stupid.

Three kinds matter on the phone. Disclosure first. My agents say they are an AI assistant at the top of the call. Several states require it, callers deserve it, and hiding it buys nothing, because people can tell anyway.

Compliance lines second. In regulated work there are sentences that must be said and sentences that must never be said. Those cannot live only in the prompt, where a persistent caller might argue the model out of them. They live in code that checks the words before the voice speaks. The prompt asks. The code enforces.

Escalation topics third. Some subjects go to a human every time, no matter how confident the model sounds. In my mass-tort engine the line is bright: anything that smells like legal advice goes to a person. The agent can confirm an appointment, collect facts, schedule a callback. The moment a caller asks whether they have a case, a human takes over. The agent is an intake assistant, not a lawyer, and the architecture remembers that even when the model forgets.

Test it like a pager is attached

I carried a pager at Google as a Senior Site Reliability Engineer, on systems that were not allowed to go down. That discipline is boring on purpose, and it is the reason my agents survive volume. I wrote the full argument in what SRE taught me about shipping AI. Voice is where it pays off hardest.

Three habits do most of the work. First, replay real calls. Before any prompt change ships, it runs against transcripts of past calls, the strange ones especially, and I compare the behavior line by line. A change that polishes the demo and breaks an old edge case gets caught on my desk, not on a caller.

Second, watch abandonment, the exact moment in each call where people hang up. Hang-ups cluster, and the cluster points at the broken layer the way an error log points at a bad deploy. When pauses grow, the cluster moves to the start of the call. When the agent repeats a question the state layer should have answered, the cluster moves to that question.

Third, every production stumble becomes a test. The call that broke the agent last week joins the replay set this week, so that failure can never ship twice.

This is how the architecture earns its keep. One afternoon a vendor’s speech service slowed down. The latency monitor flagged it, the shaping layer cut replies shorter, and the ladder moved borderline calls to humans until the vendor recovered. No heroics. The layers did their jobs, and callers heard a slightly brisker agent instead of dead air.

The operator’s checklist

You do not have to build any of this to benefit from it. You have to recognize it, because everyone now sells an “AI agent,” and a demo cannot show you layers. Ask a vendor these questions and listen for structure in the answers:

• What does the caller hear while the system thinks, and how long does it last?
• What does the agent already know about my customer when it dials, so nothing gets asked twice?
• Where do the facts live, and who updates them when a price or a deadline changes?
• Show me a call where the model failed. What did the caller hear next?
• Can a caller talk it past its compliance lines? Prove it.
• Which topics always go to a human, and how fast does the handoff happen?

A vendor with an architecture answers all six in plain language. A vendor with a prompt changes the subject.

If you build voice agents, steal this structure. None of it is secret. All of it is work. And if you own a business that needs this standing behind its phone number, that is the job I do. I’m the Fractional Chief of AI for businesses that know they’re behind. I take you from watching AI happen to running on it in 90 days. Book a strategy call.