AI Receptionist Build
The stack, the prompt structure, and the latency budget for a phone agent that books appointments in Kannada, Hindi and English. Written for someone who intends to build it, not admire it.
Five pieces, one loop
What does an AI receptionist actually consist of? Five pieces, and every turn of the conversation runs through all of them. The whole loop has to finish before the caller notices silence — which is the entire engineering problem.
Port your existing number rather than issuing a new one. A business that advertises one number for ten years should not change it because of an automation project.
What goes in each slot
The categories are stable. The products inside them are not, so treat the following as a shortlist to evaluate, current as of . It is not a ranking and it is not a recommendation.
- Telephony. Twilio and Plivo are the usual programmable-voice choices internationally. For an Indian landline or virtual number, Exotel, Knowlarity and Ozonetel are the established local options. What matters is whether the provider will stream raw audio over a socket rather than only playing files.
- Speech to text. Deepgram and Google Cloud Speech-to-Text both offer streaming recognition. Whisper and its derivatives can be self-hosted. For Indian languages specifically, look at what the Indian labs are shipping — AI4Bharat and Sarvam AI both work on Indic speech — and check current language coverage yourself, because it moves.
- The model. Any of the current general-purpose families will hold a booking conversation: OpenAI's GPT, Anthropic's Claude, Google's Gemini. Pick on first-token latency and on how reliably it calls a tool, not on reasoning benchmarks. This job is not hard reasoning.
- Text to speech. ElevenLabs, Cartesia and Google Cloud Text-to-Speech all stream audio as it generates. The binding constraint is not naturalness, it is which languages the voice you like actually speaks.
- Orchestration. You can wire the four together yourself, or use a layer built for it — Vapi, Retell AI, LiveKit Agents and Pipecat all exist for this. Assembling it yourself takes longer and teaches you where the latency lives, which you will need to know either way.
Verify every name on that list against its own documentation before you commit. A stack list is a snapshot, and this one will be wrong eventually.
The budget you are working against
How fast does a phone agent have to reply? On a call, silence reads as a dropped connection. People hang up at around a second and a half of nothing. That is the real deadline — not a benchmark, a human reflex.
| Stage | Target budget | What blows it |
|---|---|---|
| Speech to text | ~300 ms | Batch instead of streaming. Waiting for end-of-utterance. |
| Model first token | ~400 ms | A long system prompt. A big model where a small one closes. |
| Speech first audio | ~300 ms | Generating the whole sentence before speaking any of it. |
| Network and overhead | ~200 ms | A region far from the caller. Two hops where one would do. |
| Total | under 1.2 s | Anything above ~1.5 s and callers start talking over it. |
This is the budget to design against and measure yourself, on your own numbers and your own callers. Anyone quoting you platform latencies without saying which region, which codec and which language they tested on is quoting a marketing page.
Four blocks, in this order
Identity and boundary
Who it is, which business, and — stated plainly — what it must not do. No pricing it hasn't been given. No promises about availability it can't check. Callers find the edge of that boundary quickly.
The single objective
One goal: get a name, a reason, and a booked slot. An agent with three goals achieves none on a two-minute call.
Language rule
Name the three languages explicitly — Kannada, Hindi and English — and tell it to mirror whichever the caller opens in, switching mid-sentence if they do. Do not announce the switch. Kannada-English and Hindi-English mixing are not fallbacks here, they are the common case — write them as the expected behaviour, not the exception.
Escape hatch
The condition for handing to a human, stated as a rule the model can actually evaluate. Two failed understandings, or any mention of a complaint. An agent that cannot give up is worse than no agent.
Keep the whole prompt short. Every token in the system prompt is time the caller spends listening to nothing.
Three languages on one number
How do you handle Kannada, Hindi and English on the same line? The first decision is architectural, and you make it before you write a prompt: one multilingual pipeline for all three, or a language-detection step in front of three per-language pipelines.
Per-language pipelines usually transcribe their own language better, but they need to know the language before they can start, and a sentence that changes language halfway is exactly what defeats that. A single multilingual pipeline gives up some accuracy per language and survives the switch. For a Bengaluru number, where a caller can open in Kannada, give a date in English and finish in Hindi, the switch is the common event, so the single pipeline is usually the right trade. Decide it deliberately either way.
Then four details, each of which will bite you:
- The transcript needs one script. Kannada arriving as ಕನ್ನಡ script in one turn and romanised in the next gives the model two different-looking inputs for the same words. Pick one representation and normalise into it.
- The voice has to speak all three. Text-to-speech voices are frequently English-only or Hindi-only. A Kannada reply rendered by an English voice is worse than answering in English would have been.
- Numbers, dates and times are where it breaks first. A slot confirmed in Kannada has to be spoken as Kannada numerals and Kannada weekday names, not English ones dropped into a Kannada sentence.
- Do not ask the caller to choose a language. "Press 1 for Kannada" is an IVR menu, and the IVR is the thing you are replacing. The agent should infer it from the first sentence and be wrong occasionally rather than make everyone navigate a menu.
Where these break in production
- Code-switching mid-sentence. "Tuesday chalega?" is one sentence in two languages. Transcription trained mostly on single-language audio drops one of them. Test this before anything else — it is the difference between a demo and a deployment in India.
- Kannada-English is not Hinglish. "Tuesday agutta?" is the Bengaluru version of the same sentence, and a stack that handles Hindi-English may not handle it — there is less Kannada audio behind most speech models than there is Hindi. Test the two separately. Passing one tells you nothing about the other.
- Names. A model trained mainly on Western speech will mangle Indian names. Have it spell back and confirm rather than guess.
- Background noise. Real calls come from streets and workshops, not studios. Test on a speakerphone in traffic, not at your desk.
- The double-booking race. Two callers, same slot, same moment. The calendar write must be atomic or you will hand two people the same appointment and find out from them.
- Silence handling. Callers pause to think. If the agent fills every gap it will talk over people constantly. Wait longer than feels right.
Record your own calls with it for a week and listen to every one. Not a sample — all of them. The failures are never where you expect, and the first ten calls will teach you more than any amount of prompt tuning.
Send me how many calls a week you miss after hours. That number decides whether this is worth building at all — sometimes it honestly isn't, and I'll say so. House of Mohny works out of Bengaluru.
@houseofmohny on Instagram →