← Lab
Experiment·Voice & Multimodal AI·5 min read

Arabic & Darija Voice Agent

What it takes to run a speech-to-speech agent in a dialect that general models were not trained for.

Arabic & Darija Voice Agent

Dialects are not accents. Regional varieties of a language differ enough in vocabulary and structure that a model trained on the standard form performs poorly, and the failure lands on transcription — the first step, where errors propagate into everything after.

Transcription first, because everything depends on it

A generic model may sit near an 18% word error rate on dialect call audio. Roughly one word in five wrong breaks intent classification downstream, so no amount of prompt work rescues it. Tuning on real recordings from the actual domain can bring that to about 7%, which is workable.

Keep intent separate from generation

Routing a request is pattern matching; answering it is reasoning. A dedicated classifier over the transcript is faster, cheaper and more measurable than asking the language model to do both — and when it is wrong, you can see exactly where.

The latency budget decides the architecture

01Detect end of turnSilence does not reliably mean finished. This tuning affects perceived quality more than model choice.
02TranscribeDialect-tuned, measured on held-out real calls.
03Classify intentA small dedicated model, not the LLM.
04Generate and synthesiseStream synthesis from the first token so the reply starts before it is finished.
05Hand offBelow a confidence threshold, transfer to a person with the transcript attached.

Everything above has to fit inside roughly a second and a half, because past that people start talking over the system. Components that cannot meet their allowance get replaced rather than tuned.

What to take away
  • Dialect-tuned transcription is the precondition, not an optimisation.
  • Evaluate speech models on your own recordings, always.
  • Use a dedicated intent classifier rather than the language model.
  • Design backwards from the interruption threshold.

Field notes on building production AI systems — collected, verified and written up so they are useful to anyone working on the same problems.

smoke
“Innovation distinguishes between a leader and a follower.”

Need an AI system that holds up in production? Tell me what breaks today and I'll tell you what I'd build.

Call Me now !

mail

abdelmajidzaddi088@gmail.com

address

Mohammedia, Morocco

phone

+212 695 557 631