How we cut AI call costs by 60% with TinyML

When we launched Callbook AI, the goal was clear: AI calls that sounded human. We soon found a silent source of rapidly increasing costs: voicemail. This is the story behind that problem, the cost structure of an AI call, the limitations of conventional detection in Latin America, and the TinyML system we built to make every campaign more efficient.

1. Our largest early challenge: voicemail

By mid-2024, our first customers were making dozens of calls each day. A typical outbound campaign reached about 30% of dialed numbers. The remaining 70% ended in a hang-up or voicemail. One finding stood out:

More than half of those unsuccessful calls, about 50%, reached voicemail.

Each time this happened, the platform ran the entire AI flow: telephony connection, transcription, language-model analysis, and voice generation. The system incurred nearly all of the operating cost without speaking to a person. Monthly bills grew quickly, and customers began questioning the return on their campaigns.

2. The cost structure of an AI call

Each component uses a different billing model:

1. Telephony providers such as Twilio, Vonage, and Zadarma

• Billing is based on seconds or full minutes.
• Some plans bill a two-second call as a full 60 seconds.

1. Speech to text (STT)

• The service processes and transcribes the captured audio stream.
• Billing is based on minutes of transcribed audio.

1. Large language model (LLM)

• The model analyzes the transcript and selects the next action.
• Pricing varies by model, token use, and latency requirements.

1. Text to speech (TTS)

• The service produces the spoken response.
• Billing is based on generated audio.

A typical voicemail path

Call connects → telephony bills the minute → STT transcribes ten seconds of a recorded message → the LLM identifies voicemail → TTS says goodbye → the call ends.

Every step consumes billable resources, which makes a voicemail interaction cost nearly as much as a successful call.

3. Why Twilio AMD struggled in Latin America

Twilio provides Answering Machine Detection (AMD). Its performance reflected three regional challenges:

• Models trained primarily for English-speaking markets, where voicemail tone, beep duration, and pause patterns are relatively consistent.
• Greater variation across Latin America, including multiple languages, music, several beeps, and custom recordings from regional carriers.
• Detection errors and long detection times allowed voicemail audio to enter the complete AI pipeline.

A common workaround in AI calling platforms is a prompt such as:

“If you detect voicemail, end the call and say goodbye.”

At that point the LLM is already running and consuming inference time. TTS then generates the final message. The customer still pays for both services.

4. TinyML processing at the edge

We designed a tiny machine learning detector that runs locally during the first second of every call. It classifies the audio before the cloud-based speech and language pipeline starts.

5. Building the detector with TinyML and Edge Impulse

The following overview focuses on the essential process we used to teach the model to distinguish a voicemail recording from a person answering.

5.1 Collecting and organizing examples

• Real recordings: hundreds of voicemail clips from carriers in Colombia, Mexico, Argentina, and Chile.
• Natural variation: messages with different speech patterns, tones, and background music.
• Manual classification: each clip was reviewed and labeled as either a human answer or voicemail.

5.2 Audio signals used by the model

We analyze simple properties of the sound that can be calculated immediately:

• Pauses and beeps: recorded messages often contain regular cuts and fixed tones.
• Speech rhythm: human speech tends to be more irregular than a recorded carrier message.

5.3 A very small model

• The detector occupies only a few hundred kilobytes.
• It analyzes the first second of audio before any cloud processing and classifies the call as human or voicemail.
• A voicemail classification ends the call immediately.

5.4 Training and validation

• Training: 80% of examples taught the model, and 20% formed the validation set.
• Validation accuracy: more than 96% across the held-out examples.
• False negatives: undetected voicemail remained below 1.5% in the validation sample.

6. Production results and key metrics

After integrating the lightweight detector, the production evaluation showed:

• 91.4% voicemail recall: the model correctly identified about 91 out of every 100 calls that reached voicemail.
• 0.35 model loss: the training metric indicated a useful balance between learning the patterns and generalizing beyond individual examples.

Why do these numbers matter?

• Recall above 90% significantly reduces the voicemail calls that reach expensive cloud processing.
• Balanced model loss helps the detector handle new noise, carrier tones, and waiting audio beyond the training set.

In production, that efficiency produced:

A 60% reduction in spending associated with calls that reached voicemail.

Ending voicemail calls at the edge, before STT, LLM, and TTS processing, removed more than half of the unnecessary processing cost. Customers received lower campaign costs and a clearer return on their AI calling investment.