Real-Time AI Call Center: A Reference Architecture

In a call center, the moments that decide a call happen while it is still running: the customer hesitates, gets irritated, signals interest. Most speech analytics products record the call and analyze it afterwards. By then, the only thing left to do is write a report.

Over the past years I built a real-time AI call center platform that listens to calls as they happen, analyzes them in parallel and shows the agent concrete suggestions while the customer is still on the line. When we started, there was surprisingly little public material on how to structure such a system. Most of what exists are vendor pages, patents and forum threads.

This article is the reference architecture I would have liked to read at the start: the layers, the trade-offs in each one, and the decisions that matter more than the choice of model.

The goal: value while the customer is still on the line

It helps to separate three generations of AI in contact centers:

Recording and sampling. Calls are recorded and a quality team listens to a small sample. Most calls are never reviewed.
Post-call analytics. Every call is transcribed and analyzed in batch. Useful for trends, but the insight arrives after the call is over.
Real-time assist. The call is transcribed and analyzed as it happens, and the results reach the agent within seconds.

Everything below is about the third generation. Its defining constraint is latency: a correct suggestion that arrives three seconds late is useless, because the customer has already moved on.

The architecture at a glance

 Customer audio ─┐
                 ├─► Media layer (WebRTC / SIP bridge)
 Agent audio ────┘              │
                                ▼
                    Streaming speech-to-text
                    (partial + final segments)
                                │
                                ▼
                    Event stream of transcript segments
                                │   + audio features (tone, pace)
          ┌─────────────────────┼─────────────────────┐
          ▼                     ▼                     ▼
      Sentiment         Intent / behavior    Purchase propensity
          └─────────────────────┼─────────────────────┘
                                ▼
                    Recommendation engine ──► Agent screen (live)

 After the call: final transcript ──► Summary + report ──► CRM

Five layers, each with its own job:

Media layer: gets both sides of the conversation as separate audio streams.
Streaming speech-to-text: turns audio into text continuously, in small increments.
Parallel analysis: sentiment, intent and purchase propensity, computed side by side from the text and the audio itself.
Recommendation engine: turns analysis results into a small number of concrete, timely suggestions for the agent.
Post-call processing: writes the summary and the report, and hands them to the systems the team already uses.

1. Media layer: get two clean audio streams

Everything downstream depends on getting audio out of the call reliably, with as little delay as possible.

Browser-based calling (WebRTC). In our platform, calls run over a WebRTC-based communication layer. The agent talks from the browser, and the media server can fork each participant's audio directly into the analysis pipeline. This is the simplest path to real-time audio.
Existing telephony (SIP / PBX). Many contact centers already run on a PBX. Getting live audio out of those systems is often the hardest part of the whole project: you typically need a SIP-based recording interface or a media bridge that forwards the streams. Some hosted telephony platforms do not expose real-time audio at all. Find this out before anything else.
Keep the speakers separate. Capture the customer and the agent as two streams instead of one mixed recording. It removes the need to guess who said what (speaker diarization), which is slow and error-prone in real time, and it lets you analyze the customer's sentiment without the agent's voice mixed in.

2. Streaming speech-to-text

Speech recognition is the foundation: sentiment, intent and suggestions are only as good as the transcript underneath them.

Stream, don't batch. Send audio in short chunks and consume results as they arrive. Waiting for a sentence to finish adds seconds you cannot afford.
Partial versus final results. Streaming engines return quick partial hypotheses that change as more audio arrives, followed by a final version of each segment. Use partials for things that must feel instant (a live transcript), and finals for anything that triggers analysis or a suggestion. Otherwise suggestions flicker as the text changes.
Endpointing. How the engine decides that a phrase has ended directly affects latency. Tune it for conversational speech, which is full of short pauses.
Test on your own calls. Phone-quality audio, accents, product names and mixed-language terms all reduce accuracy. The platform's main language is Turkish, which is agglutinative and particularly hard for engines trained mostly on English. Vendor demos tell you little; a few hundred of your own recorded calls tell you a lot.

3. Parallel analysis

Each final transcript segment is published as an event. Independent workers consume the same stream and analyze it from different angles at the same time:

Sentiment. How the mood of the conversation develops, turn by turn. The text is only half the signal; tone, pace and volume carry the other half, so audio features are analyzed alongside the words.
Intent and behavior. Is the customer asking for information, objecting, comparing offers, about to leave?
Purchase propensity. An estimate of how likely the customer is to buy, based on what is said and how it is said.

Running these as separate consumers of one event stream has two advantages. They do not slow each other down, and you can add, replace or retrain one without touching the rest. A practical pattern is to let each worker keep a short rolling window of the conversation as context, instead of reprocessing the whole call on every segment.

4. The recommendation engine

This is where most of the product work happens. Raw analysis scores are not useful to an agent in the middle of a conversation; a short, well-timed hint is.

Few and specific. One suggestion that fits the moment beats five generic ones. Show the next best question, the answer to a specific objection, or the product that matches what the customer just said.
Thresholds and cooldowns. Only surface a suggestion when the underlying signal is strong enough, and avoid repeating the same hint every few seconds.
Trends over numbers. An agent cannot act on "sentiment 0.42". An arrow that shows the conversation getting worse is far more usable.
Measure acceptance. Track which suggestions agents actually use. It is the most honest quality metric you have, and the best training signal for improving the engine.

Results should reach the agent's screen over a persistent connection such as WebSocket, so nothing waits for a page refresh.

5. Post-call: summary, report, handover

When the call ends, the system has the complete, final transcript and every analysis result. This is the right moment for heavier processing:

A written summary of the call: the reason for the call, what was agreed, open points.
A structured report with the analysis results.
Handover to the systems the team already uses, such as the CRM.

This step is less visible than live suggestions, but it removes the after-call note-taking that agents otherwise do after every single call, and it makes every call documented in the same format.

Why the AI never talks to the customer

Voice bots that talk to customers directly are useful for short, rule-based tasks such as booking an appointment or checking an order status. For sales and relationship conversations we deliberately made a different choice: in our platform the AI never speaks to the customer.

The system estimates the customer's mood and purchase propensity in real time. That estimate is a valuable hint for an agent. If it were expressed directly to the customer, a wrong estimate would mean responding in the wrong tone. So the AI's voice stays on the agent's screen, and the person talking to the customer is always a human. It also makes the system much easier to introduce: agents see it as a tool that helps them, not as something that replaces them.

Latency budget

A useful way to design the system is to start from a budget for the whole path, from the customer finishing a phrase to a suggestion appearing on the agent's screen, and to give every layer a share of it:

Capture and transport: keep the audio path short; avoid extra hops between the media server and the recognizer.
Speech recognition: stream small chunks and tune endpointing.
Analysis: run workers in parallel, keep their context windows short, and prefer small, fast models for anything on the live path.
Delivery: push results over an open connection.

Heavy work, such as a large language model writing the full summary, belongs after the call, not on the live path.

Privacy and compliance

Call audio and transcripts are personal data, often in large volumes. Under GDPR, Turkey's KVKK and similar laws, a few decisions need to be made at the architecture level, not afterwards:

Inform callers at the start of the call that it is recorded and analyzed.
Redact sensitive data such as card numbers and national ID numbers from transcripts before they are stored or sent to any model.
Know where processing happens. If the speech or language model runs with a provider in another country, audio and transcripts are being transferred there. Decide this when you pick the provider.
Set retention periods for audio, transcripts and analysis results separately.

This is not legal advice; involve your data protection officer early.

Where to start

Solve audio access first. Confirm you can get two separate, real-time streams out of your telephony setup.
Measure recognition on your own calls. Pick the speech engine based on your data, not on demos.
Start with post-call summaries. They are low-risk, immediately useful and give you a labeled dataset of real calls.
Add live suggestions with a small pilot team. Their feedback will shape timing, thresholds and the UI more than any design document.
Track a few metrics from day one: average handle time, after-call work, first-contact resolution, conversion, and suggestion acceptance.

Conclusion

A real-time AI call center is less about a single clever model and more about a pipeline that stays fast end to end: clean audio, streaming recognition, parallel analysis and a recommendation engine that knows when to stay quiet. Built this way, it does not replace agents. It makes them better prepared during the call and frees them from paperwork after it.

You can see the platform itself on the project page. If you are planning something similar and want to talk through the architecture, have a look at AI and automation or get in touch.