The way we interact with AI is fundamentally changing. While text-based interfaces like ChatGPT have shown us what’s possible, in terms of interaction, it\u2019s only the beginning. Humans communicate not only by texting, but also talking \u2014 we show things, we interrupt and clarify in real-time. Voice AI brings these natural interaction patterns to our applications.<\/p>\n
Today, we’re excited to announce new capabilities that make it easier than ever to build real-time, voice-enabled AI applications on Cloudflare’s global network. These new features create a complete platform for developers building the next generation of conversational AI experiences or can function as building blocks for more advanced AI agents running across platforms.<\/p>\n
We’re launching:<\/p>\n
Cloudflare Realtime Agents – A runtime for orchestrating voice AI pipelines at the edge<\/p>\n
Pipe raw WebRTC audio as PCM in Workers – You can now connect WebRTC audio directly to your AI models or existing complex media pipelines already built on\u00a0<\/p>\n
Workers AI WebSocket support – Realtime AI inference with models like PipeCat’s smart-turn-v2<\/p>\n
Deepgram on Workers AI – Speech-to-text and text-to-speech running in over 330 cities worldwide<\/p>\n
Why realtime AI matters now <\/a><\/p>\n Today, building voice AI applications is hard. You need to coordinate multiple services such as speech-to-text, language models, text-to-speech while managing complex audio pipelines, handling interruptions, and keeping latency low enough for natural conversation.\u00a0<\/p>\n Building production voice AI requires orchestrating a complex symphony of technologies. You need low latency speech recognition, intelligent language models that understand context and can handle interruptions, natural-sounding voice synthesis, and all of this needs to happen in under 800 milliseconds \u2014 the threshold where conversation feels natural rather than stilted. This latency budget is unforgiving. Every millisecond counts: 40ms for microphone input, 300ms for transcription, 400ms for LLM inference, 150ms for text-to-speech. Any additional latency from poor infrastructure choices or distant servers transforms a delightful experience into a frustrating one.<\/p>\n That’s why we’re building real-time AI tools: we want to make real-time voice AI as easy to deploy as a static website. We’re also witnessing a critical inflection point where conversational AI moves from experimental demos to production-ready systems that can scale globally. If you\u2019re already a developer in the real-time AI ecosystem, we want to build the best building blocks for you to get the lowest latency by leveraging the 330+ datacenters Cloudflare has built.<\/p>\n Introducing Cloudflare Realtime Agents <\/a><\/p>\n Cloudflare Realtime Agents is a simple runtime for orchestrating voice AI pipelines that run on our global network, as close to your users as possible. Instead of managing complex infrastructure yourself, you can focus on building great conversational experiences.<\/p>\n When a user connects to your voice AI application, here’s what happens:<\/p>\n WebRTC connection – Audio streams from the user’s device is sent to the nearest Cloudflare location via WebRTC, using Cloudflare RealtimeKit mobile or web SDKs<\/p>\n AI pipeline orchestration – Your pre-configured pipeline runs: speech-to-text \u2192 LLM \u2192 text-to-speech, with support for interruption detection and turn-taking<\/p>\n Your configured runtime options\/callbacks\/tools run<\/p>\n Response delivery – Generated audio streams back to the user with minimal latency<\/p>\n The magic is in how we’ve designed this as composable building blocks. You’re not locked into a rigid pipeline \u2014 you can configure data flows, add tee and join operations, and control exactly how your AI agent behaves.<\/p>\n Take a look at the MyTextHandler function from the above diagram, for example. It\u2019s just a function that takes in text and returns text back, inserted after speech-to-text and before text-to-speech:<\/p>\n class MyTextHandler extends TextComponent { \tconstructor(env: Env) { \tasync onTranscript(text: string) { Your agent is a JavaScript class that extends RealtimeAgent, where you initialize a pipeline consisting of the various text-to-speech, speech-to-text, text-to-text and even speech-to-speech transformations.<\/p>\n export class MyAgent extends RealtimeAgent { \tasync init(agentId: string ,meetingId: string, authToken: string, workerUrl: string, accountId: string, apiToken: string) { \t\t\/\/ Construct a pipeline to take in meeting audio, transcribe it using \t\t\/\/ The RTK meeting object is accessible to us, so we can register handlers \t\t\/\/ Make sure to actually join the meeting after registering all handlers \tasync deinit() { View a full example in the developer docs<\/a> and get your own Realtime Agent running. View Realtime Agents<\/a> on your dashboard.<\/p>\n What makes Realtime Agents powerful is its flexibility:<\/p>\n Many AI provider options – Use the models on Workers AI, OpenAI, Anthropic, or any provider through AI Gateway<\/p>\n Multiple input\/output modes – Accept audio and\/or text and respond with audio and\/or text<\/p>\n Stateful coordination – Maintain context across the conversation without managing complex state yourself<\/p>\n Speed and flexibility – use RealtimeKit<\/a> to manage WebRTC sessions and UI for faster development, or for full control over your stack, you can also connect directly using any standard WebRTC client or raw WebSockets<\/p>\n Integrate with the Cloudflare Agents SDK<\/a><\/p>\n During the open beta starting today, Cloudflare Realtime Agents runtime is free to use and works with various AI models:<\/p>\n Speech and Audio: Integration with platforms like ElevenLabs and Deepgram.<\/p>\n LLM Inference: Flexible options to use large language models through Cloudflare Workers AI and AI Gateway, connect to third-party models like OpenAi, Gemini, Grok, Claude, or bring your own custom models.<\/p>\n Pipe raw WebRTC audio as PCM in Workers <\/a><\/p>\n For developers who need the most flexibility with their applications beyond Realtime Agents, we’re exposing the raw WebRTC audio pipeline directly to Workers.\u00a0<\/p>\n WebRTC audio in Workers works by leveraging Cloudflare\u2019s Realtime SFU, which converts WebRTC audio in Opus codec to PCM and streams it to any WebSocket endpoint you specify. This means you can use Workers to implement:<\/p>\n Live transcription – Stream audio from a video call directly to a transcription service<\/p>\n Custom AI pipelines – Send audio to AI models without setting up complex infrastructure<\/p>\n Recording and processing – Save, audit, or analyze audio streams in real-time<\/p>\n WebSockets vs WebRTC for voice AI <\/a><\/p>\n WebSockets and WebRTC can handle audio for AI services, but they work best in different situations. WebSockets are perfect for server-to-server communication and work fine when you don’t need super-fast responses, making them great for testing and experimenting. However, if you’re building an app where users need real-time conversations with low delay, WebRTC is the better choice.<\/p>\n WebRTC has several advantages that make it superior for live audio streaming. It uses UDP instead of TCP, which prevents audio delays caused by lost packets holding up the entire stream (head of line blocking<\/a> is a common topic discussed on this blog). The Opus audio codec in WebRTC automatically adjusts to network conditions and can handle packet loss gracefully. WebRTC also includes built-in features like echo cancellation and noise reduction that WebSockets would require you to build separately.\u00a0<\/p>\n With this feature, you can use WebRTC for client to server communication and leveraging Cloudflare to convert to familiar WebSockets for server-to-server communication and backend processing.<\/p>\n The power of Workers + WebRTC <\/a><\/p>\n When WebRTC audio gets converted to WebSockets, you get PCM audio at the original sample rate, and from there, you can run any task in and out of the Cloudflare developer platform:<\/p>\n Resample audio and send to different AI providers<\/p>\n Run WebAssembly-based audio processing<\/p>\n Build complex applications with Durable Objects<\/a>, Alarms<\/a> and other Workers primitives<\/p>\n Deploy containerized processing pipelines with Workers Containers<\/a><\/p>\n The WebSocket works bidirectionally, so data sent back on the WebSocket becomes available as a WebRTC track on the Realtime SFU, ready to be consumed within WebRTC.<\/p>\n To illustrate this setup, we\u2019ve made a simple WebRTC application demo<\/a> that uses the ElevenLabs API for\u00a0 text-to-speech.<\/p>\n Visit the Realtime SFU developer docs<\/a> on how to get started.<\/p>\n Realtime AI inference with WebSockets <\/a><\/p>\n WebSockets provide the backbone of real-time AI pipelines because it is a low-latency, bidirectional primitive with ubiquitous support in developer tooling, especially for server to server communication. Although HTTP works great for many use cases like chat or batch inference, real-time voice AI needs persistent, low-latency connections when talking to AI inference servers. To support your real-time AI workloads, Workers AI now supports WebSocket connections in select models.<\/p>\n Launching with PipeCat SmartTurn V2 <\/a><\/p>\n The first model with WebSocket support is PipeCat’s smart-turn-v2<\/a> turn detection model \u2014 a critical component for natural conversation. Turn detection models determine when a speaker has finished talking and it’s appropriate for the AI to respond. Getting this right is the difference between an AI that constantly interrupts and one that feels natural to talk to.<\/p>\n Below is an example on how to call smart-turn-v2 running on Workers AI.<\/p>\n “”” import asyncio # Configuration # WebSocket endpoint async def run_inference(audio_data: bytes) -> dict: response = await websocket.recv() # Response format: {‘is_complete’: True, ‘probability’: 0.87} def generate_test_audio(): return noise<\/p>\n async def demonstrate_inference(): try: except Exception as e: if __name__ == “__main__”: On Wednesday, we announced that Deepgram’s speech-to-text and text-to-speech models are available on Workers AI, running in Cloudflare locations worldwide. This means:<\/p>\n Lower latency – Speech recognition happens at the edge, close to users running in the same network as Workers<\/p>\n WebRTC audio processing without leaving the Cloudflare network<\/p>\n State-of-the-art audio ML models powerful, capable, and fast audio models, available directly through Workers AI<\/p>\n Global scale – leverages Cloudflare\u2019s global network in 330+ cities automatically<\/p>\n Deepgram is a popular choice for voice AI applications. By building your voice AI systems on the Cloudflare platform, you get access to powerful models and the lowest latency infrastructure to give your application a natural, responsive experience.<\/p>\n Interested in other realtime AI models running on Cloudflare? <\/a><\/p>\n If you’re developing AI models for real-time applications, we want to run them on Cloudflare’s network. Whether you have proprietary models or need ultra-low latency inference at scale with open source models reach out to us.<\/p>\n All of these features are available now:<\/p>\n
\n <\/p>\n
\n <\/p>\n
\n\tenv: Env;<\/p>\n
\n\t\tsuper();
\n\t\tthis.env = env;
\n\t}<\/p>\n
\n\t\tconst { response } = await this.env.AI.run(‘@cf\/meta\/llama-3.1-8b-instruct’, {
\n\t\t\tprompt: “You are a wikipedia bot, answer the user query:” + text,
\n\t\t});
\n\t\tthis.speak(response!);
\n\t}
\n}<\/p>\n
\n\tconstructor(ctx: DurableObjectState, env: Env) {
\n\t\tsuper(ctx, env);
\n\t}<\/p>\n
\n\t\t\/\/ Construct your text processor for generating responses to text
\n\t\tconst textHandler = new MyTextHandler(this.env);
\n\t\t\/\/ Construct a Meeting object to join the RTK meeting
\n\t\tconst transport = new RealtimeKitTransport(meetingId, authToken, [
\n\t\t\t{
\n\t\t\t\tmedia_kind: ‘audio’,
\n\t\t\t\tstream_kind: ‘microphone’,
\n\t\t\t},
\n\t\t]);
\n\t\tconst { meeting } = transport;<\/p>\n
\n\t\t\/\/ Deepgram, and pass our generated responses through ElevenLabs to
\n\t\t\/\/ be spoken in the meeting
\n\t\tawait this.initPipeline(
\n\t\t\t[transport, new DeepgramSTT(this.env.DEEPGRAM_API_KEY), textHandler, new ElevenLabsTTS(this.env.ELEVENLABS_API_KEY), transport],
\n\t\t\tagentId,
\n\t\t\tworkerUrl,
\n\t\t\taccountId,
\n\t\t\tapiToken,
\n\t\t);<\/p>\n
\n\t\t\/\/ on various events like participant joins\/leaves, chat, etc.
\n\t\t\/\/ This is optional
\n\t\tmeeting.participants.joined.on(‘participantJoined’, (participant) => {
\n\t\t\ttextHandler.speak(`Participant Joined ${participant.name}`);
\n\t\t});
\n\t\tmeeting.participants.joined.on(‘participantLeft’, (participant) => {
\n\t\t\ttextHandler.speak(`Participant Left ${participant.name}`);
\n\t\t});<\/p>\n
\n\t\tawait meeting.rtkMeeting.join();
\n\t}<\/p>\n
\n\t\t\/\/ Add any other cleanup logic required
\n\t\tawait this.deinitPipeline();
\n\t}
\n}<\/p>\n
\n <\/p>\n
\n <\/p>\n
\n <\/p>\n
\n <\/p>\n
\n <\/p>\n
\nCloudflare AI WebSocket Inference – With PipeCat’s smart-turn-v2
\n“””<\/p>\n
\nimport websockets
\nimport json
\nimport numpy as np<\/p>\n
\nACCOUNT_ID = “your-account-id”
\nAPI_TOKEN = “your-api-token”
\nMODEL = “@cf\/pipecat-ai\/smart-turn-v2″<\/p>\n
\nWEBSOCKET_URL = f”wss:\/\/api.cloudflare.com\/client\/v4\/accounts\/{ACCOUNT_ID}\/ai\/run\/{MODEL}?dtype=uint8”<\/p>\n
\n async with websockets.connect(
\n WEBSOCKET_URL,
\n additional_headers={
\n “Authorization”: f”Bearer {API_TOKEN}”
\n }
\n ) as websocket:
\n await websocket.send(audio_data)<\/p>\n
\n result = json.loads(response)<\/p>\n
\n return result<\/p>\n
\n noise = np.random.normal(128, 20, 8192).astype(np.uint8)
\n noise = np.clip(noise, 0, 255) <\/p>\n
\n # Generate test audio
\n noise = generate_test_audio()<\/p>\n
\n print(“\\nTesting noise…”)
\n noise_result = await run_inference(noise.tobytes())
\n print(f”Noise result: {noise_result}”)<\/p>\n
\n print(f”Error: {e}”)<\/p>\n
\n asyncio.run(demonstrate_inference())<\/p>\n
\n <\/p>\n