AssemblyAI details raw WebSocket voice agent build with no SDK
News
AssemblyAI published a tutorial for building a voice agent directly on WebSockets with streaming STT, LLM Gateway and TTS.
AssemblyAI published a tutorial on Oct. 29, 2026 that walks through building a voice agent directly on raw WebSockets, without using its SDK. The post ties together streaming speech-to-text, an HTTP LLM call and text-to-speech playback, and frames the approach as a way to keep full control over every protocol frame and state transition.
Evidence and context
The guide uses three connections and one API key: the Universal-3.6 Pro Realtime speech-to-text WebSocket, LLM Gateway over plain HTTPS, and the AssemblyAI Text-to-Speech WebSocket. According to the tutorial, the speech-to-text socket receives 16 kHz binary PCM audio and returns frames such as Begin, SpeechStarted, Turn and Termination, while the text-to-speech socket accepts Generate, Flush, Cancel, KeepAlive and Terminate and returns Audio, FlushDone, Cancelled, Termination and Error.
AssemblyAI says the example is designed for developers who want to understand or own each byte of the flow, including barge-in behavior and tool calling. The post says the sample fits in about 300 lines of Python and uses only the websocket, httpx and sounddevice libraries. It also says the same API key authenticates all three services, with LLM Gateway using an OpenAI-compatible chat-completions format.
The tutorial’s example agent is configured to answer weather questions via a get_weather tool call. In the sample code, the model request is sent with tools in the message payload, the function result is appended as a tool message, and the LLM is called again until it returns plain text. AssemblyAI’s sample then sends that text to the TTS socket, where each reply must be followed by Flush to produce audio.
The article also documents barge-in handling: when the user speaks over the agent, the program clears local playback and sends Cancel on the TTS socket, while dropping any trailing Audio frames until Cancelled arrives. It adds a backchannel filter so short acknowledgements such as “mhm,” “okay” or “yeah” do not interrupt the agent mid-sentence. The tutorial says the TTS session can stay open for the whole conversation, but should be kept alive with periodic frames during idle periods.
AssemblyAI’s post includes operational details and error handling guidance, including the note that the TTS socket sends an Error frame before failure closes and that the STT socket closes with websocket close codes and reasons. The guide also says the TTS voice in the sample uses Audrey’s voice ID, that the output format is 24 kHz pcm_s16le, and that the microphone input should match the 16 kHz sample rate configured on the speech-to-text side.
