Audience: firmware developers building voice AI clients, especially those porting from
xiaozhi. This explains why the protocol is shaped as it is and what it means for your
firmware. For the normative message reference, see protocol-spec.md.
iotta's protocol is built to be simpler and more reliable than xiaozhi.
The problem with one muxed connection. xiaozhi carries audio, control, and tool calls over a single connection, with multiple transports (WebSocket, MQTT+UDP) and multiple audio formats. When one thing goes wrong, everything can break, and state is spread across device and server.
iotta splits the job into two clean connections: 1. Audio & control (one WebSocket) — "here's what I'm capturing, here's what to play." 2. Tools for the AI (a separate MCP WebSocket) — "the AI wants to set my LED."
If the tool channel breaks, your device still talks to the server. If the server changes which tools it wants, you don't reflash — you just execute what it asks. Your firmware is smaller, has fewer edge cases, and the protocol won't change under you every release.
The whole design rests on one rule:
Everything that controls the session transport lives on the session channel. Everything the LLM might invoke lives on the tool channel.
This boundary is absolute: - If losing the tool channel would break audio, it goes on the session channel. - If the LLM should never decide it, it doesn't go in the tool list. - The server can always fall back to a session-only interaction.
Uplink codec selection is the canonical example. The server must know before the first
audio frame how to decode what's coming — it can't wait for a tool response — so it lives
in the session hello handshake, never as an MCP tool call.
Device Server
| |
+── Session WebSocket ────────────┤ (audio + control)
| binary audio frames |
| control JSON: listen, abort, |
| vad, stt, tts, context |
| |
+── Tool MCP WebSocket ───────────┤ (device ↔ LLM)
| MCP JSON-RPC 2.0 |
| device exposes tools |
| server calls them during LLM |
| |
Session channel — the audio lifeline. The synchronous, always-needed path. The device
sends hello, listen start/stop, audio frames, abort, and optional context; the
server sends hello, TTS audio frames, and vad / stt / tts / processing / error
control messages. No tool calls live here, so a broken tool channel never stops the
conversation. Your firmware only needs this one robust WebSocket to be functional. It also
doesn't matter whether the server produces that audio with a multi-step ASR→LLM→TTS pipeline
or an all-in-one speech-to-speech model — the device streams audio and plays audio back; the
runtime shape is the server's concern, not the firmware's.
Tool channel — the MCP boundary. Optional and asynchronous. The device acts as an
MCP server, exposing capabilities (set_led_color, read_temperature,
declare_capabilities(scope), …). The iotta server acts as the MCP client: during LLM
generation, when the model wants a tool, the server sends an MCP tools/call to the device,
which executes and returns the result. Tool schemas are versioned and owned by the server's
registry — the device doesn't know the schema, it just executes what it's asked.
(The exact message shapes are in protocol-spec.md.)
1. Two WebSockets instead of one. Audio never blocks tool execution and tool traffic never delays audio; TCP backpressure on one channel can't starve the other; each channel has one job, so failure modes are clear. Trade-off: marginally more bookkeeping on the device (two connections), but the async-I/O cost is negligible and the clarity is worth it.
2. The server picks; the device complies. No feature flags for ASR modes, audio-format
versions, or transports. Codec negotiation is one round: the device's hello says "I can
send opus or pcm," the server's hello says "send pcm," and the device does. There's
no negotiation loop and no protocol-version explosion — a new codec is added server-side
first, then rolled out to compatible devices via the registry. Trade-off: less room for
the device to push back — which is the point.
3. Provisioning is HTTP, not baked into the WebSocket. POST /provision registers the
device and returns its session URL + auth token; the WebSocket upgrade then uses those
credentials. The token is short-lived credential material, not a discovery mechanism — it's
issued at provisioning and validated at session open, and a device re-provisions if it
expires or is revoked. This cleanly separates a business operation (registration) from a
transport concern (session auth), and makes per-device tokens and health telemetry
(POST /telemetry) siblings of provisioning rather than part of the session protocol.
4. TLS by default (off-loopback). Any non-loopback client must use wss://; plain
ws:// is allowed only from 127.0.0.1 for local dev. WiFi devices are exposed to every
other device on the network, and a stolen bearer token impersonates a device — so TLS is the
baseline. The bundled reverse proxy terminates TLS.
5. Browser-compatible credentials. Two credential-transport paths: native clients use
Authorization / Device-Id / Protocol-Version headers; browsers (which can't set
headers) carry the same three values in Sec-WebSocket-Protocol subprotocols, base64url
encoded. The same protocol serves ESP32 firmware and a JavaScript web simulator, and it's
backward-compatible within protocol v1. For firmware: use headers; the browser path is for
other clients.
6. Server-owned tool schemas. Schemas live in a versioned registry on the server, not on the device. One source of truth, explicit versioning ("this turn used tool schema v2" is recorded), and the device never validates schemas — it executes a call and returns a result or an error; the server validates against the schema.
| xiaozhi | iotta | Why |
|---|---|---|
listen: state=detect (local wake word) |
not in protocol | wake logic stays on the device; the server doesn't need to know |
mcp message type (JSON-RPC muxed in) |
separate tool channel | MCP is its own WebSocket — cleaner |
llm message (emotion/expression) |
context message |
the server pushes arbitrary state; the device decides what to do |
system message (reboot/update) |
OTA is HTTP | firmware updates are provisioning-time, not in-session |
alert, custom message types |
context message |
one generic mechanism for server→device data |
| MQTT + UDP transport | WebSocket only | one transport, works with TLS and in browsers |
| audio format versions (v1/v2/v3 headers) | raw frames | no headers; the codec is negotiated, so the server knows what's coming |
multi-mode listening (auto/manual/realtime) |
one mode | the device captures; the server does VAD |
| NVS-stored OTA URL override | per-device tokens + provisioning | provision once, get a credential — no URLs stored locally |
| xiaozhi | iotta | What changed |
|---|---|---|
protocol_version in hello |
Protocol-Version header |
moved to the handshake |
supported_codecs optional |
canonical | the server always returns the chosen codec; the device always knows upfront |
| codec fixed at handshake | switchable mid-session | audio_mode requests a codec change without reconnecting (e.g. for diagnostics) |
| tool schemas on device / inferred | server-owned | the device just executes — no schema validation in firmware |
| auth optional / disabled in dev | auth + TLS by default | secure by default; localhost ws:// still allowed for dev |
Assuming a working xiaozhi firmware:
| Change | Effort |
|---|---|
| Open two WebSockets instead of one (route session vs. MCP) | 1–2 days |
Remove audio-format header parsing (read input_audio from server hello) |
a few hours |
| Provisioning HTTP client (you already do this) | ~1 day |
| Remove MQTT+UDP fallback, keep WebSocket | 1–2 days |
| Simplify tool execution (execute + return; let the server validate) | a few hours |
Remove device-side VAD (listen: detect) — the server does VAD |
2–3 days |
Total: ~1–2 weeks for a full port. From scratch it's still easier than xiaozhi — there's less state to manage.
hello exchange, then audio frames every 60 ms while listeningvad / stt / tts / processing / error / context; send listen and abortOptional but recommended: the tool channel (MCP server mode + tools/call execution),
declare_capabilities, device context in listen:start, and PCM uplink + audio_mode
for diagnostics.
Device firmware Server
| POST /provision (device_id, board, fw) ──> |
|<── session_url, token, agent ───────────── |
| WebSocket upgrade (Authorization: Bearer) |
|<── 101 Switching Protocols ─────────────── |
| hello (audio config) ───────────────────> |
|<── hello (session_id, input_audio) ────── |
| |
| user presses mic |
| listen:start ────────────────────────────> |
| [binary audio frames, every 60 ms] ──────> |
|<── vad:speech_start (server heard you) |
| listen:stop ─────────────────────────────> |
|<── stt (transcript) |
|<── processing |
|<── tts:start |
|<── [binary Opus frames] (play to speaker) |
|<── tts:stop |
| ready for next turn |
Device is playing TTS; user presses the mic again.
| abort ───────────────────────────────────> server stops TTS, cancels LLM, returns to Idle
|<── [Opus frames stop]
| listen:start ────────────────────────────> start the next turn
// Server → device:
{ "jsonrpc": "2.0", "id": "12345", "method": "tools/call",
"params": { "name": "set_led_color", "arguments": { "r": 255, "g": 0, "b": 0 } } }
// Device → server (success or error):
{ "jsonrpc": "2.0", "id": "12345", "result": { "status": "ok" } }
{ "jsonrpc": "2.0", "id": "12345", "error": { "code": -32600, "message": "Invalid arguments" } }
declare_capabilities is the same shape with "name": "declare_capabilities" and a
{"scope": "audio"} argument; the device returns what that scope supports.
abort, keep sending frames until listen:stop (or a
timeout, in case listen:stop is lost), then return to idle — don't cut off mid-buffer.Content-Type: application/json on POST /provision.input_audio in the server hello. Encode uplink in the codec it names; if it's
omitted (older server), use your declared codec.POST /provision → token, session URL, agent)hellolisten:start; server replies vad:speech_startlisten:stop, receives stt then tts, and plays the audio backabort → server stops sending audio)tools/callaudio_mode codec switchesprotocol-spec.md — normative wire format and message referencedesign.md — system goals and design principlesserver/src/iotta/protocol.py — server-side credential and codec parsingserver/tests/test_protocol.py — protocol edge-case test cases