This is the normative message reference: the connections, framing, control messages,
and state machine a client implements. For why the protocol is shaped this way — and a
guide to porting firmware, including from xiaozhi — see
protocol-design.md.
iotta uses two independent connections per session:
| Connection | Transport | Purpose |
|---|---|---|
| Session channel | WebSocket | Audio (device ↔ server) and session control messages |
| Tool channel | MCP over WebSocket | LLM tool calls dispatched to the device |
The session channel carries all audio and session lifecycle events. The tool channel is a standard MCP connection; the device exposes its capabilities as MCP tools, and the server calls them during LLM generation.
The two connections are independent. Either can open or close without immediately invalidating the other, though a session without a tool channel simply cannot execute device-side tools.
The device opens a WebSocket connection to the server's session endpoint. Three values must be presented on the upgrade. A native client (ESP32 firmware, the Python client/) presents them as request headers:
| Header | Value |
|---|---|
Authorization |
Bearer <token> |
Device-Id |
Stable unique identifier for the device (e.g. MAC address or provisioned UUID) |
Protocol-Version |
1 |
The server rejects connections with missing or invalid tokens with WebSocket close code 4001 (and an unsupported/absent protocol version or missing device id with 4000) before the session begins. The Bearer value is either the shared session token or — when the server runs with per-device session tokens — the token issued to this Device-Id at provisioning; the server validates the credential against the connecting device. See Per-device session tokens in docs/devices.md.
The Device-Id value is opaque to the platform — a stable identifier the device chooses, typically its MAC address or a provisioned UUID. The server matches on it but never parses it.
A device obtains the session_url and (when session auth is enabled) the bearer token by first calling the provisioning endpoint — see Provisioning below and docs/devices.md.
Sec-WebSocket-ProtocolThe browser WebSocket API cannot set custom request headers, so a browser client (e.g. the web device simulator) presents the same three values through the one header it can influence — Sec-WebSocket-Protocol, the offered-subprotocols list. This is the established pattern (the Kubernetes API server authenticates bearer tokens over WebSocket the same way). The client offers:
new WebSocket(url, [
"iotta.v1", // protocol-version marker (v1)
"bearer." + base64url(token), // the session bearer token
"device." + base64url(device_id), // the Device-Id
]);
| Subprotocol token | Carries |
|---|---|
iotta.v1 |
Declares protocol version 1 (stands in for the Protocol-Version header). The server echoes this one back as the negotiated subprotocol so the browser's WebSocket handshake completes. |
bearer.<value> |
The session bearer token. |
device.<value> |
The Device-Id. |
<value> is base64url (no padding) of the UTF-8 string, so an arbitrary token or a device id containing characters outside the RFC 6455 subprotocol-token grammar (e.g. the colons in a MAC address) survives transport. The credential subprotocols (bearer.*, device.*) are read by the server and never echoed; only iotta.v1 is.
A header is canonical: a value present as a header takes precedence over the same value in a subprotocol, and a header-only client is wholly unaffected. (Additive and backward-compatible within protocol v1; see protocol-design.md for the rationale.)
By default the server requires the session channel to run over TLS (wss://) for any non-loopback client; a connection that is neither TLS nor from a loopback peer (127.0.0.1/::1) is refused with close code 4003. Plain ws:// is permitted only from loopback, for local development. A browser already forces this — it refuses ws:// from an https:// page (mixed content). TLS is detected either directly (a wss:// scheme) or, behind the bundled TLS-terminating reverse proxy, via the X-Forwarded-Proto: https header the proxy sets (see DEPLOY.md). The enforcement can be disabled (server.require_tls: false) for a deployment that guarantees TLS by other means.
Two frame types are used:
type field.hello exchange (see below).Audio format: - Codec: Opus (default) or PCM (see Uplink codec negotiation) - Device → server: 16 kHz, mono - Server → device: 24 kHz, mono (always Opus) - Frame duration: 60 ms
| Codec | Binary frame payload |
|---|---|
opus |
One Opus packet per frame. Lossy. The default uplink codec and the only downlink codec. |
pcm |
Raw little-endian signed 16-bit mono samples, one frame's worth per binary frame (sample_rate × frame_duration_ms ÷ 1000 samples). Lossless relative to the device's capture; no codec sits in the path. |
pcm exists for development and diagnostics — high-fidelity capture for ASR/DSP/audio-hardware analysis (see docs/roadmap.md, Future extension: device audio analysis). It is uplink-only (device → server); the downlink is always Opus. PCM is ~10–16× the bitrate of Opus, which is a non-issue on the assumed WiFi transport but is why Opus remains the default.
Device Server
| |
|-- WebSocket upgrade --------> | (Authorization, Device-Id, Protocol-Version headers)
|<- 101 Switching Protocols --- |
| |
|-- hello ------------------> |
|<- hello ------------------- |
| |
| ... session active ... |
| |
|-- [close frame] -----------> | (or server initiates)
The device sends hello immediately after the WebSocket upgrade completes. The server responds with its own hello. The session is active once both hellos have been exchanged.
hello (device → server)Sent by the device immediately after connection. Declares the device's preferred uplink audio configuration and, optionally, the codecs it is capable of producing.
{
"type": "hello",
"version": 1,
"audio": {
"codec": "opus",
"sample_rate": 16000,
"channels": 1,
"frame_duration_ms": 60,
"supported_codecs": ["opus", "pcm"]
}
}
audio is the device's preferred/default uplink configuration. audio.supported_codecs is an optional list of every uplink codec the firmware can produce. If omitted, the server assumes [audio.codec] — so a device that only speaks Opus needs no changes and is never asked for anything else.
This is a deliberately minimal advertisement — only what the server needs at handshake to choose the uplink codec, before audio starts and without the (optional) tool channel. The richer, queryable capability surface is the declare_capabilities(scope) tool (see below).
hello (server → device)Sent by the server in response. Confirms session parameters for both directions. audio describes the downlink (server → device, always Opus). input_audio is the server's selected uplink (device → server) — the device must encode its audio frames according to it.
{
"type": "hello",
"session_id": "<uuid>",
"audio": {
"codec": "opus",
"sample_rate": 24000,
"channels": 1,
"frame_duration_ms": 60
},
"input_audio": {
"codec": "opus",
"sample_rate": 16000,
"channels": 1,
"frame_duration_ms": 60
}
}
The server selects the uplink codec; the device complies. Selection is capability-bounded, not dictated:
opus; an operator may set pcm globally, or per-device).audio.supported_codecs (treating an absent list as [audio.codec]).audio.codec.input_audio. If input_audio is omitted (e.g. an older server), the device uses its declared audio.The handshake is the authoritative negotiation point: the uplink codec must be settled before any audio frames flow.
listen (device → server)Signals a change in microphone state.
{
"type": "listen",
"state": "start" | "stop"
}
start: the device has begun capturing audio and will send binary audio frames.
stop: the device has stopped capturing; no more audio frames follow until the next listen start.
On listen start, the device may include an optional context object containing any device-side data that should be available to the LLM for this turn — sensor readings, NVRAM settings, device state, or any other ambient values. The structure is freeform; the server passes it through to the LLM without interpreting it.
{
"type": "listen",
"state": "start",
"context": {
"temperature_c": 22.5,
"volume": 0.7,
"location": "kitchen"
}
}
The server uses the most recently received context for each turn. If context is omitted, the server uses whatever context was last provided in the session, or none if none has been sent.
abort (device → server)Requests immediate cancellation of any in-progress TTS playback and LLM generation.
{
"type": "abort"
}
The server stops sending audio frames and LLM generation as quickly as possible. The session returns to the listening state.
vad (server → device)Reports server-side voice activity detection events within the audio stream. Allows the device to show responsive UI (e.g. a "heard you" indicator) as soon as speech is detected, independent of ASR completion.
{
"type": "vad",
"state": "speech_start" | "speech_end"
}
speech_start: the server has detected the beginning of a speech segment in the incoming audio.
speech_end: the server has detected that the speech segment has ended. ASR processing begins at this point.
processing (server → device)Signals that the server has finished ASR and is now running the LLM and TTS pipeline. Allows the device to display a "thinking" state during the gap between end of speech and start of audio playback.
{
"type": "processing"
}
stt (server → device)Delivers the ASR transcript of the most recent user utterance. Intended for display.
{
"type": "stt",
"text": "what is the weather like today"
}
tts (server → device)Controls TTS playback state on the device.
{
"type": "tts",
"state": "start" | "stop" | "sentence_start"
}
When state is sentence_start, a text field is included with the sentence about to be spoken, for subtitle display:
{
"type": "tts",
"state": "sentence_start",
"text": "The weather today is partly cloudy."
}
The server sends tts start before the first audio frame of a response, and tts stop after the last.
context (server → device)Pushes arbitrary informational data from the server to the device. No response is expected. The device uses the payload however is appropriate — updating local display state, storing values, informing its next interaction. The structure is freeform.
{
"type": "context",
"data": {
"user_name": "Brad",
"conversation_count": 42
}
}
The server may send context at any point during a session, including during TTS playback or between turns.
error (server → device)Signals a non-fatal error. The session remains open.
{
"type": "error",
"code": "<string>",
"message": "<human-readable description>"
}
audio_mode (server → device)Switches the uplink codec mid-session, without reconnecting. Used to put a connected device into (or out of) a high-fidelity diagnostic mode. The handshake is the normal place to choose the uplink codec; audio_mode exists for the less common case of flipping a live session.
{
"type": "audio_mode",
"input_audio": {
"codec": "pcm",
"sample_rate": 16000,
"channels": 1,
"frame_duration_ms": 60
}
}
audio_mode (device → server)The device acknowledges the directive. It must not change its encoding until it has emitted this ack, so the server knows exactly which binary frame is the first in the new format.
{
"type": "audio_mode",
"state": "applied" | "unsupported",
"input_audio": { "codec": "pcm", "sample_rate": 16000, "channels": 1, "frame_duration_ms": 60 }
}
applied: subsequent uplink frames use input_audio. The server switches its decode path on receipt of this ack.unsupported: the device cannot produce the requested codec; the server keeps the current mode. (The server should only request codecs the device advertised, so this is a safety net.)The server requests audio_mode only for codecs the device declared at handshake. A device that does not understand audio_mode at all should ignore it, and the server treats the absence of an ack as "no change."
The session channel and the tool channel carry deliberately different kinds of traffic. The boundary matters because the tool channel is optional and the tool registry is the AI↔firmware contract (see docs/design.md, goal #3).
| Belongs on the session channel (control messages) | Belongs on the tool channel (MCP tools) |
|---|---|
Anything that controls the session transport itself — audio codec/mode (hello, audio_mode), listening state, abort, VAD, TTS playback state |
Device hardware functions the LLM may invoke during generation (set volume, read a sensor, actuate hardware) |
| Must work even when the tool channel is down | Platform introspection the server invokes directly, e.g. declare_capabilities — kept out of the LLM's tool list |
The rule: if losing the tool channel must not disable it, or the LLM should never decide it, it is a session-channel control message — not a tool. (Uplink codec selection is the canonical example; the rationale — the cleanliness principle — is in protocol-design.md.)
┌──────────────────────────────────────────────────┐
│ │
▼ │
Connecting │
│ (WebSocket open + hello exchanged) │
▼ │
Idle ◄──────────── abort received ──────────── Speaking
│ ▲
│ listen:start received │
▼ │
Listening ── vad:speech_start ──► Hearing │
│ │ │
│ └── vad:speech_end │
│ listen:stop received │ │
└──────────────────────►▼ │
Processing ── tts:start sent ┘
│
│ (ASR → LLM → TTS pipeline runs here)
| State | Description |
|---|---|
Connecting |
WebSocket open; waiting for hello exchange to complete |
Idle |
Session active; waiting for the device to begin listening |
Listening |
Mic open; no speech detected yet |
Hearing |
Server VAD has detected active speech in the audio stream |
Processing |
Speech ended or capture stopped; ASR → LLM → TTS pipeline running |
Speaking |
Server is streaming TTS audio to the device |
The device may send abort from any state. The server transitions immediately to Idle on receipt.
The tool channel is a standard MCP connection over WebSocket, opened by the device to the server's MCP endpoint. The device acts as an MCP server (exposing its hardware capabilities as tools); the iotta server acts as the MCP client (calling tools during LLM generation).
The tool channel is independent of the session channel. The device should open it after the session hello exchange completes and keep it open for the duration of the session.
The Device-Id header (same value as on the session channel) must be present on the MCP WebSocket upgrade so the server can associate the tool channel with an active session.
Tool schema definitions are managed by the iotta tool schema registry, not discovered dynamically from the device. The device executes tools; the server owns the schemas.
Not every tool the device exposes is for the LLM. The registry distinguishes two classes:
declare_capabilities(scope) is the reference platform tool: the server calls it (as MCP client) to query what the device supports within a scope — e.g. "audio" (codecs, sample rates), "sensors", "display", "ota". The scope parameter lets one tool answer a growing capability surface without proliferating tools or churning the protocol.
This complements — does not replace — the minimal audio.supported_codecs advertisement in hello: the handshake carries only what must be known before audio starts and without the tool channel; declare_capabilities is the richer, on-demand surface for everything else. None of it is required for basic uplink codec negotiation.
Before opening a session, a device contacts the provisioning endpoint over HTTP to register and learn how to connect. This is a plain request/response endpoint, independent of the two WebSocket channels.
Device Server
|-- POST /provision --------> | {device_id, board, firmware_version?, capabilities?}
|<- 200 provisioning record - | {agent, connection{session_url, protocol_version, token?}, firmware_update?}
| |
| (if firmware_update present and wanted)
|-- GET /provision/firmware/{board}/{version} --> |
|<- 200 application/octet-stream (X-Firmware-Sha256) |
| |
| ... then open the session channel using `connection` ...
The response's connection block carries the session WebSocket URL, the protocol version, and — when the server requires session auth — the bearer token the device then presents on the session-channel upgrade. firmware_update is present only when a newer firmware version has been released for the device's board; the device downloads it from the firmware URL and verifies the X-Firmware-Sha256 digest before flashing.
How a device is allowed to register (open auto-enrollment vs. an operator allowlist, optional enrollment token, per-board/per-device rules) and how firmware versions are managed is the device registry, specified in docs/devices.md. The wire shape above is all the firmware needs to implement; the policy is a server-side concern.
POST /telemetryA device reports a runtime health snapshot, on a cadence it chooses, independent of any open session. This is the runtime companion to /provision (which carries the device's static facts at enrollment).
Device Server
|-- POST /telemetry --------> | {device_id, health{...}, firmware_version?}
|<- 200 ack ---------------- | {device_id, health_reported_at}
health is an open object — the firmware reports whatever it measures (e.g. uptime_s, free_heap, rssi, battery_pct, crash_count). The platform stores the latest snapshot on the device record (plus a bounded history time series behind it) and does not constrain its keys. A monotonic crash_count/reboot counter is the field the server's crash-rate signal differences across the history window.Bearer token it presents on the session channel (per-device when enabled, else the shared token). An operator's master token also works; an unknown or decommissioned device is refused (403).last_seen) and may carry firmware_version to keep the device's reported version current between provisions.See Device health telemetry in docs/devices.md.
The protocol version is declared by the device in the Protocol-Version upgrade header and echoed in the hello exchange. The current version is 1.
If the server does not support the requested version, it rejects the connection with HTTP 400 before the upgrade completes.
The audio-negotiation additions (audio.supported_codecs, input_audio, audio_mode, the PCM codec) and the browser-credential additions (the iotta.v1 / bearer.* / device.* subprotocols) are both additive and backward-compatible within version 1 — no version bump:
supported_codecs and ignores input_audio/audio_mode is treated as Opus-only; a server that omits input_audio means "use my declared audio."The TLS requirement (wss:// off-loopback) is a transport-policy default, not a protocol change. (Rationale: protocol-design.md.)