Skip to content

mqtt-udp.md

Latest commit

 

History

History
409 lines (307 loc) · 9.52 KB

mqtt-udp.md

File metadata and controls

409 lines (307 loc) · 9.52 KB

MQTT + UDP Hybrid Communication Protocol

This document describes the MQTT + UDP hybrid protocol used between the device and the server, based on the current implementation: MQTT carries control messages, UDP carries real-time audio.

1. Overview

The protocol uses two channels:

  • MQTT - control messages, state synchronization, JSON payloads.
  • UDP - real-time audio, encrypted.

1.1 Key characteristics

  • Dual channel design - control is separated from data so audio has low latency.
  • Encrypted transport - UDP audio is encrypted with AES-CTR.
  • Sequence numbers - guard against replay and reordering.
  • Automatic reconnect - MQTT reconnects on disconnect.

2. End-to-end Flow

sequenceDiagram
    participant Device as ESP32 device
    participant MQTT as MQTT broker
    participant UDP as UDP server

    Note over Device, UDP: 1. Establish MQTT connection
    Device->>MQTT: MQTT Connect
    MQTT->>Device: Connected

    Note over Device, UDP: 2. Request audio channel
    Device->>MQTT: Hello message (type: "hello", transport: "udp")
    MQTT->>Device: Hello response (UDP endpoint + encryption keys)

    Note over Device, UDP: 3. Establish UDP connection
    Device->>UDP: UDP Connect
    UDP->>Device: Connected

    Note over Device, UDP: 4. Audio streaming
    loop Audio stream
        Device->>UDP: Encrypted audio (Opus)
        UDP->>Device: Encrypted audio (Opus)
    end

    Note over Device, UDP: 5. Control messages
    par Control
        Device->>MQTT: Listen / TTS / MCP messages
        MQTT->>Device: STT / TTS / MCP / Alert responses
    end

    Note over Device, UDP: 6. Teardown
    Device->>MQTT: Goodbye
    Device->>UDP: Disconnect
Loading

3. MQTT Control Channel

3.1 Connection

The device connects to the broker using:

  • Endpoint - broker host and port.
  • Client ID - device identifier.
  • Username / Password - credentials.
  • Keep Alive - heartbeat interval (default 240 s).

3.2 Hello exchange

3.2.1 Device -> Server

{
  "type": "hello",
  "version": 3,
  "transport": "udp",
  "features": {
    "mcp": true,
    "aec": true
  },
  "audio_params": {
    "format": "opus",
    "sample_rate": 16000,
    "channels": 1,
    "frame_duration": 60
  }
}

features.mcp is always set; features.aec is set when CONFIG_USE_SERVER_AEC is enabled.

3.2.2 Server -> Device

{
  "type": "hello",
  "transport": "udp",
  "session_id": "xxx",
  "audio_params": {
    "format": "opus",
    "sample_rate": 24000,
    "channels": 1,
    "frame_duration": 60
  },
  "udp": {
    "server": "192.168.1.100",
    "port": 8888,
    "key": "0123456789ABCDEF0123456789ABCDEF",
    "nonce": "0123456789ABCDEF0123456789ABCDEF"
  }
}

Field reference:

  • udp.server - UDP server address.
  • udp.port - UDP server port.
  • udp.key - AES key, hex-encoded.
  • udp.nonce - AES nonce, hex-encoded.

3.3 JSON message types

3.3.1 Device -> Server

  1. Listen

    {
  "session_id": "xxx",
  "type": "listen",
  "state": "start",
  "mode": "manual"
}

  2. Abort

    {
  "session_id": "xxx",
  "type": "abort",
  "reason": "wake_word_detected"
}

  3. MCP

    {
  "session_id": "xxx",
  "type": "mcp",
  "payload": {
    "jsonrpc": "2.0",
    "id": 1,
    "result": {}
  }
}

  4. Goodbye

    {
  "session_id": "xxx",
  "type": "goodbye"
}

3.3.2 Server -> Device

Semantics match the WebSocket protocol. Supported types:

  • STT - speech recognition result.
  • TTS - TTS lifecycle (start, stop, sentence_start).
  • LLM - emotion update for the UI.
  • MCP - IoT control.
  • System - system control, e.g. "command": "reboot".
  • Alert - show an alert on the UI; fields: status, message, emotion.
  • Goodbye - server-initiated shutdown of the audio session. The device responds by closing the UDP channel without sending its own goodbye.
  • Custom (optional, enabled via CONFIG_RECEIVE_CUSTOM_MESSAGE).

Example alert:

{
  "session_id": "xxx",
  "type": "alert",
  "status": "Warning",
  "message": "Battery low",
  "emotion": "sad"
}

4. UDP Audio Channel

4.1 Establishing the channel

After the device receives the MQTT hello response, it:

  1. Parses the UDP host and port.
  2. Parses the AES key and nonce.
  3. Initializes the AES-CTR context.
  4. Opens the UDP socket.

4.2 Audio packet format

4.2.1 Encrypted audio packet

|type 1B|flags 1B|payload_len 2B|ssrc 4B|timestamp 4B|sequence 4B|
|payload payload_len bytes|

Field reference:

  • type: packet type, always 0x01.
  • flags: flags, currently unused.
  • payload_len: payload length (network byte order).
  • ssrc: synchronization source identifier.
  • timestamp: timestamp (network byte order).
  • sequence: sequence number (network byte order).
  • payload: encrypted Opus audio data.

4.2.2 Encryption

Uses AES-CTR with:

  • Key: 128-bit, provided by the server.
  • Nonce: 128-bit, provided by the server.
  • Counter: built from the timestamp and sequence number.

4.3 Sequence number management

  • Sender: local_sequence_ is incremented monotonically.
  • Receiver: remote_sequence_ validates continuity.
  • Anti-replay: packets with sequence numbers below the expected value are dropped.
  • Tolerance: small gaps are logged as warnings but still accepted.

4.4 Error handling

  1. Decryption failure - log an error and drop the packet.
  2. Sequence gap - log a warning, continue processing the packet.
  3. Malformed packet - log an error and drop.

5. State Management

5.1 Connection state

stateDiagram
    direction TB
    [*] --> Disconnected
    Disconnected --> MqttConnecting: StartMqttClient()
    MqttConnecting --> MqttConnected: MQTT Connected
    MqttConnecting --> Disconnected: Connect failed
    MqttConnected --> RequestingChannel: OpenAudioChannel()
    RequestingChannel --> ChannelOpened: Hello exchange success
    RequestingChannel --> MqttConnected: Hello timeout / failed
    ChannelOpened --> UdpConnected: UDP connect success
    UdpConnected --> AudioStreaming: Start audio
    AudioStreaming --> UdpConnected: Stop audio
    UdpConnected --> ChannelOpened: UDP disconnect
    ChannelOpened --> MqttConnected: CloseAudioChannel()
    MqttConnected --> Disconnected: MQTT disconnect
Loading

5.2 State check

The device determines whether the audio channel is available with:

bool IsAudioChannelOpened() const {
    return udp_ != nullptr && !error_occurred_ && !IsTimeout();
}

6. Configuration Parameters

6.1 MQTT settings

Read from storage:

  • endpoint - broker address.
  • client_id - client identifier.
  • username - user name.
  • password - password.
  • keepalive - keep-alive interval (default 240 s).
  • publish_topic - publish topic.

6.2 Audio parameters

  • Format: Opus
  • Sample rate: 16 kHz device / 24 kHz server
  • Channels: 1 (mono)
  • Frame duration: 60 ms

7. Error Handling and Reconnection

7.1 MQTT reconnect

  • Automatic retry on connect failure.
  • Optional error reporting.
  • Clean-up runs on disconnect.

7.2 UDP connection

  • No automatic retry; depends on re-negotiation via MQTT.
  • Status can be queried at any time.

7.3 Timeouts

The base Protocol class provides timeout detection:

  • Default timeout: 120 s.
  • Based on the time since the last incoming packet.
  • After a timeout the channel is marked unavailable.

8. Security

8.1 Transport encryption

  • MQTT: supports TLS/SSL (port 8883).
  • UDP: AES-CTR on audio payloads.

8.2 Authentication

  • MQTT: user name / password.
  • UDP: keys are distributed via the MQTT channel.

8.3 Anti-replay

  • Monotonically increasing sequence numbers.
  • Stale packets are dropped.
  • Timestamps are validated.

9. Performance Notes

9.1 Concurrency

A mutex protects the UDP connection:

std::lock_guard<std::mutex> lock(channel_mutex_);

9.2 Memory management

  • Network objects are created and destroyed dynamically.
  • Audio packets are managed with smart pointers.
  • Encryption contexts are released promptly.

9.3 Network optimizations

  • UDP connection reuse.
  • Reasonable packet sizes.
  • Sequence continuity checks.

10. Comparison with WebSocket

Feature MQTT + UDP WebSocket
Control channel MQTT WebSocket
Audio channel UDP (encrypted) WebSocket (binary)
Latency Low (UDP) Medium
Reliability Medium High
Complexity High Low
Encryption AES-CTR TLS
Firewall friendliness Low High

11. Deployment Notes

11.1 Network

  • Ensure UDP ports are reachable.
  • Configure firewall rules accordingly.
  • Plan for NAT traversal if needed.

11.2 Server infrastructure

  • MQTT broker configuration.
  • UDP server deployment.
  • Key management.

11.3 Monitoring

  • Connection success rate.
  • Audio transmission latency.
  • Packet loss.
  • Decryption failures.

12. Summary

The MQTT + UDP hybrid protocol achieves efficient audio communication through:

  • Split architecture - separate control and data channels with clear responsibilities.
  • Encryption - AES-CTR protects audio payloads.
  • Sequence management - prevents replay and reordering.
  • Automatic recovery - MQTT reconnects on failure.
  • Performance - UDP keeps audio latency low.

The protocol is a good fit for low-latency voice interaction, at the cost of higher network complexity than pure WebSocket.