remote_session: keep guest content off the cloud relay; soften ICE setup

The cloud websocket is treated as untrusted in our threat model, but
remote_session.created was leaking the cursor bitmap (and clipboard /
audio metadata) inside its runtime snapshot. Add a SnapshotScope to
RuntimeManager::RemoteRuntimeSnapshot and have the cloud-facing path
use kPublic so only running / display / frame / framebuffer_ready
travel over WSS; cursor continues to seed over the WebRTC control DC
(SetDataChannelOpenHandler already does this).

Also fix the WebRTC handshake stalls we were seeing on CN networks:

- AcceptOffer waited up to 5s for both local SDP *and* full ICE
  gathering; if STUN was UDP-blackholed (very common with stun.l.google
  on mainland ISPs) gathering never completed and the answer came back
  empty with "timed out creating WebRTC answer". Make gathering soft -
  return whatever host candidates we have once SDP is ready, log a
  WARN, and trickle later candidates via a new LocalIceCandidateHandler.
- Stretch the SDP wait to 10s as a safety margin.
- Default STUN list now skews toward CN-reachable hosts
  (stun.qq.com, stun.miwifi.com, stun.cloudflare.com); operators can
  still override the whole list with TENBOX_STUN_SERVERS. The same
  list is advertised to the browser via ice_servers so both peers
  probe a consistent set.
- cloud_tunnel forwards every host-side candidate as a
  remote_signal.candidate push, reusing the existing cloud relay
  forward path for browser-side trickle.

Bumps to 0.7.20.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: 78

VERSION

@@ -1 +1 @@
-0.7.19
+0.7.20

src/daemon/cloud_tunnel.cpp

@@ -1625,6 +1625,32 @@ std::shared_ptr<WebRtcPeer> CloudTunnel::CreateRemotePeer(
         peer->SetDataChannelHandler([this, vm_id](const nlohmann::json& message) {
             HandleDataChannelMessage(vm_id, message);
         });
+        // Trickle host-side ICE candidates back to the browser through the
+        // existing `remote_signal.candidate` cloud relay envelope. The
+        // browser-side hostBus router (`remote_signal.candidate` listener
+        // in RemoteDesktopPanel) feeds them to RTCPeerConnection.addIceCandidate.
+        // Without this, AcceptOffer's softened gathering window would only
+        // ship whatever candidates were collected within ~10s; on STUN-blocked
+        // networks the answer sometimes returns with zero candidates and
+        // the browser would have nothing to connect to.
+        peer->SetLocalIceCandidateHandler(
+            [this, session_id, vm_id](nlohmann::json candidate) {
+                if (fd_ < 0) return;
+                nlohmann::json payload = {
+                    {"session_id", session_id},
+                };
+                for (auto it = candidate.begin(); it != candidate.end(); ++it) {
+                    payload[it.key()] = it.value();
+                }
+                (void)SendJson({
+                    {"id", GenerateUuid()},
+                    {"type", "remote_signal.candidate"},
+                    {"host_id", host_id_},
+                    {"vm_id", vm_id},
+                    {"session_id", session_id},
+                    {"payload", std::move(payload)},
+                });
+            });
         // When the control DC opens, push the latest cached cursor so the
         // browser is in sync even if MOVE_CURSOR events fired (and were
         // dropped) while the channel was still in DTLS/SCTP setup.
@@ -1802,10 +1828,20 @@ nlohmann::json CloudTunnel::CreateRemoteSession(const std::string& vm_id, const
     auto json = ToJson(*session);
     json["video_bitrate_bps"] = video_bitrate_bps;
     json["video_pixel_format"] = RemoteVideoPixelFormatName(video_pixel_format);
+    // Advertise the same STUN list the daemon itself is using so both
+    // peers agree on the candidate-gathering servers (see ResolvedStunServers
+    // for why the defaults skew toward CN-reachable hosts; operators can
+    // override via TENBOX_STUN_SERVERS).
+    nlohmann::json ice_urls = nlohmann::json::array();
+    for (const auto& url : ResolvedStunServers()) ice_urls.push_back(url);
     json["ice_servers"] = nlohmann::json::array({
-        {{"urls", nlohmann::json::array({"stun:stun.l.google.com:19302"})}},
+        {{"urls", std::move(ice_urls)}},
     });
-    json["runtime"] = runtime_manager_.RemoteRuntimeSnapshot(vm_id);
+    // Public snapshot: scrub cursor pixels / clipboard / audio so the cloud
+    // relay never observes guest-visible content. Browser receives those
+    // exclusively through the WebRTC control DataChannel.
+    json["runtime"] = runtime_manager_.RemoteRuntimeSnapshot(
+        vm_id, RuntimeManager::SnapshotScope::kPublic);
     json["webrtc_ready"] = NativeWebRtcAvailable();
     json["message"] = NativeWebRtcAvailable()
         ? "remote session signaling is ready"

src/daemon/remote_webrtc.cpp

@@ -5,6 +5,7 @@
 #include <rtc/plihandler.hpp>
 
 #include <atomic>
+#include <cctype>
 #include <chrono>
 #include <condition_variable>
 #include <cstdlib>
@@ -15,9 +16,11 @@
 #include <inttypes.h>
 #include <mutex>
 #include <optional>
+#include <string>
 #include <string_view>
 #include <thread>
 #include <utility>
+#include <vector>
 
 namespace tenbox::daemon {
 
@@ -70,6 +73,50 @@ const char* PeerStateName(int state) {
     }
 }
 
+// Returns the configured STUN URL list. `TENBOX_STUN_SERVERS` is a
+// comma-separated list (whitespace tolerated) of full URLs such as
+// "stun:stun.qq.com:3478,stun:stun.miwifi.com:3478". Empty entries
+// are skipped.
+//
+// Defaults are skewed for mainland China deployments because that's where
+// most of the production fleet lives today and Google STUN
+// (stun.l.google.com:19302) is regularly UDP-blackholed by CN ISPs,
+// which previously made WebRTC setup time out at "creating answer".
+// Order matters - libdatachannel probes them in sequence:
+//   1. Tencent  - rock solid in CN, used by every domestic Live SDK
+//   2. Xiaomi   - secondary domestic option, embedded in MiWiFi routers
+//   3. Cloudflare - global fallback for users outside CN; CF anycast is
+//                   reachable with low latency from the mainland too,
+//                   so it doubles as a third tier when the first two are
+//                   busy or filtered.
+// Operators can override the whole list with TENBOX_STUN_SERVERS, e.g.
+// to point at a self-hosted coturn or to add TURN once we have one.
+std::vector<std::string> ConfiguredStunServers() {
+    const char* raw = std::getenv("TENBOX_STUN_SERVERS");
+    std::vector<std::string> out;
+    if (raw && raw[0] != '\0') {
+        std::string_view view(raw);
+        size_t start = 0;
+        while (start <= view.size()) {
+            size_t comma = view.find(',', start);
+            size_t end = comma == std::string_view::npos ? view.size() : comma;
+            size_t s = start;
+            size_t e = end;
+            while (s < e && std::isspace(static_cast<unsigned char>(view[s]))) ++s;
+            while (e > s && std::isspace(static_cast<unsigned char>(view[e - 1]))) --e;
+            if (e > s) out.emplace_back(view.substr(s, e - s));
+            if (comma == std::string_view::npos) break;
+            start = comma + 1;
+        }
+    }
+    if (out.empty()) {
+        out.emplace_back("stun:stun.qq.com:3478");
+        out.emplace_back("stun:stun.miwifi.com:3478");
+        out.emplace_back("stun:stun.cloudflare.com:3478");
+    }
+    return out;
+}
+
 std::string StripLipSyncGroups(std::string sdp) {
     std::string filtered;
     filtered.reserve(sdp.size());
@@ -101,7 +148,16 @@ class NativeWebRtcPeer final : public WebRtcPeer {
         : frame_reader_(std::move(frame_reader)),
           preferred_video_format_(preferred_video_format) {
         rtc::Configuration config;
-        config.iceServers.emplace_back("stun:stun.l.google.com:19302");
+        for (const auto& url : ConfiguredStunServers()) {
+            try {
+                config.iceServers.emplace_back(url);
+            } catch (const std::exception& e) {
+                std::fprintf(stdout,
+                             "[WARN]  remote_webrtc: ignoring invalid STUN url %s: %s\n",
+                             url.c_str(), e.what());
+                std::fflush(stdout);
+            }
+        }
         config.disableAutoNegotiation = true;
         peer_ = std::make_shared<rtc::PeerConnection>(std::move(config));
         peer_->onStateChange([this](rtc::PeerConnection::State state) {
@@ -131,11 +187,32 @@ class NativeWebRtcPeer final : public WebRtcPeer {
             cv_.notify_all();
         });
         peer_->onLocalCandidate([this](rtc::Candidate candidate) {
-            std::lock_guard<std::mutex> lock(mu_);
-            candidates_.push_back({
+            nlohmann::json entry = {
                 {"candidate", std::string(candidate)},
                 {"sdpMid", candidate.mid()},
-            });
+            };
+            LocalIceCandidateHandler handler;
+            {
+                std::lock_guard<std::mutex> lock(mu_);
+                candidates_.push_back(entry);
+                handler = local_ice_handler_;
+            }
+            // Trickle every host-side candidate to the embedder so the
+            // browser can start probing as soon as gathering produces
+            // host / srflx entries, instead of having to wait for the
+            // initial answer's `candidates[]` (which we cap at a short
+            // gathering window so STUN-blackholed networks don't stall
+            // session creation).
+            if (handler) {
+                try {
+                    handler(std::move(entry));
+                } catch (const std::exception& e) {
+                    std::fprintf(stdout,
+                                 "[ERROR] remote_webrtc: local ice handler threw: %s\n",
+                                 e.what());
+                    std::fflush(stdout);
+                }
+            }
         });
         peer_->onGatheringStateChange([this](rtc::PeerConnection::GatheringState state) {
             if (state == rtc::PeerConnection::GatheringState::Complete) {
@@ -183,13 +260,36 @@ class NativeWebRtcPeer final : public WebRtcPeer {
             peer_->setRemoteDescription(rtc::Description(sdp, "offer"));
             peer_->setLocalDescription(rtc::Description::Type::Answer);
 
+            // Wait long enough to (a) absolutely require the answer SDP and
+            // (b) opportunistically pick up gathered candidates so the
+            // initial answer carries something useful.
+            //
+            // `description_ready_` is hard: without local SDP we have no
+            // answer to hand back, so we error out. `gathering_complete_`
+            // is intentionally soft - if STUN servers are unreachable
+            // (common on locked-down networks: 19302/UDP black-holed by
+            // ISP, no TURN configured), libdatachannel keeps waiting on
+            // the binding response indefinitely. Returning the SDP with
+            // whatever host candidates we already have lets LAN peers
+            // connect, and the `LocalIceCandidateHandler` continues to
+            // trickle later srflx/relay candidates as they arrive.
             std::unique_lock<std::mutex> lock(mu_);
-            const bool ready = cv_.wait_for(lock, std::chrono::seconds(5), [this] {
+            cv_.wait_for(lock, std::chrono::seconds(10), [this] {
                 return description_ready_ && gathering_complete_;
             });
-            if (!ready || local_sdp_.empty()) {
+            if (local_sdp_.empty()) {
                 return WebRtcAnswer{.ok = false, .error = "timed out creating WebRTC answer"};
             }
+            if (!gathering_complete_) {
+                std::fprintf(stdout,
+                             "[WARN]  remote_webrtc: returning answer before ICE "
+                             "gathering finished (%zu host candidate(s) so far); "
+                             "remaining candidates will trickle. Check STUN "
+                             "reachability or set TENBOX_STUN_SERVERS to a "
+                             "reachable server list.\n",
+                             candidates_.size());
+                std::fflush(stdout);
+            }
             return WebRtcAnswer{
                 .ok = true,
                 .sdp = local_sdp_,
@@ -242,6 +342,33 @@ class NativeWebRtcPeer final : public WebRtcPeer {
         dc_handler_ = std::move(handler);
     }
 
+    void SetLocalIceCandidateHandler(LocalIceCandidateHandler handler) override {
+        // Snapshot any candidates already gathered before the embedder
+        // installed the trickle handler (typical race: handler is
+        // attached right after CreateWebRtcPeer but onLocalCandidate
+        // fires from the libdatachannel worker as soon as
+        // setLocalDescription is called). Replaying ensures the browser
+        // ends up with the same candidate set regardless of timing.
+        std::vector<nlohmann::json> backlog;
+        {
+            std::lock_guard<std::mutex> lock(mu_);
+            local_ice_handler_ = handler;
+            if (handler && !candidates_.empty()) {
+                for (const auto& c : candidates_) backlog.push_back(c);
+            }
+        }
+        for (auto& c : backlog) {
+            try {
+                handler(std::move(c));
+            } catch (const std::exception& e) {
+                std::fprintf(stdout,
+                             "[ERROR] remote_webrtc: local ice handler threw on backlog: %s\n",
+                             e.what());
+                std::fflush(stdout);
+            }
+        }
+    }
+
     void SetDataChannelOpenHandler(DataChannelOpenHandler handler) override {
         // Snapshot any already-open channels so a handler installed after
         // open() still gets a chance to seed initial state. Channels we
@@ -1153,6 +1280,7 @@ class NativeWebRtcPeer final : public WebRtcPeer {
     std::vector<std::shared_ptr<rtc::DataChannel>> data_channels_;
     DataChannelMessageHandler dc_handler_;
     DataChannelOpenHandler dc_open_handler_;
+    LocalIceCandidateHandler local_ice_handler_;
     std::shared_ptr<rtc::Track> video_track_;
     std::shared_ptr<rtc::Track> audio_track_;
     std::thread video_thread_;
@@ -1187,4 +1315,8 @@ bool NativeWebRtcAvailable() {
     return true;
 }
 
+std::vector<std::string> ResolvedStunServers() {
+    return ConfiguredStunServers();
+}
+
 }  // namespace tenbox::daemon

src/daemon/remote_webrtc.h

@@ -71,6 +71,14 @@ using DataChannelMessageHandler = std::function<void(const nlohmann::json&)>;
 // silently dropped by SendOnDataChannel.
 using DataChannelOpenHandler = std::function<void(const std::string& label)>;
 
+// Fired for every host-side ICE candidate as it is gathered. The
+// embedder is expected to forward the candidate to the remote peer over
+// whatever signaling channel is in use (cloud websocket relay in the
+// production setup). Candidates included in the initial WebRtcAnswer
+// are *also* delivered through this handler if it is installed before
+// AcceptOffer; deduplication is the embedder's responsibility.
+using LocalIceCandidateHandler = std::function<void(nlohmann::json candidate)>;
+
 class WebRtcPeer {
 public:
     virtual ~WebRtcPeer() = default;
@@ -80,6 +88,7 @@ class WebRtcPeer {
     virtual void SetVideoBitrate(uint32_t bitrate_bps) = 0;
     virtual void SetDataChannelHandler(DataChannelMessageHandler handler) = 0;
     virtual void SetDataChannelOpenHandler(DataChannelOpenHandler handler) = 0;
+    virtual void SetLocalIceCandidateHandler(LocalIceCandidateHandler handler) = 0;
     // Send a JSON text frame to a specific data channel (typically "control"
     // for clipboard / status events). Returns false if the channel doesn't
     // exist or isn't open yet; the caller is expected to drop the message.
@@ -91,4 +100,11 @@ std::shared_ptr<WebRtcPeer> CreateWebRtcPeer(
     PixelFormat preferred_video_format = PixelFormat::kYuv420p);
 bool NativeWebRtcAvailable();
 
+// Returns the resolved STUN URL list (TENBOX_STUN_SERVERS override or the
+// CN-leaning defaults). Exposed so the cloud tunnel can advertise the
+// same servers to the browser-side RTCPeerConnection - keeping both sides
+// of the connection probing the same set avoids one peer giving up on
+// srflx because it picked a server the other peer cannot reach.
+std::vector<std::string> ResolvedStunServers();
+
 }  // namespace tenbox::daemon

src/daemon/runtime_manager.cpp

@@ -1079,7 +1079,8 @@ bool RuntimeManager::SetRemoteVideoPixelFormat(const std::string& vm_id, PixelFo
     return true;
 }
 
-nlohmann::json RuntimeManager::RemoteRuntimeSnapshot(const std::string& vm_id) const {
+nlohmann::json RuntimeManager::RemoteRuntimeSnapshot(const std::string& vm_id,
+                                                     SnapshotScope scope) const {
     auto session = FindSession(vm_id);
     if (!session) return {{"running", false}};
     bool framebuffer_ready = false;
@@ -1088,15 +1089,25 @@ nlohmann::json RuntimeManager::RemoteRuntimeSnapshot(const std::string& vm_id) c
         framebuffer_ready = session->framebuffer && session->framebuffer->IsValid();
     }
     std::lock_guard<std::mutex> lock(session->console_mutex);
-    return {
+    nlohmann::json snapshot = {
         {"running", true},
         {"display", session->display_state},
         {"frame", session->last_frame},
         {"framebuffer_ready", framebuffer_ready},
-        {"cursor", session->cursor},
-        {"audio", session->last_audio},
-        {"clipboard", session->last_clipboard},
     };
+    // Guest-visible state (cursor bitmap, clipboard activity metadata,
+    // audio activity metadata) MUST NOT appear in a Public snapshot — that
+    // payload travels over the cloud websocket relay, which our threat model
+    // treats as untrusted. Browsers receive these via the WebRTC `control`
+    // DataChannel after DTLS/SRTP is up; the daemon seeds the latest cursor
+    // on channel-open in cloud_tunnel so a late-attaching peer is not stuck
+    // waiting for the next MOVE_CURSOR (virtio_gpu source-side dedup).
+    if (scope == SnapshotScope::kInternal) {
+        snapshot["cursor"] = session->cursor;
+        snapshot["audio"] = session->last_audio;
+        snapshot["clipboard"] = session->last_clipboard;
+    }
+    return snapshot;
 }
 
 nlohmann::json RuntimeManager::CursorSnapshot(const std::string& vm_id) const {

src/daemon/runtime_manager.h

@@ -87,7 +87,14 @@ class RuntimeManager {
                          RemoteVideoFrame* frame,
                          bool need_full_frame,
                          std::chrono::milliseconds wait_timeout = std::chrono::milliseconds(0));
-    nlohmann::json RemoteRuntimeSnapshot(const std::string& vm_id) const;
+    // Snapshot scope. `kPublic` strips every field that carries guest-visible
+    // content (cursor pixels, clipboard metadata, audio metadata). It is the
+    // only variant we are allowed to put on a wire that the cloud relay can
+    // observe in plaintext, because our threat model treats the relay as
+    // untrusted. `kInternal` keeps everything for in-process callers.
+    enum class SnapshotScope { kPublic, kInternal };
+    nlohmann::json RemoteRuntimeSnapshot(const std::string& vm_id,
+                                         SnapshotScope scope = SnapshotScope::kInternal) const;
     // Returns the most recently observed cursor JSON for `vm_id`, or an empty
     // object if no cursor event has been seen yet (or the VM is gone). Used
     // by the cloud tunnel to seed a freshly opened control DataChannel,