RustDesk Server: SSO/OAuth login breaks connectivity (WebSocket/secure tcp timeout) after successful install

[HoBeom]

✅ Have you read and understood the above guidelines?

yes

🔎 Did you run the script with verbose mode enabled?

No (this issue will likely be closed automatically)

📜 What is the name of the script you are using?

RustDesk Server

📂 What was the exact command used to execute the script?

bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/rustdeskserver.sh)"

⚙️ What settings are you using?

• Default Settings
• Advanced Settings

🖥️ Which Linux distribution are you using?

Debian 12

📈 Which Proxmox version are you on?

8.4.16

📝 Provide a clear and concise description of the issue.

The rustdeskserver LXC script installs successfully (hbbs/hbbr + rustdesk-api), but after enabling SSO/OAuth login via rustdesk-api and logging in from RustDesk Desktop, remote connectivity breaks.

• Logged out: remote connections work normally.
• After SSO/OAuth login succeeds: clients frequently become offline / rendezvous fails, with errors like:
  "Failed to secure tcp: deadline has elapsed"
  "Rendezvous server connection failed, try later"

This makes the login feature unusable even though installation completes without errors.

I suspect a WebSocket / reverse-proxy related issue triggered after login (client starts using ws endpoints), and/or a compatibility mismatch between rustdesk-api (lejianwen) and hbbs/hbbr packages installed by the script.

🔄 Steps to reproduce the issue.

Step 1: Install the container using the community-scripts rustdeskserver LXC script.
Step 2: Configure RustDesk Desktop clients to use the self-hosted ID/Relay server (works normally while logged out).
Step 3: Install/enable rustdesk-api (installed by the script) and open the admin UI.
Step 4: Configure OAuth/SSO (tested with Google and GitHub) and login successfully from the RustDesk Desktop client.
Step 5: After login succeeds, try remote control: connections fail and/or clients become offline.

❌ Paste the full error output (if available).

RustDesk Desktop client errors after successful login:

• "Failed to secure tcp: deadline has elapsed"
• "Rendezvous server connection failed, try later"

WebSocket endpoints behind Nginx Proxy Manager (OpenResty) behave inconsistently:

• HTTP/2 requests return 502
• Forcing HTTP/1.1 shows WebSocket upgrade (101 Switching Protocols)

🖼️ Additional context (optional).

Related upstream discussion that seems relevant:
lejianwen/rustdesk-api#396

• Please confirm whether this is a known issue / incompatibility when combining rustdesk-api (lejianwen) with hbbs/hbbr packages installed by the script.
• Alternatively, consider installing a compatible hbbs/hbbr build set when rustdesk-api is installed.

[HoBeom]

Update / Workaround (no script changes required)

I was able to get login + post-login connectivity working by updating hbbs/hbbr inside the LXC to the lejianwen/rustdesk-server deb builds (hbbs/hbbr/utils), without modifying this ProxmoxVE script.

After swapping hbbs/hbbr binaries with lejianwen/rustdesk-server deb packages (hbbs/hbbr/utils) and restarting services, the issue disappears.

Commands (run inside the RustDesk LXC):

set -euo pipefail
apt-get update
apt-get install -y curl ca-certificates jq

WORK="/root/rustdesk_lejianwen_upgrade"
mkdir -p "$WORK"
cd "$WORK"

REL_JSON="$(curl -fsSL https://api.github.com/repos/lejianwen/rustdesk-server/releases/latest)"

HBBS_URL="$(echo "$REL_JSON" | jq -r '.assets[] | select(.name|test("rustdesk-server-hbbs.*amd64\\.deb$")) | .browser_download_url' | head -n1)"
HBBR_URL="$(echo "$REL_JSON" | jq -r '.assets[] | select(.name|test("rustdesk-server-hbbr.*amd64\\.deb$")) | .browser_download_url' | head -n1)"
UTIL_URL="$(echo "$REL_JSON" | jq -r '.assets[] | select(.name|test("rustdesk-server-utils.*amd64\\.deb$")) | .browser_download_url' | head -n1)"

curl -fL "$UTIL_URL" -o utils_amd64.deb
curl -fL "$HBBR_URL" -o hbbr_amd64.deb
curl -fL "$HBBS_URL" -o hbbs_amd64.deb

dpkg -i ./utils_amd64.deb ./hbbr_amd64.deb ./hbbs_amd64.deb || apt-get -f install -y

systemctl daemon-reload
systemctl restart rustdesk-hbbs rustdesk-hbbr rustdesk-api

/usr/bin/hbbs -V || true
/usr/bin/hbbr -V || true
ss -lntup | egrep ':(21114|21115|21116|21117|21118|21119)\b' || true

This looks like a hbbs/hbbr build compatibility issue with rustdesk-api login flows.