Skip to content

Commit cd635fd

Browse files
committed
fix: allow Telegram webhook URLs to include deployment protection bypass param
Telegram's setWebhook API was believed to silently drop URLs containing query parameters, but live testing against the Bot API confirmed this is false — getWebhookInfo returns the full URL including query params. Remove the Telegram special-case from buildChannelWebhookUrl() that routed Telegram through buildPublicDisplayUrl (no bypass). Remove stripBypassParam() from reconcile.ts. All channels (Slack, Telegram, Discord) now use buildPublicUrl which appends x-vercel-protection-bypass when VERCEL_AUTOMATION_BYPASS_SECRET is configured. Update preflight remediation text, verifier contract, tests, and all docs (CLAUDE.md, CONTRIBUTING.md, deployment-protection.md, channels-and-webhooks.md, architecture-tradeoffs.md, environment-variables.md) to remove the false "silently drops" claim. Verified: npm test, typecheck, lint, and check:verify-contract all pass.
1 parent d75d2db commit cd635fd

File tree

14 files changed

+49
-103
lines changed

14 files changed

+49
-103
lines changed

.claude/skills/vercel-openclaw-testing/SKILL.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -409,7 +409,7 @@ These patterns are distilled from 100+ commits of production fixes. When investi
409409

410410
**Root causes found historically:**
411411
- Startup scripts called `deleteWebhook` on every sandbox boot, clearing the webhook URL. Old snapshots contained these scripts, and new scripts weren't synced fast enough after restore. **Fix:** remove `deleteWebhook` from startup scripts entirely.
412-
- Telegram's `setWebhook` API silently rejects URLs containing `x-vercel-protection-bypass` query parameter. **Fix:** use separate `buildPublicDisplayUrl` without bypass param for Telegram.
412+
- Telegram's `setWebhook` API was previously believed to reject URLs with query parameters, but testing confirmed it preserves them. All channels now use `buildPublicUrl` with the bypass param.
413413
- Queue publish failures on webhook route returned 500 to Telegram, which then exponentially backed off webhook deliveries (Telegram reduces delivery frequency on repeated failures). **Fix:** always return 200 from webhook route, even on internal errors.
414414
- Reconnecting Telegram created backlogged failed updates that prevented new webhook delivery. **Fix:** pass `drop_pending_updates: true` to `setWebhook`.
415415

@@ -483,7 +483,7 @@ Step-by-step investigation playbooks for common production issues.
483483
5. **Check for fast-path failures:** Search for `fast_path_failed` — this means Vercel Queues rejected the publish and the system fell back to store-based queuing
484484
6. **Check for job parking:** Search for `job_parked` or `job_retry_parked` — these indicate deferred or retrying jobs with `nextAttemptAt` timestamps
485485
7. **Check gateway token:** `GET /api/status` — look at `lifecycle.consecutiveTokenRefreshFailures` and `lifecycle.breakerOpenUntil` for token refresh circuit breaker state
486-
8. **Check webhook URL:** For Telegram, verify the registered URL doesn't contain `x-vercel-protection-bypass` (Telegram silently rejects it)
486+
8. **Check webhook URL:** For Telegram, verify the registered URL via `getWebhookInfo` matches the expected delivery URL
487487
9. **Trigger self-heal:** Run destructive smoke with `selfHealTokenRefresh` phase, or `POST /api/admin/launch-verify` with destructive mode
488488

489489
### "Sandbox stuck in restoring/error"

CLAUDE.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ Channel phases (`channelRoundTrip`, `channelWakeFromSleep`) call `POST /api/admi
7979
| `/api/admin/stop` | Stop the sandbox (v2 auto-snapshots on stop) |
8080
| `/api/admin/snapshot` | Stop the sandbox (same as stop for now; v2 auto-snapshots on stop) |
8181
| `/api/admin/snapshots/delete` | Delete a past snapshot from Vercel and local history |
82-
| `/api/admin/channel-secrets` | Configure smoke credentials and dispatch server-signed synthetic channel webhooks. Raw secrets are never returned. Smoke dispatch URLs use `buildPublicUrl()` (bypass included when configured) for all channels, including Telegram — this is intentionally different from provider-facing Telegram webhook registration, which omits the bypass parameter. |
82+
| `/api/admin/channel-secrets` | Configure smoke credentials and dispatch server-signed synthetic channel webhooks. Raw secrets are never returned. Smoke dispatch URLs use `buildPublicUrl()` (bypass included when configured) for all channels. |
8383
| `/api/admin/channel-forward-diag` | Read channel forward diagnostic from store |
8484
| `/api/channels/slack/install` | Slack OAuth install initiation |
8585
| `/api/channels/slack/install/callback` | Slack OAuth callback |
@@ -133,7 +133,7 @@ Responsibilities:
133133
- expose `buildPublicUrl(path: string, request?: Request): string`
134134
- append `x-vercel-protection-bypass=<VERCEL_AUTOMATION_BYPASS_SECRET>` when the bypass secret is available (regardless of auth mode)
135135

136-
Channel webhook URL construction lives in `src/server/channels/webhook-urls.ts`. The convenience wrappers in `src/server/channels/state.ts` (`buildSlackWebhookUrl`, `buildTelegramWebhookUrl`) delegate to `buildChannelWebhookUrl()` in that module. Slack delivery URLs use `buildPublicUrl` (bypass secret appended when available) for runtime webhook forwarding, but the Slack app manifest uses `buildPublicDisplayUrl` (no bypass secret) because the bypass query parameter interferes with Slack's Event Subscriptions URL verification. Telegram intentionally does not include the bypass query parameter — Telegram URLs use `buildPublicDisplayUrl` (no bypass secret) because Telegram validates webhooks via the `x-telegram-bot-api-secret-token` header, and including the bypass query parameter causes `setWebhook` to silently drop the registration.
136+
Channel webhook URL construction lives in `src/server/channels/webhook-urls.ts`. The convenience wrappers in `src/server/channels/state.ts` (`buildSlackWebhookUrl`, `buildTelegramWebhookUrl`) delegate to `buildChannelWebhookUrl()` in that module. All channel delivery URLs use `buildPublicUrl` (bypass secret appended when available). The Slack app manifest uses `buildPublicDisplayUrl` (no bypass secret) because the bypass query parameter interferes with Slack's Event Subscriptions URL verification.
137137

138138
Admin-visible surfaces (preflight payload, status responses, UI) must use `buildPublicDisplayUrl()` instead of `buildPublicUrl()`. The display variant omits the `x-vercel-protection-bypass` query parameter so secrets are never leaked to the browser or API consumers.
139139

CONTRIBUTING.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,7 @@ Full reference:
150150
| `OPENCLAW_PACKAGE_SPEC` | No | OpenClaw version to install. When unset, the runtime falls back to a pinned known-good version (currently `openclaw@2026.3.28`). On Vercel deployments, the deployment contract **warns** — it does not fail — when unset or unpinned. Pin to an exact version like `openclaw@1.2.3` for deterministic sandbox resumes. |
151151
| `OPENCLAW_SANDBOX_VCPUS` | No | vCPU count for sandbox create and resume (valid: 1, 2, 4, 8; default: 1). Keep fixed during benchmarks. |
152152
| `OPENCLAW_SANDBOX_SLEEP_AFTER_MS` | No | How long the sandbox stays alive after last activity, in milliseconds (60000–2700000; default: 1800000 = 30 min). Heartbeat and touch-throttle intervals are derived proportionally. Existing running sandboxes cannot be shortened in place. If you increase this value, the next touch/heartbeat can top the sandbox timeout up to the new target. If you decrease it, the lower value becomes exact on the next create or restore. |
153-
| `VERCEL_AUTOMATION_BYPASS_SECRET` | No | Enables protected webhook delivery when Deployment Protection is on. Slack URLs use it when configured; Telegram intentionally does not include the bypass query parameter and relies on webhook-secret validation instead. On protected deployments, Telegram needs a Deployment Protection Exception. |
153+
| `VERCEL_AUTOMATION_BYPASS_SECRET` | No | Enables protected webhook delivery when Deployment Protection is on. All channel webhook URLs (Slack, Telegram, Discord) include the bypass parameter when configured. |
154154
| `NEXT_PUBLIC_APP_URL` | No | Base origin override |
155155
| `NEXT_PUBLIC_BASE_DOMAIN` | No | Preferred external host for webhook URLs |
156156
| `BASE_DOMAIN` | No | Legacy alias for `NEXT_PUBLIC_BASE_DOMAIN` |
@@ -173,7 +173,7 @@ When you add or change Redis keys, route every key through `src/server/store/key
173173
| `/api/admin/stop` | Stop the sandbox (v2 auto-snapshots on stop) |
174174
| `/api/admin/snapshot` | Stop the sandbox (same as stop for now; v2 auto-snapshots) |
175175
| `/api/admin/snapshots/delete` | Delete a past snapshot from Vercel and local history |
176-
| `/api/admin/channel-secrets` | Configure smoke credentials and dispatch signed synthetic channel webhooks. Smoke dispatch uses `buildPublicUrl()` (bypass included) for all channels including Telegram — distinct from provider-facing registration which omits bypass for Telegram. |
176+
| `/api/admin/channel-secrets` | Configure smoke credentials and dispatch signed synthetic channel webhooks. Smoke dispatch uses `buildPublicUrl()` (bypass included) for all channels. |
177177
| `/api/admin/channel-forward-diag` | Read channel forward diagnostic from store |
178178
| `/api/cron/watchdog` | Cron watchdog for health repair and scheduled OpenClaw cron wake |
179179
| `/api/admin/watchdog` | Read cached watchdog report or run a fresh one |

docs/architecture-tradeoffs.md

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -102,9 +102,7 @@ Unlike Slack Socket Mode (where the SDK handles reconnection and Slack retries r
102102

103103
### Telegram and deployment protection
104104

105-
Telegram webhook URLs cannot include the `x-vercel-protection-bypass` query parameter. Including it causes `setWebhook` to silently drop the registration. This means Telegram webhooks are incompatible with Vercel Deployment Protection unless a Deployment Protection Exception is configured for the webhook path.
106-
107-
See [Deployment Protection](deployment-protection.md) for the full breakdown.
105+
Telegram webhook URLs now include the `x-vercel-protection-bypass` query parameter, the same as Slack and Discord. Telegram's `setWebhook` API preserves query parameters in the registered URL, so the bypass secret is delivered with every webhook callback. This means Telegram webhooks work with Vercel Deployment Protection when `VERCEL_AUTOMATION_BYPASS_SECRET` is configured.
108106

109107
## Always-on sandbox vs sleep/wake
110108

@@ -151,7 +149,7 @@ Setup: set `ADMIN_SECRET` (simple) or configure OAuth client credentials (more s
151149

152150
### Why the app can't use Vercel Deployment Protection as auth
153151

154-
Deployment Protection was attempted and abandoned. It blocks ALL unauthenticated requests — including channel webhooks from Slack, Telegram, and other platforms. It is also unavailable on Hobby plans. The bypass secret works for Slack but not for Telegram (see above).
152+
Deployment Protection was attempted and abandoned. It blocks ALL unauthenticated requests — including channel webhooks from Slack, Telegram, and other platforms. It is also unavailable on Hobby plans. The bypass secret works for all channels when configured.
155153

156154
## Firewall: enforced vs absent
157155

docs/channels-and-webhooks.md

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ Configure Telegram credentials from the admin panel. The app stores the bot toke
9696

9797
OpenClaw's config includes the app's public Telegram webhook route as `webhookUrl`. When the sandbox boots, OpenClaw itself also calls `setWebhook` with this URL, so the app's endpoint — not the sandbox's — is what Telegram calls.
9898

99-
Telegram registration URLs must not include the bypass query parameter. Telegram validates webhooks via the `x-telegram-bot-api-secret-token` header, and including the bypass parameter can cause `setWebhook` to silently drop registration.
99+
Telegram validates webhooks via the `x-telegram-bot-api-secret-token` header. Registration URLs include the bypass query parameter when configured, allowing Telegram to work with Vercel Deployment Protection.
100100

101101
### How Telegram messages flow
102102

@@ -108,9 +108,7 @@ When a Telegram update arrives at the webhook:
108108

109109
## Protected deployments
110110

111-
Slack can use a bypass-capable delivery URL on protected deployments. Telegram intentionally cannot.
112-
113-
This is because Telegram's `setWebhook` registration silently fails when extra query parameters are present in the URL. To make Telegram work on a protected deployment, configure a Deployment Protection Exception or use another protection-compatible path.
111+
All channels (Slack, Telegram, Discord) use bypass-capable delivery URLs on protected deployments when `VERCEL_AUTOMATION_BYPASS_SECRET` is configured.
114112

115113
Admin-visible URLs — in the admin panel, preflight payload, status responses, and docs examples — must stay display-safe and never expose the bypass secret. The app enforces this by using `buildPublicDisplayUrl()` for all operator-visible surfaces and reserving `buildPublicUrl()` for outbound delivery only.
116114

@@ -145,9 +143,9 @@ The admin panel shows a channel as blocked when deployment prerequisites are sti
145143

146144
This means config looks good, but the current deployment has not yet proven the full delivery path. Preflight only checks config. Safe mode proves boot or resume plus a real completion, but destructive launch verification is still required before `channelReadiness.ready` becomes `true`.
147145

148-
### Slack works but Telegram registration fails on a protected deployment
146+
### Channel webhooks fail on a protected deployment
149147

150-
Telegram is hitting Vercel's Deployment Protection, not app auth. Unlike Slack, Telegram cannot use the bypass query parameter because `setWebhook` silently drops registrations with extra parameters. Configure a Deployment Protection Exception for the Telegram webhook path, or disable Deployment Protection if your use case allows it.
148+
Channel webhooks are hitting Vercel's Deployment Protection. Enable Protection Bypass for Automation in your Vercel project settings and set `VERCEL_AUTOMATION_BYPASS_SECRET`. All channels (Slack, Telegram, Discord) include the bypass parameter in their delivery URLs when configured.
151149

152150
### Launch verification phases look mostly healthy but overall result is false
153151

docs/deployment-protection.md

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,23 +4,21 @@
44

55
## Channel behavior
66

7-
- Slack webhook URLs include the bypass query parameter when the secret is configured.
8-
- Telegram intentionally does not include the bypass query parameter. Telegram validates via the `x-telegram-bot-api-secret-token` header, and adding the bypass query parameter can cause `setWebhook` to silently drop registration. On protected deployments, Telegram needs a Deployment Protection Exception or another protection-compatible path.
7+
All channel webhook URLs (Slack, Telegram, Discord) include the `x-vercel-protection-bypass` query parameter when `VERCEL_AUTOMATION_BYPASS_SECRET` is configured. This allows webhooks from all platforms to pass through Vercel Deployment Protection.
98

109
## Delivery URLs vs operator-visible URLs
1110

1211
These are intentionally different surfaces:
1312

14-
- Slack delivery URLs may include `x-vercel-protection-bypass` when `VERCEL_AUTOMATION_BYPASS_SECRET` is configured.
15-
- Telegram intentionally does not include the bypass query parameter because Telegram webhook registration can silently fail when it is present.
13+
- Delivery URLs include `x-vercel-protection-bypass` when `VERCEL_AUTOMATION_BYPASS_SECRET` is configured.
1614
- Admin-visible payloads, rendered UI, connectability output, and docs examples must use display URLs that never expose the bypass secret.
1715

1816
Examples:
1917

2018
```
2119
Delivery URL (Slack): https://app.example.com/api/channels/slack/webhook?x-vercel-protection-bypass=[redacted]
2220
Display URL (Slack): https://app.example.com/api/channels/slack/webhook
23-
Delivery URL (Telegram): https://app.example.com/api/channels/telegram/webhook
21+
Delivery URL (Telegram): https://app.example.com/api/channels/telegram/webhook?x-vercel-protection-bypass=[redacted]
2422
Display URL (Telegram): https://app.example.com/api/channels/telegram/webhook
2523
```
2624

docs/environment-variables.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,4 +81,4 @@ See [Deployment Protection](deployment-protection.md) for full details on bypass
8181

8282
| Variable | Purpose |
8383
| -------- | ------- |
84-
| `VERCEL_AUTOMATION_BYPASS_SECRET` | Lets protected webhook requests reach the app when Vercel Deployment Protection is enabled. Telegram intentionally does not include the bypass query parameter — use a Deployment Protection Exception for Telegram on protected deployments. |
84+
| `VERCEL_AUTOMATION_BYPASS_SECRET` | Lets protected webhook requests reach the app when Vercel Deployment Protection is enabled. All channel webhook URLs include the bypass parameter when configured. |

scripts/check-verifier-contract.mjs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -230,9 +230,9 @@ for (const envName of contractEnvNames) {
230230

231231
const wordingRequirements = [
232232
{
233-
snippet: "Telegram intentionally does not include the bypass query parameter",
233+
snippet: "All channel webhook URLs",
234234
label: "Telegram bypass behavior",
235-
files: ["docs/deployment-protection.md", "CLAUDE.md", "CONTRIBUTING.md"],
235+
files: ["docs/deployment-protection.md", "CONTRIBUTING.md"],
236236
},
237237
{
238238
snippet: "deployment contract **warns** — it does not fail",

src/server/channels/telegram/reconcile.test.ts

Lines changed: 0 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -4,37 +4,10 @@ import test from "node:test";
44
import {
55
reconcileTelegramIntegration,
66
reconcileTelegramWebhook,
7-
stripBypassParam,
87
TELEGRAM_RECONCILE_KEY,
98
} from "@/server/channels/telegram/reconcile";
109
import { withHarness } from "@/test-utils/harness";
1110

12-
test("stripBypassParam removes only x-vercel-protection-bypass", () => {
13-
assert.equal(
14-
stripBypassParam(
15-
"https://app.example.com/api/channels/telegram/webhook?x-vercel-protection-bypass=topsecret&foo=1",
16-
),
17-
"https://app.example.com/api/channels/telegram/webhook?foo=1",
18-
);
19-
20-
assert.equal(
21-
stripBypassParam(
22-
"https://app.example.com/api/channels/telegram/webhook?foo=1",
23-
),
24-
"https://app.example.com/api/channels/telegram/webhook?foo=1",
25-
);
26-
27-
assert.equal(
28-
stripBypassParam(
29-
"https://app.example.com/api/channels/telegram/webhook?x-vercel-protection-bypass=secret",
30-
),
31-
"https://app.example.com/api/channels/telegram/webhook",
32-
);
33-
34-
// Malformed URL returns as-is
35-
assert.equal(stripBypassParam("not-a-url"), "not-a-url");
36-
});
37-
3811
test("reconcileTelegramIntegration sets webhook, syncs commands, and records timestamp", async () => {
3912
await withHarness(async (h) => {
4013
await h.mutateMeta((meta) => {

src/server/channels/telegram/reconcile.ts

Lines changed: 11 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,10 @@
11
import { setWebhook, getMyCommands } from "@/server/channels/telegram/bot-api";
22
import type { TelegramBotCommand } from "@/server/channels/telegram/bot-api";
33
import { getTelegramBotCommands, syncTelegramCommands } from "@/server/channels/telegram/commands";
4-
import { setTelegramChannelConfig } from "@/server/channels/state";
4+
import { buildTelegramWebhookUrl, setTelegramChannelConfig } from "@/server/channels/state";
55
import { logInfo, logWarn } from "@/server/log";
66
import { getInitializedMeta, getStore } from "@/server/store/store";
77

8-
/**
9-
* Strip the Vercel protection bypass query parameter from a URL.
10-
* Telegram's setWebhook silently drops registrations when the URL
11-
* contains this parameter.
12-
*/
13-
export function stripBypassParam(url: string): string {
14-
try {
15-
const parsed = new URL(url);
16-
if (parsed.searchParams.has("x-vercel-protection-bypass")) {
17-
parsed.searchParams.delete("x-vercel-protection-bypass");
18-
return parsed.toString();
19-
}
20-
} catch {
21-
// Not a valid URL — return as-is.
22-
}
23-
return url;
24-
}
25-
268
export const TELEGRAM_RECONCILE_KEY =
279
"telegram:integration:last-reconciled-at";
2810
export const TELEGRAM_WEBHOOK_RECONCILE_KEY = TELEGRAM_RECONCILE_KEY;
@@ -73,9 +55,15 @@ export async function reconcileTelegramIntegration(options?: {
7355
}
7456
}
7557

76-
// Strip the bypass query param if present — Telegram silently rejects
77-
// webhook URLs that contain it.
78-
const webhookUrl = stripBypassParam(config.webhookUrl);
58+
// Prefer the dynamically-built URL (includes bypass param when configured).
59+
// Fall back to stored config.webhookUrl when the origin cannot be resolved
60+
// (e.g. in tests or environments without NEXT_PUBLIC_APP_URL).
61+
let webhookUrl: string;
62+
try {
63+
webhookUrl = buildTelegramWebhookUrl();
64+
} catch {
65+
webhookUrl = config.webhookUrl;
66+
}
7967
await setWebhook(config.botToken, webhookUrl, config.webhookSecret);
8068

8169
let commandsSynced = false;
@@ -118,7 +106,7 @@ export async function reconcileTelegramIntegration(options?: {
118106
await getStore().setValue(TELEGRAM_RECONCILE_KEY, checkedAt);
119107

120108
logInfo("channels.telegram_integration_reconciled", {
121-
webhookUrl: stripBypassParam(config.webhookUrl),
109+
webhookUrl,
122110
commandsSynced,
123111
commandCount,
124112
});

0 commit comments

Comments
 (0)