Azure OAuth not working with multiple accounts

[MarcSkovMadsen]

We have for the last two years been trying to setup Azure Oauth for our Panel apps without success. Many problems have been solved along the way. But the problem remaining is that Azure Oauth does not work when having multiple azure accounts (personal + admin).

The problem is well known and described here https://panel.holoviz.org/how_to/authentication/trouble_shooting.html#cause-multiple-accounts.

You can see the problem in action below where it oauth only works for me if I use incognito mode.

bokeh.min.js?v=2630124d82b67a204b9c9da925d3d456e9ec585bb0cf8f839a1c704dbee3f200b70d237916a9fd429755022dc83ac334505cd7385ebb4453a7b17aa1d3699c35:249 WebSocket connection to 'wss://...REMOVED.../panel-oauth-test/panel_app/ws' failed: 
connect @ bokeh.min.js?v=2630124d82b67a204b9c9da925d3d456e9ec585bb0cf8f839a1c704dbee3f200b70d237916a9fd429755022dc83ac334505cd7385ebb4453a7b17aa1d3699c35:249
bokeh.min.js?v=2630124d82b67a204b9c9da925d3d456e9ec585bb0cf8f839a1c704dbee3f200b70d237916a9fd429755022dc83ac334505cd7385ebb4453a7b17aa1d3699c35:249 [bokeh 3.9.0] Failed to connect to Bokeh server: Could not open websocket
_on_error @ bokeh.min.js?v=2630124d82b67a204b9c9da925d3d456e9ec585bb0cf8f839a1c704dbee3f200b70d237916a9fd429755022dc83ac334505cd7385ebb4453a7b17aa1d3699c35:249
bokeh.min.js?v=2630124d82b67a204b9c9da925d3d456e9ec585bb0cf8f839a1c704dbee3f200b70d237916a9fd429755022dc83ac334505cd7385ebb4453a7b17aa1d3699c35:248 [bokeh 3.9.0] Failed to load Bokeh session pJKROf71epgaXLhoJlnQRikGVYIql2VFTj0IIGl1A9gV: Error: Could not open websocket
n.add_document_from_session @ bokeh.min.js?v=2630124d82b67a204b9c9da925d3d456e9ec585bb0cf8f839a1c704dbee3f200b70d237916a9fd429755022dc83ac334505cd7385ebb4453a7b17aa1d3699c35:248
bokeh.min.js?v=2630124d82b67a204b9c9da925d3d456e9ec585bb0cf8f839a1c704dbee3f200b70d237916a9fd429755022dc83ac334505cd7385ebb4453a7b17aa1d3699c35:164 Error rendering Bokeh items: Error: Could not open websocket
    at p._on_error (bokeh.min.js?v=2630124d82b67a204b9c9da925d3d456e9ec585bb0cf8f839a1c704dbee3f200b70d237916a9fd429755022dc83ac334505cd7385ebb4453a7b17aa1d3699c35:249:5776)
    at Promise.socket.onerror (bokeh.min.js?v=2630124d82b67a204b9c9da925d3d456e9ec585bb0cf8f839a1c704dbee3f200b70d237916a9fd429755022dc83ac334505cd7385ebb4453a7b17aa1d3699c35:249:1728)
k @ bokeh.min.js?v=2630124d82b67a204b9c9da925d3d456e9ec585bb0cf8f839a1c704dbee3f200b70d237916a9fd429755022dc83ac334505cd7385ebb4453a7b17aa1d3699c35:164
favicon.ico:1  Failed to load resource: the server responded with a status of 404 ()

The recommended workarounds of deleting all browsing data, using incognito mode or another browser are not working solutions. They cause a lot of problems for most users who would not understand and accept these kinds of problems they don't experience any where else.

For us it means we have not been able to introduce oauth even though trying for the last two years. For others it could easily mean rejection of the whole framework by IT.

Please fix. Thanks

[MarcSkovMadsen]

Below is generated in collaboration with Claude. I've tried to validate but I'm not experienced in this area.

---

Root cause found: oversized Sec-WebSocket-Protocol header → proxy returns HTTP 400

After a deep investigation I can now explain this "Could not open websocket" failure precisely. The headline: it is not specific to "multiple accounts" — that's a correlation. It's a request-header-size limit hit by the Bokeh session token, which Panel/Bokeh sends back in the Sec-WebSocket-Protocol header on the /ws upgrade.

Setup

• Panel 1.9.3 / Bokeh 3.9.0, --oauth-provider=azure, --num-procs=5, no encryption key.
• Behind ingress-nginx (defaults), TLS host serving HTTP/2.

What actually happens

1. After OAuth login, the page loads fine (HTTP 200).
2. Bokeh's session token embeds the originating request's cookies and headers. With Azure OAuth those cookies (access_token + id_token + refresh_token + user + oauth_expiry) are large.
3. The browser sends that token back in the Sec-WebSocket-Protocol header on the /ws request.
4. nginx's default large_client_header_buffers is 4 8k — a single header field must fit in one 8 KB buffer. The token header overflows it, so nginx returns HTTP 400 and never proxies the WS to the Bokeh server (the backend logs show nothing for these requests).

Evidence (from a Chromium net-export capture of the failing /ws)

WEBSOCKET_UPGRADE_FAILURE: "Error during WebSocket handshake: Unexpected response code: 400"
net_error: ERR_INVALID_RESPONSE (-320)

Request header:
  Sec-WebSocket-Protocol:  bokeh, <JWT ~8,366 bytes>      # header line ≈ 8,397 bytes  (> 8,192)

Decoding that JWT:

payload = { session_id, session_expiry, __bk__zlib_ : 6,163 B }
zlib-decompressed __bk__zlib_  ->  { cookies: 5,729 B, headers: 1,028 B, arguments, app_path }

So ~5.7 KB of OAuth cookies are round-tripped through the WebSocket subprotocol header, pushing it past 8 KB.

Why it "works in incognito" / looks like a multi-account issue

The failing session was only ~181 bytes over the 8 KB limit. An incognito session (or an account with smaller tokens / fewer group claims) produces cookies ~0.8 KB smaller, so the compressed token stays just under 8 KB and the WS connects. That thin margin is exactly why it presents as flaky and "account-dependent" — but the real variable is token size vs. the proxy's header-buffer limit (closely related to #7909).

Fixes / workarounds

• Proxy: raise the request-header buffer. For ingress-nginx, in the controller ConfigMap (it is not a per-Ingress annotation): large-client-header-buffers: "4 32k" (the buffer size must exceed the token, so ≥16k).
• App-side: stop Bokeh embedding the large OAuth cookies in the token, e.g. panel serve … --exclude-cookies access_token id_token refresh_token, and/or reduce Azure token size via group overage. This shrinks the token from ~8.4 KB to a few hundred bytes.

Suggestions for Panel

1. Don't round-trip large auth cookies through the WS subprotocol token by default. When OAuth is enabled, the access_token/id_token/refresh_token cookies are predictably large and are already re-sent in the Cookie header on the /ws request; embedding them again in the Sec-WebSocket-Protocol token reliably exceeds common proxy header limits (nginx default 8 KB). Consider excluding these cookies from the token by default, compressing/omitting them, or at least emitting a startup warning when the generated token approaches typical proxy limits. Documenting --exclude-cookies in the auth troubleshooting guide would help too.
2. Return a clean 403 for unauthenticated WebSocket upgrades. When a /ws request arrives without a valid authenticated user, the @authenticated-wrapped handler attempts an HTTP redirect to the login page, which is invalid for a WebSocket and raises an uncaught RuntimeError: Method not supported for Web Sockets:

   File ".../panel/io/server.py", line 751, in open
       return super().open()
   File ".../tornado/web.py", line 3406, in wrapper
       self.redirect(url)
   File ".../tornado/websocket.py", line 638, in _raise_not_supported_for_websockets
       raise RuntimeError("Method not supported for Web Sockets")
   

   Panel should reject the WS with a clean status instead of attempting a redirect.

[MarcSkovMadsen]

A note on edge vs firefox from investigations with Claude.

---

Why Edge's headers are bigger than Firefox's

Edge is built on Chromium, and Chromium browsers send extra headers that Firefox doesn't:

• User-Agent — Edge's is longer (…Chrome/148.0.0.0 Safari/537.36 Edg/148.0.0.0) than Firefox's.
• "Client hint" headers — Chromium sends sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform (browser/OS metadata).
  Firefox doesn't send these at all.

Those add up to ~238 bytes more in Edge. Bokeh copies them into the session token, so Edge's token ends up 8,397
bytes (just over 8,192) while Firefox's is 8,161 (just under).

[MarcSkovMadsen]

I can confirm that excluding some cookies --exclude-cookies access_token refresh_token works around the problem

[MarcSkovMadsen]

Root cause: the WebSocket token (Sec-WebSocket-Protocol header) exceeds the proxy's request-header buffer

After a fairly deep investigation I can now explain this precisely, and I think the title/framing is a bit misleading: it is almost always a request-header size problem, not a "multiple accounts" problem.

What actually happens

1. After OAuth login the page loads fine (HTTP 200).
2. Bokeh embeds the originating request's cookies + headers into the session token, and the browser sends that token back in the Sec-WebSocket-Protocol header when it opens the /ws connection.
3. With OAuth the embedded cookies (access_token + id_token + refresh_token) are large, so this single header can exceed the proxy's per-request-header limit — for nginx large_client_header_buffers, default 4 8k.
4. nginx then rejects the upgrade with HTTP 400 before it reaches the Panel server (nothing in the Panel logs), and the browser shows Could not open websocket.

Evidence

From a browser net-export capture of the failing /ws:

WEBSOCKET_UPGRADE_FAILURE: "Error during WebSocket handshake: Unexpected response code: 400"
net_error: ERR_INVALID_RESPONSE (-320)

Sec-WebSocket-Protocol:  bokeh, <JWT ~8,366 bytes>     # header line ≈ 8,397 bytes  (> 8,192)
   JWT payload -> __bk__zlib_ (zlib) -> { cookies: 5,729 B, headers: 1,028 B, ... }

Why "multiple accounts" / "only incognito works" is a red herring

Extra Microsoft accounts live as cookies on login.microsoftonline.com, not on the app's domain, so they don't enlarge the app's request headers. Incognito / a different browser / clearing cookies only appear to fix it because they carry slightly smaller headers and slip back under the limit.

Concrete example from the same machine and the same OAuth cookies:

Browser	Sec-WebSocket-Protocol header line	Result
Edge (normal)	8,397 B	❌ over nginx's 8,192 B → HTTP 400
Firefox (normal)	8,161 B	✅ under the limit

The only difference was the browser's own headers (Chromium sends extra sec-ch-ua* client hints + a longer UA, which Bokeh also packs into the token). It was ~181 bytes over — which is exactly why this feels random and account-dependent. This is the same class of problem as #7909.

Fixes / workarounds (any one of these unblocks it)

• Raise the proxy's request-header buffer (recommended): nginx large_client_header_buffers 4 32k;. Note this is a request-header setting — raising response buffers like proxy_buffer_size does nothing. On ingress-nginx it must go in the controller ConfigMap (large-client-header-buffers); it is not a per-Ingress annotation.
• Shrink the token: panel serve … --exclude-cookies access_token id_token refresh_token (they're still sent in the Cookie header for auth; just not packed into the token). Confirmed to resolve it.
• Reduce token size: fewer scopes, and Azure group overage so large group lists aren't inlined.

PRs

• Forward additional oauth_extra_params to the OAuth authorization endpoint #8635 — forward arbitrary oauth_extra_params (e.g. prompt) to the authorize endpoint, so prompt=select_account can be used to control which account is chosen (the genuine multiple-account UX concern).
• Set Cache-Control: no-store on the Bokeh document page #8636 — Cache-Control: no-store on the document page, to avoid a separate failure mode where a cached page serves a stale token after logout/login.
• docs: Fix the "Could not open websocket" / multiple-accounts OAuth troubleshooting #8637 — docs: rewrite the "Could not open websocket" troubleshooting to describe the real cause and fixes, and document prompt for account selection.

Two things I'd like maintainer input on before opening PRs

1. Warn when the session token is large enough to risk proxy rejection. Today this surfaces only as a cryptic Could not open websocket. Panel could log a one-time warning (pointing at --exclude-cookies / proxy config) when the generated token exceeds, say, ~7 KB. Worth doing? What threshold?
2. Return a clean 403 for unauthenticated WebSocket upgrades. When a /ws arrives without a valid user, the @authenticated-wrapped handler tries to HTTP-redirect a WebSocket and raises an uncaught RuntimeError: Method not supported for Web Sockets. It should reject with a clean status instead.

Happy to provide the full (sanitized) capture and the decoded token breakdown.

[philippjfr]

I think warning would be great! And giving a 403 response also sounds good. Thanks for the detailed investigation!