[BUG] Bilibili analysis hangs indefinitely on "Updating format list" (Windows subprocess pipe deadlock)

[KK-Hung]

Description
When clicking Analyze on a Bilibili video URL, YTSage gets stuck permanently on the "Updating format list" status. It never times out, never throws an error, and never displays the format table. Analyzing YouTube URLs under the same environment works perfectly.
Steps to Reproduce
Open YTSage, confirm Generic Mode is enabled
Paste any Bilibili video link, e.g. https://www.bilibili.com/video/BV1dnuW6EEWz
Click the Analyze button
Observe the status bar shows "Updating format list" and waits indefinitely
Expected Behavior
Like YouTube links, analysis should complete within seconds and then display video info, format list, and subtitle options.
Actual Behavior
The status bar stays on "Updating format list" forever. Clicking Analyze multiple times spawns multiple --dump-single-json subprocesses, but earlier ones never return any result or error.
Environment
OS: Windows 10 (Build 26200)
YTSage version: Latest (built-in yt-dlp updated to nightly@2026.08.04.234419)
yt-dlp path: C:\Users\kk_hu\AppData\Local\YTSage\bin\yt-dlp.exe
Deno path: C:\Users\kk_hu\AppData\Local\YTSage\bin\deno.exe
Network: Direct connection to Bilibili, no proxy
Key Log Comparison
YouTube analysis (normal, full callback):
plain
08:36:13.976 Executing yt-dlp command: ['--dump-single-json', '--flat-playlist', '--no-warnings', 'https://www.youtube.com/watch?v=qNsq7k8qJPQ']
08:37:03.173 SponsorBlock categories selected: []
08:37:16.867 Selected subtitles: ['en-US - Manual']
...subsequent format list and download flow appear normally
Bilibili analysis (abnormal, subprocess never returns):
plain
08:38:41.539 Executing yt-dlp command: ['--dump-single-json', '--flat-playlist', '--no-warnings', 'https://www.bilibili.com/video/BV1dnuW6EEWz']
08:38:50.823 Executing yt-dlp command: ['--dump-single-json', '--flat-playlist', '--no-warnings', 'https://www.bilibili.com/video/BV1dnuW6EEWz/?spm_id_from=...']
...after this, the log only shows repeated check_ytdlp_binary and check_deno_binary calls; no analysis success or failure callback ever appears
What I Have Ruled Out
yt-dlp itself is fine: running -F --simulate in YTSage's Custom Commands correctly prints the Bilibili format list
Running yt-dlp --dump-single-json directly in a terminal also returns valid JSON
Not a Cookie issue: cleared logs, disabled cookies, problem persists
Not a yt-dlp version issue: tested both stable and nightly channels, same behavior
Not a Generic Mode issue: confirmed enabled
Initial Analysis
I suspect a subprocess pipe deadlock in ytsage_gui_analysis.py's _analyze_url_with_subprocess on Windows. Bilibili's --dump-single-json output is typically much larger than YouTube's (includes anthology info, chapters, danmaku metadata, etc.), which may fill the default subprocess.PIPE buffer. This causes yt-dlp to block on the write side while YTSage blocks on the read side, resulting in a silent deadlock with no exception thrown.
Attachments
ytsage.log (clean log with cookies disabled)
ytsage_error.log (empty, no errors logged)

ytsage.log
ytsage_error.log

[konnen916]

Good report, the "what I have ruled out" list saved a lot of time here.

I think the pipe deadlock hypothesis does not hold, but there is a Windows specific mechanism that produces exactly this symptom, including the part where it survives the timeout.

Why it is probably not a buffer deadlock

_analyze_url_with_subprocess uses subprocess.run(..., capture_output=True), which goes through Popen.communicate(). On Windows communicate() starts a separate reader thread per pipe (CPython subprocess.py, _communicate under if _mswindows:), so stdout and stderr drain concurrently and a large payload cannot fill one buffer and block the child.

Measured on CPython 3.13, child writing 64 MB to stdout and 64 MB to stderr simultaneously through subprocess.run(capture_output=True):

rc=0  stdout=64 MB  stderr=64 MB  elapsed=0.77s

The classic deadlock needs Popen plus wait(), or reading one stream to completion before the other. subprocess.run is not that shape.

For what it is worth, --dump-single-json --flat-playlist on the URL in this report returns 5 bytes for me, since Bilibili answers HTTP 412 from my region, so I cannot reproduce the large payload case directly.

A mechanism that does match

The call passes timeout=300, so a plain hang should raise TimeoutExpired after five minutes and surface as an error in the UI. It does not. CPython's own subprocess.run explains why that can happen on Windows:

except TimeoutExpired as exc:
    process.kill()
    if _mswindows:
        # Windows accumulates the output in a single blocking
        # read() call run on child threads, with the timeout
        # being done in a join() on those threads.  communicate()
        # _after_ kill() is required to collect that and add it
        # to the exception.
        exc.stdout, exc.stderr = process.communicate()
    else:
        process.wait()
    raise

On Windows the timeout path kills the direct child and then calls communicate() a second time, which joins the reader threads. Those threads exit only on EOF. If the child spawned a grandchild that inherited the pipe write handles, killing the parent does not close them, EOF never arrives, the join never returns, and subprocess.run blocks indefinitely without raising anything.

That is a hang that outlives its own timeout and logs nothing, which matches an empty ytsage_error.log and no failure callback. Note the POSIX branch is just process.wait(), so this failure mode is Windows only, which also fits.

Two checks that would confirm or kill this

1. @KK-Hung, on one attempt, did you leave it for a full five minutes without clicking Analyze again? "Never times out" and "no status update for five minutes" look identical from the UI, and separating them narrows this a lot. Clicking Analyze again starts a second AnalysisThread, so it is worth testing with exactly one click.

2. While it is stuck, check whether a yt-dlp.exe is still alive:

wmic process where "name='yt-dlp.exe'" get ProcessId,ParentProcessId,CommandLine

A surviving process whose parent is gone would confirm the inherited handle path.

If it holds up, the fix is not in how the output is read. It is to stop the grandchild inheriting the pipes, or to drive this with Popen and kill the whole process tree on timeout instead of relying on what subprocess.run does internally.

[KK-Hung]

I ran both checks:
Process check: While the UI shows "Updating format list...", Get-CimInstance Win32_Process returns no running yt-dlp.exe at all. The child process seems to have already exited.
Timeout check: Clicked Analyze exactly once, left it untouched for the full 5 minutes (300s). No timeout error, no status change, no UI update. It stays on "Updating format list..." indefinitely.
So the symptom is: yt-dlp finishes (or dies) quickly, but subprocess.run or the AnalysisThread never returns, and the timeout never fires. The process is gone but the caller is still blocked.
Let me know if you need any other diagnostics.

[konnen916]

Thanks for running both. That kills my theory, and it also rules out the subprocess entirely.

No surviving yt-dlp.exe means communicate() is not blocked on an inherited handle, and no TimeoutExpired after a full 300 seconds means subprocess.run already returned. So the child ran, finished, and the code moved on. I anchored on the wrong function.

The status text is the giveaway, and I should have followed it first. "Updating format table" is analyzing_updating_table in en.json, emitted at ytsage_gui_analysis.py:264:

self.status_update.emit( _("main_ui.analyzing_updating_table") )
self.progress_update.emit(95)

# Emit all results at once
self.analysis_complete.emit(result_data)

That is the last line of the analysis thread. So the thread reaches the end and finishes. What is stuck is the main thread, in the slot that receives analysis_complete and fills the table.

That slot is _populate_format_table in ytsage_gui_format_table.py, and per row it does:

• allocates a QCheckBox, a container QWidget and a QHBoxLayout, then setCellWidget, which puts a real widget in the viewport for every row
• runs up to five any( term.lower() in quality_lower for term in [...] ) scans, over lists of roughly 20 to 35 localized terms, calling .lower() on every term again on every row

For a YouTube video that is maybe thirty rows and nobody notices. Bilibili returns a lot more formats, which is the one thing you already identified as different. That is not a deadlock, it is the GUI thread doing a very large amount of synchronous work with no exception to raise and nothing to log, which matches every symptom you reported: no error, no timeout, empty ytsage_error.log, child long gone.

One question that separates the two cleanly, and it takes ten seconds. While it is stuck, try dragging the window. If the title bar goes grey and Windows says "Not Responding", the main thread is busy and this is the explanation. If the window still moves and repaints normally, it is waiting on something and I am wrong again.

The other useful number, if you have a moment:

yt-dlp -F "https://www.bilibili.com/video/BV1dnuW6EEWz" | find /c /v ""

against the same for a YouTube URL. If Bilibili is returning hundreds of formats where YouTube returns tens, that is the whole story.

If it holds, the fix is cheap and does not involve the subprocess at all: wrap the loop in setUpdatesEnabled(False) and setSortingEnabled(False), hoist the lowercased term lists to module level instead of rebuilding them per row, and replace the checkbox-in-a-container with a checkable QTableWidgetItem so no widget is allocated per row.