fix(cookbook): recover completed downloads from DOWNLOAD_OK in background reconciler#4000
Merged
pewdiepie-archdaemon merged 1 commit intoJun 15, 2026
Conversation
…ound reconciler The dashboard background status reconciler (_pollBackgroundStatus) only recovered "done" for dependency installs when the backend reported a finished task as "stopped". A real model download whose tmux pane is gone after DOWNLOAD_OK (so the dead-session check misses the landed snapshot) fell through to `task.type === 'download' ? 'crashed'`, so a completed download was shown as crashed (and stalled on the Serve tab). Recover "done" from the terminal DOWNLOAD_OK sentinel, mirroring the dep-install recovery already present. The background poll runs blind, so it keys off the conclusive exit-0 sentinel only — not the `/snapshots/` path, which can be printed mid-stream for multi-file downloads and would risk marking an incomplete download done. Fixes odysseus-dev#3897 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
maxmilian
force-pushed
the
fix/cookbook-download-reconcile-crashed-3897
branch
from
June 11, 2026 23:17
42f223a to
8174bf7
Compare
maxmilian
marked this pull request as ready for review
June 11, 2026 23:31
This was referenced Jun 12, 2026
ashvinctrl
added a commit
to ashvinctrl/odysseus
that referenced
this pull request
Jun 12, 2026
… stopped When a download's tmux pane is gone, the status endpoint trusted only the HF-cache probe to tell completed from stopped. The probe derives its cache root from its own environment, but the download runner exports HF_HOME=<local_dir> (the odysseus-dev#2722 fix), so custom-dir downloads land in <local_dir>/hub where the probe never looks - and ollama pulls don't touch the HF cache at all. Finished downloads were reported as stopped forever, and tasks already persisted as completed were demoted back to stopped on the next poll. This is the backend half of odysseus-dev#3897, deliberately left out of the frontend fix in odysseus-dev#4000. - honor the conclusive runner markers first: DOWNLOAD_OK -> completed (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error - pass the task's local_dir through to the cache probes so they check the cache the download actually wrote to, keeping the env-var fallback for default-cache downloads - move the probe scripts and marker classification into routes/cookbook_output.py (dependency-free) with behavioral tests Fixes odysseus-dev#4017
pewdiepie-archdaemon
pushed a commit
that referenced
this pull request
Jun 15, 2026
… stopped (#4025) When a download's tmux pane is gone, the status endpoint trusted only the HF-cache probe to tell completed from stopped. The probe derives its cache root from its own environment, but the download runner exports HF_HOME=<local_dir> (the #2722 fix), so custom-dir downloads land in <local_dir>/hub where the probe never looks - and ollama pulls don't touch the HF cache at all. Finished downloads were reported as stopped forever, and tasks already persisted as completed were demoted back to stopped on the next poll. This is the backend half of #3897, deliberately left out of the frontend fix in #4000. - honor the conclusive runner markers first: DOWNLOAD_OK -> completed (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error - pass the task's local_dir through to the cache probes so they check the cache the download actually wrote to, keeping the env-var fallback for default-cache downloads - move the probe scripts and marker classification into routes/cookbook_output.py (dependency-free) with behavioral tests Fixes #4017
pewdiepie-archdaemon
approved these changes
Jun 15, 2026
pewdiepie-archdaemon
left a comment
Collaborator
There was a problem hiding this comment.
This still covers a frontend reconciliation edge after the backend dead-download classifier work: if the background status poll sees a stopped download whose retained output has DOWNLOAD_OK, the local task should recover to done instead of crashed. I temporarily merged current origin/dev locally and verified the combined result: node --check static/js/cookbookRunning.js, git diff --check origin/dev...HEAD, and /home/diogenes/selfhost/odysseus-ui/venv/bin/python -m pytest tests/test_cookbook_dependency_completion_regression.py tests/test_cookbook_dead_download_status.py -> 20 passed.
entitycs
pushed a commit
to entitycs/odysseus
that referenced
this pull request
Jun 15, 2026
… stopped (odysseus-dev#4025) When a download's tmux pane is gone, the status endpoint trusted only the HF-cache probe to tell completed from stopped. The probe derives its cache root from its own environment, but the download runner exports HF_HOME=<local_dir> (the odysseus-dev#2722 fix), so custom-dir downloads land in <local_dir>/hub where the probe never looks - and ollama pulls don't touch the HF cache at all. Finished downloads were reported as stopped forever, and tasks already persisted as completed were demoted back to stopped on the next poll. This is the backend half of odysseus-dev#3897, deliberately left out of the frontend fix in odysseus-dev#4000. - honor the conclusive runner markers first: DOWNLOAD_OK -> completed (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error - pass the task's local_dir through to the cache probes so they check the cache the download actually wrote to, keeping the env-var fallback for default-cache downloads - move the probe scripts and marker classification into routes/cookbook_output.py (dependency-free) with behavioral tests Fixes odysseus-dev#4017
entitycs
pushed a commit
to entitycs/odysseus
that referenced
this pull request
Jun 15, 2026
…ound reconciler (odysseus-dev#4000) The dashboard background status reconciler (_pollBackgroundStatus) only recovered "done" for dependency installs when the backend reported a finished task as "stopped". A real model download whose tmux pane is gone after DOWNLOAD_OK (so the dead-session check misses the landed snapshot) fell through to `task.type === 'download' ? 'crashed'`, so a completed download was shown as crashed (and stalled on the Serve tab). Recover "done" from the terminal DOWNLOAD_OK sentinel, mirroring the dep-install recovery already present. The background poll runs blind, so it keys off the conclusive exit-0 sentinel only — not the `/snapshots/` path, which can be printed mid-stream for multi-file downloads and would risk marking an incomplete download done. Fixes odysseus-dev#3897 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kootenayalex
pushed a commit
to kootenayalex/odysseus-mlx
that referenced
this pull request
Jun 16, 2026
… stopped (odysseus-dev#4025) When a download's tmux pane is gone, the status endpoint trusted only the HF-cache probe to tell completed from stopped. The probe derives its cache root from its own environment, but the download runner exports HF_HOME=<local_dir> (the odysseus-dev#2722 fix), so custom-dir downloads land in <local_dir>/hub where the probe never looks - and ollama pulls don't touch the HF cache at all. Finished downloads were reported as stopped forever, and tasks already persisted as completed were demoted back to stopped on the next poll. This is the backend half of odysseus-dev#3897, deliberately left out of the frontend fix in odysseus-dev#4000. - honor the conclusive runner markers first: DOWNLOAD_OK -> completed (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error - pass the task's local_dir through to the cache probes so they check the cache the download actually wrote to, keeping the env-var fallback for default-cache downloads - move the probe scripts and marker classification into routes/cookbook_output.py (dependency-free) with behavioral tests Fixes odysseus-dev#4017
kootenayalex
pushed a commit
to kootenayalex/odysseus-mlx
that referenced
this pull request
Jun 16, 2026
…ound reconciler (odysseus-dev#4000) The dashboard background status reconciler (_pollBackgroundStatus) only recovered "done" for dependency installs when the backend reported a finished task as "stopped". A real model download whose tmux pane is gone after DOWNLOAD_OK (so the dead-session check misses the landed snapshot) fell through to `task.type === 'download' ? 'crashed'`, so a completed download was shown as crashed (and stalled on the Serve tab). Recover "done" from the terminal DOWNLOAD_OK sentinel, mirroring the dep-install recovery already present. The background poll runs blind, so it keys off the conclusive exit-0 sentinel only — not the `/snapshots/` path, which can be printed mid-stream for multi-file downloads and would risk marking an incomplete download done. Fixes odysseus-dev#3897
buluma
pushed a commit
to buluma/odysseus
that referenced
this pull request
Jul 1, 2026
… stopped (odysseus-dev#4025) When a download's tmux pane is gone, the status endpoint trusted only the HF-cache probe to tell completed from stopped. The probe derives its cache root from its own environment, but the download runner exports HF_HOME=<local_dir> (the odysseus-dev#2722 fix), so custom-dir downloads land in <local_dir>/hub where the probe never looks - and ollama pulls don't touch the HF cache at all. Finished downloads were reported as stopped forever, and tasks already persisted as completed were demoted back to stopped on the next poll. This is the backend half of odysseus-dev#3897, deliberately left out of the frontend fix in odysseus-dev#4000. - honor the conclusive runner markers first: DOWNLOAD_OK -> completed (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error - pass the task's local_dir through to the cache probes so they check the cache the download actually wrote to, keeping the env-var fallback for default-cache downloads - move the probe scripts and marker classification into routes/cookbook_output.py (dependency-free) with behavioral tests Fixes odysseus-dev#4017
buluma
pushed a commit
to buluma/odysseus
that referenced
this pull request
Jul 1, 2026
…ound reconciler (odysseus-dev#4000) The dashboard background status reconciler (_pollBackgroundStatus) only recovered "done" for dependency installs when the backend reported a finished task as "stopped". A real model download whose tmux pane is gone after DOWNLOAD_OK (so the dead-session check misses the landed snapshot) fell through to `task.type === 'download' ? 'crashed'`, so a completed download was shown as crashed (and stalled on the Serve tab). Recover "done" from the terminal DOWNLOAD_OK sentinel, mirroring the dep-install recovery already present. The background poll runs blind, so it keys off the conclusive exit-0 sentinel only — not the `/snapshots/` path, which can be printed mid-stream for multi-file downloads and would risk marking an incomplete download done. Fixes odysseus-dev#3897 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ToyVo
pushed a commit
to ToyVo/odysseus
that referenced
this pull request
Jul 7, 2026
… stopped (odysseus-dev#4025) When a download's tmux pane is gone, the status endpoint trusted only the HF-cache probe to tell completed from stopped. The probe derives its cache root from its own environment, but the download runner exports HF_HOME=<local_dir> (the odysseus-dev#2722 fix), so custom-dir downloads land in <local_dir>/hub where the probe never looks - and ollama pulls don't touch the HF cache at all. Finished downloads were reported as stopped forever, and tasks already persisted as completed were demoted back to stopped on the next poll. This is the backend half of odysseus-dev#3897, deliberately left out of the frontend fix in odysseus-dev#4000. - honor the conclusive runner markers first: DOWNLOAD_OK -> completed (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error - pass the task's local_dir through to the cache probes so they check the cache the download actually wrote to, keeping the env-var fallback for default-cache downloads - move the probe scripts and marker classification into routes/cookbook_output.py (dependency-free) with behavioral tests Fixes odysseus-dev#4017
ToyVo
pushed a commit
to ToyVo/odysseus
that referenced
this pull request
Jul 7, 2026
…ound reconciler (odysseus-dev#4000) The dashboard background status reconciler (_pollBackgroundStatus) only recovered "done" for dependency installs when the backend reported a finished task as "stopped". A real model download whose tmux pane is gone after DOWNLOAD_OK (so the dead-session check misses the landed snapshot) fell through to `task.type === 'download' ? 'crashed'`, so a completed download was shown as crashed (and stalled on the Serve tab). Recover "done" from the terminal DOWNLOAD_OK sentinel, mirroring the dep-install recovery already present. The background poll runs blind, so it keys off the conclusive exit-0 sentinel only — not the `/snapshots/` path, which can be printed mid-stream for multi-file downloads and would risk marking an incomplete download done. Fixes odysseus-dev#3897 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A completed model download was being shown as crashed on the dashboard. The background status reconciler (
static/js/cookbookRunning.js,_pollBackgroundStatus) only recovereddonefor dependency installs when the backend reported a finished task asstopped. A real model download whose tmux pane is gone afterDOWNLOAD_OK(so the dead-session check misses the landed snapshot) fell through totask.type === 'download' ? 'crashed', so a download you watched reachDOWNLOAD OKshowed up as crashed after navigating away (andstalledon the Serve tab). This mirrors the existing dep-install recovery into the download branch.Target branch
dev, notmain.Linked Issue
Fixes #3897
Type of Change
Checklist
devuvicorn app:app) and verified the change end-to-end — see the before/after below.How to Test
DOWNLOAD OK./api/cookbook/tasks/statuspoll reports the task asstopped.DOWNLOAD_OK, the reconciler recovers it to finished/done instead of crashed.Regression test (source-invariant pin, same style as the existing dep-install test — this repo has no JS test runner):
node --check static/js/cookbookRunning.jspasses.Root cause
_pollBackgroundStatus(): onlive.status === 'stopped', onlydepDone(payload._dep+_depInstallSucceeded) recovereddone; a model download had no equivalent recovery, sotask.type === 'download'mapped straight tocrashed.Fix
Add
downloadDone = task.type === 'download' && String(task.output || '').includes('DOWNLOAD_OK')and treatdepDone || downloadDoneas recovered.DOWNLOAD_OKis emitted by the runner only after it exits 0 (if [ $_ec -eq 0 ]; then echo "DOWNLOAD_OK"), so it's a conclusive terminal marker.This background poll keys off
DOWNLOAD_OKonly — deliberately not the/snapshots/path that the reconnect loop also accepts. The reconnect loop runs with the live stream in view and can debounce ambiguous markers; this poll runs blind after a refresh, and/snapshots/can be printed mid-stream for multi-file downloads, so accepting it here could mark an incomplete downloaddone. Keying off the exit-0 sentinel keeps the blind path conservative.Visual / UI changes
This is a status-computation change inside a
static/js/module, not a styling change: a finished download resolves to the existingdone/finishedbadge instead of thecrashedone — no new colors, classes, fonts, spacing, SVG, or component patterns, and the same rendering path is reused.Verified end-to-end against a locally-run app (
uvicorn app:app): adownloadtask whose output carriesDOWNLOAD_OKis driven into the dead-session (stopped) state that the/api/cookbook/tasks/statuspoll reports, then the Running tab is rendered.Before (current
dev) — the completed download is mislabeled crashed:After (this PR) — the same task recovers to finished:
finished/donebadge + green check; no new colors/classes/fonts/spacing/SVG.Scope
Deliberately does not touch the backend status detection. If the backend separately reports
stoppedfor GGUF cache layouts where it should reportcompleted, that's a distinct follow-up; this PR fixes the frontend mislabel of a download that already emittedDOWNLOAD_OK.