Skip to content

fix(cookbook): recover completed downloads from DOWNLOAD_OK in background reconciler#4000

Merged
pewdiepie-archdaemon merged 1 commit into
odysseus-dev:devfrom
maxmilian:fix/cookbook-download-reconcile-crashed-3897
Jun 15, 2026
Merged

fix(cookbook): recover completed downloads from DOWNLOAD_OK in background reconciler#4000
pewdiepie-archdaemon merged 1 commit into
odysseus-dev:devfrom
maxmilian:fix/cookbook-download-reconcile-crashed-3897

Conversation

@maxmilian

@maxmilian maxmilian commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

A completed model download was being shown as crashed on the dashboard. The background status reconciler (static/js/cookbookRunning.js, _pollBackgroundStatus) only recovered done for dependency installs when the backend reported a finished task as stopped. A real model download whose tmux pane is gone after DOWNLOAD_OK (so the dead-session check misses the landed snapshot) fell through to task.type === 'download' ? 'crashed', so a download you watched reach DOWNLOAD OK showed up as crashed after navigating away (and stalled on the Serve tab). This mirrors the existing dep-install recovery into the download branch.

Target branch

  • This PR targets dev, not main.

Linked Issue

Fixes #3897

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)

Checklist

  • I searched open issues and open PRs — this is not a duplicate.
  • This PR targets dev
  • My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
  • I actually ran the app (uvicorn app:app) and verified the change end-to-end — see the before/after below.

How to Test

  1. Open Cookbook, download a model and wait for DOWNLOAD OK.
  2. Navigate away (e.g. to the Serve tab) so the download's tmux pane is reaped and the next /api/cookbook/tasks/status poll reports the task as stopped.
  3. Return to the Download/Running tab.
    • Before: the finished model is shown as crashed (and stalled on Serve).
    • After: because the retained output still carries DOWNLOAD_OK, the reconciler recovers it to finished/done instead of crashed.

Regression test (source-invariant pin, same style as the existing dep-install test — this repo has no JS test runner):

python -m pytest tests/test_cookbook_dependency_completion_regression.py

node --check static/js/cookbookRunning.js passes.

Root cause

_pollBackgroundStatus(): on live.status === 'stopped', only depDone (payload._dep + _depInstallSucceeded) recovered done; a model download had no equivalent recovery, so task.type === 'download' mapped straight to crashed.

Fix

Add downloadDone = task.type === 'download' && String(task.output || '').includes('DOWNLOAD_OK') and treat depDone || downloadDone as recovered. DOWNLOAD_OK is emitted by the runner only after it exits 0 (if [ $_ec -eq 0 ]; then echo "DOWNLOAD_OK"), so it's a conclusive terminal marker.

This background poll keys off DOWNLOAD_OK only — deliberately not the /snapshots/ path that the reconnect loop also accepts. The reconnect loop runs with the live stream in view and can debounce ambiguous markers; this poll runs blind after a refresh, and /snapshots/ can be printed mid-stream for multi-file downloads, so accepting it here could mark an incomplete download done. Keying off the exit-0 sentinel keeps the blind path conservative.

Visual / UI changes

This is a status-computation change inside a static/js/ module, not a styling change: a finished download resolves to the existing done/finished badge instead of the crashed one — no new colors, classes, fonts, spacing, SVG, or component patterns, and the same rendering path is reused.

Verified end-to-end against a locally-run app (uvicorn app:app): a download task whose output carries DOWNLOAD_OK is driven into the dead-session (stopped) state that the /api/cookbook/tasks/status poll reports, then the Running tab is rendered.

Before (current dev) — the completed download is mislabeled crashed:

odysseus-3897-before-crashed

After (this PR) — the same task recovers to finished:

odysseus-3897-after-finished
  • Screenshot of the change in the running app (before/after above).
  • Style match: reuses the existing finished/done badge + green check; no new colors/classes/fonts/spacing/SVG.
  • No new component patterns — the existing card rendering path is reused.
  • Not a bulk auto-generated PR — single focused fix; root cause filed and claimed on download marking as crashed #3897 first.

Scope

Deliberately does not touch the backend status detection. If the backend separately reports stopped for GGUF cache layouts where it should report completed, that's a distinct follow-up; this PR fixes the frontend mislabel of a download that already emitted DOWNLOAD_OK.

@github-actions github-actions Bot added the ready for review Description complete — ready for maintainer review label Jun 11, 2026
…ound reconciler

The dashboard background status reconciler (_pollBackgroundStatus) only
recovered "done" for dependency installs when the backend reported a
finished task as "stopped". A real model download whose tmux pane is
gone after DOWNLOAD_OK (so the dead-session check misses the landed
snapshot) fell through to `task.type === 'download' ? 'crashed'`, so a
completed download was shown as crashed (and stalled on the Serve tab).

Recover "done" from the terminal DOWNLOAD_OK sentinel, mirroring the
dep-install recovery already present. The background poll runs blind, so
it keys off the conclusive exit-0 sentinel only — not the `/snapshots/`
path, which can be printed mid-stream for multi-file downloads and would
risk marking an incomplete download done.

Fixes odysseus-dev#3897

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxmilian
maxmilian force-pushed the fix/cookbook-download-reconcile-crashed-3897 branch from 42f223a to 8174bf7 Compare June 11, 2026 23:17
@maxmilian
maxmilian marked this pull request as ready for review June 11, 2026 23:31
ashvinctrl added a commit to ashvinctrl/odysseus that referenced this pull request Jun 12, 2026
… stopped

When a download's tmux pane is gone, the status endpoint trusted only the
HF-cache probe to tell completed from stopped. The probe derives its cache
root from its own environment, but the download runner exports
HF_HOME=<local_dir> (the odysseus-dev#2722 fix), so custom-dir downloads land in
<local_dir>/hub where the probe never looks - and ollama pulls don't touch
the HF cache at all. Finished downloads were reported as stopped forever,
and tasks already persisted as completed were demoted back to stopped on
the next poll. This is the backend half of odysseus-dev#3897, deliberately left out of
the frontend fix in odysseus-dev#4000.

- honor the conclusive runner markers first: DOWNLOAD_OK -> completed
  (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error
- pass the task's local_dir through to the cache probes so they check the
  cache the download actually wrote to, keeping the env-var fallback for
  default-cache downloads
- move the probe scripts and marker classification into
  routes/cookbook_output.py (dependency-free) with behavioral tests

Fixes odysseus-dev#4017
@alteixeira20 alteixeira20 added the bug Something isn't working label Jun 12, 2026
pewdiepie-archdaemon pushed a commit that referenced this pull request Jun 15, 2026
… stopped (#4025)

When a download's tmux pane is gone, the status endpoint trusted only the
HF-cache probe to tell completed from stopped. The probe derives its cache
root from its own environment, but the download runner exports
HF_HOME=<local_dir> (the #2722 fix), so custom-dir downloads land in
<local_dir>/hub where the probe never looks - and ollama pulls don't touch
the HF cache at all. Finished downloads were reported as stopped forever,
and tasks already persisted as completed were demoted back to stopped on
the next poll. This is the backend half of #3897, deliberately left out of
the frontend fix in #4000.

- honor the conclusive runner markers first: DOWNLOAD_OK -> completed
  (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error
- pass the task's local_dir through to the cache probes so they check the
  cache the download actually wrote to, keeping the env-var fallback for
  default-cache downloads
- move the probe scripts and marker classification into
  routes/cookbook_output.py (dependency-free) with behavioral tests

Fixes #4017

@pewdiepie-archdaemon pewdiepie-archdaemon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still covers a frontend reconciliation edge after the backend dead-download classifier work: if the background status poll sees a stopped download whose retained output has DOWNLOAD_OK, the local task should recover to done instead of crashed. I temporarily merged current origin/dev locally and verified the combined result: node --check static/js/cookbookRunning.js, git diff --check origin/dev...HEAD, and /home/diogenes/selfhost/odysseus-ui/venv/bin/python -m pytest tests/test_cookbook_dependency_completion_regression.py tests/test_cookbook_dead_download_status.py -> 20 passed.

@pewdiepie-archdaemon
pewdiepie-archdaemon merged commit 65c7321 into odysseus-dev:dev Jun 15, 2026
26 checks passed
entitycs pushed a commit to entitycs/odysseus that referenced this pull request Jun 15, 2026
… stopped (odysseus-dev#4025)

When a download's tmux pane is gone, the status endpoint trusted only the
HF-cache probe to tell completed from stopped. The probe derives its cache
root from its own environment, but the download runner exports
HF_HOME=<local_dir> (the odysseus-dev#2722 fix), so custom-dir downloads land in
<local_dir>/hub where the probe never looks - and ollama pulls don't touch
the HF cache at all. Finished downloads were reported as stopped forever,
and tasks already persisted as completed were demoted back to stopped on
the next poll. This is the backend half of odysseus-dev#3897, deliberately left out of
the frontend fix in odysseus-dev#4000.

- honor the conclusive runner markers first: DOWNLOAD_OK -> completed
  (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error
- pass the task's local_dir through to the cache probes so they check the
  cache the download actually wrote to, keeping the env-var fallback for
  default-cache downloads
- move the probe scripts and marker classification into
  routes/cookbook_output.py (dependency-free) with behavioral tests

Fixes odysseus-dev#4017
entitycs pushed a commit to entitycs/odysseus that referenced this pull request Jun 15, 2026
…ound reconciler (odysseus-dev#4000)

The dashboard background status reconciler (_pollBackgroundStatus) only
recovered "done" for dependency installs when the backend reported a
finished task as "stopped". A real model download whose tmux pane is
gone after DOWNLOAD_OK (so the dead-session check misses the landed
snapshot) fell through to `task.type === 'download' ? 'crashed'`, so a
completed download was shown as crashed (and stalled on the Serve tab).

Recover "done" from the terminal DOWNLOAD_OK sentinel, mirroring the
dep-install recovery already present. The background poll runs blind, so
it keys off the conclusive exit-0 sentinel only — not the `/snapshots/`
path, which can be printed mid-stream for multi-file downloads and would
risk marking an incomplete download done.

Fixes odysseus-dev#3897

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kootenayalex pushed a commit to kootenayalex/odysseus-mlx that referenced this pull request Jun 16, 2026
… stopped (odysseus-dev#4025)

When a download's tmux pane is gone, the status endpoint trusted only the
HF-cache probe to tell completed from stopped. The probe derives its cache
root from its own environment, but the download runner exports
HF_HOME=<local_dir> (the odysseus-dev#2722 fix), so custom-dir downloads land in
<local_dir>/hub where the probe never looks - and ollama pulls don't touch
the HF cache at all. Finished downloads were reported as stopped forever,
and tasks already persisted as completed were demoted back to stopped on
the next poll. This is the backend half of odysseus-dev#3897, deliberately left out of
the frontend fix in odysseus-dev#4000.

- honor the conclusive runner markers first: DOWNLOAD_OK -> completed
  (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error
- pass the task's local_dir through to the cache probes so they check the
  cache the download actually wrote to, keeping the env-var fallback for
  default-cache downloads
- move the probe scripts and marker classification into
  routes/cookbook_output.py (dependency-free) with behavioral tests

Fixes odysseus-dev#4017
kootenayalex pushed a commit to kootenayalex/odysseus-mlx that referenced this pull request Jun 16, 2026
…ound reconciler (odysseus-dev#4000)

The dashboard background status reconciler (_pollBackgroundStatus) only
recovered "done" for dependency installs when the backend reported a
finished task as "stopped". A real model download whose tmux pane is
gone after DOWNLOAD_OK (so the dead-session check misses the landed
snapshot) fell through to `task.type === 'download' ? 'crashed'`, so a
completed download was shown as crashed (and stalled on the Serve tab).

Recover "done" from the terminal DOWNLOAD_OK sentinel, mirroring the
dep-install recovery already present. The background poll runs blind, so
it keys off the conclusive exit-0 sentinel only — not the `/snapshots/`
path, which can be printed mid-stream for multi-file downloads and would
risk marking an incomplete download done.

Fixes odysseus-dev#3897
buluma pushed a commit to buluma/odysseus that referenced this pull request Jul 1, 2026
… stopped (odysseus-dev#4025)

When a download's tmux pane is gone, the status endpoint trusted only the
HF-cache probe to tell completed from stopped. The probe derives its cache
root from its own environment, but the download runner exports
HF_HOME=<local_dir> (the odysseus-dev#2722 fix), so custom-dir downloads land in
<local_dir>/hub where the probe never looks - and ollama pulls don't touch
the HF cache at all. Finished downloads were reported as stopped forever,
and tasks already persisted as completed were demoted back to stopped on
the next poll. This is the backend half of odysseus-dev#3897, deliberately left out of
the frontend fix in odysseus-dev#4000.

- honor the conclusive runner markers first: DOWNLOAD_OK -> completed
  (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error
- pass the task's local_dir through to the cache probes so they check the
  cache the download actually wrote to, keeping the env-var fallback for
  default-cache downloads
- move the probe scripts and marker classification into
  routes/cookbook_output.py (dependency-free) with behavioral tests

Fixes odysseus-dev#4017
buluma pushed a commit to buluma/odysseus that referenced this pull request Jul 1, 2026
…ound reconciler (odysseus-dev#4000)

The dashboard background status reconciler (_pollBackgroundStatus) only
recovered "done" for dependency installs when the backend reported a
finished task as "stopped". A real model download whose tmux pane is
gone after DOWNLOAD_OK (so the dead-session check misses the landed
snapshot) fell through to `task.type === 'download' ? 'crashed'`, so a
completed download was shown as crashed (and stalled on the Serve tab).

Recover "done" from the terminal DOWNLOAD_OK sentinel, mirroring the
dep-install recovery already present. The background poll runs blind, so
it keys off the conclusive exit-0 sentinel only — not the `/snapshots/`
path, which can be printed mid-stream for multi-file downloads and would
risk marking an incomplete download done.

Fixes odysseus-dev#3897

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ToyVo pushed a commit to ToyVo/odysseus that referenced this pull request Jul 7, 2026
… stopped (odysseus-dev#4025)

When a download's tmux pane is gone, the status endpoint trusted only the
HF-cache probe to tell completed from stopped. The probe derives its cache
root from its own environment, but the download runner exports
HF_HOME=<local_dir> (the odysseus-dev#2722 fix), so custom-dir downloads land in
<local_dir>/hub where the probe never looks - and ollama pulls don't touch
the HF cache at all. Finished downloads were reported as stopped forever,
and tasks already persisted as completed were demoted back to stopped on
the next poll. This is the backend half of odysseus-dev#3897, deliberately left out of
the frontend fix in odysseus-dev#4000.

- honor the conclusive runner markers first: DOWNLOAD_OK -> completed
  (keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error
- pass the task's local_dir through to the cache probes so they check the
  cache the download actually wrote to, keeping the env-var fallback for
  default-cache downloads
- move the probe scripts and marker classification into
  routes/cookbook_output.py (dependency-free) with behavioral tests

Fixes odysseus-dev#4017
ToyVo pushed a commit to ToyVo/odysseus that referenced this pull request Jul 7, 2026
…ound reconciler (odysseus-dev#4000)

The dashboard background status reconciler (_pollBackgroundStatus) only
recovered "done" for dependency installs when the backend reported a
finished task as "stopped". A real model download whose tmux pane is
gone after DOWNLOAD_OK (so the dead-session check misses the landed
snapshot) fell through to `task.type === 'download' ? 'crashed'`, so a
completed download was shown as crashed (and stalled on the Serve tab).

Recover "done" from the terminal DOWNLOAD_OK sentinel, mirroring the
dep-install recovery already present. The background poll runs blind, so
it keys off the conclusive exit-0 sentinel only — not the `/snapshots/`
path, which can be printed mid-stream for multi-file downloads and would
risk marking an incomplete download done.

Fixes odysseus-dev#3897

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

download marking as crashed

3 participants