Skip to content

Restored install that pauses before its first download-started signal cannot be restarted (restart_failed no-ops, marker loses resume metadata) #9480

Description

@lstein

Summary

An install job restored from an on-disk install marker that pauses before its first download-started signal ends up in a PAUSED state that cannot be restarted: restart_failed() silently no-ops, and the PAUSED marker rewrite erases the persisted resume metadata. The only escape is cancelling the install and re-importing from scratch.

Found during the adversarial review of #9432 (see the review body there); the bug predates that PR, but #9432's stricter 416 gate made the trigger reachable with ordinary local state.

Mechanism

  • install_job.download_parts is populated only by _download_started_callback (model_install_default.py:1453). An install restored by _restore_incomplete_installs (model_install_default.py:214) starts with an empty download_parts — the marker's file metadata goes to job._resume_metadata instead.
  • The 416 resume-mismatch branch in _do_download pauses and raises before _signal_job_started (download_default.py:434-453). So if the first part of a resumed install hits it, no started callback ever fires and the install job's download_parts stays empty.
  • _download_cancelled_callback correctly detects resume_required — but on the multifile job's parts (model_install_default.py:1498) — and sets the install PAUSED. So far so good.
  • restart_failed() however reads the install job's download_parts (model_install_default.py:625) — empty → early return. The UI's restart action does nothing; the job is stuck PAUSED forever.
  • Compounding it: the PAUSED marker rewrite (_write_install_marker, gate at model_install_default.py:142) writes files: [], permanently discarding the persisted etag/canonical_url/expected_total_bytes/download_path metadata that the next restore/resume would have used.

Reproduction (manual)

  1. Start a multi-file install (e.g. an HF diffusers model). Let the first file download partially, then quit the app. An install marker with per-file resume metadata is written to the install tmpdir.
  2. While the app is stopped, append a few bytes of garbage to the first .downloading file in the tmpdir so it is larger than the remote file.
  3. Start the app and resume the install. The server answers the Range: bytes=N- request with 416 and a mismatched Content-Range: bytes */<total>, so the part pauses before any started signal ("Resume refused by server. Restart required."), and the install goes PAUSED.
  4. Trigger the restart action (the route that calls restart_failed). Observe: it returns immediately, nothing is enqueued, the job stays PAUSED. The marker on disk now contains "files": [].

Reproduction (test sketch)

The core assertion needs no HTTP at all:

def test_restart_failed_with_empty_install_parts(mm2_installer) -> None:
    # Simulate a marker-restored install: parts live only on the multifile job.
    install_job = ModelInstallJob(id=1, source=URLModelSource(url=...), config_in=ModelRecordChanges(), local_path=tmpdir)
    part = DownloadJob(source=..., dest=tmpdir)
    part.resume_required = True
    install_job._multifile_job = MultiFileDownloadJob(id=2, dest=tmpdir, download_parts=[part])
    assert install_job.download_parts == []          # restored state
    mm2_installer.restart_failed(install_job)
    assert install_job.status == InstallStatus.PAUSED  # FAILS today only in that nothing was enqueued/no state change

An end-to-end variant can drive it through DownloadQueueService with a TestAdapter returning 416 + Content-Range: bytes */8 against a larger pre-seeded .downloading file (see the tests added in #9432 for the pattern), with the install job constructed via the marker-restore path.

Suggested fix

  • restart_failed() (and anything else that reads job.download_parts for liveness) should fall back to job._multifile_job.download_parts when the install job's list is empty.
  • _write_install_marker should not overwrite a marker's existing files metadata with [] when the install job has no parts — skip the field or merge with what's on disk.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions