Summary
An install job restored from an on-disk install marker that pauses before its first download-started signal ends up in a PAUSED state that cannot be restarted: restart_failed() silently no-ops, and the PAUSED marker rewrite erases the persisted resume metadata. The only escape is cancelling the install and re-importing from scratch.
Found during the adversarial review of #9432 (see the review body there); the bug predates that PR, but #9432's stricter 416 gate made the trigger reachable with ordinary local state.
Mechanism
install_job.download_parts is populated only by _download_started_callback (model_install_default.py:1453). An install restored by _restore_incomplete_installs (model_install_default.py:214) starts with an empty download_parts — the marker's file metadata goes to job._resume_metadata instead.
- The 416 resume-mismatch branch in
_do_download pauses and raises before _signal_job_started (download_default.py:434-453). So if the first part of a resumed install hits it, no started callback ever fires and the install job's download_parts stays empty.
_download_cancelled_callback correctly detects resume_required — but on the multifile job's parts (model_install_default.py:1498) — and sets the install PAUSED. So far so good.
restart_failed() however reads the install job's download_parts (model_install_default.py:625) — empty → early return. The UI's restart action does nothing; the job is stuck PAUSED forever.
- Compounding it: the PAUSED marker rewrite (
_write_install_marker, gate at model_install_default.py:142) writes files: [], permanently discarding the persisted etag/canonical_url/expected_total_bytes/download_path metadata that the next restore/resume would have used.
Reproduction (manual)
- Start a multi-file install (e.g. an HF diffusers model). Let the first file download partially, then quit the app. An install marker with per-file resume metadata is written to the install tmpdir.
- While the app is stopped, append a few bytes of garbage to the first
.downloading file in the tmpdir so it is larger than the remote file.
- Start the app and resume the install. The server answers the
Range: bytes=N- request with 416 and a mismatched Content-Range: bytes */<total>, so the part pauses before any started signal ("Resume refused by server. Restart required."), and the install goes PAUSED.
- Trigger the restart action (the route that calls
restart_failed). Observe: it returns immediately, nothing is enqueued, the job stays PAUSED. The marker on disk now contains "files": [].
Reproduction (test sketch)
The core assertion needs no HTTP at all:
def test_restart_failed_with_empty_install_parts(mm2_installer) -> None:
# Simulate a marker-restored install: parts live only on the multifile job.
install_job = ModelInstallJob(id=1, source=URLModelSource(url=...), config_in=ModelRecordChanges(), local_path=tmpdir)
part = DownloadJob(source=..., dest=tmpdir)
part.resume_required = True
install_job._multifile_job = MultiFileDownloadJob(id=2, dest=tmpdir, download_parts=[part])
assert install_job.download_parts == [] # restored state
mm2_installer.restart_failed(install_job)
assert install_job.status == InstallStatus.PAUSED # FAILS today only in that nothing was enqueued/no state change
An end-to-end variant can drive it through DownloadQueueService with a TestAdapter returning 416 + Content-Range: bytes */8 against a larger pre-seeded .downloading file (see the tests added in #9432 for the pattern), with the install job constructed via the marker-restore path.
Suggested fix
restart_failed() (and anything else that reads job.download_parts for liveness) should fall back to job._multifile_job.download_parts when the install job's list is empty.
_write_install_marker should not overwrite a marker's existing files metadata with [] when the install job has no parts — skip the field or merge with what's on disk.
Summary
An install job restored from an on-disk install marker that pauses before its first download-started signal ends up in a PAUSED state that cannot be restarted:
restart_failed()silently no-ops, and the PAUSED marker rewrite erases the persisted resume metadata. The only escape is cancelling the install and re-importing from scratch.Found during the adversarial review of #9432 (see the review body there); the bug predates that PR, but #9432's stricter 416 gate made the trigger reachable with ordinary local state.
Mechanism
install_job.download_partsis populated only by_download_started_callback(model_install_default.py:1453). An install restored by_restore_incomplete_installs(model_install_default.py:214) starts with an emptydownload_parts— the marker's file metadata goes tojob._resume_metadatainstead._do_downloadpauses and raises before_signal_job_started(download_default.py:434-453). So if the first part of a resumed install hits it, no started callback ever fires and the install job'sdownload_partsstays empty._download_cancelled_callbackcorrectly detectsresume_required— but on the multifile job's parts (model_install_default.py:1498) — and sets the install PAUSED. So far so good.restart_failed()however reads the install job'sdownload_parts(model_install_default.py:625) — empty → early return. The UI's restart action does nothing; the job is stuck PAUSED forever._write_install_marker, gate atmodel_install_default.py:142) writesfiles: [], permanently discarding the persistedetag/canonical_url/expected_total_bytes/download_pathmetadata that the next restore/resume would have used.Reproduction (manual)
.downloadingfile in the tmpdir so it is larger than the remote file.Range: bytes=N-request with416and a mismatchedContent-Range: bytes */<total>, so the part pauses before any started signal ("Resume refused by server. Restart required."), and the install goes PAUSED.restart_failed). Observe: it returns immediately, nothing is enqueued, the job stays PAUSED. The marker on disk now contains"files": [].Reproduction (test sketch)
The core assertion needs no HTTP at all:
An end-to-end variant can drive it through
DownloadQueueServicewith aTestAdapterreturning416+Content-Range: bytes */8against a larger pre-seeded.downloadingfile (see the tests added in #9432 for the pattern), with the install job constructed via the marker-restore path.Suggested fix
restart_failed()(and anything else that readsjob.download_partsfor liveness) should fall back tojob._multifile_job.download_partswhen the install job's list is empty._write_install_markershould not overwrite a marker's existingfilesmetadata with[]when the install job has no parts — skip the field or merge with what's on disk.