Skip to content

REVIVE button silently no-ops when resuming a run with no hyperparams in /api/status #6

Description

@mmoalem

REVIVE button silently no-ops when resuming a run with no hyperparams in /api/status

Symptom

After restoring a session from an external backup (e.g. HF Hub sync, as the Kaggle notebook setup does), clicking REVIVE on a killed/error/completed run does nothing — no modal opens, no network request fires, no visible error. The button itself is enabled (has_restart_cmd is true) and onclick="openResumeModal()" is correctly wired, but the handler throws before reaching classList.add('active').

Browser console shows:

Uncaught TypeError: Cannot read properties of undefined (reading 'length')
    at window.openResumeModal (index.html:4507)

Root cause

dashboard/index.html has 8 call sites of this shape:

const info = _modelSeqInfo[someKey] || _modelSeqInfo['sa3'];
// ... info.length used unconditionally right after

The hardcoded 'sa3' fallback assumes every deployment registers a model under the literal key 'sa3'. That's not true for deployments that only register versioned/qualified model names (in my case: sa3-medium, sa3-sm-music, sa3-sm-sfx — no plain sa3).

This only bites when the primary key lookup also misses. That happens in openResumeModal() specifically because _currentStatus.hyperparams can be entirely absent: _get_status() in dashboard/server.py returns an early stub response ({"running": false, "status": ..., "message": "Log file not found", "has_restart_cmd": ...}) whenever run["log_path"] doesn't exist on disk — which is the normal case after restoring run state from an external backup, since .log files aren't part of any state that gets backed up/restored (only .safetensors/.json are, per typical sync setups). With hyperparams missing, hp.base_model is undefined, the primary lookup misses, and the 'sa3' fallback also misses → _modelSeqInfo[...] is undefined.length throws.

Repro

  1. Train a run, register a model under a key other than exactly sa3 (e.g. sa3-medium).
  2. Lose/delete the run's log file on disk (or restore runs.json + checkpoints from a backup without the .log file — anything that makes Path(run["log_path"]).exists() false).
  3. Restart the dashboard server; the run shows status killed/error with a REVIVE button.
  4. Click REVIVE.

Expected: resume modal opens.
Actual: nothing happens; console shows the TypeError above.

Fix

Replace the hardcoded 'sa3' fallback with a small helper that falls back to 'sa3' if present, then to whatever model is registered first — so the lookup can never resolve to undefined as long as at least one model is registered:

function _seqInfoFor(key) {
  return _modelSeqInfo[key] || _modelSeqInfo['sa3'] || _modelSeqInfo[Object.keys(_modelSeqInfo)[0]];
}

...and point all 8 _modelSeqInfo[x] || _modelSeqInfo['sa3'] call sites at it. Patch attached (fix.patch, applies cleanly to dashboard/index.html at 06b61cb). Tested: reproduces the exact throw on the old code path, confirmed resolved on the patched path, normal case (hyperparams present, valid base_model) unaffected.

Happy to open this as a PR directly if preferred — let me know.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions