REVIVE button silently no-ops when resuming a run with no hyperparams in /api/status
Symptom
After restoring a session from an external backup (e.g. HF Hub sync, as the Kaggle notebook setup does), clicking REVIVE on a killed/error/completed run does nothing — no modal opens, no network request fires, no visible error. The button itself is enabled (has_restart_cmd is true) and onclick="openResumeModal()" is correctly wired, but the handler throws before reaching classList.add('active').
Browser console shows:
Uncaught TypeError: Cannot read properties of undefined (reading 'length')
at window.openResumeModal (index.html:4507)
Root cause
dashboard/index.html has 8 call sites of this shape:
const info = _modelSeqInfo[someKey] || _modelSeqInfo['sa3'];
// ... info.length used unconditionally right after
The hardcoded 'sa3' fallback assumes every deployment registers a model under the literal key 'sa3'. That's not true for deployments that only register versioned/qualified model names (in my case: sa3-medium, sa3-sm-music, sa3-sm-sfx — no plain sa3).
This only bites when the primary key lookup also misses. That happens in openResumeModal() specifically because _currentStatus.hyperparams can be entirely absent: _get_status() in dashboard/server.py returns an early stub response ({"running": false, "status": ..., "message": "Log file not found", "has_restart_cmd": ...}) whenever run["log_path"] doesn't exist on disk — which is the normal case after restoring run state from an external backup, since .log files aren't part of any state that gets backed up/restored (only .safetensors/.json are, per typical sync setups). With hyperparams missing, hp.base_model is undefined, the primary lookup misses, and the 'sa3' fallback also misses → _modelSeqInfo[...] is undefined → .length throws.
Repro
- Train a run, register a model under a key other than exactly
sa3 (e.g. sa3-medium).
- Lose/delete the run's log file on disk (or restore
runs.json + checkpoints from a backup without the .log file — anything that makes Path(run["log_path"]).exists() false).
- Restart the dashboard server; the run shows status
killed/error with a REVIVE button.
- Click REVIVE.
Expected: resume modal opens.
Actual: nothing happens; console shows the TypeError above.
Fix
Replace the hardcoded 'sa3' fallback with a small helper that falls back to 'sa3' if present, then to whatever model is registered first — so the lookup can never resolve to undefined as long as at least one model is registered:
function _seqInfoFor(key) {
return _modelSeqInfo[key] || _modelSeqInfo['sa3'] || _modelSeqInfo[Object.keys(_modelSeqInfo)[0]];
}
...and point all 8 _modelSeqInfo[x] || _modelSeqInfo['sa3'] call sites at it. Patch attached (fix.patch, applies cleanly to dashboard/index.html at 06b61cb). Tested: reproduces the exact throw on the old code path, confirmed resolved on the patched path, normal case (hyperparams present, valid base_model) unaffected.
Happy to open this as a PR directly if preferred — let me know.
REVIVE button silently no-ops when resuming a run with no
hyperparamsin/api/statusSymptom
After restoring a session from an external backup (e.g. HF Hub sync, as the Kaggle notebook setup does), clicking REVIVE on a
killed/error/completedrun does nothing — no modal opens, no network request fires, no visible error. The button itself is enabled (has_restart_cmdis true) andonclick="openResumeModal()"is correctly wired, but the handler throws before reachingclassList.add('active').Browser console shows:
Root cause
dashboard/index.htmlhas 8 call sites of this shape:The hardcoded
'sa3'fallback assumes every deployment registers a model under the literal key'sa3'. That's not true for deployments that only register versioned/qualified model names (in my case:sa3-medium,sa3-sm-music,sa3-sm-sfx— no plainsa3).This only bites when the primary key lookup also misses. That happens in
openResumeModal()specifically because_currentStatus.hyperparamscan be entirely absent:_get_status()indashboard/server.pyreturns an early stub response ({"running": false, "status": ..., "message": "Log file not found", "has_restart_cmd": ...}) wheneverrun["log_path"]doesn't exist on disk — which is the normal case after restoring run state from an external backup, since.logfiles aren't part of any state that gets backed up/restored (only.safetensors/.jsonare, per typical sync setups). Withhyperparamsmissing,hp.base_modelisundefined, the primary lookup misses, and the'sa3'fallback also misses →_modelSeqInfo[...]isundefined→.lengththrows.Repro
sa3(e.g.sa3-medium).runs.json+ checkpoints from a backup without the.logfile — anything that makesPath(run["log_path"]).exists()false).killed/errorwith a REVIVE button.Expected: resume modal opens.
Actual: nothing happens; console shows the TypeError above.
Fix
Replace the hardcoded
'sa3'fallback with a small helper that falls back to'sa3'if present, then to whatever model is registered first — so the lookup can never resolve toundefinedas long as at least one model is registered:...and point all 8
_modelSeqInfo[x] || _modelSeqInfo['sa3']call sites at it. Patch attached (fix.patch, applies cleanly todashboard/index.htmlat06b61cb). Tested: reproduces the exact throw on the old code path, confirmed resolved on the patched path, normal case (hyperparams present, valid base_model) unaffected.Happy to open this as a PR directly if preferred — let me know.