feat: add watch mode for continuous checkpoint evaluation by marta-sd · Pull Request #857 · NVIDIA-NeMo/Evaluator

marta-sd · 2026-03-17T09:42:42Z

No description provided.

copy-pr-bot · 2026-03-17T09:42:47Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

agronskiy

Couple of small things, generally LGTM after those fixed -- unblocking with one-shot approve

agronskiy · 2026-03-23T14:19:03Z

+
+WATCH_STATE_FILE = os.getenv(
+    "NEMO_EVALUATOR_WATCH_STATE_FILE",
+    Path.home() / ".nemo-evaluator" / "watch-state" / "watch-state.v1.jsonl",


bug: when the env var is set, the getenv would get str, not Path, and down at the calling site it at line 38 I think it will crash because str does not have those methods.

Good catch!

fixed in a9855d6

agronskiy · 2026-03-23T14:28:32Z

+        assert local_source.is_dir()
+    remote_destination_str = f"{username}@{hostname}:{remote_target}"
+    local_sources_str = " ".join(map(str, local_sources))
+    rsync_upload_command = f"rsync -qcaz {local_sources_str} {remote_destination_str}"


question: on mac, oftentimes the local paths have paths w/ spaces (yeah we linux users should think about those mac ppl), would not that break? I suggest hacvin command as a list with sub-parts - this way the .Run below will pass them to bash properly tokenized.

This code was just moved from one place to another, so I assume it's functional, but better be on the safe side 👍 Done in 59dba76

agronskiy · 2026-03-23T14:31:35Z

+    """
+    if socket is None:
+        return
+    ssh_command = f"ssh -O exit -S {socket} {username}@{hostname}"


question: I would recommand to check on all the commands to be passed as lists -- shlex.split later will break if socket path on Mac contains spaces

...and it's inconsistent with how we open it 👍 Fixed in 6fda968

agronskiy · 2026-03-23T14:33:42Z

+    if socket is None:
+        return
+    ssh_command = f"ssh -O exit -S {socket} {username}@{hostname}"
+    completed_process = subprocess.run(args=shlex.split(ssh_command))


bug(?): the default of stderr param is None, I think this would lead to double-raise later when stderr.decode is called

Actually there's no reason for us to throw an error here. I replaced with similar log.error message to the one we have when opening failed (6fda968)

Signed-off-by: Grzegorz Chlebus <gchlebus@nvidia.com> Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

- Add nel-watch script entry in pyproject.toml - Remove watch subcommand from nel CLI (cli/main.py) - Simplify cli/watch.py: remove --tmux, --env-file, _build_watch_command, _launch_in_tmux; add main() entrypoint; make CLI a thin 1:1 wrapper over the Python API - Remove tmux/env-file/build_watch_command tests from test_watch.py - Remove unused MagicMock import Addresses team feedback: Unix philosophy (do one thing well), thin CLI wrapper, separate entrypoint for watch functionality. Signed-off-by: Grzegorz Chlebus <gchlebus@nvidia.com> Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

…n API) Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

…e execution.sbatch_dependency Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

marta-sd · 2026-03-24T14:28:34Z

/ok to test bfed76c

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

marta-sd · 2026-03-25T12:59:52Z

/ok to test 9df4cf9

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

marta-sd · 2026-03-25T13:18:14Z

/ok to test 25c436f

github-actions Bot added documentation Improvements or additions to documentation nemo-evaluator-launcher tests labels Mar 17, 2026

marta-sd force-pushed the martas/watcher branch from 222a5b7 to 3e9acd5 Compare March 17, 2026 09:43

marta-sd changed the title ~~feat: add watch mode for continuous checkpoint evaluatiob~~ feat: add watch mode for continuous checkpoint evaluation Mar 18, 2026

marta-sd force-pushed the martas/watcher branch from 3e9acd5 to 7f80004 Compare March 18, 2026 14:39

marta-sd commented Mar 18, 2026

View reviewed changes

Comment thread docs/references/api/nemo-evaluator-launcher/api.md

marta-sd force-pushed the martas/watcher branch 2 times, most recently from 25a57eb to b311bd6 Compare March 19, 2026 15:07

marta-sd commented Mar 19, 2026

View reviewed changes

Comment thread ...ges/nemo-evaluator-launcher/src/nemo_evaluator_launcher/common/container_metadata/loading.py

marta-sd commented Mar 19, 2026

View reviewed changes

Comment thread packages/nemo-evaluator-launcher/src/nemo_evaluator_launcher/common/execdb.py

agronskiy approved these changes Mar 23, 2026

View reviewed changes

gchlebus and others added 17 commits March 24, 2026 12:05

feat: add watch mode for continuous checkpoint evaluation

8aaa7c4

Signed-off-by: Grzegorz Chlebus <gchlebus@nvidia.com> Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

docs: add nel-watch documentation (how-to guide, CLI reference, Pytho…

9d759d9

…n API) Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

rewrite wip

659fd86

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

config structures

a1e6e2c

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

move things around

aaccbb3

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

another pass

d5528f0

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

fixes

e26a5c9

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

add option to specify --dependency flag for sbatch job

d108468

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

extract ssh utilities function

21b353e

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

add template file

75de737

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

refactor RunConfig so that from_hydra actually returns RunConfig type

c3b5376

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

consolidate ssh utils further

bcd4a90

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

update slurm tests

5140e63

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

add cluster logic

893a485

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

add cluster logic

5d0f058

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

manage output_dir for all jobs

959e2f0

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

marta-sd added 2 commits March 24, 2026 12:06

fix tutorial linking

ed4a587

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

use newly added execution.sbatch_extra_flags field instead of separat…

ba136ad

…e execution.sbatch_dependency Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

marta-sd force-pushed the martas/watcher branch from bb467ab to ba136ad Compare March 24, 2026 11:13

update tests

bfed76c

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

marta-sd marked this pull request as ready for review March 24, 2026 14:28

marta-sd requested review from a team as code owners March 24, 2026 14:28

copy-pr-bot Bot temporarily deployed to test March 24, 2026 14:30 Inactive

copy-pr-bot Bot had a problem deploying to nemo-ci March 24, 2026 14:30 Failure

copy-pr-bot Bot temporarily deployed to nemo-ci March 24, 2026 14:30 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci March 24, 2026 14:32 Inactive

marta-sd added 4 commits March 25, 2026 12:00

use timestamp in dir name

29c6609

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

cp headers

33a89bc

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

espace curly brackets in docs

65b159c

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

update tests

9df4cf9

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

copy-pr-bot Bot temporarily deployed to test March 25, 2026 13:01 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci March 25, 2026 13:01 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci March 25, 2026 13:03 Inactive

remove broken link

25c436f

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

copy-pr-bot Bot temporarily deployed to test March 25, 2026 13:19 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci March 25, 2026 13:19 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci March 25, 2026 13:21 Inactive

marta-sd merged commit 9e3f445 into main Mar 25, 2026
48 checks passed

marta-sd deleted the martas/watcher branch March 25, 2026 14:02

marta-sd mentioned this pull request Apr 9, 2026

feat: add watch mode for continuous checkpoint evaluation #799

Closed

Conversation

marta-sd commented Mar 17, 2026

Uh oh!

copy-pr-bot Bot commented Mar 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

agronskiy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marta-sd Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marta-sd commented Mar 24, 2026

Uh oh!

marta-sd commented Mar 25, 2026

Uh oh!

marta-sd commented Mar 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

marta-sd Mar 24, 2026 •

edited

Loading