-
Notifications
You must be signed in to change notification settings - Fork 13
Description
When running adaptive scheduler, sometimes a learner completes, but the job does not shut down:
{"job_id": "289289", "log_fname": "adaptive-scheduler-1-289289.log", "job_name": "adaptive-scheduler-1", "event": "trying to get learner", "timestamp": "2020-01-14 18:10.46"} {"event": "sent start signal, timeout after 10s.", "timestamp": "2020-01-14 18:10.46"} {"reply": "[I DELETED MY PATH]", "event": "got reply", "timestamp": "2020-01-14 18:10.46"} {"event": "got fname", "timestamp": "2020-01-14 18:10.46"} {"event": "picked a learner", "timestamp": "2020-01-14 18:10.46"} {"event": "started logger on hostname [DELETED HOSTNAME]", "timestamp": "2020-01-14 18:10.46"} {"npoints": 100, "event": "npoints at start", "timestamp": "2020-01-14 18:10.46"} {"status": "finished", "event": "runner status changed", "timestamp": "2020-01-14 18:10.46"} {"elapsed_time": "0:00:00.000688", "overhead": 0, "npoints": 100, "cpu_usage": 1.6, "mem_usage": 2.2, "event": "current status", "timestamp": "2020-01-14 18:10.46"} {"event": "goal reached! \ud83c\udf89\ud83c\udf8a\ud83e\udd73", "timestamp": "2020-01-14 18:10.46"} {"fname": "[I DELETED MY PATH]", "event": "sent stop signal, timeout after 10s", "timestamp": "2020-01-14 18:10.46"}

Also, when running parse_log_files(), the log files of the running jobs don't show up