Can training be stopped from another process (via callback)?

[haimat]

I would like to control YOLOv5 training from another process (i.e. a custom training GUI), so I am playing with the utils.callbacks.Callbacks class utilized in train.py. I know that one can get infos on training status and progress via that callback class, however, is there also a way to stop the training process before all epochs are done?

If the Callbacks class is not what I am looking for in this context, what other approach should I follow to stop training from another process? I would rather not send a SIGKILL signal to the training process ...

[glenn-jocher]

@haimat the most frequent callback is probably on_batch_end, so you might put a listener there.

[haimat]

What if we would add a new member stop_training = False to the yolov5.utils.callback.Callbacks parent class. Then from within the callbacks child class one could simply set callback.stop_training = True whenever he wants. And in the train() method we could then do this:

callbacks.run('on_train_batch_end', ni, model, imgs, targets, paths, plots, opt.sync_bn)
if callbacks.stop_training:
    return  # or whatever would be best to gracefully stop the training loop

Wouldn't that be all we need for this feature?

[glenn-jocher]

@haimat yeah that looks like a decent implementation. Can you submit a PR for it?

[haimat]

@glenn-jocher Sure, I have created this PR - is it OK for you?

[glenn-jocher]

@haimat great! I will take a look today.

[haimat]

@glenn-jocher Don't want to bother you, just asking because I am developing this feature at the moment in our application - can you estimate when you might approve that PR so it will be part of the official master?