Stop the training when key metric in the server does not increase after a given number of global iterations

[gonzaiq]

I would like my training to stop automatically if the key metric received by the server does not increase for more than a given number of epochs. Is it possible with one of the current versions of NvFlare ? (I am using version 2.3.8).
If not, would it be interesting to add it as a new feature ?

[YuanTingHsieh]

Hi @gonzaq94 thanks for the good idea!

It should be doable in 2.3.8.
You could write your own controller to make this happen, for example you can add metrics summation inside
https://github.com/NVIDIA/NVFlare/blob/2.3/nvflare/app_common/workflows/scatter_and_gather.py#L332
(self.accu_metrics += result["metrics"])

And then inside the control_flow method after https://github.com/NVIDIA/NVFlare/blob/2.3/nvflare/app_common/workflows/scatter_and_gather.py#L267
You add a condition (if self.accu_metrics >= self.max_metrics: break)

You also need to get max metrics everytime you finish a round (self.max_metrics = max(self.max_metrics, self.accu_metrics))
Remember to also reset the self.accu_metrics to 0 when each round starts.

We might provide an official example on how to do this later so it is easier for users to use, stay tuned!