When running Bris-inference for multiple starting states, sometimes the last forecasts are missing.
This is probably related to the way we split the data when running data parallelism. Currently, the input data is splitted equally on all processes, and the remainder is never run. So if we have 3 starting states and run two models in parallel, one forecast will be missing.
A better solution would be to split the remainders and run them on some of the processes. Even though the load will be (slightly) imbalanced, we will at least get all the forecasts.
When running Bris-inference for multiple starting states, sometimes the last forecasts are missing.
This is probably related to the way we split the data when running data parallelism. Currently, the input data is splitted equally on all processes, and the remainder is never run. So if we have 3 starting states and run two models in parallel, one forecast will be missing.
A better solution would be to split the remainders and run them on some of the processes. Even though the load will be (slightly) imbalanced, we will at least get all the forecasts.