Result inconsistency with NeuralProphet

[AhmedGabal]

I have an issue where running the same model multiple times yields very different results, not even similar to each other. I believe this happens because the model learns differently each time due to the neural network initializing the weights with a different set of numbers each run. Consequently, the gradient starts from a different point every time, leading to varying outcomes. For example, when I run the same model with identical hyperparameters three times, the results are (-5, 13, 50). The result of 13 is close to the actual value, but the other two runs produce significantly worse results, if i use the random seed i will get every time like the first one (-5) which is not accurate as well , but the model learn different things each run time.

Has anyone else experienced this inconsistency with NeuralProphet, and found a solution?

[adi250491]

Yes I had this issue and found out that the seed reset function in neuralprophet did not reset the random state of all the involved libraries. I created my own reset random state function. Call this function with the seed of your choice before fitting your model and it will give consistent results everytime.

def reset_random_state(seed): os.environ['PYTHONHASHSEED']=str(seed) torch.manual_seed(seed) np.random.seed(seed) random.seed(seed)

[AhmedGabal]

Thanks @adi250491 for your reply, and your effort to send me this solution
however it is not working with me ,and i am not concern about consistent results every time, my concern is the result is fluctuate every run time, let ,me explain this more below:
1st > i built my own neural network with encoder decoder and multi head attention and ever time i run the same code i got the result with low variance between the results in each run for example first run the result is 12 , second 15 third 8 so the variance is very low between from run to another
2nd> when i do the same test with neuralprophet with same hyperparameters at each run i got very high variances between the results in each run for example first run the result is 12 , second 36 third -1 so the variance is very very high.

so if we used the random seed you will get every time one result from 3 , in the 1st example if you take any result from 3 runs and we assume the actual is 10 you will get little bit consistent Mae (2-5), however in the 2nd example with neural prophet we will get random mae (2-26) which is very worse.

this actually my point

[mahdi-990]

can you mention if this problem was solved or not ?