Ramp Function Diverges

[jabowery]

As a low-effort test of the ORELM, I substituted for NYC Taxi traffic, the attached ramp function from 0 to 20000 in increments of 10000, backed up the nyc_taxi.csv and copied therampified_nyc_taxi.csv in its place. It seems not to converge.

This isn't necessarily an issue with the code. It is interesting in that it raises the issue of parameter seeking in the ORELM algorithm. What is it about the ramp function that causes ORELM to diverge and how would one find the parameter(s) values that converge on a solution?

rampified_nyc_taxi.zip

For comparison, this is with a sine function rather than ramp function -- although in this case, I synchronized the time of day with the phase:

sine_nyc_taxi.zip

[chickenbestlover]

As I described in OR-ELM paper, one disadvantage of OR-ELM is that It is very vulnerable to adjustment of a hyper parameter called "forgetting factor".

This divergence is not a problem that occurs only in ORELM, but in fact it is a common problem in forgetting factor applied recursive least square based algorithms.

If the factor is too low, it may diverge; A rule of thumb choosing the forgetting factor is just using 0.99. However, the optimal value may vary depending on the data set.

Therefore my suggestion is to adjust forgetting factor; in this case, OutWeightFF.
Increasing the value close to 1 can improve performance (max value is 1).

net = initializeNet(nDimInput=X.shape[1], nDimOutput=1, numNeurons=23, algorithm=algorithm, LN=True, InWeightFF=1, OutWeightFF=0.915, HiddenWeightFF=1, AE=True, ORTH=False)