Applying NeuralProphet at Scale

[sargupta]

Prerequisites

• [X ] Put an X between the brackets on this line if you have done all of the following:
  • Checked the Answered Questions on the Github Disscussion board: https://github.com/ourownstory/neural_prophet/discussions
    If you have the same question but the Answer does not solve your issue, please continue the conversation there.
  • Checked that your issue isn't already filed: https://github.com/ourownstory/neural_prophet/issues
    If you have the same issue but there is a twist to your situation, please add an explanation there.
  • Considered whether your issue might need further discussing before being defined as a feature request:
    Please post an idea or feedback

Is your feature request related to a problem? Please describe.

My dataset looks like this

+-------+--------+----------+
|     ID|     Val|      Date|
+-------+--------+----------+
|Ax3838J|81119.73|2021-07-01|
|Ax3838J|81289.62|2021-07-02|
|Ax3838J|81385.62|2021-07-03|
|Ax3838J|81385.62|2021-07-04|
|Ax3838J|81385.62|2021-07-05|
|Bz3838J|81249.76|2021-07-02|
|Bz3838J|81324.28|2021-07-03|
|Bz3838J|81329.28|2021-07-04|
|Bz3838J|81329.28|2021-07-05|
|Bz3838J|81329.28|2021-07-06|
+-------+--------+----------+

In real, there are 2.7 million IDs and total 56 million rows.
I am using Azure Databricks (PySpark) and trying to apply fbprophet on a sampled dataset of 10000 rows and it's already taking 5+ hours.

I am considering applying NeuralProphet and StatsForecast but not sure how can I apply the forecast model for each individual ID to do the forecasting on ID basis.

Any suggestions?

NB: while applying fbprophet, val becomes 'y' and Date becomes ds in the respective order.

Here is what I have tried so far:

def forecast_balance(history_pd: pd.DataFrame) -> pd.DataFrame:

    anonym_cis = history_pd.at[0,'ID']
    
    # instantiate the model, configure the parameters
    model = Prophet(
        interval_width=0.95,
        growth='linear',
        daily_seasonality=True,
        weekly_seasonality=True,
        yearly_seasonality=False,
        seasonality_mode='multiplicative'
    )

    # fit the model
    model.fit(history_pd)

    # configure predictions
    future_pd = model.make_future_dataframe(
        periods=30,
        freq='d',
        include_history=False
    )

    # make predictions
    results_pd = model.predict(future_pd)
    results_pd.loc[:, 'ID'] = anonym_cis

    # . . .


    # return predictions
    return results_pd[['ds', 'ID', 'yhat', 'yhat_upper', 'yhat_lower']]

result_schema =StructType([
  StructField('ds',DateType()),
  StructField('CIS_ANONYM',IntegerType()),
  StructField('yhat',FloatType()),
  StructField('yhat_upper',FloatType()),
  StructField('yhat_lower',FloatType())
  ])

historic_data = df.filter(F.col('ds') < '2022-02-20')
 
group_results = (
    historic_data
    .groupBy('ID')
    .applyInPandas(forecast_balance, schema=result_schema)
    )
 

Describe the solution you'd like

I would like to apply NeuralProphet model for each ID to make individual id based forecasting.
At present I can not find anything close to that.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

[ourownstory]

Hi @sargupta

You should be able to apply NeuralProphet in a similar way as you do with Prophet.

In real, there are 2.7 million IDs and total 56 million rows.

Do I understand right, that on average, your time-series have a length of 20 time-stamps?
That means you have a very wide dataset (large number of short series).

Suggestion:
Now, If your series all share similar dynamics, you could train a global model with shared weights.
This is something that NeuralProphet can do, and it may be much more efficient (see tutorial ).

Sidenote:
We are currently using a setup with a dict of dataframes .
This setup may not be the most efficient for such a wide dataset, but should work.
Try it out, and if you run into issues, let us know, and we may consider changing it to accept dataframes with an ID column instead.