benchmark

[akthammomani]

Hi @ourownstory,

I'm having errors when try to import the following:

from neuralprophet.benchmark import Dataset, NeuralProphetModel, SimpleExperiment, CrossValidationExperiment
from neuralprophet.benchmark import SimpleBenchmark, CrossValidationBenchmark

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-83bb49d3db36> in <module>
     2 import pandas as pd
     3 from neuralprophet import NeuralProphet, set_log_level
----> 4 from neuralprophet.benchmark import Dataset, NeuralProphetModel, SimpleExperiment, CrossValidationExperiment
     5 from neuralprophet.benchmark import SimpleBenchmark, CrossValidationBenchmark

ModuleNotFoundError: No module named 'neuralprophet.benchmark'

Please let me know your comments or if you have any directions or if it's not available yet.

Thank you

BR//Aktham

[ourownstory]

Hi @akthammomani
Did you install with pip from pypi or explicitly from this github?
You can check your version number. 0.2.8 Is pypi and 0.2.9 is from here.
The benchmark framework is currently only available when directly installing from github.

[akthammomani]

Thanks @ourownstory for the quick reply. Installation was completed using github and the version is 0.2.9 by following below:

git clone https://github.com/ourownstory/neural_prophet 
cd neural_prophet 
pip install .[live]

Please let me know if you have any recommendations? should I start new env. and install the library again?

Thank you
BR//Aktham

[akthammomani]

@ourownstory, I went a head and installed the Library again using below in a fresh env:

git clone <copied link from github>
cd neural_prophet
pip install .

so when trying below:

from neuralprophet import NeuralProphet, set_log_level
from neuralprophet.benchmark import Dataset, NeuralProphetModel, SimpleExperiment, CrossValidationExperiment

I started to see a new error as shown below:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_43524/177920820.py in <module>
      1 from neuralprophet import NeuralProphet, set_log_level
----> 2 from neuralprophet.benchmark import Dataset, NeuralProphetModel, SimpleExperiment, CrossValidationExperiment

~\Anaconda3\envs\nprophet_v1\lib\site-packages\neuralprophet\benchmark.py in <module>
    158 
    159 
--> 160 @dataclass
    161 class SimpleBenchmark:
    162     """

~\Anaconda3\envs\nprophet_v1\lib\site-packages\neuralprophet\benchmark.py in SimpleBenchmark()
    171     """
    172 
--> 173     model_classes_and_params: List[tuple[Model, dict]]
    174     datasets: List[Dataset]
    175     metrics: list[str]

TypeError: 'type' object is not subscriptable

Please let me know if you have any suggestions.... Thank you very much

BR//Aktham

[ourownstory]

Hi @akthammomani This second error is actually a bug. Thank you for bringing it to my attention. Will treat it in #455

Good to hear that the new install resolved the import eror.

[akthammomani]

@ourownstory, Is there like a workaround to start using the benchmark framework?

Thank you

[ourownstory]

I already merged the fix! re-install and try again :)

[akthammomani]

Thank you, @ourownstory,

Do you know if the future regressors are included @ benchmarking framework ? Or do you recommend to complete benchmarking without any future regressors?

[ourownstory]

Hi @akthammomani
The bench framework is still in development and currently only supports univariate time series, and autoregression is also only half tested. Sorry for the inconvenience. It should be ready in a month.

If you like to use regressors etc, I recommend writing your own loop over configurations, using the inbuilt crossvalidation tool:
https://github.com/ourownstory/neural_prophet/blob/master/tutorials/feature-use/crossvalidation.ipynb

[ourownstory]

@akthammomani in general, if you have relevant regressors, it is good to use them though!

[akthammomani]

Thanks @ourownstory for the support.

[codeananda]

@ourownstory I'm using 0.5 now installed via Github and am getting the error

from neuralprophet.benchmark import (
    CrossValidationExperiment, 
    Dataset,
    NeuralProphetModel, 
    SimpleExperiment
)

ModuleNotFoundError: No module named 'neuralprophet.benchmark'

Also, I cannot see CrossValidationExperiment anywhere in the repo. What has happened to it and benchmark.py? The docs still show it as existing.

[nachi-lieder]

Hi, did you manage to resolve this? I am with the same issue. Also with the benchmark , and also with the functionality of highlight_forecast in the plot.

[codeananda]

Nope. I just had to implement it manually.
@karl-richter is this something you're familiar with?