Skip to content

[FEAT] TabPFN test on VN1-competition#601

Closed
elephaint wants to merge 3 commits intomainfrom
feat/tabpfn-exp
Closed

[FEAT] TabPFN test on VN1-competition#601
elephaint wants to merge 3 commits intomainfrom
feat/tabpfn-exp

Conversation

@elephaint
Copy link
Contributor

Work done by @marcopeix, few small changes from my end.

Run the following commands from a terminal, using a nixtla environment (see Contributing.md), from folder experiments\vn1-competition,:

  1. make download_data
  2. pip install tabpfn-time-series datasets
  3. python experiments/vn1-competition/test_tabpfn.py

Following results should display (takes a loooonng time):

image

@elephaint elephaint requested a review from marcopeix February 11, 2025 08:28
@github-actions
Copy link
Contributor

github-actions bot commented Feb 11, 2025

Experiment Results

Experiment 1: air-passengers

Description:

variable experiment
h 12
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 12.6793 11.0623 47.8333 76
mape 0.027 0.0232 0.0999 0.1425
mse 213.936 199.132 2571.33 10604.2
total_time 1.2887 1.4137 0.0045 0.0034

Plot:

Experiment 2: air-passengers

Description:

variable experiment
h 24
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 58.1031 58.4587 71.25 115.25
mape 0.1257 0.1267 0.1552 0.2358
mse 4040.22 4110.79 5928.17 18859.2
total_time 0.906 0.8872 0.0037 0.0034

Plot:

Experiment 3: electricity-multiple-series

Description:

variable experiment
h 24
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 178.293 268.13 269.23 1331.02
mape 0.0234 0.0311 0.0304 0.1692
mse 121589 219485 213677 4.68961e+06
total_time 1.0852 2.2406 0.0047 0.0041

Plot:

Experiment 4: electricity-multiple-series

Description:

variable experiment
h 168
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 465.532 346.972 398.956 1119.26
mape 0.062 0.0436 0.0512 0.1583
mse 835120 403760 656723 3.17316e+06
total_time 0.9753 1.9046 0.0053 0.0044

Plot:

Experiment 5: electricity-multiple-series

Description:

variable experiment
h 336
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 558.673 459.757 602.926 1340.95
mape 0.0697 0.0565 0.0787 0.17
mse 1.22723e+06 739114 1.61572e+06 6.04619e+06
total_time 1.0205 1.0297 0.005 0.0044

Plot:

@marcopeix
Copy link
Contributor

Minor comment: we should place this in experiments/vn1-competition/src/.
@cchallu, do we want to merge this?

@marcopeix
Copy link
Contributor

Closing as I'm not sure if we want to merge it, but keeping the branch just in case.

@marcopeix marcopeix closed this Mar 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants