Description
Problem Description:
It appears that including step_scale() in the recipe disrupts prediction generation within modeltime. I encountered this issue when evaluating models using modeltime_accuracy() - where model fit was significantly worse with step_scale() - and also in modeltime_calibrate(), where forecasts displayed a clear systematic error. As a result, the discrepancy between forecasts from recipes with and without step_scale() is visually evident in the plotted forecasts.
Sample code:
library(ggplot2)
library(recipes)
#> Loading required package: dplyr
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
#>
#> Attaching package: 'recipes'
#> The following object is masked from 'package:stats':
#>
#> step
library(parsnip)
library(rsample)
library(workflows)
library(modeltime)
library(timetk)
data(economics)
eco_splits <- time_series_split(economics, date, asses = '10 years', cumulative = T)
spec_lm <- linear_reg(mode = "regression", engine = "lm")
recep <- recipe(pce ~ ., data = economics) %>%
update_role(date, new_role = 'time_index') %>%
step_arrange(date) %>%
step_mutate(date_num = as.numeric(date))
recep_with_scale <- recep %>%
step_scale(all_numeric_predictors())
wflw_fit <- workflow() %>%
add_recipe(recep) %>%
add_model(spec_lm) %>%
fit(training(eco_splits))
wflw_fit_with_scale <- workflow() %>%
add_recipe(recep_with_scale) %>%
add_model(spec_lm) %>%
fit(training(eco_splits))
calibration_tbl <- modeltime_table(wflw_fit, wflw_fit_with_scale) %>%
mutate(.model_id = c('without_scale', 'with_scale')) %>%
modeltime_calibrate(new_data = testing(eco_splits))
calibration_tbl %>%
modeltime_accuracy()
#> # A tibble: 2 × 9
#> .model_id .model_desc .type mae mape mase smape rmse rsq
#> <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 without_scale LM Test 1510. 14.3 37.4 15.4 1588. 0.981
#> 2 with_scale LM Test 11778. 114. 292. 200 11794. 0.975
forcast_tbl <- calibration_tbl %>%
modeltime_forecast(new_data = testing(eco_splits),
actual_data = economics
)
forcast_tbl %>% plot_modeltime_forecast(.interactive = F)
#> Warning in max(ids, na.rm = TRUE): no non-missing arguments to max; returning
#> -Inf
sessionInfo()
#> R version 4.4.1 (2024-06-14)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 22.04.5 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
#>
#> locale:
#> [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
#> [4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
#> [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
#> [10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: Europe/Warsaw
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] timetk_2.9.0 modeltime_1.3.1 workflows_1.1.4 rsample_1.2.1
#> [5] parsnip_1.2.1 recipes_1.1.0 dplyr_1.1.4 ggplot2_3.5.1
#>
#> loaded via a namespace (and not attached):
#> [1] tidyselect_1.2.1 timeDate_4041.110 farver_2.1.2
#> [4] fastmap_1.2.0 reprex_2.1.1 digest_0.6.37
#> [7] rpart_4.1.23 timechange_0.3.0 lifecycle_1.0.4
#> [10] StanHeaders_2.32.10 yardstick_1.3.1 survival_3.7-0
#> [13] magrittr_2.0.3 compiler_4.4.1 rlang_1.1.4
#> [16] tools_4.4.1 utf8_1.2.4 yaml_2.3.10
#> [19] data.table_1.16.2 knitr_1.49 labeling_0.4.3
#> [22] DiceDesign_1.10 withr_3.0.2 purrr_1.0.2
#> [25] nnet_7.3-19 grid_4.4.1 tune_1.2.1
#> [28] fansi_1.0.6 xts_0.14.1 colorspace_2.1-1
#> [31] future_1.34.0 globals_0.16.3 scales_1.3.0
#> [34] iterators_1.0.14 MASS_7.3-61 cli_3.6.3
#> [37] rmarkdown_2.29 generics_0.1.3 RcppParallel_5.1.9
#> [40] rstudioapi_0.17.1 future.apply_1.11.3 stringr_1.5.1
#> [43] tidymodels_1.2.0 splines_4.4.1 dials_1.3.0
#> [46] parallel_4.4.1 infer_1.0.7 vctrs_0.6.5
#> [49] hardhat_1.4.0 Matrix_1.7-1 listenv_0.9.1
#> [52] foreach_1.5.2 gower_1.0.1 tidyr_1.3.1
#> [55] glue_1.8.0 modeldata_1.4.0 parallelly_1.39.0
#> [58] codetools_0.2-20 stringi_1.8.4 lubridate_1.9.3
#> [61] gtable_0.3.6 munsell_0.5.1 GPfit_1.0-8
#> [64] tibble_3.2.1 furrr_0.3.1 pillar_1.9.0
#> [67] workflowsets_1.1.0 htmltools_0.5.8.1 ipred_0.9-15
#> [70] lava_1.8.0 R6_2.5.1 lhs_1.2.0
#> [73] evaluate_1.0.1 lattice_0.22-6 backports_1.5.0
#> [76] broom_1.0.7 class_7.3-22 Rcpp_1.0.13-1
#> [79] prodlim_2024.06.25 xfun_0.49 forcats_1.0.0
#> [82] zoo_1.8-12 fs_1.6.5 pkgconfig_2.0.3Created on 2024-11-14 with reprex v2.1.1
Expected Behavior:
Forecasts with step_scale() should ideally align with those without scaling when properly calibrated.
Observed Behavior:
Forecasts generated with step_scale() in the recipe display systematic errors and notably different results compared to those without scaling, as evident from accuracy metrics and forecast plots.
Additional Context:
It would be helpful to understand whether step_scale() is introducing unexpected data transformations that interfere with forecast accuracy in modeltime, or if additional steps are needed to ensure compatibility.
Activity