Base + Meta learner updates

[mitchellmanware]

@kyle-messier @sigmafelix
To improve overall model performance (goal: rsq > 0.8), adopting a base/meta learner ensemble mimicking that utilized in Di el al. 2019. New workflow will:

1. utilize a gam to model monitor PM2.5 on base learner predictions
2. predict base gam on full prediction grid
3. use full spacetime prediction grid estimates to calculate temporal (t-1; t-3; ...) and spatial (exact; 1km neighbors; 10km neighbors) lag estimates
4. combine spatiotemporal lag estimates with raw covariate set
5. run elnet, brulee, and lightgbm meta learners with new covariates
6. utilize gam to model monitor PM2.5 on meta learner predictions
7. predict meta gam on full prediction grid.

To do:

• Rebuild containers from mm-0701
• Integrate target_calculate_predict (full prediction grid covariates; Prediction grid covariates progress #429)
• Update monitor and prediction grid TRI covariates to extract and then re-bake recipe, respectively
• Dispatch and debug target_calculate_predict (@sigmafelix expect questions/comments/help with chopin implementation)
• Dispatch and debug fit_learner_base_* with new full_data parameter for "trained" workflows
• Dev and implement gam learner function on base learner predictions
• Predict gam model on prediction grid
• Calculate spatiotemporal lag covariates from gam-predicted prediction grid values
• Combine spatiotemporal lag covariates with original covariate dataset for monitors
• Dispatch elnet brulee and lightgbm (or bart ?) meta learners
• Implement bart or Bayesian gam learner function on meta learner predictions
• Predict PM2.5 values or posterior distribution for full prediction grid

[kyle-messier]

🚀 🤠

[mitchellmanware]

@sigmafelix
In the target_calculate_predict targets for the prediction grid covariates, is the [[1]] supposed to be a dev sample of just one of the grids? Or is this index required to extract from the output of the chopin function?

    targets::tar_target(
      sf_pred_calc_split,
      command = {
        init_grid <-
          chopin::par_pad_grid(
            sf_pred_calc_grid,
            mode = "grid",
            nx = 10L,
            ny = 5L,
            padding = 100
          )[[1]]
        grid_sample <- dplyr::sample_frac(sf_pred_raw_grid, 0.01)
        init_grid_intersect <-
          init_grid[grid_sample, ]
        init_grid_intersect
      },
      iteration = "vector",
      description = "sf split grid polygons | prediction"
    ),

[sigmafelix]

@mitchellmanware
chopin::par_pad_grid() returns a list of the two sf/terra grid polygon sets in $original and $padded. [[1]] calls the original (i.e., not padded) grid polygons to subset the full spatial grid points by non-overlapping grids.

[mitchellmanware]

@sigmafelix
Great, thank you!!

[mitchellmanware]

#429

[kyle-messier]

@mitchellmanware Thanks for creating the checklist. I suggest we follow the workflow of Di el al. 2019, with an important modification. For Part 2, our meta-learner needs to allow for uncertainty quantification. We either do this by continuing our Monte Carlo replicates or a Bayesian model such as BART or Bayesian GAM. I think a single point estimation in Part 1 with a GAM is good since it's main goal is to get spatial and temporal lag predictions into the covariate set and a handful of point predictions is more than sufficient. Thanks! Also @sigmafelix @Sanisha003