Compare two prediction errors from RF in toy data

[Fuhan-Yang]

Here is a EDA of the prediction errors from random forest using a toy data. The data is created in the way suited for linear regression. Then linear regression and random forest are used to fit the training data (80% of the total data) and do out-of-sample prediction. The prediction mean along with the prediction interval are plotted. The intervals were calculated in two ways for random forest: V_UIJ used by Wager et al and out-of-bag error by Lu et al., given number of trees varying within [100,1000, 5000, 10000]. We can see the prediction intervals are pretty consistent as the increase of trees. The OOB method creates wider interval than V_UIJ. Note that the width of the interval is the same across all the predictions for OOB, which assumes that the uncertainty is consistent across time. Applying this in vaccine coverage, this assumes that the prediction uncertainty of end-of-season coverage is the same as predicting the coverage one month after the forecast date. @swo

[swo]

This is still kind of a complicated example. See #279 for something very very simple.

I'm also still confused about how we'd use any method that does test/training split or in- vs. out-of-bag distinctions. We're interested in forecasting, so the target isn't in-bag or out-of-bag; it's just not in the dataset.

My emerging conclusion is to either (1) take the interval over trees or (2) switch to gradient boosting.

[swo]

We agreed to do an interval over trees' prediction