helper function for k-fold-cv

[avehtari]

The package is named as loo, but when psis-loo fails and k-fold-cv is used, then corresponding function could form a object which could be given to compare function.

[bnicenboim]

I wrote all sort of helper functions to deal with k-fold CV for my submission to stancon.

It needs a vector that indicates with 1 what was held out, and 0 what was used for training. And then for the model, one can skip the held out data:

model {
...
for (n in 1:N_obs) 
    if(holdout[n] == 0)
      target += model_lpdf[n];
}

and the generative quantities has a vector[N_obs] log_lik. My helper functions extract the relevant log_lik, and calculate the elpd like it is done with loo.

If it's suitable, I could contribute it to the loo package (with a little bit of guidance).

[avehtari]

Thanks for the comment. Your helper functions are not directly related to this issue, but they would be helpful to make it easier for people to use k-fold-cv. Could you open a new issue for your proposal (something like "k-fold-cv helper functions"). I already wrote the below comment before I realized that this should be in a separate issue...

I think it would be better to use index vector (below trainidx). Then your code would change to

model {
...
for (n in trainidx) 
      target += model_lpdf[n];
}

and some models could be written simply as

model {
...
y[trainidx] ~ normal(mu, sigma);
}

[bnicenboim]

ok, I created a new issue, and yes, I forgot that Stan allows to do that now :)