Creating inits from datasets

[mhollanders]

Hi,

I'm trying to do some SBC with JAGS models with discrete parameters which require inits that are consistent with the data. Therefore, each dataset has its own corresponding set of inits. Is there a way to tell SBC to have separate inits for each generated dataset?

EDIT: I see there's an init_factory argument for the cmdstanr backend, but I don't know how to use it or if it's even implemented for the rjags backend.

Thanks,

Matt

[martinmodrak]

Yes, this is a bit of an unsolved problem - I am a bit unsure on what the best approach to handling inits is. The init_factory approach for cmdstanr is experimental and was an attempt to test one of the possible ways to handle this and is implemented only for this backend (generally, inits tend to have backend-specific format so some backend-specific handling is going to be required).

For future myself, I'll just explain my thinking, see if others can provide some feedback to make a good design choice here.

There are at least two reasonable ways to approach the problem. The init_factory approach means providing a function that takes in a dataset and guesses good inits. This is "clean" in the sense that inits cannot be built with knowledge of the underlying simulated values of the parameters, which mimics real-world model fitting. It also lets this guessing run in parallel (if you fit in parallel), so dataset creation is still quick even if the guess is expensive (my use case where I tested this was running optimization to get an initial guess) The downside is that the guessing part can get quite complicated and you may actually want to use the simulated values to get around some difficult corner cases. This leads to a second possible approach - add an extra field to SBC_datasets to track optional, per-dataset information passed to the backend. The main advantage is that this could in principle be a shared implementation for all backends and handle other stuff than inits as well (e.g. dataset-specific length of warmup, control arguments etc.). Alternatively, one could add attributes to elements of $generated to achieve a similar effect.

As a quick workaround, you may want to implement your own "jags with inits" backend which would take inits as part of generated - simply copying the code for SBC_backend_rjags and SBC_fit.SBC_backend_rjags and handling the new element of generated should do the trick (you then return SBC_rjags_fit and use the rest of the default handling).

[mhollanders]

I like the approach of having a third element in datasets. In ecological models, the initial values for discrete latent parameters have to be informed by the data; for instance, an alive individual in mark-recapture cannot be initialised as dead, because an observation of that individual is inconsistent with it being dead. Would there not have to be some extra code to tell the backend (JAGS) to look for a specific list element in generated as well?

[lunafazio]

I'd also be inclined to follow the approach of augmenting the datasets object. I've already been doing this, although in my case it was for passing some data-dependent stanvars code for brms. In any case, it would be nice to have an "officially" supported extra slot since it would reduce the load on the user side when trying to figure out whether doing so on their own may risk breaking something or deciding how to name that additional slot (known to be quite a challenging problem 🙂).

[mhollanders]

Hey Luna, nice to see you here. Any chance you could share some of that code so I can adapt it for my problem?