Add other data to schema?

[ColCarroll]

Per #219, other data may be involved in inference. In the eight schools model, for example, the data block contains:

data {
  int<lower=0> J; // number of schools 
  real y[J]; // estimated treatment effects
  real<lower=0> sigma[J]; // s.e. of effect estimates 
}

right now, we only extract "observed_data", which are observations of parameters in the model. More specifically, the posterior_predictive/ group should match with the observed_data/ group. Open for suggestions on whether and how to store the rest of the data.

[ahartikainen]

I think we should have some special group that contains model specific information.

{library, version, algorithm, init settings}

This is to make sampled data "reproducible" or at least in mcmc sense. So we could have "generic" kw and then special kw for library specific settings.

For Stan this could include stan model + cppcode, init info, warm-up info (inv_metric, stepsize, starting location for sample, seed), thinning info etc.

Do we have some good name for this and can we make it different kind of group than posterior etc are (e.g. in repr it could be shown as "model information, see model_settings.

Also, who has better sugggestion for the name than model_setting

[OriolAbril]

I think there are 2 different kinds of data involved whose possible names could be model_parameters and sampling_parameters/sampler_parameters (or maybe params to make it shorter). In my opinion, both additions to the InferenceData object would be really valuable.

Model parameters

These would be parameters of the model which is not actually observed data but they are needed to run the model, as @ColCarroll was pointing out. J and sigma would fall in this category. In some cases the border between observed data and model parameters could be a little blurry, so the best definition would be probably "arguments samples with the observed kwarg in pymc3" or what is the same "defined in the data section of the stan code, and sampled with ~ in the model section".

They are currently stored in the observed_data group automatically for emcee, and in general for pystan too. In pymc3 however, they are not stored at all for pymc3 objects, as observed_data comes from data sampled with observed=... kwarg. I think there are two approaches, either storing this info always in observed_data or create a new group for them. I prefer the second option, but it may be harder to explain.

Sampler parameters

This should include the information of sampler algorithm name and parameters like @ahartikainen was saying. Currently, only the library and its version is being stored as attrs of each dataset. That comes to my mind right now, the only complication with this information would be when merging inference data objects.

---

Both groups of information would make the MCMC runs more reproducible and as a side effect they would ease the writing and usage of sampling wrappers as they basically reproduce the fit on a subsample of the observed data.

[ahartikainen]

For merging, let's just decide a good way to do it (e.g. iloc 0 to 1000 ... pystan infom 1000 to 2000 pymc info if combining pystan + pymc. Then just keep on updating this depending how we concat. Slicing adds some extra complexity)

[OriolAbril]

@aloctavodia proposed constants for what I described as model_parameters and it sounds better to me.

[OriolAbril]

I want to leave a link to the current specification on InferenceData schema here: https://arviz-devs.github.io/arviz/schema/schema.html

Moreover, I realized that @tyarkoni will probably have useful insight on sampler parameters/backend parameters and the like. Anything you would like to share will be more than welcome.