Suggestion: Remove AIC, BIC, and DIC

[ParadaCarleton]

AIC, BIC, and DIC have all been fully supplanted by WAIC, WBIC, and LOO-IC, and there's no longer any real reason to use them. I would suggest removing them from the package to avoid confusing beginners. Someone seeing them used in the package may be led to believe that using them is a good idea, that they provide information different from that provided by the newer information criteria, or that there must be some reason they're in the package. BIC is especially problematic: The name simultaneously gives new users the false impression that the BIC is actually Bayesian and that WAIC/LOO-IC are not Bayesian.

It might be a good idea to implement WBIC to have a replacement for BIC, since BIC has a different use case from the other information criteria in this package (asymptotically maximizing the probability of selecting the correct model, rather than maximizing the expected predictive accuracy of the model). I would also suggest referring to it as the Watanabe-Schwarz Information Criterion to avoid giving the impression that WBIC is "Especially" Bayesian.

[goedman]

@itsdfish

Thank you Carlos. I tend to agree with you that today WAIC, LOO-IC (and WBIC, I don't know this one) are the information criteria to use. I'm including Chis on this response because he specifically asked to include these.

As an interim solutions we can also include a clear warning, similar to your above input or Richard McElreath's positioning of these older ic's. From Chris' thumb up I assume he also agrees with you message

I will take a look at WBIC although right now I have a bit too much on my plate.

EDIT: Given that this package is GPL3 licensed it might still be better to develop a new, MIT licensed package directly from the descriptions of the different ic's. From the StatisticalRethinking introductory/learning point of view MIT licensing is not very important, but for general use it is.

Rob

[itsdfish]

Rob, I agree also.

I was in the process of reimplementing PSIS-LOO but ran into some difficulties. Do you know anyone who is familiar with the algorithm?

[goedman]

What about Aki Vehtari? Are you using "Vehtari, A., A. Gelman, and J. Gabry. 2015. Efficient implementation of leave-one-out cross-validation and WAIC for evaluating fitted Bayesian models"?

[itsdfish]

Yeah. I am using that article. My sticking point is the procedure for fitting the inverse Pareto.

[ParadaCarleton]

Yeah. I am using that article. My sticking point is the procedure for fitting the inverse Pareto.

Funny coincidence -- I'm actually reimplementing it as well, since the R LOO package is looking at using Julia in its backend to speed up its calculations. I've got the code for fitting the inverse Pareto written up already, and I added some more comments that should make it easier to understand.

[ParadaCarleton]

Note that the generalized Pareto distribution is like a slightly wonky monomial, and you can easily fit a monomial between any pair of points. The GPD fitting procedure takes advantage of this in a way that works a bit like the Theil-Sen estimator. You start by taking the first quartile as point A, and then for each individual point B in your sample, you fit a GPD going through A and B. This gives you a set of pointwise estimates for the shape parameter k. When this is done, you take the average of all of these pointwise estimators, weighted by the likelihood that they would produce this data set. At the end, an (optional) adjustment to k-hat is used, where you average k-hat together with a weakly informative prior; this reduces variance at the cost of a little bit of bias.

function gpdfit(x::AbstractArray, min_grid_pts::Int = 30, sort_x::Bool = true)
    
    # x must be sorted, but we can skip if x is already sorted
    if sort_x
        sort!(x)
    end

    n = length(x)
    m = min_grid_pts + isqrt(n) # isqrt = floor sqrt
    prior = 3
    n_0 = 10  # determines how strongly to nudge kHat towards .5
    
    # Estimate parameters

    xStar = x[(n+2) ÷ 4]  # take first sample quartile
    # build pointwise estimates of k and θ by using each element of the sample.
    vectorθ = @. 1 / x[n] + (1 - sqrt(m / (collect(1:m) - .5))) / prior / xStar
    vectorK = mean(log1p.(- vectorθ .* x'), dims = 2)  # take mean of each row
    logLikelihood = @. log(- vectorθ / vectorK) - vectorK - 1  # Calculate log-likelihood at each estimate
    weights = @. 1 / sum(exp(logLikelihood - logLikelihood')) # Calculate weights from log-likelihood

    θHat = weights ⋅ vectorθ  # Take the dot product of weights and pointwise estimates of θ to get the full estimate

    kHat = mean(log1p.(- θHat .* x))
    σ = -kHat / θHat
    # Drag towards .5 to reduce variance for small n
    if wip
      kHat = (kHat * n + .5 * n_0) / (n + n_0) 
    end

    return kHat, σ

end

(Note that I haven't tested this code yet.)

[itsdfish]

@ParadaCarleton, thank you for sharing your code. Given that you know more about PSIS-LOO and related metrics than I do, it might make sense to collaborate. Our main goal was avoid the GNU license and use the MIT license instead. I know that the community has expressed interest in having these metrics, but no one until now has had the expertise to put it together.

[ParadaCarleton]

@ParadaCarleton, thank you for sharing your code. Given that you know more about PSIS-LOO and related metrics than I do, it might make sense to collaborate. Our main goal was avoid the GNU license and use the MIT license instead. I know that the community has expressed interest in having these metrics, but no one until now has had the expertise to put it together.

I'd love to help with this -- I'm pretty sure I can get PSIS-LOO working, along with some related measures like WAIC/WSIC. I do have a bit less experience with specific implementations of MCMC samplers (especially Stan -- most of my work has been in PyMC3+Turing) and Julia itself. If I could get some kind of function that would take objects produced by Stan, Turing, or Soss and convert them into standard objects I could work with, I'd be happy to build the code that takes these and actually calculates the information criteria for them.

[itsdfish]

@ParadaCarleton, that sounds great. I will provide a simple example that can be adapted to Stan, Turing and Soss. I can post something here within the next 24 - 36 hours.

Our original plan was to develop a generic method that would operate on arrays and then add conversion methods that would interface with MCMCChains and any other data type. For example:

A generic method:

function psisloo(x::Array, ...) 
        ...
end

A method to interface with MCMCChains

function psisloo(x::Chain, ...) 
        # some code to extract and format array
        ...
        # generic method
        psisloo(extracted_array)
end

With this approach, the model comparison metrics will work with any data format. We can define the most common data formats, such as MCMCChains. Others can develop custom functions as needed. Does this still sound like a good approach to you?

[ParadaCarleton]

@ParadaCarleton, that sounds great. I will provide a simple example that can be adapted to Stan, Turing and Soss. I can post something here within the next 24 - 36 hours.

Our original plan was to develop a generic method that would operate on arrays and then add conversion methods that would interface with MCMCChains and any other data type. For example:

A generic method:

function psisloo(x::Array, ...) 
        ...
end

A method to interface with MCMCChains

function psisloo(x::Chain, ...) 
        # some code to extract and format array
        ...
        # generic method
        psisloo(extracted_array)
end

With this approach, the model comparison metrics will work with any data format. We can define the most common data formats, such as MCMCChains. Others can develop custom functions as needed. Does this still sound like a good approach to you?

I think this sounds good, although I would make the generic method use the AbstractMCMC API rather than just an array, and convert other objects to AbstractMCMC objects. The problem with a basic array is that it would get very confusing to keep track of all the different things.

[goedman]

@ParadaCarleton

I'm trying to wrap my mind about your 2 (very interesting) questions:

1. use the AbstractMCMC API
2. Switching to Makie/AlgebraOfGraphics

On the first item, in Stan you can:

chns = read_samples(m1_1s; output_format=:mcmcchains)

where m1_1s is the sampled Stan model. Typically m1_1 is the Stan language model.

This would provide a chns object with typeof(chns) <: AbstractMCMC.AbstractChains. Would that be enough for your approach to LOO & company?

I'm not sure if e.g. stepper functionality makes sense although maybe it is possible to come up with an mcmcsample method.

Your second question, on the MCMCChains.jl repo, is also very interesting. My main objection to MCMCChains.jl has always been the AxisArray base. But with your suggested approach we could, once convinced the chains are proper, use Makie Layers.

Maybe this combination could result in 2 new packages BayesMakiePlots.jl and a new, MIT licensed, version of StatsModelsComparisons.jl (or a better name for the latter repo)?

[ParadaCarleton]

Regarding the first question, I actually made a mistake -- since I wasn't familiar with the AbstractMCMC or MCMCChains APIs, I didn't realize they don't really do what I need. I'm still trying to figure out how you get the log-pdf from Stan, Turing, and Soss models after leaving one observation out, which is the main thing I need to build PSIS-LOO.

As for the second thing, that's a longer-term interest that could be discussed in another thread -- for now, I'm just trying to get PSIS-LOO working.

[goedman]

Would it help if I augment the arsenic example in StatsModelComparisons.jl (using Stan) to generate an MCMCChain object with the log-pdfs and maybe Chris can do the same with Turing (using VarInfo?). I can take a look at Soss as well or maybe ask Chad?

i would store the log-pdfs in a separate section of the MCMCChain.

I fully agree with the second part of your reply. Although I find the pk-plots useful.

[ParadaCarleton]

Would it help if I augment the arsenic example in StatsModelComparisons.jl (using Stan) to generate an MCMCChain object with the log-pdfs and maybe Chris can do the same with Turing (using VarInfo?). I can take a look at Soss as well or maybe ask Chad?

i would store the log-pdfs in a separate section of the MCMCChain.

I fully agree with the second part of your reply. Although I find the pk-plots useful.

I think Soss support would be great eventually, but Soss is definitely the least popular of the PPLs -- we can focus on getting Turing and Stan working first, then find a way to include Soss. It looks like the relevant objects are the MCMCChain object, which holds a sample from the posterior distribution, and the VarInfo object, which holds information about the full model. I'm going through ArviZ.jl at the moment to see how they did it.

As for turning this into a full package, the inference algorithm itself is pretty simple, but looking through the LOO package in R, things like "Check to make sure the user didn't make a mistake when they input their data" or "Clean the data so it's usable" seem to make up most of the actual code and could be pretty time-consuming to write.

[ParadaCarleton]

I'm currently implementing PSIS-LOO for matrices, following the API for the loo package in R as closely as possible. Functions converting from ArviZ.jl and Stanfit objects will probably end up being useful, although InferenceData will probably be getting a revamp fairly soon to use Julia objects, so I'd hold off on ArviZ objects.