License issue?

[devmotion]

I'm not sure (I am always confused how licenses interact or are compatible with each other) but the comment in

StatsModelComparisons.jl/src/psisloo.jl

Lines 1 to 7 in 0789687

	# Copyright (c) 2015 Aki Vehtari, Tuomas Sivula
	# Original Matlab version by Aki Vehtari. Translation to Python
	# by Tuomas Sivula.
	# This software is distributed under the GNU General Public
	# License (version 3 or later); please refer to the file
	# License.txt, included with the software, for details.

seems to indicate that at least parts of the package have to use the GPLv3? This seems a bit problematic and is not mentioned in the LICENSE.md. I guess it would be possible to resolve this problem by reimplementing the algorithm without looking at the original code (and probably also not the existing implementation in this package if it was based on the original code), eg., just by studying the original paper.

[itsdfish]

As far as I can tell, the main difference between GNUv3 and MIT is that GNUv3 requires authors of derivative works to make the source code open source. The derivative code inherits the GNUv3 license. This is mostly a problem for people who want to incorporate the code into commercial products---if the source code is open source, there is no incentive to purchase. Are there other problems that I am not aware of?

[devmotion]

IMO the main problem is that this implies that not all parts of the package can be licensed under MIT and this is not clear from the license file (actually, it's incorrect currently). Additionally, my impression is that some developers are reluctant to include or use any GPL code in their packages due to the additional restrictions and since commonly packages use the MIT license in the Julia ecosystem.

[itsdfish]

I can see how that might be an annoyance. The annoyance would be minor if it does not affect other packages. For example, if MCMCChains uses StatsModelComparisons as a dependency, does that change the license for MCMCChains?

[itsdfish]

By the way, commonly used packages like Distributions.jl and StatsFuns.jl are subject to the same licensing issues.

[devmotion]

AFAIK only the parts involving Rmath use a different license - and this is stated in e.g. https://github.com/JuliaStats/Distributions.jl/blob/master/LICENSE.md. So the GPL only applies if you use Rmath-dependent functionality.

[itsdfish]

There appears to be debate about the scope of the license. Some argue that the license transfers to the entirety of a derive work. Under this interpretation, the GNU license would apply to RMath.jl because it uses software that is under a GNU license. As a consequence, the license would transfer to StatsFuns.jl, then to Distributions.jl, then to DistributionsAD.jl and then to Turing and so forth, until nearly all of the Julia ecosystem is under a GNU license. Others, by contrast, have adopted a more restrictive interpretation. In either case, I would follow the precedent set by Distributions.jl, which is to note the specific parts of the code that are directly associated with a GNU license.

[goedman]

Yes, I'm thinking about how to do this for all 3 main packages in StatisticalRethinking. From Richard McElreath I have explicit permission, so I'm not too concerned there and it is not MIT licensed.

But in both StatsModelsComparisons.jl and StructuralCausalModels.jl there are some parts where David's remark is absolutely correct and you end up in these murky discussions as to how copyleft and derivative work flows in layered software.

I'll check if I have left sufficient copyleft hints behind and add a more general hint in LICENSE.md.

If I can't figure that out, I'll have to change the LICENSE.md in a similar way as I did for StatisticalRethinking.jl. I understand that (or even the copyleft hints already) might be a problem for some folks and the Turing project.

[ParadaCarleton]

ParetoSmooth is fully MIT-licensed, and can be used to replace the parts of this package using code licensed under GPL. So long as older versions can be used under the conditions of GPL, and all contributors agree to relicensing, you can switch the license over to MIT after removing any dependencies/code licensed under GPL (which I believe is just the PSIS.jl code).

[goedman]

Thank you. My plan is to deprecate this package (StatsModelComparisons.jl) and move WAIC related code over to StatisticalRethinking. Deprecating means that the package will remain available for up to 2 years or until Julia 2.0 is released.

All in all, this is a substantial amount of work and a breaking change for both the StatisticalRethinkingJulia and StanJulia suites of packages. In addition to switching over to ParetoSmooth I'm also targeting AlgebraOfGraphics.jl and MCMCDiagnosticsTools.jl. And I'm in the process of adding DiffEqBayesStan.jl to StanJulia.

But before that, I'll first do the KeyAxis.jl based implementation of loo_compare and the release of DiffEqBayesStan.jl.

Particularly the choice of the axis in the KeyedArrays is interesting to say the least. StanJulia, for 10 years now, used [draws, params, chains]. My simple underlying reason for that is Julia being column major and most often I selected columns.
I ran into issues with multilevel models. My first attempt was NamedTuples (e.g. some of the log_lik examples) and more recently Tables.jl based chains. Both are a step forwards but not ideal.

I've seen some Julia posts suggestion [params, draws, chains] (as you have chosen as well), while a recent move in R (package posterior and RVar-s) opt for [draws, chains, params].

I guess the jury is still out. I wonder of KeyAxis.jl can help here using views based on permutedims.

[ParadaCarleton]

Thank you. My plan is to deprecate this package (StatsModelComparisons.jl) and move WAIC related code over to StatisticalRethinking. Deprecating means that the package will remain available for up to 2 years or until Julia 2.0 is released.

All in all, this is a substantial amount of work and a breaking change for both the StatisticalRethinkingJulia and StanJulia suites of packages. In addition to switching over to ParetoSmooth I'm also targeting AlgebraOfGraphics.jl and MCMCDiagnosticsTools.jl. And I'm in the process of adding DiffEqBayesStan.jl to StanJulia.

But before that, I'll first do the KeyAxis.jl based implementation of loo_compare and the release of DiffEqBayesStan.jl.

Particularly the choice of the axis in the KeyedArrays is interesting to say the least. StanJulia, for 10 years now, used [draws, params, chains]. My simple underlying reason for that is Julia being column major and most often I selected columns.
I ran into issues with multilevel models. My first attempt was NamedTuples (e.g. some of the log_lik examples) and more recently Tables.jl based chains. Both are a step forwards but not ideal.

I've seen some Julia posts suggestion [params, draws, chains] (as you have chosen as well), while a recent move in R (package posterior and RVar-s) opt for [draws, chains, params].

I guess the jury is still out. I wonder of KeyAxis.jl can help here using views based on permutedims.

I believe someone was interested in adding WAIC to StatsBase, to supplement their AIC calculations; I'd open an issue there. @robertfeldt

[goedman]

Thank you Carlos. WAIC is pretty simple given the availability of logsumexp(). And in StatisticalRethinking, waic(_compare) is used in a particular way (as is psis_loo) as one can simply pass in a vector of models. I tend to overload existing methods for that purpose.

As a 1st exercise with AxisKeys.jl I added an option to specify a KeyedArray as output_format to read_samples() in StanSample.jl (v3.2.0). I'm quite impressed how powerful that is and how simple it is to select chains, append chains, select matrices for vectored parameters, etc:

julia>  read_samples(m10_4s; output_format=:keyedarray)
3-dimensional KeyedArray(NamedDimsArray(...)) with keys:
↓   param ∈ 9-element Vector{Symbol}
→   iteration ∈ 1000-element UnitRange{Int64}
□   chain ∈ 4-element UnitRange{Int64}
And data, 9×1000×4 Array{Float64, 3}:
[:, :, 1] ~ (:, :, 1):
                   (1)         (2)         (3)         …  (998)         (999)         (1000)
   Symbol("a.1")    -0.986444   -0.8161     -0.441504        -0.672502     -0.505936      -0.817204
   Symbol("a.2")     5.15659     8.55964    16.7775           9.53766       8.25182        6.73213
   Symbol("a.3")    -0.893064   -1.03065    -1.00922         -0.955229     -1.14963       -0.78207
   ⋮                                                   ⋱                    ⋮         
   Symbol("a.6")     0.14984    -0.184515    0.267662         0.511867      0.240681       0.380556
   Symbol("a.7")     1.77497     1.81025     1.4951           1.93943       2.04955        1.60187
  (:bp)              0.769698    1.16076     1.10907          0.689747      1.09935        0.504433
  (:bpC)            -0.130559   -0.474077   -0.517344        -0.478045     -0.447211       0.296149

[:, :, 2] ~ (:, :, 2):
...

# Overloaded matrix() from Tables interface, select all parameters starting with a String or Symbol followed by [, . or _.

julia> ma = matrix(ka, "a");

julia> params, ndraws, nchains = size(ma);

# Append all chains

julia> rma = reshape(ma, params, draws*nchains);
julia> mean(rma, dims=2) |> display
7×1 Matrix{Float64}:
 -0.7417288872399986
 10.948280419999978
 -1.05673619375
 -1.0532829562499972
 -0.7410015863050001
  0.21910729936674975
  1.811969725750005

Next up is the PR for loo_compare.