@@ -31,20 +31,22 @@ A struct containing the results of model comparison.
31
31
substantial overlap, which creates a downward biased estimator. LOO-CV differences are
32
32
*not* asymptotically normal, so these standard errors cannot be used to
33
33
calculate a confidence interval.
34
- - `gmp::NamedTuple`: The geometric mean of the posterior probability assigned to each data
35
- point by each model. This is equal to `exp(cv_avg/n)` for each model. By taking the
36
- exponent of the average score, we can take outcomes on the log scale and shift them back
37
- onto the probability scale, making the results more easily interpretable. This measure
38
- is only meaningful for classifiers, i.e. variables with discrete outcomes; it is not
39
- possible to interpret GMP values for continuous outcome variables.
34
+ - `gmpd::NamedTuple`: The geometric mean of the predictive distribution. It equals the
35
+ geometric mean of the probability assigned to each data point by the model, that is,
36
+ `exp(cv_avg)`. This measure is only meaningful for classifiers (variables with discrete
37
+ outcomes). We can think of it as measuring how often the model was right: A model that
38
+ always predicts incorrectly will have a GMPD of 0, while a model that always predicts
39
+ correctly will have a GMPD of 1. However, the GMPD gives a model "Partial points"
40
+ between 0 and 1 whenever the model assigns a probability other than 0 or 1 to the
41
+ outcome that actually happened.
40
42
41
43
See also: [`PsisLoo`](@ref)
42
44
"""
43
45
struct ModelComparison{RealType<: Real , N}
44
46
pointwise:: KeyedArray{RealType, 3, <:NamedDimsArray, <:Any}
45
47
estimates:: KeyedArray{RealType, 2, <:NamedDimsArray, <:Any}
46
48
std_err:: NamedTuple{<:Any, Tuple{Vararg{RealType, N}}}
47
- gmp :: NamedTuple{<:Any, Tuple{Vararg{RealType, N}}}
49
+ gmpd :: NamedTuple{<:Any, Tuple{Vararg{RealType, N}}}
48
50
end
49
51
50
52
@@ -114,8 +116,8 @@ function loo_compare(
114
116
log_norm = logsumexp (cv_elpd)
115
117
weights = @turbo warn_check_args= false @. exp (cv_elpd - log_norm)
116
118
117
- gmp = @turbo @. exp (cv_elpd / data_size)
118
- gmp = NamedTuple {name_tuple} (gmp )
119
+ gmpd = @turbo @. exp (cv_elpd / data_size)
120
+ gmpd = NamedTuple {name_tuple} (gmpd )
119
121
120
122
@turbo warn_check_args= false @. cv_elpd = cv_elpd - cv_elpd[1 ]
121
123
@turbo warn_check_args= false avg_elpd = cv_elpd ./ data_size
@@ -125,7 +127,7 @@ function loo_compare(
125
127
statistic= [:cv_elpd , :cv_avg , :weight ],
126
128
)
127
129
128
- return ModelComparison (pointwise_diffs, total_diffs, se_total, gmp )
130
+ return ModelComparison (pointwise_diffs, total_diffs, se_total, gmpd )
129
131
130
132
end
131
133
0 commit comments