Evaluating bitrounded results with metrics?

[observingClouds]

To ease the decision on specific keepbits for a simulation, it would be great to include some metrics into this package that can quantify the differences between bit-rounded and original values. The simplest being:

• mean
• standard deviation

These could also be enhanced by plotting routines that loop over possible keepbits.

[aaronspring]

do you mean temporal metrics, eg. temporal mean?

I would go for normalized error (ori - bitrounded)/ori and error normalized by temp std (ori - bitrounded)/ori.std("time").

we could use metrics from https://xskillscore.readthedocs.io/en/stable/api.html#distance-metrics

In general, I am not sure whether we can provide a general solution here. for simple metrics such as mean and std, the user can easily do this in xr. maybe what I lack is a short API proposal like compare(ori, rounded, metrics=[])->"xr.plot(col="metric",row='?")" I think this will be hard to generalize

see also milankl/BitInformation.jl#25 (comment)

I'm not sure I managed to get my point across: If you approach the compression of fgco2 with the prior knowledge that your error should not be higher than 1% then you don't need the bitinformation algorithm. You can directly infer that for a max 1% error you'll need 6 mantissa bits

[aaronspring]

lets definately have a notebook how to deal with this. or even integrate just one plot into the quick-start

[milankl]

Note that both mean and standard deviation are not particularly insightful metrics for assessing bitrounding. Following the IEEE-754 standard, bitrounding is bias-free with respect to 0 and also for absolute values. Having said that, this is true for any data, as long as the assumption of a uniform distribution within ulp is true. If, for example, for some weird reason all your data is just larger than ulp/2 (ulp=unit in the last place, the distance between two adjacent representable numbers after rounding) there'll be a bias as every value is rounded up. But in practice, a bias-free rounding mode will not affect the mean.

A similar argument holds for standard deviation: As long as the standard deviation is much larger than ulp then rounding will not affect it.

[rsignell-usgs]

@christophernhill, what was the metric you were mentioning the other day where you experienced non-physical behavior for rounded values?
Was it eddy covariance?

[christophernhill]

Yes <u'T'> = <uT> - <u><T> Chris

…

On Thu, Jun 2, 2022 at 3:59 PM Rich Signell ***@***.***> wrote: @christophernhill, what was the metric you were mentioning the other day where you experienced non-physical behavior for rounded values? Was it eddy covariance? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[observingClouds]

Thanks @christophernhill for these insights. In case you find spurious behaviour about the bit rounded values, please also consider reporting those issues upstream at https://github.com/milankl/BitInformation.jl. Currently a lot of those issues are discussed in milankl/BitInformation.jl#25

How much "information" did you preserve for u and T? 99%, 100% or something else?

[milankl]

I've just tested this with the following code

using BitInformation, Statistics, UnicodePlots

function covariance(u,T)
    uT_cov = mean(u .* T) - mean(u)*mean(T)
    uT_cov_rounded = mean(round(u,7) .* round(T,7)) - mean(round(u,7))*mean(round(T,7))
    return uT_cov,uT_cov_rounded,uT_cov-uT_cov_rounded
end

and

function cov_test(n)
    errors = zeros(n)
    for i in 1:n
        u = randn(5)
        T = 300 .+ 5u
        _,_,errors[i] = covariance(u,T)
    end
    return errors
end

the errors are normally distributed with zero mean, and do not change whether one calculates <u'T'> or <uT> - <u><T>:

julia> histogram(errors,nbins=30,vertical=true)
        ┌                          ┐ 
   1896  ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀█▇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀██⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀██⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀██⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀▇██▇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀████⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀████⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀████⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀▄████▃⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀██████⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀██████⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀▇██████▆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀⠀████████▁⠀⠀⠀⠀⠀⠀⠀⠀⠀  
         ⠀⠀⠀⠀⠀⠀⠀▆█████████⠀⠀⠀⠀⠀⠀⠀⠀⠀  
      0  ⠀⠀⠀▁▁▃███████████▆▃▁▁⠀⠀⠀⠀⠀  
        └                          ┘ 
        ⠀-1.2⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀1.4⠀ 
        ⠀⠀⠀⠀⠀⠀μ ± σ: 0.0 ± 0.23⠀

So while the rounding errors amplify in the product (eps(BFloat16) = 0.0078125, because I rounded to 7 mantissa bits in covariance) they are unbiased. So I reckon what you see as "unphysical" is not actually a flaw in the method but just a manifestation that a product of two variables with uncertainty is just even more uncertain? Please correct me if there's more to it!

[aaronspring]

Yes

<u'T'> = -

Do you also get unacceptable errors when regridding the field instead of bitrounding?

[christophernhill]

@aaronspring and others - just came across this. I'll try and remind myself what we did in detail. We weren't doing a super systematic analysis!