hist function reports incorrect bins

[exalate-issue-sync]

hist() in the python api (tested on 3.24.0.1) seems to show incorrect breaks.

For example, you can see that sepal_len has a min value of 4.3 but the first break point which marks the first bin is 5.02

{code:python}>>> iris = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris_wheader.csv")

iris.describe()
iris[0].hist(breaks=5,plot=False)
======================================================
sepal_len sepal_wid petal_len petal_wid class
type real real real real enum
mins 4.3 2.0 1.0 0.1
mean 5.8433 3.0539 3.7586666 1.1986666
maxs 7.9 4.4 6.9 2.5
======================================================
breaks counts mids_true mids density
5.02 nan nan nan nan
5.74 32 2.15 5.38 0.296296
6.46 41 2.55 6.1 0.37963
7.18 42 2.9 6.82 0.388889
7.9 35 3.25 7.54 0.324074
{code}

[exalate-issue-sync]

Sebastien Poirier commented: Some additional observations from a first investigation:

• we should try to obtain results similar to what is obtained with the {{hist}} R function (at least for the properties we’re returning):

{code:r}> hist(iris$Sepal.Length, breaks=5, plot=FALSE)

$breaks
[1] 4 5 6 7 8

$counts
[1] 32 57 49 12

$density
[1] 0.2133333 0.3800000 0.3266667 0.0800000

$mids
[1] 4.5 5.5 6.5 7.5

$xname
[1] "iris$Sepal.Length"

$equidist
[1] TRUE{code}

compare with:

{noformat}> h2o.hist(as.h2o(iris$Sepal.Length), breaks=5, plot=FALSE)

$breaks
[1] 5.02 5.74 6.46 7.18 7.90

$counts
[1] 32 41 42 35

$density
[1] 0.2962963 0.3796296 0.3888889 0.3240741

$mids
[1] 5.38 6.10 6.82 7.54

$xname
[1] "as.h2o(iris$Sepal.Length)"{noformat}

• the {{AstHist}} implementation does sth like this (translated to Python):
  {noformat}interval = (max-min)/nbreaks
  breaks = [min + (n+1)interval for n in range(nbreaks)]{noformat}
  problem is that everything between {{min}} and {{breaks[0]}} is ignored...
  should probably be sth like:
  {noformat}interval = (max-min)/(nbreaks-1)
  breaks = [min + ninterval for n in range(nbreaks)]{noformat}

• there’s basically no proper test for our current {{hist}} implementation:
  ** no backend test for {{AstHist}}
  ** no proper python test : {{pyunit_hist.py}} and {{pyunit_h2oH2OFrame_hist.py}} are both trivial tests, not checking the results.
  ** some decent R tests {{runit_NOPASS_hex_1829_hist.R}}, but marked as {{NOPASS}}, and indeed, they are failing…

• {{mids_true}} column in the Python results (see ticket description above) is 100% buggy: we should either decide to fix it or remove it.
  From my understanding of the implementation, it is supposed to return the “true” midpoint for each bin.
  For example you have a bin [4, 5] but the real values in this bin are between 4.1 and 4.8, then for this bin we should have {{mids = 4.5}} and {{mids_true = 4.45}}.
  However the current impl. is buggy:
  {code:python}In [3]: iris[0].hist(breaks=5,plot=False)
  Out[3]:
  breaks counts mids_true mids density

---

5.02       nan       nan     nan     nan
5.74        32         2.15    5.38    0.296296
6.46        41         2.55    6.1     0.37963
7.18        42         2.9     6.82    0.388889
7.9         35         3.25    7.54    0.324074{code}

{noformat}2.15 = 4.3 / 2 (4.3 = min first bin)
2.55 = 5.1 / 2 (5.1 = min second bin)
2.9 = 5.8 / 2
3.25 = 6.5 / 2
I think you got the pattern.{noformat}

Overall, many things to fix in {{hist}}.

[h2o-ops]

JIRA Issue Migration Info

Jira Issue: PUBDEV-7331
Assignee: New H2O Bugs
Reporter: Lauren DiPerna
State: Open
Fix Version: N/A
Attachments: N/A
Development PRs: N/A