Skip to content

canvas.raster mean aggregation on integer arrays causes artifacts #655

Open
@philippjfr

Description

@philippjfr

The regridding support in datashader is based on numba code which expects float arrays. Therefore when a user passes integer arrays to canvas.raster they are first cast to floats, then regridded and finally cast back to integers. Unfortunately this introduces highly undesirable artifacts which are very evident when using 'mean' aggregation. This is because numerical precision issues will cause values to fluctuate slightly around the integer values even in regions which are uniform in value, and when casting the floats with minor numerical precision differences to integers the minor differences are potentially amplified to a whole integer difference.

The question then is how we should handle mean/median/var/std aggregation on an integer array. I see a number of options:

  1. 'mean'/'var'/'median'/'std' aggregation on an integer arrays returns floats
  2. canvas.raster always returns float arrays
  3. After 'mean' aggregation the float array is rounded before casting back to integers

My inclination is to go with option 1) but would welcome other suggestions.

Cc: @mmccarty

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions