GDALRasterSources performance

[pomadchin]

During the RasterSources benchmark https://github.com/pomadchin/vlm-performance/ on EMR we received no performance improvements with GDAL usage. On the contrary, it created more complexity in jobs configuration. Without tuning GDAL_CAHEMAX options and wo (for warp operations), EMR nodes are dying / hanging and => tasks are failing. This is also can be the problem of the GDAL 2.3 usage on yarn and according to OSGeo/gdal@5351335 GDAL doesn't respect cgroups. But it is unclear how yarn limits resources, it may be not using cgroups to limit CPU / RAM and this can be useless in this case.

Related to https://github.com/azavea/raster-foundry-platform/issues/743
and to https://github.com/azavea/geotrellis/issues/68

[rossbernet]

@notthatbreezy
@jcahail
@echeipesh

We talked about this yesterday in cross team planning.

But we wanted to confirm the expected outcome, timeline, and priority for RF.

Does it make sense to wait until https://github.com/azavea/raster-foundry-platform/issues/743 would be resolved, because this has the potential to fix the RF problems.

It's an option to spend at least two points this sprint, but we confirm with you all.

[notthatbreezy]

There are a few issues being worked on that probably need to be resolved for us to get more information to make working on GDAL related performance issues easier as far as RF goes.

• Ensure that the healthcheck fails when the tile servers turn into zombies: https://app.zenhub.com/workspaces/raster-foundry-team-workspace-5878d68005907de543b50bbd/issues/raster-foundry/raster-foundry/4945
• Ensure that core dumps are written to S3 (and push this to prod): https://app.zenhub.com/workspaces/raster-foundry-team-workspace-5878d68005907de543b50bbd/issues/azavea/raster-foundry-platform/735
• Then start investigating an issue where large mosaics (many scenes) throw an error: https://app.zenhub.com/workspaces/raster-foundry-team-workspace-5878d68005907de543b50bbd/issues/azavea/raster-foundry-platform/738

I think at least the first two need to be done before we can start looking into garbage collection more otherwise we won't really have enough information to determine if we're improving things.

[pomadchin]

RPMs for the EMR test are here: s3://geotrellis-test/rpms/v1.0/