Options for the fsspec sync hanging bug

[norlandrhagen]

I wanted to get some thoughts on what peoples current strategies are to get around the fsspec hanging bug.

It seems like the two working approach's are:

• Running pangeo-forge-recipes with an Xarray fork that disables garbage collection. @jbusecke has experience using this.
• Running pangeo-forge-recipes using @moradology's https://github.com/moradology/httpfs-sync library.

Some questions:

• Should we try to get a PR to Xarray to add a flag to turn off GC?
• a-la httpfs-sync, @ranchodeluxe has an example implementing sync for s3fs. With these two we cover reading from https & s3. Is this enough?

Wondering if folks have any more thoughts on this or general recommendations.

[jbusecke]

Thanks for tracking this @norlandrhagen.

I have not 100% convinced myself, but I believe that for the CMIP case at least I am still getting more datasets to pass (without download errors) by turning off the gc in xarray via gregs hack. Will test this more thoroughly and report back.