Get rid of dbmfile-based dogpile cache

[amotl]

Describe the bug
The dogpile cache based on dbmfile is brittle.

To reproduce
Run Wetterdienst from different Python environments and see accessing the shared cache break more often than not, see #217, #232, #233, #242 and #244. Also, #236 seems to be related as well.

Expected behavior
Wetterdienst should work in all circumstances, even when switching between different Python environments.

Additional context
While the documentation about pickle [1] promises that

The pickle serialization format is guaranteed to be backwards compatible across Python releases provided a compatible pickle protocol is chosen and pickling and unpickling code deals with Python 2 to Python 3 type differences if your data is crossing that unique breaking change language boundary.

it apparently still has problems in our context. While I currently don't have a clue why, I figure it might be coming from marshalling/unmarshalling data frames from different versions of Pandas. We can either investigate this further or use a different means of data storage and/or serialization protocol for the dogpile cache.

[1] https://docs.python.org/3/library/pickle.html

[amotl]

At #254 (comment), we found the dbmfile-based cache also has serious concurrency issues. When looking at future usage of the REST API, this is yet another reason to finally improve the situation here.

[gutzbenj]

Dear @amotl ,

as you have already mentioned here [1], we should change to another backend as apart from the latest problems we faced the dbmbackend is not supported for Windows. Available backends of dogpile.cache are described at [2]. The dogpile.cache redis backend offers options for cases of concurrency and may be the suitable replacement.

distributed_lock – boolean, when True, will use a redis-lock as the dogpile lock. Use this when multiple processes will be talking to the same redis instance. When left at False, dogpile will coordinate on a regular threading mutex.

For future developments we may as well want to follow the project at [3] that has shown improvements in memory consumption with other serialization methods.

[1]

wetterdienst/wetterdienst/util/cache.py

Line 12 in e03a62e

# TODO: Make backend configurable, e.g. better use Redis.

[2] https://dogpilecache.sqlalchemy.org/en/latest/api.html#module-dogpile.cache.backends.memory
[3] https://github.com/jvanasco/dogpile_backend_redis_advanced

[kmuehlbauer]

Hi @gutzbenj and @amotl, concerning retrieving/caching of remote assets, did you already consider using fsspec (https://filesystem-spec.readthedocs.io/en/latest/index.html)?

[amotl]

Dear Kai,

thanks for bringing this to our attention. The option to cache files from a target filesystem locally [1] would be appropriate, right? In this regard, we would have to check if the target filesystem can be just something accessed over HTTP through the regular Python requests module when downloading artefacts from opendata.dwd.de [2], right?

The feature documentation [3] says:

Consistent API to many different storage backends. The general API and functionality were proven with the projects s3fs and gcsfs (along with hdfs3 and adlfs), within the context of Dask and independently.

as well as [4]:

Any path of any file-system can be mapped to a local directory using pyfuse and fsspec.fuse.run(). This feature is experimental, but basic file listing with details, and read/write should generally be available to the extent that the remote file-system provides enough information.

So, we will have to check how to wrap fsspec's target filesystem concept around a general directory index structure as served by HTTP servers?

That would indeed be very elegant in order to shift the caching away from the read-through caching currently employed by dogpile.cache upon Python function calls. Instead, it would be based upon fetching remote resources from HTTP servers.

Please add further adjustments if you believe I am thinking into the wrong direction here.

With kind regards,
Andreas.

[1] https://filesystem-spec.readthedocs.io/en/latest/features.html#caching-files-locally
[2] https://opendata.dwd.de/
[3] https://filesystem-spec.readthedocs.io/en/latest/features.html
[4] https://filesystem-spec.readthedocs.io/en/latest/features.html#mount-anything-with-fuse

[amotl]

How to wrap fsspec's target filesystem concept around a general directory index structure as served by HTTP servers?

Indeed, there is a generic fsspec.implementations.http.HTTPFileSystem implementation, which would probably fit the bill perfectly.

Unfortunately, it is not listed at [5], so I missed it first hand. test_http.py implements the respective test case for that.

[5] https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations

[kmuehlbauer]

I'm stumbling over fsspec every once in a while since several months, but did not have a deeper look into it until now. The user documentation still has room for improvements when it comes to specific examples. What I tried so far is fetching DWD radar data with a very simple approach:

import fsspec
import wradlib as wrl
fname = "https://opendata.dwd.de/weather/radar/sites/sweep_vol_z/ess/hdf5/"
files = list(fsspec.get_mapper(fname))
with fsspec.open(fname+files[-1]) as f:
    ds = wrl.io.open_odim(f, loader="h5netcdf")
    display(ds[0].data)

Alone this is really nice for just grabbing some data from remote https.

I also tried to comprehend the full possibilities (caching, fusing into local filesystem etc) but did not come as far as i wished. But it seems like it is adopted widely around the big data science packages (pandas, xarray, pangeo).

[amotl]

Dear Kai,

I quickly put something up at [1]. This is excellent.

Thanks again and with kind regards,
Andreas.

[1] https://gist.github.com/amotl/0a2eb63708b8a0cf4dc457b1e6a87455