Create a daemon process to fetch geo datasets information in the background

[olegs]

Once we have caching database, it would be good to have a background activity process to fetch datasets information in background.

[MomirMilutinovic]

The GEO FTP server presents a useful tool for fetching new datasets that are missing from GEOmetadb. The datasets on this FTP server are organized in contiguous batches of accession numbers (e.g., GSE310nnn). Leveraging the fact that GEO assigns accession numbers sequentially, this structure may allow us to efficiently download fresh datasets.

For example, the link ftp.ncbi.nlm.nih.gov/geo/series/GSE310nnn, corresponds to every dataset with accession numbers from GSE310000 to GSE310999. The datasets in this specific folder were all submitted in the second half of November.

Downloading the MINiML files for all 217 datasets in this directory, took a bit over 7 minutes, resulting in 233MB of downloads. Therefore, it seems that using the FTP server to backfill GEOmetadb in the background is a viable approach.

Edit: The command I used for this test was

wget --reject-regex="/matrix/|/suppl/|/soft/" -r -nH --cut-dirs=2 ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE310nnn/

Edit 2: Corrected the number of datasets from 399 to 217

[MomirMilutinovic]

It seems the scraping process can be optimized by leveraging multiple parallel HTTP connections to the NCBI server. I've tested an aiohttp-based pipeline for backfilling the geometadb database, which successfully processed the same 217 datasets in approximately one minute. Given this performance, scaling the daemon to cover the entirety of GEO should be highly feasible.