Use directory (datatable entries) sizes in TPV

[bernt-matthias]

Run time often depends on the size of reference data. Is it possible to use the size of some folder (stored in a datatable) in TPV? Would be useful for some tools.

I imagine that this would be useful for quite a few tools and could be a better solution compared to extending data tables one by one galaxyproject/tools-iuc#7681

[cat-bro]

kraken2 's shared db rule uses ref size here https://github.com/galaxyproject/tpv-shared-database/blob/main/tools.yml#L1632

[pvanheus]

When I discussed this with @bgruening he was of the opinion that this might be overwhelming to compute for each job that was executed, thus the plan to compute it in the DM.

[bernt-matthias]

In this rule only the size of a single file is checked which should be fast - plus its cached. Has somebody done some benchmarks?

Guess if more rules use such a scheme it might make sense to provide this centrally -- or should we got the other way round and provide it such that more rules can use it (easily).

[pvanheus]

Hm, the use of a cache changes my thinking though. What is the lifetime of this cache? If it is shared between job invocations it will indeed be efficient enough to be used.

[bgruening]

I still think that kraken2 indices are not changing too often and we can hardcode commonly used indices here, for indices that we don't have, we can calculate the size as we do know.

@pvanheus I think the cache is per job-handler. Means one read of all kraken2 indices per handler-restart.

[nuwang]

Can confirm that cache is per job-handler. TPV compiles all python code fragments on first job dispatch and thereafter, relies on cached fragments.

[pvanheus]

Thanks @nuwang - this indeed makes the size-calculating work in the kraken2 DM update likely obsolete. Just a question on the rule that is currently in the shared rules: https://github.com/galaxyproject/tpv-shared-database/blob/main/tools.yml#L1673C28-L1673C30 seems to set 48 GB as a lower cap for kraken2 jobs. This is much too high for e.g. the Mycobacterium database, which is only about 8 GB. the memory usage of a kraken2 job should be a bit larger than the on-disk database size (see: https://pmc.ncbi.nlm.nih.gov/articles/PMC6883579/), so the ref_size * 1.2 seems right without adding 48? (Especially if math.ceil rather than int is used on the raw on-disk size)

[nuwang]

@pvanheus I think that @bgruening makes a good point about adding the column to the data table. We could go ahead with that, and use a runtime calculation in tpv as the fallback. That way, existing users can still use the rule as is, while users who install fresh kraken indices with a datamanager already have that size pre-calculated - saving them some wasteful computation on handler restarts. I don't know what the rationale for the 48 is, but yes, the rule did originate from usegalaxy.org.

[cat-bro]

A potential disadvantage of using file size calculated by the handler is that it requires all reference data to be available for measurement by the handler. There might be some databases that are so big that the admin only wants to keep them on the remote pulsar node where the jobs will run.

[mvdbeek]

Getting the size via lstat should work via the catalog metadata, so handlers won't be pulling the database in. But it is of course true that cvmfs would need to be mounted, unless you copy over the loc files in another way.

[bernt-matthias]

Yes, that is what I thought: https://cvmfs.readthedocs.io/en/stable/cpt-details.html#getattr-and-lookup

[bgruening]

How many indices do we expect? Can we keep this (I assume) small list here in the YAML?