- 
                Notifications
    You must be signed in to change notification settings 
- Fork 28
Description
A faster binhash using xxhash algorithm has been added to Yamanifest (see PR).
As commented on the above PR, I've run some tests comparing binhash and the new option binhash-xxh. It doesn't seem to make a significant difference to small manifests or when running payu setup on the login node. But for manifests with a large number of inputs such as ACCESS-OM2 01deg_jra55_iaf_bgc when running on the payu setup on PBS nodes, it would cut fast hash calculation down by ~50%. It's still in the realm of seconds e.g. from 10 seconds to 5 seconds (time it takes also depends on resources in the job).
It may be worth updating the default fast hash to binhash-xxh, as the time saving will add up. If we did update the default fast hash, we should let users know as manifests will change in payu/dev, and all the md5 hashes would be recalculated which could take a few minutes for large manifests.
It's currently undocumented but it is possible to manually configure the hashes when latest version of yamanifest is in the conda environments, e.g
manifest:
  fasthash:
  - binhash-xxh