Thanks for this beautiful start of the benchmarking and having a common way ;
Currently, Pangaea uses setup.py and requirements.txt (with conda file) for dependency management. There are quite a few nice tools that have evolved, and robust dependency management remains crucial for reproducibility. However, the repo currently lacks tags and releases, and uses a static requirements.txt without any version constraints. Because of this, I'm uncertain whether I'd be able to replicate this benchmark setup. Additionally, Docker support would be really helpful.
Do you have plans to upgrade the dependency management or would you accept contributions toward this? I think that for the objective of evaluating models, the environment needs to be reproducible. uv could be a good option to pin packages, and tagging would be beneficial. Deps pinning is crucial in anycases ! It would have been helpful to have tagging or versioning information so that we could identify which version was used or at what state of the code the results presented in the paper were produced. Currently, I'm unable to pinpoint this information. Something to keep in mind for next version may be ?
Thanks for this beautiful start of the benchmarking and having a common way ;
Currently, Pangaea uses setup.py and requirements.txt (with conda file) for dependency management. There are quite a few nice tools that have evolved, and robust dependency management remains crucial for reproducibility. However, the repo currently lacks tags and releases, and uses a static requirements.txt without any version constraints. Because of this, I'm uncertain whether I'd be able to replicate this benchmark setup. Additionally, Docker support would be really helpful.
Do you have plans to upgrade the dependency management or would you accept contributions toward this? I think that for the objective of evaluating models, the environment needs to be reproducible. uv could be a good option to pin packages, and tagging would be beneficial. Deps pinning is crucial in anycases ! It would have been helpful to have tagging or versioning information so that we could identify which version was used or at what state of the code the results presented in the paper were produced. Currently, I'm unable to pinpoint this information. Something to keep in mind for next version may be ?