Rollout pandas version 2

[victorlin]

Pandas version 2 is supported by Augur and it's likely that our runtimes will eventually resolve to this new version with potentially breaking changes.

Tasks

• Merge Support pandas v2 augur#1671
• 26.2.0 release augur#1684
• Wait for pandas to resolve to version 2 in conda-base and docker-base. Pending at least
  • Support pandas version 2 blab/evofr#42
  • (docker-base only) Isolate fauna in venv
  • (conda-base only) Support pandas version 2 in pathogen-embed
• Be on the watch for potential unexpected pandas upgrade related changes in our ecosystem
  • ncov-ingest (pinned to v1, installed via pip in custom Dockerfile layered atop docker-base)

Related issues

• test failure beyond pandas 2.0 augur#1303
• Numpy deprecation warnings for pandas imports augur#1316
• parse: Support pandas 2.x augur#1436
• augur/parse.py: fix test failure with pandas 2.2. augur#1471

[corneliusroemer]

The latest pyarrow no longer supports pandas <2 it seems so this is another good reason to allow pandas 2.

[corneliusroemer]

An even bigger motivation: we can't support python 3.12 until we support pandas v2

Would be really cool if you could figure this out @victorlin 😄

[corneliusroemer]

Any progress on this? We'd like to use augur in some parts of Loculus and we've been using some Python 3.12 features (f-strings, type hints) that make downgrading a pain. Would be really cool if we could get Python 3.12 support done, especially now that it's not even the latest anymore (since Python 3.13 release).

[victorlin]

@corneliusroemer thanks for the bump – I've been busy with some Auspice stuff but will try to take a look next week. If anyone else wants to look in the meantime, feel free to do so!

[victorlin]

I just tested locally and Augur+pandas are installable on Python 3.12 as-is. The lack of 3.12 support on pandas 1.x just means that it has to be built from source instead of installing pre-built wheels. I haven't done any actual testing, but maybe this is good enough to get things going on your end.

[corneliusroemer]

Thanks @victorlin - we'll have a look, this means we have to pip-install augur in the conda environment.yaml. Seems to work as a workaround for now, good idea! Though we'll also have to downgrade pandas from 2->1 then.

[corneliusroemer]

I've just had another look at supporting pandas v2, tests pass up to latest pandas v2.2.3

I think we should just go and officially sort it, all the tests pass: unit and functional. If something does come up, we can fix it. Would be good to run pathogen-ci with it put that's about it.

Update: pathogen-CI works with v2.2.3 of pandas 🎉
nextstrain/augur#1672 (comment)

[victorlin]

@corneliusroemer thanks for looking! Did you mean to create 2 PRs for this (nextstrain/augur#1671, nextstrain/augur#1672)? The only difference between the two is that one supports pandas v1 while the other doesn't. I think it would be good to keep support for v1 if trivial. The tests are passing so it seems fine?

I think pandas changes should be good as long as tests are passing. But I'll want to review the Cram stuff in more detail. Could you rebase the pandas PR so that it's not based on the changes from nextstrain/augur#1667?

[corneliusroemer]

Yes the two PRs are absolutely on purpose. I agree we should keep v1 compatibility for now. The reason I made the second PR excluding V1 is that pathogen CI will still use pandas v1 in tests unless you exclude v1. Does that make sense?

The reason is that we use the conda base and then pip install augur into it in CI. Since conda base already has pandas v1 and v1 is allowed pip keeps it around and doesn't upgrade to V2. So the actual V2 PR doesn't actually check V2 in pathogen CI. Hence the need for the test PR.

I expect that we can upgrade pandas to V2 in conda base once augur supports it so excluding V1 is just a temporary test PR