compatibility with xr.DataTree

[mathause]

• Closes #xxx
• Tests added
• Fully documented, including CHANGELOG.rst

Adds compatibility with xr.DataTree. Works (at least locally) but is not finished and needs some ugly workarounds due to some design decisions in the new version... The new code path is only tested in upstream-dev ci.

I am not sure we should support both, datatree and xarray datatree, but this was actually the smaller part of the whole operation.

[codecov]

Codecov Report

Attention: Patch coverage is 79.68750% with 13 lines in your changes missing coverage. Please review.

Project coverage is 80.29%. Comparing base (615a0c7) to head (77d950a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
mesmer/core/_datatreecompat.py	43.47%	13 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #607      +/-   ##
==========================================
- Coverage   80.44%   80.29%   -0.16%     
==========================================
  Files          49       50       +1     
  Lines        3079     3121      +42     
==========================================
+ Hits         2477     2506      +29     
- Misses        602      615      +13     

Flag	Coverage Δ
unittests	80.29% <79.68%> (-0.16%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mathause]

The upstream packages seem flaky

[veni-vidi-vici-dormivi]

Wow thanks, that is a lot of work! I think it's okay not to support xarray-datatree at all then, since it is not recommended to use it anyway.

[mathause]

Thanks. It was mostly annoying because 98 % I did not understand what is happening 🤷‍♂️

As long as we need to patch xr.map_over_datasets we can just as well support datatree (because we need to import from _datatreecompat anyway.

It's annoying that the new version has become so bothersome to work with 😒

[mathause]

This takes all the fun out of using xr.DataTree and cannot be expected from a user. The other 'issues' (missing methods) will probably be fixed in time but this seems to be a design decision. 👎

[veni-vidi-vici-dormivi]

Yes I agree.

[mathause]

I think what we have to do is write a helper function calc_anomaly(dt, ref_period) with this inside.

[mathause]

I have not yet updated the example scripts and we need to fix the pin for xarray (so it's not only tested in upstream-dev ci).

[mathause]

I opened pydata/xarray#10042 - there seems to be some willingness to enable skipping empty nodes. With this change we could avoid the skip_empty_nodes helper function and I'd say that's the moment we can "fully" support xr.DataTree.

But not super convinced this PR gets a fast turnaround 😒

[mathause]

Ah yes and after 507cd1c this definitely requires xarray main, i.e. we require

xarray >=2023.07, < 2024.10, > 2025.1

(thats probably not a valid specifier)

[yquilcaille]

Should we merge this branch? I'm quite blocked now, because I'm trying to use the former datatree, but knowing that we will use this PR...

[yquilcaille]

Actually, there are no imports in the init.py of mesmer.core. Should we add a commit doing the following?

# flake8: noqa

from ._data import *
from .datatree import *
from .geospatial import *
from .grid import *
from .mask import *
from .regionmaskcompat import *
from .utils import *
from .volc import *
from .weighted import *

[mathause]

Should we merge this branch? I'm quite blocked now, because I'm trying to use the former datatree, but knowing that we will use this PR...

I can merge (but see my next comment), or you can use the old version and I'll update it later.

[mathause]

Actually, there are no imports in the init.py of mesmer.core. Should we add a commit doing the following?

# flake8: noqa

from ._data import *
from .datatree import *
from .geospatial import *
from .grid import *
from .mask import *
from .regionmaskcompat import *
from .utils import *
from .volc import *
from .weighted import *

Users don't really need to access functions in core so there is no need to add these. Except for the ones in datatreecompat. We should make them top-level functions (mesmer.DataTree). (However, I would very much prefer that we don't need to release this and am still hoping for pydata/xarray#10042.)

[mathause]

This is not moving forward on xarrays site, so I am merging. I will merge +- as is. Open a PR for a calc_anomaly helper function, then drop support for the old datatree package (i.e. require xarray >= 2025.02).

[mathause]

I also have to find out if map_over_datasets is only used internally. If not I'll make it a top-level function...

[mathause]

And we can always remove the compatibility map_over_datasets if that get's resolved before we release.