test_unet_mlm_matches_example_json AssertionError: Generated STAC Item does not match the saved example.

[rbavery]

🐛 Bug Report

In this test, the expected metadata stored in item_pytorch_geo_unet.json does not match the parsed metadata from the unet model weights from Torchgeo

tests/torch/test_unet_mlm.py:23: in test_unet_mlm_matches_example_json
    assert item == expected, "Generated STAC Item does not match the saved example."
E   AssertionError: Generated STAC Item does not match the saved example.

Band names changed: The model now reports B8 instead of B8A
Value scaling removed: The model previously had z-score normalization [{'mean': 0, 'stddev': 3000, 'type': 'z-score'}] but now returns an empty list []

🔬 How To Reproduce

Steps to reproduce the behavior:

1. make test

Environment

• OS: Mac
• 3.11
• latest/main

📈 Expected behavior

test passes. Or do we need this test? It seems to be checking behavior we don't control in TorchGeo, they can change how metadata gets stored. I'd be fine with removing and testing this a different way with mocks so that we don't have an upstream dependency on torchgeo for this test.

📎 Additional context

This test functionality was added by CRIM I believe, could y'all take a look and confirm if we need this test as is or if it should be changed?

[fmigneault]

@rbavery
Are you looking into fixing it? (just want to avoid duplicate work if you already started).

[rbavery]

I have not, not sure if the correct approach here is to investigate how torchgeo metadata was updated. Because I think keeping our test in sync with torchgeo metadata could be a chore and there are quicker ways to test we can source metadata from model weights.

[fmigneault]

I agree that the torch utilities to extra MLM metadata are provided as best-effort. If torchgeo drastically change their metadata structure, I would expect that users interested in these tools (which could be us depending on projects) would provide PRs fixing the mapping.

However, in this specific case, the error seems related to the following, which is manually injected as MLM extra metadata. I do not know why a different behaviour happens, but I think it is not a torchgeo issue specifically (maybe a pydantic change?).

mlm/stac_model/examples.py

Lines 66 to 76 in 41a2cef

	value_scaling = [
	cast(
	ValueScalingObject,
	dict(
	type="z-score",
	mean=mean,
	stddev=stddev,
	),
	)
	for mean, stddev in zip(stats_mean, stats_stddev, strict=False)
	]

I can try investigating later this week.