Skip to content

Handle BOM-prefixed EcoHarmonogram responses in v3#6984

Merged
markvp merged 1 commit into
mampfes:release/3.0.0from
EvotecIT:fix/ecoharmonogram-v3-json
Jul 21, 2026
Merged

Handle BOM-prefixed EcoHarmonogram responses in v3#6984
markvp merged 1 commit into
mampfes:release/3.0.0from
EvotecIT:fix/ecoharmonogram-v3-json

Conversation

@PrzemyslawKlys

@PrzemyslawKlys PrzemyslawKlys commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary

  • decode EcoHarmonogram response text as UTF-8 with BOM support before parsing JSON
  • retry the provider request once when a response is not valid JSON
  • add focused coverage for normal JSON, BOM-prefixed JSON, and a transient invalid response

The integration-wide transient-fetch recovery previously included here has been removed at maintainer request. That work is preserved separately for a future v3.1 PR; this PR now contains only the provider-specific parser fix suitable for release/3.0.0.

Validation

  • 3 focused EcoHarmonogram response tests passed
  • 3,070 passed, 4 skipped, and 2,139 deselected in the complete offline suite
  • Ruff, Ruff format, codespell, Bandit, and mypy hooks passed

Related to #6981 and the v3 architecture in #6562.

@PrzemyslawKlys PrzemyslawKlys changed the title Handle BOM-prefixed Ecoharmonogram responses in v3 Harden transient source recovery and EcoHarmonogram parsing in v3 Jul 20, 2026

@markvp markvp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, and for the thorough diagnosis and tests. The EcoHarmonogram parsing fix is spot on: I confirmed it's a genuine v3 regression. On master the client used requests with response.encoding = "utf-8-sig", which handled the BOM; the pipeline migration moved it onto the curl_cffi session, whose .json() parses raw bytes and chokes on that same payload. Decoding utf-8-sig explicitly is the right fix, and I like that the error message doesn't leak the response body. This half is a must-have for 3.0.

Could I ask you to split the PR, though? The transient-recovery hardening (the SourceShell.fetch -> bool change, UpdateFailed on first refresh, and the YAML "advance only when all sources succeed" behaviour) isn't actually v3-specific: the same "a failed fetch still marks the day as fetched" behaviour is present on master today, so it's a long-standing robustness improvement rather than a v3 regression. It also changes coordinator and YAML fetch semantics that deserve their own review, and it would widen the blast radius of the release branch.

So my suggestion:

  • Keep this PR to the EcoHarmonogram fix, targeting release/3.0.0.
  • Move the recovery hardening to a separate PR against 3.1 (it's an additive change, so a minor). Given the v2 counterpart in #6983, that one may even be a candidate for a 2.x patch.

Happy to help line up the 3.1 PR once this one is scoped down. Thanks again.

@markvp markvp added this to the 3.0.0 milestone Jul 21, 2026
@PrzemyslawKlys
PrzemyslawKlys force-pushed the fix/ecoharmonogram-v3-json branch from ab7a5e4 to 3fe4aee Compare July 21, 2026 06:05
@PrzemyslawKlys PrzemyslawKlys changed the title Harden transient source recovery and EcoHarmonogram parsing in v3 Handle BOM-prefixed EcoHarmonogram responses in v3 Jul 21, 2026
@PrzemyslawKlys

Copy link
Copy Markdown
Contributor Author

Applied the requested scope split. This branch is now rebased on the current release/3.0.0 and contains only the EcoHarmonogram BOM/JSON parser fix plus its three response tests. The generic transient-fetch recovery is no longer in this PR and is preserved separately for a future v3.1 PR. The focused tests pass, and the complete offline suite reports 3,070 passed and 4 skipped.

@markvp markvp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for splitting this down to just the EcoHarmonogram fix — exactly the 3.0 scope. The utf-8-sig BOM decode plus the single retry reads cleanly, the regression test is now wired into pytest.ini so it actually runs, and CI is green. Good for 3.0.

@markvp
markvp merged commit 8a061b9 into mampfes:release/3.0.0 Jul 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants