General refactor load stac#1484
Draft
JeroenVerstraelen wants to merge 9 commits into
Draft
Conversation
The original from_date, to_date calculation was implemented for this issue: #609 However, these dates are rarely used inside the JVM. They are only used to query the OpenSearch client and sometimes to set the layer metadata. The FixedOpenSearchClient ignores the dateRange parameter completely so only the temporal extent of the layer metadata is relevant to us. We just need to ensure our GeopysparkCubeMetadata is correct and we can pass its temporal_extent to scala. The only source of truth for the GeopysparkCubeMetadata should be the temporal extent of the STAC items we extracted. If for some reason these do not give us a temporal extent then we either have an empty datacube or faulty items. In both cases it should be okay to fallback to the user requested temporal extent. However I feel like we should handle 'faulty' features a bit more explicitly. Since for calculating the temporal extent of an ItemCollection we use `item.properties.get("start_datetime")` and `"end_datetime` but when constructing the OpenSearch Features we send to the JVM we use `itm.properties.get("datetime")`. So what happens when the STAC items have a datetime but no start and end datetime, then we construct a RasterCube with valid dates but our python metadata is out of sync.
Copy original load_stac.py to load_stac_legacy.py to enable golden master testing during refactor. This file will be used as the reference implementation to ensure behavioral equivalence.
Split load_stac.py into focused modules: - stac/exceptions.py: NoDataAvailableException, LoadStacException - stac/fixed_features_open_search_client.py: OpenSearch client handling - stac/item_collection.py: ItemCollection, PropertyFilter, ItemDeduplicator, and related STAC item processing This modularization improves code organization and maintainability while preserving all original functionality.
Major changes: - Import STAC classes from new stac/ module instead of defining locally - Simplify load_stac.py by ~800 lines through extraction - Maintain identical public API and behavior - All original functionality preserved, just reorganized This refactor makes the codebase more maintainable and testable while ensuring no behavioral changes to the load_stac process.
JeroenVerstraelen
marked this pull request as draft
January 6, 2026 14:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.