Few updates to S2 conversion [NOT TE BE MERGED] #76
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rces and adjusted parameters
…culate_shard_dimension utility
…utility functions
- Introduced a new dataset tree structure for Sentinel-2 data, detailing conditions, quality, and measurements. - Added a comprehensive test script to verify the sharding fix for GeoZarr conversion. - Implemented tests for shard dimension calculations and encoding creation with sharding enabled/disabled. - Enhanced output for better debugging and validation of shard dimensions against chunk dimensions.
…c for sharded variables
…essing - Created the `s2_optimization` module for optimizing Sentinel-2 Zarr datasets. - Implemented CLI commands for converting Sentinel-2 datasets to optimized structures. - Developed band mapping and resolution definitions for Sentinel-2 optimization. - Added the `S2OptimizedConverter` class for handling the conversion process. - Implemented data consolidation logic to reorganize Sentinel-2 structure. - Created multiscale pyramid generation for optimized data. - Added downsampling operations for various data types (reflectance, classification, quality masks). - Implemented validation logic for optimized Sentinel-2 datasets. - Developed unit tests for band mapping, converter functionality, and resampling operations.
…ods and testing framework
- Implement unit tests for initialization, pyramid levels structure, chunk alignment, and shard dimension calculations. - Create tests for encoding generation, dataset writing, and level dataset creation with various resolutions. - Include integration tests for realistic measurements data and edge cases handling. - Ensure coverage for time separation logic and coordinate preservation during processing.
…ding and rechunking
…sions in S2MultiscalePyramid
…hance progress tracking for Zarr writes
…lization and improved memory management
… improved parallel processing
…n S2OptimizedConverter
…s for multiscale pyramid creation
…e bands and quality data
…metadata handling in S2 multiscale pyramid creation
…ary datatree loading and streamline dataset writing return
… remove unused downsampling operations
…dling in S2 converter
… multiscale classes
…ee initialization in S2 converter and multiscale classes
…issing intermediary groups in Zarr structure
…2OptimizedConverter
…mprove test for geo metadata integration with level creation
…nd chunk dimensions
…st script - Deleted the comprehensive optimization plan for the Sentinel-2 Zarr conversion, which included details on the current state, proposed structure, technical specifications, implementation plan, and expected benefits. - Removed the test script for verifying the sharding fix in GeoZarr conversion, which included tests for shard dimensions and encoding creation.
- Removed --skip-geometry and --skip-meteorology options from CLI as they are no longer needed. - Updated S2OptimizedConverter to initialize and use CRS for datasets, improving metadata handling. - Modified S2MultiscalePyramid to accept CRS and ensure geographic metadata is written correctly. - Improved logging messages for clarity and consistency throughout the conversion process.
emmanuelmathot
changed the title
emmanuelmathot
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement sharding support for GeoZarr conversion, enhancing the CLI with new commands for optimized Sentinel-2 datasets. Improve multiscale pyramid creation, downsampling methods, and metadata handling while ensuring compatibility with chunk dimensions. Refactor code for better readability and maintainability, and add comprehensive tests to validate the new features.