feat(consumer): Add new metadata, refactor #66

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

devsjc merged 22 commits into main from satellite_metadata

Dec 22, 2025

Contributor

devsjc commented Dec 12, 2025 •

edited

Loading

Adds new data to the output datasets

Removes the ability to rescale the data
Updates the configuration method (no more CLI entrypoint)
Refactors the service somewhat
Enables sharding

There are a fair few changes in here - I would recommend going through one commit at a time for ease of following them.

devsjc requested a review from dfulu

December 12, 2025 16:48

devsjc force-pushed the satellite_metadata branch from 7865b8e to 31ad8d3 Compare

December 14, 2025 12:12

dfulu and others added 4 commits

December 14, 2025 12:13


          feat(storage): Add new data vars and metadata

9beb789


          chore(repo): Refactor and simplify service

63b54e0


          feat(service): Enable icechunk

0f2ba51


          feat(process): Enable sharding

5102edd

devsjc force-pushed the satellite_metadata branch from 31ad8d3 to 5102edd Compare

December 14, 2025 12:14

dfulu reviewed

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved


          fix(repo): Correctly use importlib

afd40f3

dfulu reviewed

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

dfulu reviewed

View reviewed changes

src/satellite_consumer/cmd/main.py Outdated Show resolved Hide resolved

dfulu reviewed

View reviewed changes

src/satellite_consumer/download_eumetsat.py Outdated Show resolved Hide resolved

dfulu reviewed

View reviewed changes

src/satellite_consumer/download_eumetsat.py Show resolved Hide resolved


          feat(download): Save raw files in subfolders

485a4df

dfulu reviewed

View reviewed changes

src/satellite_consumer/process.py Outdated Show resolved Hide resolved

dfulu reviewed

View reviewed changes

src/satellite_consumer/process.py Outdated Show resolved Hide resolved

dfulu reviewed

View reviewed changes

src/satellite_consumer/process.py Outdated Show resolved Hide resolved

devsjc and others added 3 commits

December 15, 2025 10:46


          chore(download): Address PR comments

0ac74a1


          chore(process): Specify earth disk

d090601

Co-authored-by: James Fulton <[email protected]>


          feat(write): Properly check for existing times

102dc2d

devsjc force-pushed the satellite_metadata branch from d02c13a to 102dc2d Compare

December 15, 2025 12:50

dfulu and others added 5 commits

December 16, 2025 10:31


          fix(process): Efficient satpy usage (#67)

bdec6dd

* Refactor satpy usage for speed

* lint

* Add attrs


          Speed up nan-frac calculation (#68)

abbba0e

* Speed up nan-frac calculation

* linting

* enforce type


          chore(process): Speed up coordinate reordering (#69)

8c3ecc0

speed up spatial coord sorting


          chore(pyproject): Satpy pin

47a526a


          chore(process): More NaN leeway

8d5c5a4

devsjc commented

View reviewed changes

src/satellite_consumer/storage.py Show resolved Hide resolved

dfulu mentioned this pull request

Determine cause of satellite coordinate shift #62

Closed

dfulu reviewed

View reviewed changes

src/satellite_consumer/process.py Outdated Show resolved Hide resolved


          chore(consume): Don't break on skipped low-quality files

1896f6d

dfulu and others added 7 commits

December 18, 2025 07:45


          chore(process): Refactor geos bound calculation (#70)

2fedeaf

* Use area attr to calculate geos bounds

* linting

* igore type

* chore(repo): Linting

---------

Co-authored-by: devsjc <[email protected]>


          fix(consume): Don't break on validation error

326c33d


          fix(process): Allow ValidationError through

05bea39


          Add projection coordinates (#72)

5f2be9a


          fix(process): Save instrument as fixed length string

98679c5


          chore(process): Increase NaN allowable threshold

75926ed


          chore(process): Allow 20% NaNs on earth disk

f53e236

devsjc merged commit 159f5e9 into main

3 checks passed

devsjc deleted the satellite_metadata branch

December 22, 2025 13:04

dfulu mentioned this pull request

Store area/lon_0 parameter as a coordinate #28

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet