Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speeds up satellite collection building by parallelizing independent network and
I/O work, and adds a nested progress display so long-running overpass fetches show
live feedback in the terminal.
Performance
platforms) acquisition-plan URLs are now scraped in parallel via a shared
_scrape_esa_planshelper. Spec order is preserved so downstream platformtagging is unaffected.
sync_scratch_directoryaredownloaded on a thread pool instead of sequentially; cached files are untouched
and result order is preserved (failed downloads skipped).
build_sentinel_collectionruns on a thread pool. Results are concatenated and re-sorted by
begin_date, soordering is order-independent and each file writes a distinct geojson (no write
collisions).
Cold-start collection build is roughly 3–5× faster. Tide (single batched NOAA call)
and cloudiness (already fans out internally with its own rate limiter) were
intentionally left unchanged to avoid triggering API rate limits.
Progress bars
utils/progress.py: a master bar across all selected satellites plus oneper-satellite sub bar whose label reflects the current step (spinner style, no
fill %).
next_*_passentry function (sentinel,landsat,nisar) takes anoptional
step_cb=Nonecallback and reports coarse phase transitions (e.g.Loading collection → Finding intersects → Cloudiness → Tide → Formatting).
(
sys.__stdout__) and is disabled when stdout is not a TTY (cron, redirectedoutput, the email pipeline). In that case the existing
LOGGER.infostep linesremain the feedback path — automated-run behavior is unchanged. ANSI control
codes never reach
run_output.txt.rich.logging.RichHandlersowarnings/errors surface cleanly above the live region; the original log handlers
are restored on exit.
Other
pyogrio/fiona"Created N records").richtorequirements.txt.Testing
pytest— 102/102 pass (test mocks updated for the newstep_cbkwarg and adebuglogger method).flake8andblack --checkclean on all changed files.restored) and the non-TTY no-op fallback.