Skip to content

Parallelize collection building and add nested progress bars for overpass fetching - #78

Open
ehavazli wants to merge 2 commits into
mainfrom
speedup
Open

Parallelize collection building and add nested progress bars for overpass fetching#78
ehavazli wants to merge 2 commits into
mainfrom
speedup

Conversation

@ehavazli

@ehavazli ehavazli commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Speeds up satellite collection building by parallelizing independent network and
I/O work, and adds a nested progress display so long-running overpass fetches show
live feedback in the terminal.

Performance

  • Concurrent ESA plan scrapes — Sentinel-1 (2 platforms) and Sentinel-2 (3
    platforms) acquisition-plan URLs are now scraped in parallel via a shared
    _scrape_esa_plans helper. Spec order is preserved so downstream platform
    tagging is unaffected.
  • Concurrent KML downloads — missing KML files in sync_scratch_directory are
    downloaded on a thread pool instead of sequentially; cached files are untouched
    and result order is preserved (failed downloads skipped).
  • Concurrent KML parse/read — per-file parse/read in build_sentinel_collection
    runs on a thread pool. Results are concatenated and re-sorted by begin_date, so
    ordering is order-independent and each file writes a distinct geojson (no write
    collisions).

Cold-start collection build is roughly 3–5× faster. Tide (single batched NOAA call)
and cloudiness (already fans out internally with its own rate limiter) were
intentionally left unchanged to avoid triggering API rate limits.

Progress bars

  • New utils/progress.py: a master bar across all selected satellites plus one
    per-satellite sub bar whose label reflects the current step (spinner style, no
    fill %).
  • Each next_*_pass entry function (sentinel, landsat, nisar) takes an
    optional step_cb=None callback and reports coarse phase transitions (e.g.
    Loading collection → Finding intersects → Cloudiness → Tide → Formatting).
  • Non-interactive safe: the bar renders only to the real terminal
    (sys.__stdout__) and is disabled when stdout is not a TTY (cron, redirected
    output, the email pipeline). In that case the existing LOGGER.info step lines
    remain the feedback path — automated-run behavior is unchanged. ANSI control
    codes never reach run_output.txt.
  • While the bar is active, logging is routed through rich.logging.RichHandler so
    warnings/errors surface cleanly above the live region; the original log handlers
    are restored on exit.

Other

  • Silenced noisy third-party INFO chatter (pyogrio/fiona "Created N records").
  • Added rich to requirements.txt.

Testing

  • pytest — 102/102 pass (test mocks updated for the new step_cb kwarg and a
    debug logger method).
  • flake8 and black --check clean on all changed files.
  • Verified both the TTY-enabled render path (master + sub labels, log handlers
    restored) and the non-TTY no-op fallback.

@ehavazli
ehavazli requested review from cmspeed and ifenni August 4, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant