Skip to content

feat: idempotent ingestion (skip unchanged, upsert reprocessed, report failures) - #3

Merged
betolink merged 3 commits into
mainfrom
feat/idempotent-ingest
Nov 16, 2026
Merged

betolink merged 3 commits into
mainfrom
feat/idempotent-ingest

Conversation

@betolink

@betolink betolink commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

What

request() becomes idempotent by default for CMR granules, and reports what
happened.

A virtual store is append-only, so re-running a batch duplicates its time steps
and a reprocessed granule cannot replace its predecessor. This PR keys each
granule by its concat-coordinate value (derived from UMM metadata, no open
required), records the ingested keys in the repository metadata, and classifies
the batch against the store:

  • unchanged (same key, same CMR revision-id; href fallback) → skipped,
    zero writes
    ;
  • reprocessed (same key, new revision) → region-written in place
    (to_icechunk(region=…)), so no duplicate time step;
  • new → appended;
  • unkeyed (no derivable time) → appended.

One session, one commit. The ledger lives in the repo metadata
(earthaccess:vds/ingested) and is merged, never replaced.

job.info()["summary"]  # {'appended': 28, 'upserted': 3, 'skipped': 0, 'unkeyed': 0}

Options

option default meaning
dedupe "upsert" skip unchanged, overwrite reprocessed, append new
"skip" never overwrite
"off" old append-only behaviour
key concat_dim coordinate identifying a slot (only time is derivable from CMR metadata)
on_error "raise" "skip": record a failed granule, ingest the rest, end PARTIAL
verify "metadata" "etag": one HTTP HEAD per granule, records/compares the origin ETag

Stores written without a ledger degrade gracefully to overwrite/append.

Failures

  • ProgressState keeps a bounded errors list (granule, phase, error)
    and the summary; job.info() exposes both.
  • on_error="skip" falls back to per-granule opens, records failures, and
    VirtualJob reports PARTIAL.
  • pandera SchemaErrors from preprocess= land in the same report.

Tests

  • tests/test_idempotency.py (13): rerun is a no-op; reprocessed → in-place
    upsert (size unchanged, ref updated); new → append; mixed batch; skip/off;
    ledger persisted; missing store.
  • tests/test_on_error.py (3): skip → PARTIAL + errors + survivors;
    raise → FAILED; rejected with tree=True.
  • Full suite 92 passed, mypy clean, ruff clean, pre-commit clean.

Docs

README: 3 option rows + a short idempotency section. The long-form walkthrough
(batching, pandera.xarray, SMAP preprocess=) is a docs follow-up.

Limitations (by design)

  • One granule per key. A batch with two granules sharing a key would map
    both to the same store index and silently overwrite one with the other; once
    the store has state, such a batch is rejected with guidance (key=… to pick
    the slot, or dedupe="off").
  • Icechunk only. Kerchunk has no region writes, so dedupe is ignored there
    (append is a read-combine-rewrite).
  • The ledger lives in repository metadata and grows with the key count
    (~100 B/key; 3.6k keys for 10 years of daily MUR).
  • Each request opens the store once to read the concat coordinate (a small,
    concrete array).
  • on_error="skip" falls back to serial per-granule opens on failure.

Not covered by CI

  • Real S3 store writes (CI has no credentials) — the store tests use local
    icechunk repositories.
  • Real CMR metadata variety (synthetic granules only) and concurrent commits
    from several writers.

verify="etag" — catching silent file replacement

revision-id is CMR's metadata revision. If a DAAC replaces a file in place
without updating CMR, neither the revision nor ProductionDateTime changes.
With verify="etag" the writer probes the origin:

  • one HEAD per granule through the EDL session, allow_redirects=True
    (PODAAC answers 303 to CloudFront/S3; the final response carries the
    origin ETag — verified: "42f39d…-3");
  • the ETag is not a content hash (S3 multipart ETags carry a -N suffix,
    CloudFront passes the origin value through) but it changes when the object
    changes, which is what the ledger needs;
  • probes run only for granules the metadata considers unchanged; a differing
    ETag promotes the key to an in-place upsert, a missing baseline records one,
    and a failed probe is counted as unverified (the key is left untouched);
  • probe-only runs persist the baseline without rewriting references;
  • probes are HTTPS-only today (S3-only granules report unverified), and cost
    one HEAD per granule — hence opt-in.

…t failures)

request() now makes a store append-only *and* idempotent for CMR granules:

- dedupe='upsert' (default): keys are the concat-coordinate values derived from
  UMM metadata; the ledger of ingested keys lives in the repository metadata
  (earthaccess:vds/ingested). Unchanged granules are skipped, a reprocessed one
  (CMR revision-id change, href fallback) is region-written in place, and only
  new keys are appended.
- dedupe='skip' never overwrites; dedupe='off' restores the old append-only
  behaviour. Stores written without a ledger degrade to overwrite/append.
- on_error='skip' falls back to per-granule opens, records the failures, ingests
  the survivors and ends the job PARTIAL.
- ProgressState keeps a bounded error list and a summary; job.info() exposes
  summary/errors, and VirtualJob reports PARTIAL when any granule failed.
- writer: region upserts + ledger merge; engine: lenient open path.
- README: options table + idempotency section.

Tests: tests/test_idempotency.py (13), tests/test_on_error.py (3).
- A batch with two granules sharing a key would map both to the same store
  index and silently overwrite one with the other. Once the store has state,
  plan_batch now raises with guidance (pass key= or dedupe='off').
- When the concat coordinate cannot be read but the ledger knows the key, skip
  it instead of appending a duplicate.
- Tests: duplicate-key rejection on re-run; ledger-without-index skip.
…ication

- fingerprint adds umm.DataGranule.ProductionDateTime, so a reprocessed granule
  is detected even when CMR keeps the same revision-id (where the product
  carries it)
- verify='etag' (opt-in): one HTTP HEAD per granule through the EDL session,
  following the DAAC's 303 redirect, records/compares the origin ETag. This
  catches a file replaced in place without a CMR update — the only case the
  metadata cannot see. Probe failures count as 'unverified' and leave the key
  untouched.
- probe-only runs persist the ETag baseline without rewriting references
  (write_ingested_only).
- README: verify row. Tests: production fingerprint, apply_etags promotion /
  baseline / unverified, the probe helper (redirects, errors, HTTPS-only), and
  end-to-end replacement / unverified / no-probe paths.
@betolink
betolink merged commit ba4d839 into main Nov 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant