Repository navigation
feat: idempotent ingestion (skip unchanged, upsert reprocessed, report failures) - #3
Merged
Merged
Conversation
…t failures) request() now makes a store append-only *and* idempotent for CMR granules: - dedupe='upsert' (default): keys are the concat-coordinate values derived from UMM metadata; the ledger of ingested keys lives in the repository metadata (earthaccess:vds/ingested). Unchanged granules are skipped, a reprocessed one (CMR revision-id change, href fallback) is region-written in place, and only new keys are appended. - dedupe='skip' never overwrites; dedupe='off' restores the old append-only behaviour. Stores written without a ledger degrade to overwrite/append. - on_error='skip' falls back to per-granule opens, records the failures, ingests the survivors and ends the job PARTIAL. - ProgressState keeps a bounded error list and a summary; job.info() exposes summary/errors, and VirtualJob reports PARTIAL when any granule failed. - writer: region upserts + ledger merge; engine: lenient open path. - README: options table + idempotency section. Tests: tests/test_idempotency.py (13), tests/test_on_error.py (3).
- A batch with two granules sharing a key would map both to the same store index and silently overwrite one with the other. Once the store has state, plan_batch now raises with guidance (pass key= or dedupe='off'). - When the concat coordinate cannot be read but the ledger knows the key, skip it instead of appending a duplicate. - Tests: duplicate-key rejection on re-run; ledger-without-index skip.
…ication - fingerprint adds umm.DataGranule.ProductionDateTime, so a reprocessed granule is detected even when CMR keeps the same revision-id (where the product carries it) - verify='etag' (opt-in): one HTTP HEAD per granule through the EDL session, following the DAAC's 303 redirect, records/compares the origin ETag. This catches a file replaced in place without a CMR update — the only case the metadata cannot see. Probe failures count as 'unverified' and leave the key untouched. - probe-only runs persist the ETag baseline without rewriting references (write_ingested_only). - README: verify row. Tests: production fingerprint, apply_etags promotion / baseline / unverified, the probe helper (redirects, errors, HTTPS-only), and end-to-end replacement / unverified / no-probe paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
request()becomes idempotent by default for CMR granules, and reports whathappened.
A virtual store is append-only, so re-running a batch duplicates its time steps
and a reprocessed granule cannot replace its predecessor. This PR keys each
granule by its concat-coordinate value (derived from UMM metadata, no open
required), records the ingested keys in the repository metadata, and classifies
the batch against the store:
revision-id; href fallback) → skipped,zero writes;
(
to_icechunk(region=…)), so no duplicate time step;One session, one commit. The ledger lives in the repo metadata
(
earthaccess:vds/ingested) and is merged, never replaced.Options
dedupe"upsert""skip""off"keyconcat_dimtimeis derivable from CMR metadata)on_error"raise""skip": record a failed granule, ingest the rest, endPARTIALverify"metadata""etag": one HTTP HEAD per granule, records/compares the origin ETagStores written without a ledger degrade gracefully to overwrite/append.
Failures
ProgressStatekeeps a boundederrorslist (granule,phase,error)and the summary;
job.info()exposes both.on_error="skip"falls back to per-granule opens, records failures, andVirtualJobreportsPARTIAL.panderaSchemaErrors frompreprocess=land in the same report.Tests
tests/test_idempotency.py(13): rerun is a no-op; reprocessed → in-placeupsert (size unchanged, ref updated); new → append; mixed batch;
skip/off;ledger persisted; missing store.
tests/test_on_error.py(3):skip→PARTIAL+ errors + survivors;raise→FAILED; rejected withtree=True.Docs
README: 3 option rows + a short idempotency section. The long-form walkthrough
(batching,
pandera.xarray, SMAPpreprocess=) is a docs follow-up.Limitations (by design)
both to the same store index and silently overwrite one with the other; once
the store has state, such a batch is rejected with guidance (
key=…to pickthe slot, or
dedupe="off").dedupeis ignored there(append is a read-combine-rewrite).
(~100 B/key; 3.6k keys for 10 years of daily MUR).
concrete array).
on_error="skip"falls back to serial per-granule opens on failure.Not covered by CI
icechunk repositories.
from several writers.
verify="etag"— catching silent file replacementrevision-idis CMR's metadata revision. If a DAAC replaces a file in placewithout updating CMR, neither the revision nor
ProductionDateTimechanges.With
verify="etag"the writer probes the origin:HEADper granule through the EDL session,allow_redirects=True(PODAAC answers
303to CloudFront/S3; the final response carries theorigin ETag — verified:
"42f39d…-3");-Nsuffix,CloudFront passes the origin value through) but it changes when the object
changes, which is what the ledger needs;
ETag promotes the key to an in-place upsert, a missing baseline records one,
and a failed probe is counted as
unverified(the key is left untouched);unverified), and costone HEAD per granule — hence opt-in.