Consumer ingestion updates

Author: Amit Kothari (from Dev Box)

docs/consumer-ingestion-guide.md

@@ -53,7 +53,9 @@ This is the exact logical flow a consumer should understand before operating or
 5. **Path selection**:
   - non-partitioned: latest row by `RunId`
   - partitioned: latest `RunId` per `Partition` (snapshot entries)
-6. **Checkpoint gate**: skip ingestion if `(DatasetName, Partition, RunId)` already exists in watermark.
+6. **Checkpoint gate**: skip ingestion if the effective watermark key already exists:
+   - starter-kit style: `(RunId, Partition)` within the dataset-scoped run
+   - shared-store style: `(DatasetName, RunId, Partition)`
 7. **Load pattern**: ingest into staging table, validate, swap/rename to main table, create/update view.
 8. **Commit checkpoint**: watermark is saved only after successful load.
 
@@ -318,7 +320,7 @@ EXEC sp_rename '[dbo].[Staging_CDM_Products]', '[Main_CDM_Products]'
 
 **Checkpoint storage for SQL:**
 - Create a `Checkpoint` table in same SQL database
-- Schema: `(DataProduct, DatasetName, Partition, RunId, LoadedAt, Status, ErrorMsg)`
+- Schema: `(DataProduct, DatasetName, RunId, Partition, LoadedAt, Status, ErrorMsg)`
 
 #### Substitution pattern: Kusto → Snowflake
 
@@ -350,17 +352,17 @@ ALTER TABLE CDM_Products
 
 #### General rules for any substitution
 
-1. **Idempotency**: Your target writer must handle rerun of same `(DatasetName, Partition, RunId)` without duplication.
-2. **Checkpoint deduplication**: Use all three components `(DatasetName, RunId, Partition)` as the dedup key - use `Partition` column value directly, not parsed from paths.
+1. **Idempotency**: Your target writer must handle rerun of the same effective key without duplication:
+  - dataset-scoped style: `(RunId, Partition)`
+  - shared-store style: `(DatasetName, RunId, Partition)`
+2. **Checkpoint deduplication**: Use the effective key for your execution model:
+   - dataset-scoped style: `(RunId, Partition)`
+   - shared-store style: `(DatasetName, RunId, Partition)`
+   - in both cases, use the `Partition` column value directly, not parsed from paths
 3. **Staging pattern**: Use explicit staging layers to ensure atomicity of final write.
 4. **Validation**: Always validate (row count, checksum) before committing checkpoint.
-5. **Checkpoint storage**: Persist `(DataProductName, DatasetName, RunId, Partition, LoadedAt, Status)` in durable storage, using the exact `Partition` value from the index file.
-6. **Test reruns**: Verify that reruns do not increase row counts, and checkpoint prevents re-ingestion of the same `(DatasetName, RunId, Partition)` combination.
-
-### Enhancement path B: Add first-class Delta ingestion support
-
-Recommended approach:
-
+5. **Checkpoint storage**: Persist the effective dedup key plus status metadata in durable storage, using the exact `Partition` value from the index file.
+6. **Test reruns**: Verify that reruns do not increase row counts, and checkpoint prevents re-ingestion of the same effective key.
 1. Add explicit ingestion policy per dataset (`SnapshotOnly`, `DeltaOnly`, `SnapshotPlusDelta`).
 2. For `SnapshotPlusDelta`, process latest snapshot baseline then apply unseen deltas in ordered `RunId`.
 3. Extend checkpoint schema to include `IngestionMode` and optional sequence ordering metadata.