SubstreamPartitionRouter does not propagate the substream's incremental cursor to the parent's re-execution — is this expected?

[Andreas Skielboe (askielboe)]

CDK version: declarative-manifest 7.18.1 (Airbyte Cloud)

Setup

I have a two-stream declarative source:

• sales (parent) — calls GET /api/sales?since=YYYY-MM-DD, returns a flat JSON array. Has incremental_sync with DatetimeBasedCursor on the started field, 30-day lookback_window, and start_time_option injecting since as a request parameter.
• sales_details (substream) — calls GET /api/sales/{id} for each ID from the parent via SubstreamPartitionRouter. Returns a single JSON object per sale with the same started field.

The API has no pagination — since is the only filtering mechanism.

Observed behavior

The SubstreamPartitionRouter re-executes the parent stream definition independently to generate partition IDs. This re-execution always starts from config.start_date, regardless of whether the substream has its own incremental_sync with persisted cursor state. I've confirmed this by:

1. Running with sales_details as Full Refresh (no incremental_sync) — all records re-fetched every sync, generation increments each run.
2. Adding incremental_sync to sales_details (same cursor config as parent, without start_time_option since the detail endpoint has no since parameter) — same result, all records re-fetched.
3. Enabling the sales parent as a synced stream alongside sales_details — the parent's own pipeline correctly advances its cursor, but the substream's re-execution of the parent ignores this and starts from config.start_date.

In all cases, the substream fetches details for every historical sale on every sync. For my use case (~100 tenants, each with hundreds of sales), this means thousands of unnecessary API calls per sync run.

Question

Is this the intended behavior? If so, is there a supported pattern for making the parent's re-execution within SubstreamPartitionRouter aware of either the parent's or substream's persisted cursor state?

[devin-ai-integration[bot]]

Hi Andreas Skielboe (@askielboe), thank you for the very detailed write-up — this is exactly the right question to ask, and there's a supported way to wire it up.

Short answer

What you're observing is the default behavior: when the parent stream is re-executed inside SubstreamPartitionRouter to generate partitions, by default it does not carry over any persisted cursor state, so it starts from config.start_date every sync. To make the parent re-execution use the persisted parent-cursor state across syncs, set incremental_dependency: true on the parent stream config inside the substream's partition router.

Example

sales_details:
  type: DeclarativeStream
  retriever:
    type: SimpleRetriever
    requester:
      path: "/api/sales/{{ stream_slice.sale_id }}"
      # ...
    partition_router:
      type: SubstreamPartitionRouter
      parent_stream_configs:
        - stream: "#/definitions/streams/sales"
          parent_key: id
          partition_field: sale_id
          incremental_dependency: true   # <-- the key flag
  incremental_sync:
    type: DatetimeBasedCursor
    cursor_field: started
    # ... (no start_time_option needed since /api/sales/{id} has no `since`)

With incremental_dependency: true:

• The parent (sales) re-execution inside the router uses the persisted parent-cursor state, so on subsequent syncs it requests ?since=<last_high_watermark> instead of ?since=<config.start_date>.
• The substream's per-partition state is also tracked, so individual sales/{id} calls are skipped when the substream cursor has already advanced past them.
• The parent state is persisted as part of the substream's state under a parent_state key (you'll see it in the emitted state messages), so you can verify it's working by inspecting state output from a sync run.

Important prerequisite

This pattern only behaves correctly if the parent record's cursor field advances whenever any of its children change. From the docs:

The API must ensure that the parent record's cursor is updated whenever child records are added or updated. If this requirement is not met, child records added to older parent records will be lost.

In your case, both sales and sales_details use started as the cursor field. If started is the sale's creation/start timestamp and never changes after the sale begins, then a sales_details record can't be modified later in a way that would be missed — so incremental_dependency is safe. If, however, sales_details can be updated after the parent's started timestamp (e.g., line items added later), you would either need to (a) use a different cursor field that tracks "last modified" on the parent, or (b) keep your existing 30-day lookback_window on the parent, which the router will respect.

If you don't want to track parent state

If using incremental_dependency doesn't fit your use case, the alternatives are:

• global_substream_cursor: true — uses a single global cursor for the substream, which prunes which sales_details requests need to be made on subsequent syncs based on the substream's own state. Trade-off: less granularity, only suitable when partitions can't fall behind each other.
• Default (per-partition with global fallback) — what you're already using. The substream's own incremental state already prevents child records from being re-emitted, but it cannot prevent the parent re-execution from listing all parent IDs unless incremental_dependency is set.

References

• Partition Router → SubstreamPartitionRouter
• Incremental syncs → Nested Streams (Incremental Dependency / Global Substream Cursor)

If you give incremental_dependency: true a try and still see the parent re-fetching from config.start_date after the first successful sync, please reply with a state snippet from your last sync (with any sensitive values redacted) and we'll take a closer look.

---

Need more help? Join the Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this discussion URL.

---

Devin session

[Andreas Skielboe (askielboe)]

Thanks, will give it a try.

[Andreas Skielboe (askielboe)]

That seemed to work! Thank you Devin.. :)