Skip to content

Conversation

@isegall-da
Copy link
Contributor

@isegall-da isegall-da commented Jan 14, 2026

Will be useful for streaming ACS snapshots to S3

Pull Request Checklist

Cluster Testing

  • If a cluster test is required, comment /cluster_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
  • If a hard-migration test is required (from the latest release), comment /hdm_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.

PR Guidelines

  • Include any change that might be observable by our partners or affect their deployment in the release notes.
  • Specify fixed issues with Fixes #n, and mention issues worked on using #n
  • Include a screenshot for frontend-related PRs - see README or use your favorite screenshot tool

Merge Guidelines

  • Make the git commit message look sensible when squash-merging on GitHub (most likely: just copy your PR description).

Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
Signed-off-by: Itai Segall <[email protected]>
@isegall-da isegall-da changed the title WIP: stream ACS snapshot dumps (or maybe this PR will end up being only "support querying next snapshot") support querying next snapshot after a timestamp Jan 15, 2026
@isegall-da isegall-da marked this pull request as ready for review January 15, 2026 17:45
Signed-off-by: Itai Segall <[email protected]>
Copy link
Contributor

@rautenrieth-da rautenrieth-da left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be useful for streaming ACS snapshots to S3

Code looks fine, but why are we adding a new HTTP endpoint? Is the process that uploads to S3 going to implemented outside of scan? If it will live inside scan, then we don't need a new endpoint, no?

type: integer
format: int64
description: |
The endpoint will return the record time of the first snapshot for this migration id or larger.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is different from the /v0/state/acs/snapshot-timestamp endpoint, which only looks at snapshots from exactly the given migration id.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH, I think that the semantics in the existing one is less useful. People don't really care about migration IDs, it's almost an implementation detail. They just want the previous/next snapshot from a specific timestamp.

and history_id = $historyId """ ++ orderLimit

val query =
sql"select * from ((" ++ sameMig ++ sql") union all (" ++ largerMig ++ sql")) all_queries order by snapshot_record_time asc limit 1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit complicated for a table with only thousands of rows, but should make optimal use of the available primary key index.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, that was my thinking exactly

@isegall-da
Copy link
Contributor Author

why are we adding a new HTTP endpoint?

Mostly so that I can call it in the integration test TBH.
But I'd say "why not"? Do we have any reason to believe that people want to query "before" more than they want "after"?

Signed-off-by: Itai Segall <[email protected]>
@isegall-da
Copy link
Contributor Author

why are we adding a new HTTP endpoint?

Mostly so that I can call it in the integration test TBH. But I'd say "why not"? Do we have any reason to believe that people want to query "before" more than they want "after"?

@rautenrieth-da @ray-roestenburg-da I'm going to merge this as-is. Please shout if you think that having this endpoint is a mistake, we have until the next release to remove it easily if so.

@isegall-da isegall-da enabled auto-merge (squash) January 16, 2026 15:43
@isegall-da isegall-da merged commit e040089 into main Jan 16, 2026
44 checks passed
@isegall-da isegall-da deleted the isegall/acs-stream branch January 16, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants