Skip to content

Commit c07cc68

Browse files
committed
storage/pg: only report snapshot size error in strict mode
Our default snapshot size determination strategy is to use the table statistics in pg instead of performing a `SELECT COUNT(*)` on the table. This saves resources from the upstream database since it only reads a couple of rows from the statistics table at the expense of accuracy. This PR silences an error that only makes sense when the `COUNT(*)` strategy is used. Signed-off-by: Petros Angelatos <[email protected]>
1 parent 2692f35 commit c07cc68

File tree

1 file changed

+6
-5
lines changed

1 file changed

+6
-5
lines changed

src/storage/src/source/postgres/snapshot.rs

+6-5
Original file line numberDiff line numberDiff line change
@@ -516,16 +516,17 @@ pub(crate) fn render<G: Scope<Timestamp = MzOffset>>(
516516
}
517517
*rewind_cap_set = CapabilitySet::new();
518518

519-
if snapshot_staged < snapshot_total {
520-
error!(%id, "timely-{worker_id} snapshot size {snapshot_total} is somehow
521-
bigger than records staged {snapshot_staged}");
522-
snapshot_staged = snapshot_total;
519+
let strict_count = config.config.parameters.pg_snapshot_config.collect_strict_count;
520+
if strict_count && snapshot_staged != snapshot_total {
521+
panic!("timely-{worker_id} size mimatch for snapshot of source {id}. \
522+
expected: {snapshot_total} actual: {snapshot_staged}");
523523
}
524+
// Report the same known and staged records to signify that the snapshot is complete.
524525
stats_output.give(
525526
&stats_cap[0],
526527
ProgressStatisticsUpdate::Snapshot {
527528
records_known: snapshot_total,
528-
records_staged: snapshot_staged,
529+
records_staged: snapshot_total,
529530
},
530531
);
531532

0 commit comments

Comments
 (0)