feat: Phase 3d+3e — ParquetMergePipeline supervisor + publisher feedback#6354
Open
g-talbot wants to merge 2 commits intogtt/parquet-merge-pipeline-3cfrom
Open
feat: Phase 3d+3e — ParquetMergePipeline supervisor + publisher feedback#6354g-talbot wants to merge 2 commits intogtt/parquet-merge-pipeline-3cfrom
g-talbot wants to merge 2 commits intogtt/parquet-merge-pipeline-3cfrom
Conversation
4 tasks
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline
branch
from
ceba410 to
e96a920
Compare
g-talbot
changed the base branch from
gtt/parquet-merge-pipeline
to
gtt/parquet-merge-pipeline-3c
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
ceba410 to
5937440
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3de
branch
from
d82d72d to
92cb5ed
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
5937440 to
0b1c9cc
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3de
branch
from
92cb5ed to
b6f8bcc
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
0b1c9cc to
66b97e0
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3de
branch
from
b6f8bcc to
d296774
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
66b97e0 to
0f051bc
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3de
branch
from
d296774 to
ededb89
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
0f051bc to
16b46d7
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3de
branch
from
ededb89 to
761b379
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
16b46d7 to
8e19b6b
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3de
branch
from
761b379 to
9b3769e
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
8e19b6b to
f32bd64
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3de
branch
from
9b3769e to
927dc9f
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
f32bd64 to
de17c0e
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3de
branch
from
927dc9f to
c51a84b
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
de17c0e to
1f6512e
Compare
…se 3d+3e) Phase 3 pipeline integration, combined supervisor and feedback PR: - ParquetMergePipeline supervisor: spawns all merge actors (publisher → sequencer → uploader → executor → downloader → planner), health-checks with periodic supervision loop, respawn on failure with backoff, graceful shutdown via FinishPendingMergesAndShutdownPipeline that disconnects feedback and runs finalize policy. 3 tests. - Publisher feedback: add parquet_merge_planner_mailbox_opt to Publisher (feature-gated behind cfg(feature = "metrics")). After successful ParquetSplitsUpdate publish of new ingested splits, sends ParquetNewSplits to the planner. Merge outputs (non-empty replaced_split_ids) are not fed back to avoid infinite loops. - DisconnectMergePlanner extended to clear both Tantivy and Parquet planner mailboxes, supporting shutdown drain for both pipeline types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3de
branch
from
c51a84b to
804e384
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #6358 (Phase 3c). Adds the supervisor and feedback loop that make the merge pipeline runnable.
FinishPendingMergesAndShutdownPipelinethat disconnects feedback and runs finalize policy for cold windows.parquet_merge_planner_mailbox_opttoPublisher(feature-gatedcfg(feature = "metrics")). After successfulParquetSplitsUpdatepublish of new ingested splits, sendsParquetNewSplitsto the planner. Merge outputs are not fed back (guards against infinite loops).Test plan
metricsfeaturecargo clippyclean, license headers OK🤖 Generated with Claude Code