Skip to content

Remove stage delegation in favor of planning-time stage assignation - #71

Merged
gabotechs merged 1 commit into
mainfrom
remove-stage-delegation
Aug 4, 2025
Merged

Remove stage delegation in favor of planning-time stage assignation#71
gabotechs merged 1 commit into
mainfrom
remove-stage-delegation

Conversation

@gabotechs

@gabotechs gabotechs commented Aug 4, 2025

Copy link
Copy Markdown
Collaborator

In preparation of the upcoming work, removes the stage delegation mechanism in favor of something simpler: assign the StageContexts to the ArrowFlightReadExec nodes as an extra planning step instead of dynamically during the query.

This PR ships several simplifications:

  • Removes the stage delegation mechanism in favor of leaving an empty slot in the ArrowFlightReadExec that an extra assing_stages planning task will fill. All the StageContexts and the node urls are now known at planning-time.
  • Removes the delegate-actor terminology in favor of just tasks
  • Updates the physical plan assertions in the tests to also account for the new information known during planning time.

@gabotechs
gabotechs marked this pull request as ready for review August 4, 2025 06:32
@gabotechs
gabotechs force-pushed the remove-stage-delegation branch from 6c7b21f to 57216d9 Compare August 4, 2025 07:04
@robtandy

robtandy commented Aug 4, 2025

Copy link
Copy Markdown
Collaborator

This looks like the right step before the ExecutionStage work comes in a subsequent PR

Base automatically changed from add-error-serde to main August 4, 2025 13:16
@gabotechs
gabotechs force-pushed the remove-stage-delegation branch from 57216d9 to 7936ba9 Compare August 4, 2025 13:23
@gabotechs
gabotechs merged commit a25907e into main Aug 4, 2025
3 checks passed
@gabotechs
gabotechs deleted the remove-stage-delegation branch August 4, 2025 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants