Add support for starting/restarting without state in leader mode - #1122
Merged
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the second attempt at #1111, which accidentally merged the wrong version of this code. This PR reverts that change, and adds the intended version.
The existing implementation of skipping state for controller mode relies on querying the most recent epoch for the job from the checkpoints table, and then setting that as the "ignore_before_epoch" field in the config. This approach is awkward for leader mode though, as we don't have a checkpoints table, and determining the latest epoch involves going to object storage.
There are several principled approaches we could take here—for example, introducing a "state version" field, which could be bumped to start a new state lineage for the pipeline, which would require a number of changes throughout the controller, worker, and state system. However, long-term this will be solved by a more sophisticated system of pipelines and jobs, where we create a new job for a new state lineage.
While that redesign remains in the future, this PR implements the same idea in a somewhat hacky way: when restarting a leader-mode pipeline without state, we simply replace the existing job with a new one. As checkpoints are stored by job_id, this addresses the need here without any worker or state specific code. There is some risk of introducing multiple running jobs as there is no explicit synchronization with the controller, however we mitigate by restricting these operations to pipelines in Failed or Stopped.
This PR also adds a "start without state" option and button the UI, so this is not just limited to failed pipelines.