omdb: add facility for abandoning a saga #7791
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Breaking apart #4378 and copying the structure of #7695, add `omdb db saga` as a command and implement the following sub-commands: Usage: omdb db saga [OPTIONS] <COMMAND> Commands: running List running sagas fault Inject an error into a saga's currently running node This addresses part of the minimum amount required during a release deployment: 1. after quiescing (#6804), omdb can query if there are any running sagas. 2. if those running sagas are stuck in a loop and cannot be drained (#7623), and the release contains a change to the DAG that causes Nexus to panic after an upgrade (#7730), then omdb can inject a fault into the database that would cause that saga to unwind when the affected Nexus is restarted Note for 2, unwinding a saga that is stuck in this way may not be valid if there were significant changes between releases.
- change fault to inject-error - show the current sec for a saga instead of the creator - inject an error for all started (but not completed) nodes of a saga: remember, it's a dag! - add a /v1/ping endpoint to the internal api, and ping to see if the current sec is up - it's not normally safe to inject an error while the saga is running - add a bypass for this check - clearly state what errors we're injecting - inject errors using a specific uuid for omdb
Define an "abandoned" saga state. An abandoned saga will not begin to be executed by any SEC. Technicians mark sagas as abandoned using omdb; this requires the saga's current executor not to be running (otherwise it could receive a state update from Steno that will clobber the Abandoned state). This commit defines the new state in the database schema and fixes up the DB crates accordingly, but adds no affordances for applying the new saga state or considering it when deciding what sagas to recover.
This picks up the `omdb db sagas` subcommand and some other useful omdb bits (like common logic for determining whether a saga's SEC is likely to be running).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(N.B. This reuses a bunch of
omdbcode added by #7732 that I've merged into this branch. If we add more warnings or prompts to theinject-errorcommand before that PR merges, I'll probably want to reuse them here, so this PR is just a draft until 7732 is in main and I can clean up the history here. I don't expect the schema or Nexus bits to change that much more, though, unless review and testing reveal I've messed something up.)Add an
Abandonedsaga state. This state disqualifies a saga from being picked up by Nexus saga recovery. (A running saga will continue running if it is Abandoned, and continued saga execution may end up clobbering the Abandoned state entirely.) Add anomdbsubcommand to move a saga to this state (and refactor a bit to avoid duplicating code with theinject-errorsubcommand).Tested (so far) by:
svcadm restart), but is no longer recovered once abandonedFixes #7730.