chore: harden comparison harness for spot-cluster runs - #33
Merged
Conversation
Improvements to run-comparison.sh from running the comparison on an EKS spot cluster: - Tolerate transient CrashLoopBackOff. Spot-node preemption / OOM during autoscale briefly crash-loops pods; the old fail-fast bailed on the first CrashLoopBackOff. Now it bails immediately only on unrecoverable image errors, and treats CrashLoopBackOff as terminal only after CRASH_RESTART_LIMIT (default 5) restarts. - Dump the crashing pods (container states + --previous logs) before the sweep, so a genuine crash loop's reason is captured in the run output. - Add a WORKER_CPU knob to override the per-backend worker CPU request/limit, mirroring the existing WORKER_REPLICAS. Signed-off-by: Matteo Merli <mmerli@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improvements to
scripts/run-comparison.shsurfaced by running the comparison on an EKS spot cluster.CrashLoopBackOff. Spot-node preemption / OOM during Karpenter scale-up briefly crash-loops pods; the old fail-fast bailed on the firstCrashLoopBackOff. Now it bails immediately only on unrecoverable image errors (ImagePullBackOff/ErrImagePull/InvalidImageName), and treatsCrashLoopBackOffas terminal only once a container has restartedCRASH_RESTART_LIMITtimes (default 5).--previouslogs, so the failure reason lands in the run output instead of being swept away.WORKER_CPUknob to override the per-backend worker CPU request/limit, mirroring the existingWORKER_REPLICAS.Verification
Used across the YCSB comparison + scaling sweeps: the tolerance let oxia survive transient spot churn (a run that previously aborted), and the restart limit correctly distinguished a genuine crash loop from a transient one, dumping its reason.