possible fix #1154
Closed
possible fix #1154
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previous commit started stopped instances whenever check() ran, even when no jobs were waiting. This caused unnecessary instance starts. Added check to only start stopped instance if itemsInQueueForThisSlave() returns true, meaning there are actually jobs waiting for this specific node. Changes: - Check for queued jobs before calling startInstances() - Log whether jobs are queued: "jobs in queue: true/false" - Skip starting if no jobs: "No jobs waiting - leaving it stopped" - Only start if jobs waiting: "Jobs are waiting - attempting to start" This ensures stopped instances remain stopped until actually needed for work.
Changed from checking only explicit node assignment (selfLabel) to using Label.contains() which properly checks if a node can execute jobs based on label matching. This fixes the issue where stopped instances would only start for jobs explicitly tied to the node name, not for jobs that match the node's labels. Changes: - Use assignedLabel.contains(selfNode) instead of assignedLabel == selfLabel - Handle null assignedLabel (jobs that can run on any node) - Added comment explaining the label matching logic Now stopped instances will start for: - Jobs with no label requirement (assignedLabel == null) - Jobs whose labels match this node's capabilities (assignedLabel.contains(selfNode)) Before this fix, stopped instances only started for jobs explicitly tied to the specific node name.
The NoDelayProvisionerStrategy was counting offline STOPPED EC2 instances
as "available capacity", preventing provisioning from being triggered when
jobs were queued. This caused STOPPED instances to remain stopped forever,
with jobs waiting indefinitely.
Root cause:
- countProvisionedButNotExecutingNodes() counted ALL offline nodes
- STOPPED instances were included in available capacity
- When capacity >= demand, provisioning was skipped
- provisionOndemand() was never called to start the stopped instances
Fix:
- Check AWS instance state for offline nodes
- Exclude STOPPED/STOPPING instances from capacity count
- Only count instances that will come online (PENDING/RUNNING)
- Fail-safe: if state check fails, count the instance to avoid over-provisioning
This preserves the fixes from:
- JENKINS-76151: EC2RetentionStrategy still only reconnects RUNNING instances
- JENKINS-76171: Offline PENDING/RUNNING instances still counted to prevent over-provisioning
Testing:
1. Stop an EC2 instance (via AWS or Jenkins stopOnTerminate)
2. Queue a job requiring that label
3. Verify provisioning is triggered and instance starts in AWS
4. Check logs for "Excluding STOPPED instance {id} from available capacity"
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.