[JENKINS-76200] Exclude STOPPED instances from capacity calculation

The NoDelayProvisionerStrategy was counting offline STOPPED EC2 instances
as "available capacity", preventing provisioning from being triggered when
jobs were queued. This caused STOPPED instances to remain stopped forever,
with jobs waiting indefinitely.

Root cause:
- countProvisionedButNotExecutingNodes() counted ALL offline nodes
- STOPPED instances were included in available capacity
- When capacity >= demand, provisioning was skipped
- provisionOndemand() was never called to start the stopped instances

Fix:
- Check AWS instance state for offline nodes
- Exclude STOPPED/STOPPING instances from capacity count
- Only count instances that will come online (PENDING/RUNNING)
- Fail-safe: if state check fails, count the instance to avoid over-provisioning

This preserves the fixes from:
- JENKINS-76151: EC2RetentionStrategy still only reconnects RUNNING instances
- JENKINS-76171: Offline PENDING/RUNNING instances still counted to prevent over-provisioning

Testing:
1. Stop an EC2 instance (via AWS or Jenkins stopOnTerminate)
2. Queue a job requiring that label
3. Verify provisioning is triggered and instance starts in AWS
4. Check logs for "Excluding STOPPED instance {id} from available capacity"

Author: mikecirioli

src/main/java/hudson/plugins/ec2/NoDelayProvisionerStrategy.java

@@ -90,15 +90,19 @@ public NodeProvisioner.StrategyDecision apply(NodeProvisioner.StrategyState stra
      * Counts executors in EC2 nodes that have been provisioned (exist in Jenkins) but are NOT yet counted in the
      * LoadStatistics snapshot. This specifically targets the gap where nodes exist but are:
      * - Offline (just added to Jenkins, before connecting starts)
+     * - Instance is PENDING or RUNNING in AWS (will come online soon)
      *
      * We explicitly DO NOT count:
      * - Connecting nodes (already in snapshot.getConnectingExecutors())
      * - Online nodes (already in snapshot.getAvailableExecutors() or busy executors)
+     * - STOPPED instances (won't come online without explicit start action)
      *
      * This prevents over-provisioning by accounting for nodes in the critical gap between:
      * 1) Node added to Jenkins (after PlannedNode future completes)
      * 2) Node starts connecting (shows up in snapshot.getConnectingExecutor())
      *
+     * JENKINS-76200: Exclude STOPPED instances - they won't come online on their own.
+     *
      * @param label the label to match, or null for unlabeled nodes
      * @return the number of executors from provisioned EC2 nodes in the offline->connecting gap
      */
@@ -113,6 +117,7 @@ int countProvisionedButNotExecutingNodes(Label label) {
         int offlineNodes = 0;
         int connectingNodes = 0;
         int onlineNodes = 0;
+        int stoppedNodes = 0;
 
         for (Node node : nodes) {
             // Only count EC2 nodes
@@ -136,16 +141,36 @@ int countProvisionedButNotExecutingNodes(Label label) {
             }
 
             // Only count nodes that are OFFLINE (not connecting, not online)
-            // These are in the gap between being added to Jenkins and starting to connect
+            // and not STOPPED in AWS (won't come online without explicit start)
             if (computer.isOffline() && !computer.isConnecting()) {
+                // JENKINS-76200: Check if instance is STOPPED in AWS
+                if (computer instanceof EC2Computer ec2Computer) {
+                    try {
+                        InstanceState state = ec2Computer.getState();
+                        if (state == InstanceState.STOPPED || state == InstanceState.STOPPING) {
+                            stoppedNodes++;
+                            LOGGER.log(
+                                    Level.FINE,
+                                    "Excluding STOPPED instance {0} from available capacity",
+                                    ec2Computer.getInstanceId());
+                            continue; // Don't count stopped instances
+                        }
+                    } catch (Exception e) {
+                        LOGGER.log(
+                                Level.FINE,
+                                "Could not get state for " + ec2Computer.getName() + ", counting as available",
+                                e);
+                        // If we can't determine state, count it to avoid over-provisioning
+                    }
+                }
                 count += node.getNumExecutors();
             }
         }
 
         LOGGER.log(
                 Level.FINER,
-                "EC2 nodes for label {0}: total={1}, offline={2}, connecting={3}, online={4}",
-                new Object[] {label, totalEC2Nodes, offlineNodes, connectingNodes, onlineNodes});
+                "EC2 nodes for label {0}: total={1}, offline={2}, connecting={3}, online={4}, stopped={5}",
+                new Object[] {label, totalEC2Nodes, offlineNodes, connectingNodes, onlineNodes, stoppedNodes});
 
         return count;
     }