scylla: add integrated monitoring stack support

Spin up a Prometheus + Grafana + Alertmanager stack alongside any CCM
cluster using Docker containers with --net=host.

Automatic mode (--monitoring or CCM_MONITORING=1) keeps Prometheus scrape
targets in sync on every topology change. Manual mode
(ccm monitoring start/stop/sync) gives on-demand control.

Multi-cluster setups are supported via port offsets based on cluster ID.
When scylla-monitoring repo is available, real dashboards are generated;
otherwise a built-in fallback overview dashboard is used.

New CLI:
  ccm create ... --monitoring [--monitoring-dir=PATH]
  ccm monitoring start|stop|enable|disable|sync|status

Author: dkropachev

README.md

@@ -131,7 +131,7 @@ Requirements
 - **Java 8+** - only required for:
   - Using Cassandra clusters
   - Using older Scylla versions (< 6.0)
-- **Docker** (optional) - for Docker-based clusters
+- **Docker** (optional) - for Docker-based clusters and [integrated monitoring](docs/monitoring.md)
 - **Multiple loopback interfaces** - CCM runs nodes on 127.0.0.X addresses
   - On Linux: Usually available by default
   - On Mac OS X: Create aliases manually:
@@ -142,8 +142,18 @@ Requirements
     ```
 
 By default ccm will look for Java in `/usr/lib/jvm` directory. If you have a custom Java installation directory,
-you can provide `CUSTOM_JAVA_HOME` env variable. When this environment, ccm will assume Java binary is available
-under `$CUSTOM_JAVA_HOME/bin/java` path.  
+you can provide `CUSTOM_JAVA_HOME` env variable. When this environment variable is set, ccm will assume Java binary is available
+under `$CUSTOM_JAVA_HOME/bin/java` path.
+
+### Environment Variables
+
+| Variable | Description |
+|---|---|
+| `CCM_MONITORING` | When non-empty and not `0`, enable automatic monitoring for all newly created clusters (same as `--monitoring`). |
+| `SCYLLA_MONITORING_DIR` | Path to the [scylla-monitoring](https://github.com/scylladb/scylla-monitoring) checkout. Used as default for `--monitoring-dir`. |
+| `CUSTOM_JAVA_HOME` | Custom Java installation directory. |
+| `SCYLLA_EXT_OPTS` | Extra Scylla command-line options passed to every node. |
+| `SCYLLA_EXT_ENV` | Extra environment variables for Scylla processes. |
 
 Known Issues
 ------------
@@ -237,6 +247,7 @@ ccm start                       # Start all nodes
 ccm stop                        # Stop all nodes
 ccm remove                      # Remove current cluster
 ccm clear                       # Clear cluster data but keep config
+ccm monitoring <subcmd>         # Manage monitoring stack (start/stop/enable/disable/sync/status)
 ```
 
 ### Node Management
@@ -258,6 +269,42 @@ ccm add node4 -i 127.0.0.4      # Add a new node to cluster
 
 For complete command reference, run `ccm --help` or `ccm <command> --help`.
 
+Integrated Monitoring
+---------------------
+
+CCM can start and maintain a [scylla-monitoring](https://github.com/scylladb/scylla-monitoring) stack
+(Prometheus + Grafana) alongside your cluster. See [docs/monitoring.md](docs/monitoring.md) for full details.
+
+### Quick start — automatic mode
+```bash
+# Create a cluster with monitoring enabled
+ccm create mycluster --scylla -n 3 -v release:2024.2 --monitoring --monitoring-dir=/path/to/scylla-monitoring -s
+
+# Monitoring is now running and auto-updates on topology changes
+ccm monitoring status
+# Grafana:    http://localhost:3000
+# Prometheus: http://localhost:9090
+
+ccm add node4 --scylla   # targets updated automatically
+ccm stop                  # monitoring stops automatically
+```
+
+### Quick start — environment variable
+```bash
+# Enable monitoring for all new clusters without passing --monitoring every time
+export CCM_MONITORING=1
+ccm create mycluster --scylla -n 3 -v release:2024.2 -s
+```
+
+### Quick start — manual mode
+```bash
+ccm create mycluster --scylla -n 3 -v release:2024.2 -s
+ccm monitoring start --monitoring-dir=/path/to/scylla-monitoring
+ccm add node4 --scylla
+ccm monitoring sync      # manually refresh targets
+ccm monitoring stop
+```
+
 Working with Cassandra
 ----------------------
 

ccmlib/cluster.py

@@ -41,6 +41,18 @@ def __init__(self, path, name, partitioner=None, install_dir=None, create_direct
         self._debug = []
         self._trace = []
 
+        # Monitoring stack integration (Scylla-only).
+        # These fields are initialized here so they're always present on any
+        # Cluster instance (simplifies config persistence and CLI).  Actual
+        # monitoring auto-start only happens in ScyllaCluster.
+        # CCM_MONITORING env var is applied in ScyllaCluster.__init__.
+        self.monitoring_enabled = False
+        self.monitoring_stack = None  # runtime only, not persisted
+        self.monitoring_dir = None
+        self.grafana_port = 3000
+        self.prometheus_port = 9090
+        self.alertmanager_port = 9093
+
         if self.name.lower() == "current":
             raise RuntimeError("Cannot name a cluster 'current'.")
 
@@ -226,14 +238,25 @@ def version(self):
     def cassandra_version(self):
         return self.version()
 
+    def _notify_topology_change(self):
+        """Called after any operation that changes which nodes are UP.
+
+        Only updates targets if automatic monitoring is enabled AND running.
+        """
+        if self.monitoring_enabled and self.monitoring_stack and self.monitoring_stack.is_running():
+            try:
+                self.monitoring_stack.update_targets()
+            except Exception as e:
+                self.warning(f"Failed to update monitoring targets: {e}")
+
     def add(self, node: ScyllaNode, is_seed, data_center=None, rack=None):
         if node.name in self.nodes:
             raise common.ArgumentError(f'Cannot create existing node {node.name}')
         self.nodes[node.name] = node
         if is_seed:
             self.seeds.append(node)
         self._update_config()
-        
+
         # If data_center is not specified, infer it from existing nodes
         if data_center is None and len(self.nodes) > 1:
             # Get datacenter from the first existing node (excluding the one we just added)
@@ -243,7 +266,7 @@ def add(self, node: ScyllaNode, is_seed, data_center=None, rack=None):
                     if rack is None and existing_node.rack is not None:
                         rack = existing_node.rack
                     break
-        
+
         node.data_center = data_center
         node.rack = rack
         node.set_log_level(self.__log_level)
@@ -257,6 +280,7 @@ def add(self, node: ScyllaNode, is_seed, data_center=None, rack=None):
             self.debug(f"{node.name}: data_center={node.data_center} rack={node.rack} snitch={self.snitch}")
             self.__update_topology_files()
         node._save()
+        self._notify_topology_change()
         return self
 
     # nodes can be provided in multiple notations, determining the cluster topology:
@@ -420,6 +444,7 @@ def remove(self, node: ScyllaNode=None, wait_other_notice=False, other_nodes=Non
                 self.seeds.remove(node)
             self._update_config()
             node.stop(gently=False, wait_other_notice=wait_other_notice, other_nodes=other_nodes)
+            self._notify_topology_change()
         else:
             self.stop(gently=False, wait_other_notice=wait_other_notice, other_nodes=other_nodes)
 
@@ -700,7 +725,12 @@ def _update_config(self, install_dir=None):
                 'log_level': self.__log_level,
                 'use_vnodes': self.use_vnodes,
                 'id': self.id,
-                'ipprefix': self.ipprefix
+                'ipprefix': self.ipprefix,
+                'monitoring_enabled': self.monitoring_enabled,
+                'monitoring_dir': self.monitoring_dir,
+                'grafana_port': self.grafana_port,
+                'prometheus_port': self.prometheus_port,
+                'alertmanager_port': self.alertmanager_port,
             }
 
         with open(filename, 'w') as f:

ccmlib/cluster_factory.py

@@ -55,6 +55,17 @@ def load(path, name):
             if 'ipprefix' in data:
                 cluster.ipprefix = data['ipprefix']
 
+            # Restore monitoring settings
+            if 'monitoring_enabled' in data:
+                cluster.monitoring_enabled = data['monitoring_enabled']
+            if 'monitoring_dir' in data:
+                cluster.monitoring_dir = data['monitoring_dir']
+            if 'grafana_port' in data:
+                cluster.grafana_port = data['grafana_port']
+            if 'prometheus_port' in data:
+                cluster.prometheus_port = data['prometheus_port']
+            if 'alertmanager_port' in data:
+                cluster.alertmanager_port = data['alertmanager_port']
 
         except KeyError as k:
             raise common.LoadError("Error Loading " + filename + ", missing property:" + str(k))