ScaDS
diff --git a/‎.gitignore‎
Lines changed: 7 additions & 0 deletions b/‎.gitignore‎
Lines changed: 7 additions & 0 deletions
diff --git a/‎README.md‎
Lines changed: 42 additions & 2 deletions b/‎README.md‎
Lines changed: 42 additions & 2 deletions
diff --git a/‎docs/guides/c-custom-collector.md‎
Lines changed: 111 additions & 0 deletions b/‎docs/guides/c-custom-collector.md‎
Lines changed: 111 additions & 0 deletions
@@ -173,4 +173,11 @@ poetry.toml
 # LSP config files
 pyrightconfig.json
 
+# Compiled C monitoring collector and build artifacts
+jumper_collector
+*.o
+
+# Benchmark results
+jumper_extension/monitor/benchmark/results/
+
 # End of https://www.toptal.com/developers/gitignore/api/python
@@ -86,11 +86,17 @@ Try it yourself:
 
 1. **Start monitoring**:
    ```python
-   %perfmonitor_start [interval]
+   %perfmonitor_start [interval] [--monitor TYPE] [--check-sanity]
    ```
 
    `interval` is an optional argument for configuring frequency of performance data gathering (in seconds), set to 1 by default. This command launches a performance monitoring daemon.
 
+   The `--monitor` option selects a backend (`default`, `native_c`, `subprocess_python`, `thread`, `slurm_multinode`). The default backend is the native C collector (`native_c`); it is compiled on first use — you will see a `[JUmPER] Compiling native_c monitor binary...` message. If no C compiler is available or compilation fails, JUmPER automatically falls back to the `subprocess_python` monitor.
+
+   The optional `--check-sanity` flag runs a short validation of the selected backend (collecting a few samples, verifying expected metric columns are present, non-NaN, and non-zero) before real monitoring starts.
+
+   > **IMPORTANT.** `--check-sanity` was tailored for the `thread`, `subprocess_python` and `native_c` monitors. When used with any other monitor (e.g. `slurm_multinode`, or a custom monitor provided via the programmatic API) it is **expected to fail** because those backends populate a different set of per-level metric columns. A warning is printed in that case and the check still runs, but a failure does not necessarily indicate a broken monitor — only that the tailored check does not apply.
+
 2. **Run your code**
 
 3. **View performance report**:
@@ -233,6 +239,41 @@ The following metric keys can be used with `--metrics` for both direct and live
    ```
    Export performance measurements for entire notebook and cell execution history with timestamps, allowing you to project measurements onto specific cells.
 
+### Custom Monitors
+
+Any object implementing `MonitorProtocol`
+(`jumper_extension/monitor/common.py`) can be plugged in via the
+`monitor=` argument of `service.start_monitoring` — see
+[`jumper_extension/monitor/README.md`](jumper_extension/monitor/README.md)
+for a worked `SlurmMultinodeMonitor` example and the full
+[Custom Monitors guide](https://scads.github.io/jumper_jupyter_performance/latest/guides/custom-monitor/)
+for a step-by-step walkthrough.
+
+### Custom Collectors
+
+To extend the metric pipeline with a new group of logically related metrics —
+for example network I/O or a hardware sensor — without replacing the entire
+monitor backend, there are two ways to create a collector:
+
+1. **Python collector** — a `CollectorBackend` + `StorageHandler` pair loaded
+   automatically from `collectors.yaml`. Works with the `thread` and
+   `subprocess_python` monitors. See the full
+   [Custom Python Collectors guide](https://scads.github.io/jumper_jupyter_performance/latest/guides/python-custom-collector/)
+   for a step-by-step walkthrough including a `NetworkCollector` example.
+
+2. **C collector** — a C translation unit implementing the `CCollector` interface,
+   compiled directly into the `native_c` binary. See the
+   [Custom C Collectors guide](https://scads.github.io/jumper_jupyter_performance/latest/guides/c-custom-collector/)
+   for a step-by-step walkthrough.
+
+### Visualizing Custom Collector Metrics
+
+Custom collector columns are available to `%perfmonitor_plot --metrics` once
+registered in `plots.yaml`. See the
+[Visualizing Custom Collector Metrics guide](https://scads.github.io/jumper_jupyter_performance/latest/guides/visualizing-custom-collector-metrics/)
+for the full reference — built-in plot types, `default_subsets`, and a
+disk-vs-network composite panel example.
+
 ### Monitoring Right in Your Code
 Run the monitor around any code block and save its performance profile to CSV/JSON.
 
@@ -280,7 +321,6 @@ The extension supports four different levels of metric collection, each providin
 - **NVIDIA GPUs**: Full support for all monitoring levels (process, user, system, slurm) including per-process GPU memory tracking
 - **AMD GPUs**: System-level monitoring supported; per-process and per-user metrics are limited by AMD ADLX API capabilities
 
-
 ## Full Documentation
 
 - Online (latest): https://scads.github.io/jumper_jupyter_performance/latest/
 
@@ -0,0 +1,111 @@
+# Custom C Collectors
+
+The `native_c` monitor is a compiled binary that reads `/proc` directly.  Its
+metric backends are C translation units — not Python classes — and are registered
+at compile time, not via `collectors.yaml`.
+
+!!! note
+    For the **Python** monitor backends (`thread`, `subprocess_python`) see
+    [Custom Python Collectors](python-custom-collector.md).
+
+---
+
+## Source layout
+
+```
+native_c/
+├── monitor.c / monitor.h   # main loop, shared types, registry
+├── Makefile
+└── metrics/
+    ├── cpu/    cpu.c / cpu.h
+    ├── memory/ memory.c / memory.h
+    ├── io/     io.c / io.h
+    └── gpu/    gpu.c / gpu.h
+```
+
+Each backend exposes a single `const CCollector <name>_collector` symbol and
+includes `monitor.h` (available via the `-I$(DIR)` compile flag).
+
+---
+
+## `CCollector` interface
+
+```c
+typedef struct {
+    const char *name;
+    void (*setup)(void);
+    void (*snapshot)(TickContext *ctx);
+    void (*emit_columns)(FILE *fp, int level_idx, int n_cpus, int n_gpus);
+    void (*emit_sample)(FILE *fp, int level_idx, TickContext *ctx, double dt);
+    void (*post_tick)(void);
+    void (*teardown)(void);
+} CCollector;
+```
+
+| Callback | When called | Set to `NULL` if |
+|---|---|---|
+| `setup` | once, before main loop | no one-time init needed |
+| `snapshot` | every tick, before any emit | no per-tick `/proc` reads needed |
+| `emit_columns` | once per level at start (ready handshake) | backend adds no columns |
+| `emit_sample` | every tick, per level | backend adds no columns |
+| `post_tick` | after all levels emitted | no post-emit cache work needed |
+| `teardown` | on clean shutdown | no resources to release |
+
+`TickContext` carries the PID sets for the current tick (proc, user, slurm, all).
+Use `ctx->all_pids` / `ctx->n_all` to iterate over all tracked processes.
+
+---
+
+## Adding a new backend
+
+**1.** Create `metrics/<name>/<name>.h`:
+```c
+#ifndef JUMPER_METRICS_<NAME>_H
+#define JUMPER_METRICS_<NAME>_H
+#include "monitor.h"
+extern const CCollector <name>_collector;
+#endif
+```
+
+**2.** Implement `metrics/<name>/<name>.c` — define all needed callbacks and expose:
+```c
+const CCollector <name>_collector = {
+    .name          = "<name>",
+    .setup         = <name>_setup,
+    .snapshot      = <name>_snapshot,
+    .emit_columns  = <name>_emit_columns,
+    .emit_sample   = <name>_emit_sample,
+    .post_tick     = NULL,
+    .teardown      = NULL,
+};
+```
+
+**3.** Add to `Makefile`:
+```makefile
+SRCS := \
+    ...
+    metrics/<name>/<name>.c
+```
+
+**4.** Register in `monitor.c`:
+```c
+#include "metrics/<name>/<name>.h"
+
+static const CCollector * const g_registry[] = {
+    &cpu_collector, &memory_collector, &io_collector, &gpu_collector,
+    &<name>_collector,
+};
+```
+
+**5.** Add to `config/collectors/c/collectors.yaml`:
+```yaml
+collectors:
+  - cpu
+  - memory
+  - io
+  - gpu
+  - <name>
+```
+
+After these changes, rebuild with `make` (or trigger `ensure_native_c(force_build=True)`
+from Python) — the new collector will be active on the next monitor start.