Modularity & Module Boundaries

This document defines core vs feature modules, public interfaces, dependency rules, and ownership boundaries. Goal: parallel development without cross-cutting churn.

Read with Activation & runtime control, Data Layer, and Extensibility. Shared vocabulary: Core Concepts.

1. Layer model

Probing is organized in four layers. Dependencies only flow downward (higher layers may call lower; never the reverse).

Layer	Role	Changes when…
L1 Platform	SQL engine, storage, wire types, plugin traits	Federation, memtable, config, catalog
L2 Collectors	Produce rows into tables; optional HTTP via Extension	New signals (GPU, NCCL, host, Python runtime)
L3 Control plane	HTTP API, CLI, composition root	New endpoints, auth, cluster fan-out
L4 Experience	UI, skills, Python integration	Diagnostics UX, hooks, agent flows

Composition root (only place that wires everything):

probing/server/src/engine.rs — registers all ProbeDataSource + ProbeExtension for the in-process server.
Root src/lib.rs + Cargo.toml — PyO3 wheel bundles server + python extension.

2. Crate & directory map

L1 — Platform core (stable interfaces)

Unit	Path	Responsibility	Must NOT
probing-proto	`probing/proto/`	`Message<T>`, `DataFrame`, `Node`, `Query` DTOs	Import core, server, extensions
probing-memtable	`probing/memtable/`	MEMT/MEMH/MEMC read/write, mmap discovery	Know SQL, HTTP, Python
probing-core	`probing/core/`	DataFusion engine, federation, config, traits	Import server, cli, extensions
probing-macros	`probing/macros/`	`#[derive(ProbeExtension)]`	Business logic

Key core submodules:

Submodule	Path	Contract
Engine	`core/engine.rs`	`async_query`, `enable(ProbeDataSource)`
Federation	`core/federation/`	`global.*` catalog, tags `_host/_addr/_rank/_role` — see Federated query engine
Memtable SQL	`core/memtable_sql.rs`	mmap files → `TableProvider`
Config	`config.rs`	`get` / `set` / `write` KV + extension options

L2 — Collectors (feature modules)

Each collector writes data and optionally registers tables + extension config. Collectors do not call each other.

Unit	Path	Schema / tables	Extension
probing-python	`probing/extensions/python/`	`python.*` (backtrace, mmap tables)	`PythonExt`, `TorchProbeExtension`, `PprofProbeExtension`
probing-cc	`probing/extensions/cc/`	`cpu.`, `cluster.nodes`, `rdma.`, `process.`, `files.`	`CpuProbeExtension`, `RdmaProbeExtension`
probing-gpu	`probing/extensions/gpu/`	`gpu.utilization`, `gpu.devices`	`GpuProbeExtension`
probing-nccl-profiler	`probing/extensions/nccl-profiler/`	`nccl.proxy_ops`, `nccl.net_qp` (mmap)	NCCL plugin ABI only (no HTTP)

Python-side collectors (same layer, different language):

Unit	Path	Tables
Torch tracing	`python/probing/profiling/`	`python.torch_trace`, `python.comm_collective`
Tracing spans	`python/probing/tracing/`	`python.trace_event`
Parallel role	`python/probing/parallel.py`	stamps `role` on rows
User plugins	`python/probing/ext/`	`python.<custom>` via `@table`

L3 — Control plane

Unit	Path	Responsibility
probing-server	`probing/server/`	Axum routes, auth, `initialize_engine()`, cluster fan-out, torchrun heartbeat (`torchrun_cluster.rs`)
probing-cli	`probing/cli/`	HTTP client to probe; inject/list/query/repl/skill

Stable HTTP surface: probing/server/API.md, enforced by tests/regression/spec/api_spec.json.

Host-process invariant: probing-server is embedded in the instrumented application. It must never call process::exit, abort the process, or otherwise terminate the host because a probing engine, listener, or reporting component failed. Failures stay inside probing and surface through component state, logs, and readiness responses.

L4 — Experience

Unit	Path	Responsibility
web/	Dioxus WASM	Pages, visualization, Investigate agent
skills/	symlink → `python/probing/bundled_skills/`	Authoring alias for skill content SSOT
probing-skills	`probing/crates/skills/`	Shared loader, interpreter, runner (CLI / Web / MCP)
python/probing/extensions/	entry points	Skill + magic + vendor package discovery
python/probing/	Python package	Hooks, `query()`, agent install helpers

3. Public interfaces (contracts)

New work should extend one of these contracts instead of adding cross-module calls.

3.1 `ProbeDataSource` — register SQL tables

Where: probing/core/src/core/data_source.rs Register: EngineBuilder::with_data_source (wired in server/engine.rs)

Kind	Use when	Example
`Table`	Fixed schema, one table	`gpu.devices`
`Namespace`	Dynamic tables	`python.*`, mmap discovery

Rules:

Schema + scan logic live in the collector crate.
Federation: table names under known schemas (python, nccl, gpu, …) auto-mirror to global.<schema>.<table>.
Do not query other collectors from inside scan(); join at SQL layer.

3.2 `ProbeExtension` — typed config + HTTP contracts

Where: probing/core/src/core/probe_extension.rs Derive: #[derive(ProbeExtension)] in probing-macros

Capability	Mechanism
Identity	`ProbeExtension::name` supplies the registration namespace
Config keys	`ProbeExtensionConfig` publishes typed key/alias specs and implements `set` / `get` / `options`
Side effects	Background sampler start/stop in `set_*` handlers
HTTP	`ProbeExtensionCall::routes` registers method/content-type/CORS/readiness contracts; `call` executes them

Rules:

Extension name = URL segment (pythonext, rdmaextension, …).
Registration validates names, duplicate routes, and config key/alias collisions before engine publication.
Prefer tables for data, extension for control (start/stop, eval, flamegraph render).
Never todo!() in default trait methods — return EngineError.

3.3 Python `@table` — application data plugins

Where: python/probing/core/table.py, documented in Extensibility

@table("comm_collective")
@dataclass
class CommCollective: ...

Rules:

Writes go through memtable mmap → appear as python.<name>.
Stamp local_step, global_step, rank, role on training rows (see concepts).
No direct Rust imports from Python plugins.

3.4 Skill contract — diagnostic workflows

Where: python/probing/bundled_skills/<id>/SKILL.md + steps.yaml (alias: skills/), catalog catalog.yaml

Field	Purpose
`requires.any_tables`	Preconditions
`spec.steps[].sql`	Evidence queries (only interface to engine)
`interpretation.rules`	Deterministic findings
`next_steps`	Hand-off to other skills

Rules:

Skills only talk to the engine via SQL (probing query) or documented HTTP APIs.
No Rust/Python code in skills — YAML + markdown only.
Content SSOT: python/probing/bundled_skills/ (repo-root skills/ is a symlink alias).
Discovery: python/probing/extensions/ entry points + GET /apis/pythonext/skills/*.
Execution SSOT: probing-skills — CLI (probing/cli/skill/), Web WASM (web/src/agent/runner.rs), MCP (run_skill / plan_skill in server).
Python probing/skills/ is discovery + agent install only; not an execution path.

3.5 Wire protocol — CLI / Web ↔ Server

Where: probing/proto/

Endpoint	Payload
`POST /query`	`Message<Query>` → `Message<Data>`
`POST /query/dto`	Stable external DTO
`GET /apis/*`	JSON / SVG per API.md
`GET /ws`	REPL

Rules:

CLI and Web must not link probing-core at runtime (CLI: proto only; Web: HTTP + proto types).
Breaking API changes require api_spec.json + contract tests update.

3.6 Federation tags

Where: probing/core/src/core/federation/convert.rs

Every global.* row adds:

Column	Source
`_host`	Peer hostname
`_addr`	Peer `host:port`
`_rank`	`torch.distributed` rank from node registry
`_node_rank`	Node / worker group rank (`GROUP_RANK`)
`_local_rank`	GPU index on node (`LOCAL_RANK`)
`_role`	Parallel role key from node registry

Collectors must not invent alternate peer tags.

4. Dependency rules

Dependency matrix (target state)

	proto	memtable	core	cc/gpu/py	server	cli
proto	—
memtable		—
core	✓	✓	—
extensions	✓	✓	✓	—
server	✓	✓	✓	✓	—
cli	✓	opt				—
web	✓

4.1 PyPI packaging coupling (cdylib → `probing-cli`)

Maturin builds one native artifact (probing._core cdylib from root Cargo.toml). The probing console script is not a separate Rust binary on PyPI:

This is an accepted compile-time coupling for the wheel workflow (pyproject.toml [tool.maturin] + [project.scripts]). It is not the Python collector calling the CLI control plane at runtime for data paths.

Contract (keep the edge thin):

Root src/lib.rs (the maturin cdylib composition root) may call probing_cli::pyo3::cli_main and re-export it as _core.cli_main. probing-python itself does not depend on probing-cli.
Do not import other probing-cli modules (inject, skill runner internals, ctrl) from collectors or server.
Standalone Rust binary (probing/cli/src/main.rs) remains optional for non-PyPI installs; PyPI users always go through the Python script entry.

If CLI logic grows, split probing-cli-lib (shared cli_main + HTTP client) from CLI-only commands, rather than letting collectors or the cdylib spread imports across the cli crate.

Implications:

New metrics → new table (or new columns with new table name), not ad-hoc server state.
Cross-signal analysis → SQL JOIN or skill steps, not collector callbacks.
Hot/cold retention → memtable + MemTableProbeExtension config only.

6. Module boundaries by concern

Use this table to decide where a change belongs:

Concern	Owner module	Interface
SQL parsing, federation rewrite	probing-core	`Engine::async_query`
Table/column semantic docs	probing-core	Code-first `docs` registry + `resources/tables.yaml` overlay → `probe.probing.table_docs` / `column_docs`
mmap format, compaction	probing-memtable	`RowWriter`, `ColdStore`
Torchrun cluster heartbeat	probing-server	`torchrun_cluster.rs`, `cluster_report_backoff.rs`, PUT `/apis/nodes`
Mixed Python/C stack	probing-python/features	`python.backtrace`, pprof
macOS per-thread SIGUSR2	probing-core	`signal::send_sigusr2_to_thread_id`
Torch module sampling	python/probing/profiling	`python.torch_trace`
Collective wall time	python/probing/profiling/collective	`python.comm_collective`
PyTorch Flight Recorder bridge	python/probing/profiling/flight_recorder	`python.torch_nccl_flight_record`, `python.torch_nccl_pg_status`
NCCL wait decomposition	probing-nccl-profiler	`nccl.proxy_ops`
Host CPU / RDMA counters	probing-cc	`cpu.`, `rdma.`
GPU mem / util	probing-gpu	`gpu.*`
Cluster node registry	probing-core/cluster + server/report	`cluster.nodes`, PUT/GET `/apis/nodes`
Cross-rank fan-out	probing-server/cluster_fanout	`global.*`, `/apis/cluster/query`
Auth, request limits	probing-server	middleware
Inject, query CLI	probing-cli	HTTP to server
Diagnostic skills	skills/	steps.yaml
Training step matrix UI	web/src/next/pages/training.rs	GET `/apis/training/step_matrix`
Agent routing	web/agent + skills catalog	skill metadata

7. Team ownership (suggested)

Area	Paths	Can merge without
Platform	`probing/core`, `probing/memtable`, `probing/proto`, `probing/macros`	Touching collectors
Host/GPU	`probing/extensions/cc`, `probing/extensions/gpu`	Python, web
Runtime Python	`probing/extensions/python`, `python/probing/profiling`, `python/probing/tracing`	NCCL plugin, web pages
NCCL	`probing/extensions/nccl-profiler`, `python/probing/nccl`	Torch hooks
Control plane	`probing/server`, `probing/cli`	Skill content, web UI
Diagnostics	`skills/`, `probing/cli/skill`, `web/src/agent`	Collector internals
Web UI	`web/` (except agent skill loader)	Rust collectors
Docs	`docs/`	—

Merge checklist:

Does it respect dependency direction (§4)?
If adding a table → ProbeDataSource or @table only?
If adding diagnostics → skill step or new table, not server one-off?
HTTP change → API.md + api_spec.json?
Federation → uses standard tags only?

8. Known boundary violations (technical debt)

Track and fix incrementally:

Issue	Current	Target
Python ext → CLI	~~`probing-python` → `probing-cli`~~	Done — no crate dep; wheel `cli_main` wired only in root `src/lib.rs` (§4.1)
Python ext → CC	~~`probing-python` → `probing-cc`~~	Done — `send_sigusr2_to_thread_id` moved to `probing-core::signal`
Core → NCCL/HCCL	~~`probing-core` → `probing-nccl-profiler` / `probing-hccl-shim` (`builtin-schema-docs` feature) for `semantic_catalog`~~	Done — `register_docs()` called from `server/engine.rs` composition root; `probing-core` default features empty
Core → skills YAML	~~`semantic_catalog.rs` `include_str!(skills/semantic/tables.yaml)`~~	Done — overlay at `probing/core/resources/tables.yaml`; descriptions SSOT in `docs` registry
Server → python `features/*`	~~`server/profiling.rs` / training flamegraph handlers~~	Done — flamegraphs via `torchextension` / `pprofextension` `ProbeExtensionCall` only
Server → python REPL internals	~~`PythonRepl` in server~~	`/ws` uses `ReplSession` facade only
Composition sprawl	All wiring in `server/engine.rs`	Optional: manifest TOML listing enabled extensions
Skills triple loader	~~Rust + Python + Web compile-time embed~~	Done — `probing-skills` is loader/interpret/runner SSOT; Python keeps discovery entry-points + PyO3 serialize bridge; Web deserializes API into shared types
kmsg collector	Registered (Linux/kmsg feature gate)	Done

Cluster membership and external runtime tables

The current source for cluster.nodes membership is the Torchrun cluster heartbeat. L3 probing-server uses hierarchical HTTP PUT plus TCPStore side-channel keys without modifying torch rendezvous keys. The Rust ctor starts maybe_start_torchrun_cluster() by default; see Distributed membership and control plane.

pulsing.* is an externally produced mmap schema discovered through the same table contract as other vendor data. Probing does not currently merge Pulsing gossip members into cluster.nodes or bootstrap a Pulsing ActorSystem.

9. Adding a new feature (decision tree)

Anti-patterns:

Adding business logic to server/engine.rs beyond registration.
Web page importing SQL strings for tables that don't exist in catalog.
Collector calling Engine::async_query from write path.
Skill execution outside probing-skills (duplicate runners in Python or Web).

10. Related documents

Doc	Scope
Activation & runtime control	Process entry and service readiness
Data Layer	MEMT/MEMC internals
Extensibility	Public extension paths (table + skill)
Distributed	Federation & cluster
Distributed membership	Hierarchical torchrun membership and health
Distributed Profiler	Target 10K-rank timeline query and visualization
NCCL Profiler	NCCL plugin boundary
web/DESIGN.md	UI module layout
AGENTS.md	Agent skill usage

Navigation Menu

Uh oh!

Modularity & Module Boundaries

1. Layer model