feat(observoor): eBPF profiling into the engine OTel ClickHouse - #1416
Open
qu0b wants to merge 2 commits into
Open
feat(observoor): eBPF profiling into the engine OTel ClickHouse#1416qu0b wants to merge 2 commits into
qu0b wants to merge 2 commits into
Conversation
Add a shared PPROF_PORT_ID ("pprof", 6060) and wire it into the Go-based
clients so their net/http/pprof endpoint is reachable:
- geth, erigon: --pprof --pprof.addr=0.0.0.0 --pprof.port=6060 + port reg
- prysm: rename PROFILING_PORT_ID -> PPROF_PORT_ID for consistency
Caplin is intentionally excluded.
Add 'observoor' as an additional_service: a privileged, host-PID eBPF agent that collects kernel-level performance metrics (CPU, memory, disk/net I/O, scheduler latency, syscalls) for every Ethereum client process with zero client modifications. It ships data to the single engine OTel ClickHouse on the Docker host (the same stack 'otel' tracing uses) over the native protocol, writing to its own 'observoor' database — no second ClickHouse. main.star now discovers the engine ClickHouse native endpoint (gateway:19000) and passes it to observoor; enabling observoor triggers discovery but does NOT turn on per-client tracing. Replaces the earlier Pyroscope approach, which is removed entirely. Requires the engine OTel ClickHouse to publish its native port and carry single-node cluster config (see docs/observoor/ and the companion kurtosis change kurtosis-tech/kurtosis#3141).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds production-ready, low-overhead performance profiling as a single, focused addition — and exposes it for agents to consume as context.
observooradditional_serviceA privileged, host-PID observoor eBPF agent that collects kernel-level metrics — CPU (per-core), memory (RSS/VM), disk I/O, network I/O (with port labels), TCP RTT/cwnd, scheduler latency, syscalls, page faults, FDs, context switches — for every Ethereum client process, with zero client modifications.
It ships data to the single engine OTel ClickHouse on the Docker host (the same stack
oteltracing uses), writing to its ownobservoordatabase over the native protocol. No second ClickHouse, no extra backend collector. Enablingobservoordiscovers the engine ClickHouse and triggers nothing else — in particular it does not turn on per-client tracing.pprof
Adds a shared
PPROF_PORT_ID(6060) and exposes the Go clients' pprof endpoint: geth, erigon (--pprof), and prysm (port id renamed for consistency). Caplin intentionally excluded.Removed
Pyroscope is removed entirely (server-only mode couldn't pull pprof; observoor supersedes it), along with superseded in-enclave ClickHouse/Vector/eth-debug WIP.
Companion change
Requires the engine OTel ClickHouse to publish its native port and carry single-node cluster config so observoor's
ReplicatedReplacingMergeTree/ON CLUSTERschema can be created. That's a smallkurtosis otelchange: kurtosis-tech/kurtosis#3141. Seedocs/observoor/for the exact config + rationale.Verification (local, Docker)
additional_services: [observoor]; engine-OTel discovery found the host ClickHouse, observoor connected over native TCP atgateway:19000, ran its migrations, and streamed metrics for geth + lighthouse — noobservoor-clickhousecontainer in the enclave.observoordatabase (alongsideotel).otelandobservoorDBs and accepts observoor's replicatedON CLUSTERDDL over the published native port.Scope
Target clients only — EL: geth, besu, nimbus-eth1, ethrex, erigon, nethermind; CL: lighthouse, lodestar, nimbus, prysm, teku. Caplin excluded. Coverage will expand in follow-ups; this lays the basic rails.