FAQ.md

What is format of the URL accepted by chdig?

The simplest form is just - localhost

For a secure connections with user and password (note: passing the password on the command line is not safe), use:

chdig -u 'user:password@clickhouse-host.com/?secure=true'

A full list of supported connection options is available here.

Note: This link currently points to my fork, as some changes have not yet been accepted upstream.

Environment variables

A safer way to pass the password is via environment variables:

export CLICKHOUSE_USER='user'
export CLICKHOUSE_PASSWORD='password'
chdig -u 'clickhouse-host.com/?secure=true'
# or specify the port explicitly
chdig -u 'clickhouse-host.com:9440/?secure=true'

What is --config (CLICKHOUSE_CONFIG)?

This is standard config for ClickHouse client, i.e.

user: foo
password: bar
host: play
secure: true

See also some examples and possible advanced use cases here

How to configure views and panes layout (like tmuxinator)?

The chdig config (--chdig-config/CHDIG_CONFIG, defaults to ~/.config/chdig/config.yaml, ~/.chdig.yaml or /etc/chdig/config.yaml) has two sections for this:

• views - per-view settings, applied whenever the view is opened. Views are referred to by their CLI subcommand names (see chdig --help; both last_queries and last-queries are accepted):

  • filter - initial value of the view's / filter
  • query_kind - restrict a queries view to these query_kind values (Select, Insert, Create, Drop, Alter, ...), a single value or a list; combined with filter. The live queries view needs ClickHouse 23.2+ for it (system.processes.query_kind)
  • start/end - time interval override for this view (such a view ignores the global --start/--end and T/t/Alt+t seeking)
  • limit - row limit override (--limit/--queries-limit, whichever applies to the view)
  • level - maximum log level for log views, includes everything at this severity and above (i.e. error = Fatal, Critical and Error)
  • columns - columns of a queries view or of part_log in display order (the table headers: query_id, cpu, mem, query, ... resp. event_time, event_type, part_name, rows, ...; for the queries views it overrides the global view.query_columns). Any profile event can be a column too (ProfileEvents.SelectedRows), plus any query setting for the queries views (Settings.max_threads)

  The same settings can be changed at runtime in the settings dialog (F3; Alt+Up/Alt+Down switch tabs): every view (and named instance) has a tab under Views; the queries views' tabs also list their columns. Tab in the column fields completes the event/setting names from the loaded rows (pe./s. are accepted while typing).

• layout - startup pane layout, a tree of splits. Each pane is a view name or a nested split (direction, panes); ratio is the fraction of the parent split given to a pane (panes without it share the remainder equally). focus selects the initially focused view (defaults to the first one). An explicit view on the command line (e.g. chdig merges) disables the layout. The live server flamegraphs are placeable too, each in its own pane (cpu_flamegraph, real_flamegraph, memory_flamegraph, memory_sample_flamegraph, jemalloc_sample_flamegraph, memory_allocated_without_check_flamegraph, events_flamegraph, live_flamegraph, jemalloc_flamegraph).

A views: entry with a view: field defines a named instance of that view type: the key becomes the instance name, and the layout may reference it like any view name. This is the way to show several differently configured copies of one view side by side (e.g. last SELECT / INSERT / DDL queries):

views:
  last_selects:
    view: last_queries
    query_kind: Select
  last_inserts:
    view: last_queries
    query_kind: Insert

layout:
  panes: [last_selects, last_inserts]
  focus: last_inserts

Each instance has its own /-filter state; the (/) queries-limit keys stay global (an instance's limit: overrides it), and the F3 queries filter edits the builtin view's filter, not the focused instance's (use / in the pane instead). Instances are only instantiated via the layout - the F2 menu opens the builtin views.

See chdig_views_layout.yaml for a directly runnable example (queries, CPU flamegraph and server logs stacked in equal panes), and chdig_view_instances.yaml for the named-instances one (last SELECT / INSERT / DDL queries plus CPU and jemalloc flamegraphs), and chdig_view_columns.yaml for per-view columns (profile events and settings of the queries views, part log events).

views:
  queries:
    filter: "insert"
  last_queries:
    start: 4h
    end: 30m
  server_logs:
    limit: 1000

layout:
  direction: horizontal
  panes:
  - queries
  - direction: vertical
    ratio: 0.4
    panes:
    - last_queries
    - server_logs
  focus: queries

What is --connection?

--connection allows you to use predefined connections, that is supported by clickhouse-client (1, 2).

Here is an example in XML format:

<clickhouse>
    <connections_credentials>
        <connection>
            <name>prod</name>
            <hostname>prod</hostname>
            <user>default</user>
            <password>secret</password>
            <!-- <secure>false</secure> -->
            <!-- <skip_verify>false</skip_verify> -->
            <!-- <ca_certificate></ca_certificate> -->
            <!-- <client_certificate></client_certificate> -->
            <!-- <client_private_key></client_private_key> -->
        </connection>
    </connections_credentials>
</clickhouse>

Or in YAML:

---
connections_credentials:
  prod:
    name: prod
    hostname: prod
    user: default
    password: secret
    # secure: false
    # skip_verify: false
    # ca_certificate:
    # client_certificate:
    # client_private_key:

And later, instead of specifying --url (with password in plain-text, which is highly not recommended), you can use chdig --connection prod.

What is Perfetto export?

Pressing X in the queries view exports a timeline visualization to Perfetto UI — an open-source trace viewer that provides a zoomable timeline, flamegraph visualization, and SQL-queryable trace data. It runs entirely in the browser.

An embedded HTTP server starts on port 9001 (lazily, on first export) and serves the binary protobuf trace. The browser opens automatically.

The export includes data from multiple ClickHouse system tables (when available):

Source table	What it shows
In-memory queries	Query duration slices grouped by host/user
system.opentelemetry_span_log	Processor pipeline spans
system.trace_log (ProfileEvent)	Per-thread counter increments
system.trace_log (CPU/Real/Memory/ProfileEvent)	Stack trace samples (flamegraph in Perfetto)
system.text_log	Query log messages grouped by level
system.query_metric_log	Per-query metric snapshots
system.part_log	Part lifecycle events (NewPart, MergeParts, etc.)
system.query_thread_log	Per-thread execution with ProfileEvents

Tables that don't exist are silently skipped — the export works with whatever data is available.

When queries are selected with Space, only those queries are exported.

To get the richest traces, enable these ClickHouse settings for the queries you want to analyze:

SET
    opentelemetry_start_trace_probability = 1,
    opentelemetry_trace_processors = 1,
    opentelemetry_trace_cpu_scheduling = 1,
    log_query_threads = 1,
    trace_profile_events = 1,
    query_metric_log_interval = 0

• opentelemetry_start_trace_probability / opentelemetry_trace_processors / opentelemetry_trace_cpu_scheduling — enable OpenTelemetry spans for the query execution pipeline (populates system.opentelemetry_span_log)
• log_query_threads — log per-thread execution info (populates system.query_thread_log)
• trace_profile_events — record ProfileEvent counter increments with timestamps into system.trace_log, giving precise per-event timelines
• query_metric_log_interval — controls periodic metric snapshots in system.query_metric_log (sampled every N milliseconds). Set to 0 to disable if you prefer the more accurate trace_profile_events. Set to e.g. 1000 (1 second) if you want periodic snapshots — note that these are sampled and less precise than trace_profile_events, but lighter on overhead

What is flamegraph?

It is best to start with Brendan Gregg's site for a solid introduction to flamegraphs.

Below is a description of the various types of flamegraphs available in chdig:

• Real - Traces are captured at regular intervals (defined by query_profiler_real_time_period_ns/global_profiler_real_time_period_ns) for each thread, regardless of whether the thread is actively running on the CPU
• CPU - Traces are captured only when a thread is actively executing on the CPU, based on the interval specified in query_profiler_cpu_time_period_ns/global_profiler_cpu_time_period_ns
• Memory - Traces are captured after each memory_profiler_step/total_memory_profiler_step bytes are allocated by the query or server
• Live - Real-time visualization of what server is doing now from system.stack_trace

See also:

• Sampling Query Profiler

Note: for Memory chdig uses memory_profiler_step over memory_profiler_sample_probability, since the later is disabled by default

Why I see IO wait reported as zero?

• You should ensure that ClickHouse uses one of taskstat gathering methods:

  • procfs
  • netlink

• And also for linux 5.14 you should enable kernel.task_delayacct sysctl as well.

How to copy text from chdig

By default chdig is started with mouse mode enabled in terminal, you cannot copy with this mode enabled. But, terminals provide a way to disable it temporary by pressing some key (usually it is some combination of Alt, Shift or/and Ctrl), so you can find yours press them, and copy.

---

See also bugs list