Skip to content

README.md

Latest commit

 

History

History
452 lines (323 loc) · 64.2 KB

README.md

File metadata and controls

452 lines (323 loc) · 64.2 KB
Raw

YScope CLP Plugin

Plugin for analyzing logs with CLP (Compressed Log Processor): compress raw logs or Claude Code and Codex sessions, search them with KQL, and write an insights report.

CLP is the open-source platform for log archive storage, search, and analytics. Pre-release builds may also include licensed YScope extensions.

API Surface

Everything runs through one command, bin/clp, and of clp-s it exposes only:

  • list recent Claude Code and Codex session JSONL files.
  • compress one selected session with clp-s c --timestamp-key timestamp, or a session bundle: the session and every agent and workflow it launched.
  • compress log files from an arbitrary folder with clp-s c --remove-path-prefix FOLDER -f FILE_LIST OUTPUT_DIR.
  • search local CLP archives with KQL (including semantic("query")) and stdout results.
  • dump an archive's log shape dictionary with the stats.log_shapes query, and read its schema tree and archive statistics with stats.schema_tree and stats.archives.
  • count matching records and list a field's distinct values in the engine, with --count and --unique.
  • decompress a local CLP archive directory.

It does not expose reducers, network/file output handlers, results-cache writes, indexing, conversion, remote decompression, metadata sinks, or arbitrary clp-s option passthrough.

Skills

Skill Scope
compress Compress a session JSONL file into a CLP archive directory.
compress-folder Compress log files from an arbitrary folder into a CLP archive directory.
search Search CLP archives with KQL, including semantic("query").
analyze-logs The one entry point for log analysis. clp TARGET identifies which application wrote the logs and reports the route; the skill follows it. Specialised route (a Claude Code session): end-to-end session analysis over the bundle, checking seven lightly pre-trained areas, with optional 0–10 scoring. General route (anything else): trained on the spot, driven by the archive's log shape dictionary — dump the templates, classify them (cached), then run targeted queries derived from templates that are known to exist.
decompress Decompress a CLP archive directory for raw inspection.
codex-trajectory Same workflow for Codex session logs, plus Codex-specific query starters.

Future use-cases will add their own skill directories under skills-claude/ (or skills-codex/ if the use-case is agent-specific).

Writing guide

analyze-logs has a subagent write each report, and that writer reads writing-guide/rules.md before it starts: the rules for concision, readability, and claims and evidence, each one a test you can run over a draft. Its evidence — the sentences a reviewer rejected, quoted verbatim — is in writing-guide/examples.md. Both files are plain Markdown at the plugin root, with no build step, so editing them is how the plugin's writing standard changes; writing-guide/README.md says which file owns what.

Only the report's shape lives in skills-claude/references/report-style.md: the short form by default, the thorough form on request, an evidence tier on every claim, and the collapsed reference section its claims link into. bin/lib/report_save.py renders <details> and <summary> for that section and opens the enclosing block when a link lands inside it; every other tag in a report stays escaped.

Install

Hosted installer:

curl -fsSL https://installer.yscope.ai/coding-agent-plugin.sh | bash

Local marketplace:

claude plugin validate .
claude plugin validate ./plugins/clp
scripts/validate-codex-plugin.sh ./plugins/clp

claude plugin marketplace add "$PWD" --scope user
claude plugin install clp@yscope --scope user

codex plugin marketplace add "$PWD"
codex plugin add clp@yscope

Local plugin session:

claude --plugin-dir ./plugins/clp

Commands

The plugin is one command, bin/clp. clp <TARGET> is the front door: it analyses the logs you point it at. Everything else is a subcommand of it, one step of that flow or one drill-down after a finding, and bin/ holds nothing but clp, the engine shim clp-s (the binary the installer puts there; it is not a subcommand and nobody types it) and lib/, where every implementation lives.

clp with no arguments prints the usage and the subcommand table and exits 2. clp --help is the front door's own help, which carries that table with it, and any other first argument starting with - is the front door with no target, which lists the sessions it can see and stops. Otherwise the first argument is a subcommand when it is in the table below, and a TARGET when it holds a /, a . or a ~, names something that exists, or is a session-id UUID; anything else is error: unknown subcommand: <it> and exit 2. No subcommand name holds a /, a . or a ~, so a mistyped subcommand is reported as one, while ./search and logs/app.log are read as targets.

TARGET is a log file, a folder of log files, a CLP archive directory, a session bundle directory or a Claude Code session id.

Subcommand Scope
list-sessions list the Claude Code and Codex sessions on this machine
detect read the start of each log file and report what is there
compress <session|folder|status> compress logs into an archive, or report a running compression
search search archives with KQL, including semantic search
decompress decompress an archive directory into an output directory
schema every field of an archive, its type drift, its per-field counts
kql render a JSON filter as KQL, or check a query or a plan
session <turns|facts|score> measure and score one Claude Code session
bundle one session bundle: its catalog, and the records behind it
bundle-review a directory of bundles: what went wrong, where, how often
report <check|save> check a finished report against its inputs, and save it
bootstrap load an archive's structure and log shapes, and look up the cache
baseline-plan write a run's app-agnostic baseline queries to a plan file
extract write the overview queries and the report writer's inputs
run execute a query plan, reporting each entry as it completes
focus queue the focus queries for what the user picked
facts compute the facts a report may quote from the evidence
shape-cache the log shape classification cache: hit, growth, new shapes
shape-cluster merge semantically similar log shapes before classifying

clp <SUBCOMMAND> --help prints that subcommand's own options, and clp --help the front door's. kql, bundle, bundle-review, shape-cache and shape-cluster take a further word of their own (clp kql render, clp bundle BUNDLE show, clp shape-cache diff), which the sections below give.

Five subcommands are restricted passthroughs to clp-s, and they are the security boundary (flag allowlist, path validation, env hardening): list-sessions, compress session, compress folder, search and decompress. They prefer CLP_S_BIN, then plugin-local bin/clp-s, then plugin-local .clp-core/bin/clp-s, then clp-s on PATH. Every other subcommand reaches clp-s only through those five, or not at all.

clp-s projects leaf columns only: a column inside an array (message.content.text when message.content is an array of blocks) or an object column (message) projects nothing, although a filter on it matches. clp search therefore rewrites each --projection column from the archive's schema tree (stats.schema_tree, read from the archive's metadata in milliseconds) with bin/lib/projection.py: a column inside an array becomes the array, which comes back whole; an object becomes the leaf columns under it. Each rewrite is explained on stderr as a projection: ... line.

Each subcommand in detail, the front door first and then in table order. The two files that are not commands sit beside what uses them, and the subcommands with a section of their own below (the five passthroughs, bundle and bundle-review) are documented there.

  • clp <TARGET> — the front door to the analysis flow: it classifies a target, does the deterministic preparation, and prints the next command. It asks one question — which application produced these logs? — and then whether that application has a registered optimisation. claude-code does (its layout and its record graph are known in advance, and its seven areas are lightly pre-trained: the plugin ships them), so it takes the specialised route; every other application takes the general route, which discovers structure and categories and caches the classification. codex and vllm are recognised and still general: being registered buys them the right acquire — a Codex rollout compressed as a session so its payload arrays become queryable columns, a vLLM text log structurized — and nothing about the analysis. The application is read from the records, never from the path: an archive's from its merged schema tree (clp schema --field-counts), a file's from the first 128 KiB (clp detect, whose bundled-format name is the application name where it has one). The classification and its evidence are printed before any work starts. Preparation is idempotent: an archive is reused when its metadata records this session log's sha256, or exactly these source paths written after anything under them was last modified; a bundle is reused when its manifest records the same main log. For a Claude Code session log it looks beside the log before compressing anything: agent and workflow logs in the session's own directory (subagents/, workflows/) mean a bundle, which compresses the main log with them, and none mean one archive, so a session is compressed once, never as an archive and again as a bundle (ADJACENT_LOGS=). With no bundle it runs the launch count (message.content.name:Agent OR message.content.name:Workflow) on the archive, and a count above zero is reported as agents whose logs are missing. A file in the session's directory that no layout rule names is kept under the bundle's files/; only an unnamed .jsonl log stops the build, since its records would be missing from the analysis. --general forces the general route (the acquire stage stays the application's, so a session log is still compressed as a session); --app NAME asserts the application and is refused by records that say another; --dry-run prints the plan and writes nothing. Output is KEY=VALUE: TARGET, FORM, APP, APP_FROM, WHY, EVIDENCE, ROUTE, ROUTE_WHY, RAN (every command it ran), ARCHIVE, BUNDLE, ADJACENT_LOGS, LAUNCHES, PREPARED/ALREADY_PREPARED, NEXT, NEXT_SKILL and DRILL. It orchestrates the other subcommands and reimplements none, and it is a door into the flow rather than a lid on the toolbox — the subcommands used after a finding (clp search, clp bundle sql|evidence|who, clp schema) stay first-class, and the DRILL= lines name them for the artefacts in hand. Bundles go under --bundles-root, then CLP_S_BUNDLES_ROOT, then ${TMPDIR:-/tmp}/yscope-clp-bundles, one per session id. Unit tests: python3 -m unittest discover -s plugins/clp/tests.
  • clp detect — reads the first 128 KiB of each log file (read-only). JSON (two or more objects parsed) is reported with its structure, its timestamp field and its first records, every string cut to 128 characters; text is reported with its first lines, each cut to 256 characters, and whether they match a bundled --structurize format (vLLM). It suggests a clp compress folder command per group of files that need the same settings. --parser FILE dry-runs an agent-written parser on the lines read. See Log Files.
  • bin/lib/structurize.py — converts text logs to structured JSONL: the built-in vLLM formats, or any format through --parser FILE, a Python file defining parse_line(line). Used by clp compress folder --structurize; not called directly.
  • clp compress status — reports the state (running, done, failed, died) and progress of a clp compress folder run from the status file it keeps beside the archive directory; clp compress folder also prints a [compress] heartbeat with progress and an estimated time left (--heartbeat SECONDS).
  • clp schema — lists every field of an archive from its merged schema tree (stats.schema_tree), with no sampling: FIELD path=… type=… records=N per field, the KQL path first (an array element's fields are queried through the array, so message.content[].name is message.content.name; shown= gives the [] form). An object with many children that all look alike, such as data used as keys, is collapsed into one * row. With --log-shapes-file (a stats.log_shapes dump carrying node_counts), each text field also gets values= and templates=, and TEXT_FIELDS= lists the text fields, most values first. Two other readings of the same tree replace the field rows: --drift reports every field path the archive stores under more than one type (DRIFT path=… types=ClpString:116,Object:4525 records=4641, most records first) — the tree keeps one node per path and type, so a field that is a ClpString on a failed tool call and an Object on a success is already two nodes there, and a query written for one of the two shapes silently misses the records that carry the other; --field-counts reports the record root's children grouped by record count (FIELDS count=32614 share=72.8% fields=… subtree_nodes=…), where fields reading a co-equal count are a hint that the same records carry them — these are per-field counts and not a partition, since a record is counted under every field it carries, so the shares overlap and must never be summed, which the output says on itself in FIELD_COUNT_NOTE=. --drift-file and --field-counts-file write the same rows as NDJSON.
  • clp kql (+ bin/lib/kql_build.py) — renders a structured JSON filter (all/any/not, eq/contains/prefix/exists/gt…, scoped semantic) to KQL with every value quoted and escaped and every group parenthesized, so query plans never carry model-written KQL. clp kql render '<filter>' prints the KQL; clp kql check-plan FILE validates every query_plan entry, and its ranking (lib/classification.py): each entry's category, priority and stage, and each taxonomy category's priority and why. clp shape-cache merge/put refuse classifications that fail the same check. With --drift-file F (a clp schema --drift-file dump) it also catches a filter that will silently answer about part of its field: clp-s stores a node per type, and a scalar filter reaches the union of a path's scalar types but never its Object or StructuredArray nodes — toolUseResult:* matches 116 records, not the 4,641 carrying that path, because the other 4,525 hold an object. Each at-risk entry gets a DRIFT_RISK line with the type counts, the direction measured (at=path or at=descendant, since the reachable share differs by where the filter sits) and the addressable share; the entry fails when that share is below --drift-fail-below, default 0.95. A filter may declare "types": [...] on a leaf to say it targets one shape deliberately, which reports DRIFT_DECLARED and never fails, while a declaration naming a type the path does not hold — or naming only types a filter there cannot reach — is rejected as a plan error; there is no flag to silence the check. Without --drift-file it prints DRIFT_CHECK=SKIPPED, so a missing dump never reads as a clean pass. The direction split also catches a dead query: a filter on a subfield of a path that is never structural reaches nothing at all, and toolUseResult.stdout.lines:* matching zero records previously failed silently. clp kql validate-wildcards is clp search's guard against an unquoted wildcard value with a space (field:*a b*, which clp-s reads as natural language), and rejects the query before it runs.
  • clp session turns — where a Claude Code session's time went, per turn. A turn runs from one human prompt to the next (records the harness injects, such as task notifications, command output and compaction summaries, are not prompts), and its time splits into human wait (AskUserQuestion, ExitPlanMode), tool, model, idle (no event for --idle-minutes, default 10) and other, with no second counted twice. It reads whole records from an archive made by clp compress session, with or without --structurize-arrays, through clp search, and lists the longest tool waits. The record model follows TraceLab's Claude Code extractor (Apache-2.0): the assistant records that share a message id form one round, and a tool call is paired with its result by tool_use_id. Claude Code sessions only. Unit tests: python3 -m unittest discover -s plugins/clp/tests.
  • clp session measure — computes every number an analyze-logs specialised-route report can quote, in code, so the report writer only puts facts into words. It reads a bundle's catalog and, through clp session turns, clp bundle outcomes and clp bundle repo, writes a markdown facts file covering the skill's seven lightly pre-trained areas (reliability, cost, time, outcomes, harness faults, human loop, rework), each figure with its denominator. --axes adds the scoring inputs: 20 axes in four groups, each with its raw value and the components behind it. It assigns no scores — the agent does that against scoring-scale.json, because what counts as an acceptable stall rate or cache hit rate is a customer's policy, not a measurement. --axes --check-scale [--scale FILE] validates a scale instead, printing SCALE_OK or one SCALE_PROBLEM= per defect. Its stdout is grep-able KEY=VALUE lines, one AXIS <id> group=… value=… components=… per axis (value= is a bare number or n/a with a reason, never a percent or a separator), and one ALERT=<category>:<slug> value=… threshold=… per category whose headline metric crosses a bad threshold, which is what the skill uses to order the focus options — alerts are not scores, and their thresholds are printed so they are never a hidden judgement. A category it cannot compute (no bundle, no repository) is reported as such, never as zero. Claude Code sessions only.
  • clp session score — applies a scale to clp session measure --axes output and writes the scores as JSON for a dashboard, plus --format table for a terminal. It maps each raw value to 0–10 through the scale's ladder and computes the per-group means, and does nothing a ladder lookup does not cover: no penalties, no weighting, no narrative, and no overall composite field anywhere in its output. That division is deliberate — measuring is clp session measure, the thresholds are the customer's in the scale file, and what the scores mean stays with the agent. It resolves the scale by the same lookup order as the skill, records which one won in scale.source, and validates it first: a scale with defects gets its SCALE_PROBLEM lines printed, no JSON written, and exit 2, so a broken customer scale is never silently swapped for the default. An axis whose value is n/a is not scored — it goes to unscored with its reason and its group's mean rests on fewer axes, and a group with no scored axes has a null mean rather than 0. --cohort KEY=VALUE (repeatable) records what kind of work the session was, since a trend needs sessions grouped by similar shape; the JSON's cohort._unset lists the conventional keys left unsupplied so a consumer can tell an ungrouped session from one grouped as null. It also carries provenance — scale file and scale_version, bundle, and both tools' versions — so an old score stays interpretable after the scale moves.
  • scoring-scale.json — the default 0–10 scale the analyze-logs skill's specialised route applies to clp session measure --axes output: four groups with their owners, and per axis a label, what it measures, its unit, its direction and an exhaustive ladder. It is the only place default thresholds live. Every axis also states what its thresholds rest on: a basis of definitional (it follows from what the metric means), mechanism (from how the system works), observed (a named measurement, with its size — a single session says n=1) or judgement (a defensible choice with no measurement behind it), a rationale for where the rungs fall, and anchors for the values that pin them. Most defaults are judgement, and saying so is deliberate: it tells a customer which rungs to recalibrate first rather than implying an evidence base that does not exist. An axis may also carry a gated_by block naming a cohort key it depends on, so that an axis which is only meaningful under one intent is left unscored rather than scored misleadingly: C4 autonomy is gated on supervision (autonomous, supervised or mixed, declared in the scale's cohort_keys) and is scored only for a session meant to run unattended, since in a supervised session the same number is a design property and not a fault; D5 cost concentration is gated on workload_shape (interactive, batch, mixed) and scored only under interactive, because a batch or mixed workload can have one run dominate the bill by design. A gated-out axis lands in the scores JSON's unscored with gated: true and its reason, and its group's mean rests on one fewer axis; the raw value is still measured and reported, because the gate is a scoring policy and clp session measure never applies it. An axis may also carry a min_denominator, below which it is left unscored with insufficient_data: true and both counts, because a rate over a handful of observations is noise whatever ladder reads it — A5 runtime honesty keeps a strict ladder and instead declines to rate fewer than 10 workflow instances, where one misreport would otherwise read as 50% dishonesty. clp session measure --axes publishes each axis's denominator= where its divisor is genuinely a count of things (not where it is a duration), and the scores JSON carries it on scored axes too, so a consumer can weigh a score sitting just above its floor. The three reasons an axis goes unscored are therefore distinguishable: gated with its cohort key, insufficient_data with its counts, or neither when the value could not be measured at all. A top-level calibration object names how to replace them — clp bundle-review --json over your own sessions gives the percentiles, within a cohort, with the caveat that one or two sessions describe those sessions and not a fleet. --check-scale rejects a scale whose axes lack a basis or rationale. A customer overrides it by naming a path, or with ./.clp-scoring-scale.json or .claude/clp-scoring-scale.json, which take precedence in that order; the skill reports which scale it used and its scale_version.
  • clp report check — checks a general-route report against its inputs and prints FLAG lines for a figure absent from the facts and results table (with the two listed figures it sums to), a share never printed as a share, a count found only beside different wording, a timestamp the inputs lack, and a KQL filter on a field the archive does not have. The skill hands its FLAG lines back to the report writer, which fixes the real ones and leaves the ones the facts support.
  • clp report save — saves the finished report where the user chose, in one or more formats, each headed with the date it was generated and, when the facts give one, the logs' time span: Markdown, HTML (a self-contained page with light and dark themes), PDF (printed from that page by a headless Chrome, Chromium or Edge found on the machine: $CLP_PDF_BROWSER, PATH, the macOS app bundles, Playwright's Chromium, or on WSL the Windows browser), and a page fragment the Claude Code skill publishes to claude.ai with the Artifact tool. A folder gets log-insights-<name>-<YYYYmmdd-HHMM>.<ext>; an existing file is never overwritten. --list-formats says which formats this machine can write, so the skill offers PDF only when it can print one.
  • clp bootstrap — one-command bootstrap for the analyze-logs general route: the schema tree's fields (FIELD, TEXT_FIELDS=), value distributions for the most common scalar fields from a sample, the fields each template came from (TEMPLATE_FIELDS_FILE=), log shape dictionary dump, per-template frequencies from the counts stored in the archive, and the classification-cache probe, summarized as grep-able KEY=VALUE lines. Before the dump it reads stats.archives — a tenth of a second against the archive headers — for the templates and variables the archives actually store (ARCHIVE_LOG_SHAPES=, ARCHIVE_VARS=, one ARCHIVE_STAT per archive), states that count in its first progress line and sizes its estimate with it; and from the schema tree it already dumped it adds the type drift and the per-field record counts (TYPE_DRIFT_COUNT=, TYPE_DRIFT_FILE=, FIELD_COUNT_GROUPS=, FIELD_COUNT_NOTE=, FIELD_COUNTS_FILE=), where TYPE_DRIFT_COUNT=0 is the answer "nothing drifts" and UNAVAILABLE means there was no tree to read. --fields-only stops there: it reads the fields, the drift and the per-field counts, skips the dictionary and the classification cache, neither reading nor writing the cache, and prints FIELDS_ONLY=1 in place of the dictionary and cache keys. It is what the specialised route runs on a session bundle, where the areas are lightly pre-trained and a template is free text from one session.
  • clp baseline-plan — writes the app-agnostic baseline as its own plan (/tmp/clp-insights-baseline-plan.txt), run by its own pool while the templates are classified: it samples the schema's severity and logger values from the head of the archive, then adds a count per common value, a count for the residual (everything else), a then rule that fetches the records behind a small severity residual, and one scoped semantic entry. It never uses --unique, which scans every record.
  • clp extract — builds the general route's insight-pass inputs from a classification, joining its template hashes to the texts it streams from the archive's frequencies file: templates grouped by category (top N by frequency, truncated), the core plan (the core entries, one per line, high priority first, reporting entries without a valid match filter or ranking as QUERY_PLAN_INVALID=), the drill entries apart (/tmp/clp-insights-drill-plan.txt), and, with frequencies, the exact records per category with the classifier's priority and why (/tmp/log-shape-category-totals.json) and the top templates overall and per category with their counts and categories (/tmp/log-shape-top-templates.json). Bounded, so classifications holding huge near-duplicate templates stay fast. It also empties the focus inbox, so a new analysis starts with no focus.
  • clp run — the query pool. It executes the plan through clp search, rendering each entry's match filter to KQL with clp kql, printing each entry's result as it completes and recording the rendered KQL, count, share of records, status (ok, zero, error, timeout, non-selective), elapsed time, peak memory and samples in /tmp/clp-insights-query-results.ndjson. Searches run concurrently under memory control: the first runs alone, its peak resident memory is measured, and another starts only while free memory can take one more (sampled twice a second; a search is paused and requeued if memory runs short). An entry's then rule can add a follow-up query once its result is in, --retry-failed retries a failed entry once, and --jobs N pins the concurrency. With --inbox FILE the pool also takes entries from FILE while it runs, ahead of every plan entry not yet started, and stays open until FILE says {"close": true} (or --inbox-timeout, default 900 s, passes): the core plan starts while the user is still choosing a focus, and the focus runs next without a second pool counting the same free memory. --print-table renders the recorded results as Markdown, focus entries marked.
  • clp focus — turns the user's answer to "what should this analysis focus on?" into queries: it queues the chosen categories' drill entries, plus any entries the agent wrote from the user's own question or context (validated like plan entries), into the pool's inbox, closes it, and records the focus and the user's context verbatim in /tmp/clp-insights-focus.json. Run once per analysis, even for "everything", since it is what closes the inbox.
  • clp facts — computes every number of the insight report in code, from both the baseline's and the plan's results (the user's focus and context with the focus queries' results first, then totals, severity and logger breakdowns with check lines, the category table with priorities, its sum and gap, top templates per category, the fetched warnings and errors grouped by message shape, and the archive's time span from its metadata's timeRange) into /tmp/clp-insights-facts.md, so the report writer only puts them into words.
  • clp shape-cache — persistent cache of the analyze-logs general route's classification, with incremental update when an archive grows. See Shape Cache. Its fields subcommand writes, per template hash, the fields its values came from, from a dump with node_counts and the archive's schema tree.
  • clp shape-cluster (+ bin/lib/shape_cluster.py) — groups semantically similar log shapes using embeddings from the semantic server so the LLM classifies one representative per cluster. See Shape Cluster.

Session Workflow

List recent sessions:

./plugins/clp/bin/clp list-sessions

Defaults:

  • sources: Claude ~/.claude/projects and Codex ${CODEX_HOME:-~/.codex}/sessions.
  • limit: latest 5, sorted by session file mtime descending.
  • Claude subagents: excluded unless --include-subagents.
  • manifest: written to /tmp.

When presenting choices, always include IDX, AGENT, modified timestamp, raw bytes, human size, session name, project/cwd, and session ID.

Check archive root:

./plugins/clp/bin/clp compress session --show-archives-root

Default archive root is ${TMPDIR:-/tmp}/yscope-clp-archives. Use it without asking. Ask only when the user wants persistent storage or a different root.

Compress the selected row:

./plugins/clp/bin/clp compress session \
  --selection-file /tmp/clp-s-session-selection-...tsv \
  --session-index 1 \
  --timestamp-key timestamp

After compression, report these lines:

  • Raw input bytes
  • Archive bytes
  • Compression ratio
  • File size reduction
  • Time range (with a timestamp key: the earliest and latest timestamp across every record)
  • Archives dir
  • Selected session
  • Archive metadata

Sessions are always compressed with --structurize-arrays. A session record keeps its prompts, replies, tool calls and tool output in the message.content array; structured, each element's fields become schema-tree columns, so their text is templated into log shapes and each field can be queried and counted. Unstructured, the array is one opaque string. On one 97 MB session this also made the archive smaller (10.6× against 9.1×) and raised the log shape count from 5,292 to 25,524.

Use the printed top-level Archives dir for search and decompression. clp search and clp decompress resolve the inner clp-s archive directory automatically. Metadata in .yscope-clp-archive.json maps archive to session file, agent, roots, timestamp key, time range (timeRange, from clp-s --print-archive-stats, the same as clp compress folder records), SHA-256, compression stats, command, and resolved inner archive.

Log Files

Compressing log files is two steps with a decision between them. First, see what each file holds (read-only; it reads the first 128 KiB of each file, so it is instant even on many-GB logs):

./plugins/clp/bin/clp detect /var/log/myapp /data/vllm_worker_3.log

It says whether each path is a file or a folder. Per file, if two or more JSON objects parse from the start, it is JSON: the report lists every field path with its type, the field holding a timestamp in every record and the kind of value, and the first records with every string cut to 128 characters (still valid JSON). Otherwise it is text: the report shows the first 20 lines, each cut to 256 characters, and says whether they match a bundled format that --structurize converts on its own (vLLM). For text without one, the agent works out the structure from those lines and writes a parser (below). The report ends with a SUGGEST command per group of files that need the same settings and SKIP lines for files that can't go in as they are. Then compress with the flags chosen from that report:

./plugins/clp/bin/clp compress folder --path /var/log/myapp --timestamp-key ts
./plugins/clp/bin/clp compress folder --path /data/vllm_worker_3.log --structurize

--path takes a log file or a folder and can be repeated; the files found in all of them go into one list, which clp-s c -f compresses into one archive.

Defaults:

  • extensions (for folders): log,jsonl,json,txt,ndjson,out,err (override with --extensions, or use --extensions '*' to include every regular file). A file named with --path is always included.
  • jsonl detection: on. A file whose first lines are JSON objects but whose name is not .json/.jsonl/.ndjson is handed to clp-s as a staged *.jsonl copy, because clp-s picks its parser by file name.
  • recursive: yes (use --no-recursive for the top level of each folder only).
  • structurize: off (pass --structurize for text logs — see below).
  • timestamp key: none (pass the field clp detect reports as --timestamp-key KEY; required for time-range search).
  • archive root: ${TMPDIR:-/tmp}/yscope-clp-archives (override per-run with --archives-root DIR). Ask only when the user wants persistent storage or a different root.
  • --dry-run is read-only: it prints the plan and converts, stages and compresses nothing.

Text logs

clp-s only ingests JSON, so text logs are converted first. --structurize runs each text file through bin/lib/structurize.py, producing JSONL with timestamp/logger/level/message and setting --timestamp-key timestamp automatically. A file that is already JSON is compressed as it is, never converted:

./plugins/clp/bin/clp compress folder --path /var/log/vllm --structurize

structurize.py knows the vLLM formats (sflow-wrapped vllm_worker lines and raw vllm serve output). For any other text format, write a Python file that defines parse_line(line): it returns a dict with at least timestamp (ISO 8601 or epoch) and message for a line that starts a record, or None for a line that continues the previous one. Test it on the lines the detector reads, then compress with it:

./plugins/clp/bin/clp detect --parser /tmp/myapp_parser.py /var/log/myapp/app.log
./plugins/clp/bin/clp compress folder --path /var/log/myapp/app.log --structurize --parser /tmp/myapp_parser.py

Each converted file prints [structurize] <file>: N records; a file that can't be converted is skipped with a warning that gives the reason.

After compression, report:

  • Raw input bytes
  • Archive bytes
  • Compression ratio
  • File size reduction
  • Input files
  • Archives dir
  • Archive metadata

The resulting archive is compatible with clp search and clp decompress. Use the printed top-level Archives dir for search and decompression. Metadata in .yscope-clp-archive.json records the input paths and their common directory, extensions, file count, parser, timestamp key, time range (timeRange: the earliest and latest timestamp across every record, from clp-s --print-archive-stats, when compressed with --timestamp-key), compression stats, command, and resolved inner archive.

Useful commands:

./plugins/clp/bin/clp compress folder --show-archives-root
./plugins/clp/bin/clp compress folder --set-archives-root ~/clp-archives
./plugins/clp/bin/clp compress folder --path /var/log/myapp --dry-run
./plugins/clp/bin/clp compress folder --path ./logs --extensions log,txt

Search

./plugins/clp/bin/clp search /tmp/session-archive 'level:error'

Allowed controls: --tge, --tle, --ignore-case, --archive-id, --with-archive, --projection, --semantic-endpoint, --semantic-top-k, --semantic-threshold, --embedding-batch-size.

The path may hold several archives (for example a bundle of logs compressed into one --output-dir): each is searched in turn and the results are concatenated. --count and --unique rows carry archive_id, --limit caps the total, --archive-id picks one archive, and --with-archive wraps each record row as {"archive_id":…,"record":{…}} so a hit says which archive it came from (the record itself is untouched).

Use single quotes around KQL in shell commands. Numeric comparisons use infix syntax, for example durationMs >= 30000.

Session Bundles

A bundle is one session's logs kept as CLP archives, with a manifest and a SQLite catalog: archives/ (one clp-s archives dir, one archive per kind of log: main log, direct-agent transcripts, workflow-agent transcripts, workflow journals, workflow summaries), files/ (every other file of the session, such as agent meta files, tool results, file snapshots and tasks, at its path relative to the session), manifest.json (every source file: kind, size, hash, and where it went, down to the positions of its records in its archive) and catalog.sqlite (what exists and how it connects: agents, workflow runs and their resumed instances, retried attempts with a failure cause, timing, and one row per record). The catalog is built from the bundle alone: each archive is read once, in its original order, using the position clp-s keeps for each record, so clp bundle BUNDLE rebuild makes it again without the session's original files. It names records by the IDs they carry (agentId, runId, uuid) and holds no text. It also holds token usage as the API reported it, each model response counted once: totals for the main thread, every agent and attempt, and every workflow instance and run (tokens_* on nodes), and per call on the record that carries each response's final usage (tokens_* on events; tokens_input is the context sent with that call). A workflow run's own figure, which is closer to its final context size, is kept apart as attrs.runtime_tokens. clp session turns reports the main thread's tokens in total and per turn, and with --calls one line per call.

clp bundle build makes one from a Claude Code session. Every file of the session is classified by its path, and one that matches no rule stops the build; so does a line that is not a JSON record, or a last record cut off mid-write (for a session still running, build it once it has stopped), each named by file and line. NUL bytes, which lost writes leave in real session logs and which clp-s cannot read, are removed before compression (a raw NUL byte is never part of valid JSON); each file's count is printed as REPAIRED and kept in the catalog, and the source files are never changed. The build checks that each archive holds exactly the records its files contribute, and removes what it made if anything fails. It refuses an existing directory (--force replaces an existing bundle only). A session with no subagents or workflows gets a catalog with only its main thread. A uuid can occur twice, since Claude Code rewrites some records with the same uuid when a session is reopened; who --uuid and record show every copy. A catalog of an older layout is refused with the command that rebuilds it. clp bundle moves between the catalog and the archives:

./plugins/clp/bin/clp bundle BUNDLE build --session-id ed54042e-…      # make BUNDLE from a session under ~/.claude (or --session-file PATH.jsonl)
./plugins/clp/bin/clp bundle BUNDLE rebuild                            # make catalog.sqlite again from the bundle alone
./plugins/clp/bin/clp bundle BUNDLE show a1b2c3d4         # the catalog's row: status, cause, time, parents, unit, archive and query
./plugins/clp/bin/clp bundle BUNDLE evidence a1b2c3d4     # its records from the archive, sorted by time (--tail N, --all, --raw)
./plugins/clp/bin/clp bundle BUNDLE who --at 2026-08-25T17:15          # what was running then (UTC)
./plugins/clp/bin/clp bundle BUNDLE who --tool-use-id toolu_…          # the node a launch created
./plugins/clp/bin/clp bundle BUNDLE who --uuid 92b5ed72-…              # a record's event, and the nodes it points at (the agent it is in, what it launched or reports on)
./plugins/clp/bin/clp bundle BUNDLE events --agent a1b2c3d4 --tool Bash   # thin rows for records: time, tools called, error/interrupt flags, turn; filter by agent, turn, tool, errors, interrupts, type, time
./plugins/clp/bin/clp bundle BUNDLE record 92b5ed72-…                  # one event's full record, read from its archive by uuid
./plugins/clp/bin/clp bundle BUNDLE context 92b5ed72-… --before 10 --after 5   # the records around it in its own log and order, uuid-less records included
./plugins/clp/bin/clp bundle BUNDLE outcomes [--by agent]                # per turn or agent: files changed, commits, PRs and test runs (run vs confirmed by their output)
./plugins/clp/bin/clp bundle BUNDLE repo [--repo PATH] [--no-github]    # which commits (git, all refs and reflogs) and PRs (GitHub) each command actually made, matched by time
./plugins/clp/bin/clp bundle BUNDLE sql "select cause, count(*) from nodes group by 1"   # read-only

The catalog's events table has one row per user or assistant record and per record that refers to an agent or task (the completion notifications), with the tools called inside it in event_tools, and no text; a record is found by its uuid. Hook and reminder attachments, system rows, records without a uuid and journal rows are counted in the bundle table (events_unlisted_*, events_skipped_*), not listed. That makes main-thread records (which carry no agent) and single tool calls visible to SQL: select … from nodes n join events e on e.agent_id = n.agent_id ….

clp bundle-review DIR --build [--skip ID] [--json] reviews many sessions at once, for someone looking at the harness rather than one task: it bundles every Claude Code session under a Claude home into DIR (building missing or changed ones, rebuilding catalogs of an older layout, skipping a session still being written), computes a signal vector per session, and reports zero-tolerance findings (damaged logs, tool calls without a result, workflow runs reported completed with failed attempts, resumes re-running finished work, configuration errors, ...) with example ids, rate signals against the median and 90th percentile of the sessions active enough to rate, the sessions flagged most often, and failed tool calls grouped across sessions. Groups start from CLP's own templates (shape(toolUseResult)); command failures are then merged by a shared beginning and end, and other messages by embeddings from the plugin's built-in semantic endpoint (their template text is sent to it; --semantic-endpoint names another), and an unreachable endpoint is an error. --trend week|day adds the signals per ISO week or day, summed over all sessions by each event's own time, with volume, the number of sessions behind each value and a 95% interval for rates; a change is flagged only between periods with enough volume whose intervals do not overlap.

show and evidence take a node id, an agent id (or a unique prefix of six or more characters), a run id or a task id; --json gives show, who and sql as JSON.

Semantic Search

./plugins/clp/bin/clp search /tmp/session-archive 'semantic("slow database queries")'

Semantic search finds log events whose log shape is semantically similar to a natural language query, even when exact keywords differ. Use semantic("query") in KQL and combine with regular KQL using AND, e.g. 'semantic("errors") AND level:error'.

Semantic search requires an embedding server that is already running. The plugin never starts one — no Docker container is spun up and no embedding model is downloaded locally. clp search health-checks the endpoint before running a semantic search; if it is unavailable, the search fails with a clear error.

Endpoint resolution, highest precedence first:

  1. --semantic-endpoint URL (inline)
  2. CLP_SEMANTIC_ENDPOINT
  3. the semantic-endpoint config file — ~/.config/yscope-clp-plugin/semantic-endpoint, one URL per line, blank lines and #comments ignored (override the path with CLP_SEMANTIC_ENDPOINT_FILE)
  4. the built-in remote endpoint — https://ca-central-semantic-cache.yscope.ai, used if it passes the health check

An endpoint named by 1–3 that fails its health check is a hard error: clp search will not silently fall back to a different host, since that would send log text somewhere the user did not choose. Only the built-in defaults in 4 are probed and skipped on failure. A server you host yourself (including one on localhost) must be named explicitly; it is not auto-detected.

# Pin an endpoint once, for every subcommand and session:
echo 'https://embeddings.internal.example.com' \
  > ~/.config/yscope-clp-plugin/semantic-endpoint

URLs must be HTTPS, a localhost/loopback address, or a *.yscope.ai host. The same resolution drives clp shape-cluster (see below), so one setting covers both semantic search and log shape clustering.

Other semantic flags: --semantic-top-k K (default 5), --semantic-threshold T (default 0.3, range 0.0-1.0), and --embedding-batch-size N (default auto).

Decompress

./plugins/clp/bin/clp decompress \
  /tmp/session-archive \
  /tmp/session-archive-decompressed

Log Insights

analyze-logs's general route analyzes an archive by first dumping its log shape dictionary — the complete vocabulary of distinct message templates, with variables replaced by <*>:

# stats.log_shapes (shapes API, clp-core >= 0.13) dumps the dictionary as raw
# {"archive_id","count","id","shape"} lines. Shape strings encode variables in an
# archive-dependent form (raw placeholder bytes on regular archives,
# %rule.name% TextShape placeholders on clpp archives) — `clp shape-cache
# normalize` detects the encoding per line and renders both to the canonical
# {"log_shape":"...<*>..."} NDJSON.
# The legacy `stats.logtypes` spelling is rejected: shapes-API binaries silently return nothing for it.
./plugins/clp/bin/clp search /tmp/archive 'stats.log_shapes' 2>/dev/null \
  | ./plugins/clp/bin/clp shape-cache normalize > /tmp/log-shapes.ndjson
python3 -c 'import sys; print(sum(1 for line in open(sys.argv[1]) if line.strip()))' \
  /tmp/log-shapes.ndjson

This reads the dictionary rather than every record, so it is cheap regardless of archive size: a run with millions of records typically has tens to a few hundred templates. Every subsequent query is derived from a template that is known to exist, instead of guessing keywords that may not appear at all.

The skill is app-agnostic — it reads the fields (timestamp/severity/logger/message candidates and every text field) from the archive's merged schema tree, so it works on structurized text archives and native-JSON archives alike.

The skill's mechanical preamble is packaged as one subcommand:

./plugins/clp/bin/clp bootstrap /tmp/archive

It samples records for schema discovery, prints per-field value distributions, dumps + normalizes the dictionary, sums the per-template counts that clp-s stored at compression time, probes the classification cache, and prints a grep-able KEY=VALUE summary (LOG_SHAPE_COUNT=, FREQS=, CACHE_MODE=, CACHE_REASON=, TO_CLASSIFY=, MAX_CHARS=, output-file paths). The cache probe is given the field rules, which are part of its key: --field-rules F, or the rules file an earlier run of the same --out-dir left behind (RULES_SOURCE=, RULES_FILE=, RULES_DIGEST=). The rules are derived after the bootstrap, so a first run has none to probe with and CACHE_REASON=rules-changed is how a later run says the cached classification was built under other rules — a re-classification, not a first run. The first time it sees an archive it renders the dictionary once (clp shape-cache ingest) and stores each template's hash, count, length and first MAX_CHARS characters in the cache database; a later run on the same archive (SHAPES_SOURCE=stored) reads them back and skips the dump. It first prints an estimate from the archive's size (ESTIMATE_SECONDS=: about 1 minute per 300 MiB of archive on a first run, about the sample's time after), then a [bootstrap] line as each of its three stages starts and ends, a heartbeat every 30 s while one runs (--heartbeat), and the measured BOOTSTRAP_TIMINGS at the end. FREQS=UNAVAILABLE means an archive was compressed before clp-s stored those counts; recompress it to get frequencies.

Note that message:term is an exact match against the whole field value, same as field:term on any field, so it correctly returns 0 unless a message equals exactly term — the message field being stored as a CLP-string doesn't change that. Exact match is faster, so prefer it whenever you know a field's full value; wildcard only for a substring match — message:"*term*" works and returns real hits, and message content almost always needs it, since it's free text. semantic("…") also searches the log shapes directly and is a good complement to wildcard search for concept-shaped questions.

Shape Cache

Classifying templates into categories and deriving a query plan is the expensive step, and it is a property of the application, not the individual capture — the same build emits the same templates every run. clp shape-cache persists that classification, keyed by sha256 of the sorted distinct log shape strings capped at a character limit (default 500, --max-chars / $CLP_LOG_SHAPE_MAX_CHARS) and de-duplicated — the same treatment the templates get before they are embedded, so the key fingerprints the embedded vocabulary. The placeholder-rendered form is hashed, so fingerprints are stable across binary generations. (Consequence of the limit: a template whose tail changes beyond it does not change the fingerprint, so it does not register as growth.)

The applied field rules are in the key too, as a canonical digest folded into it (--field-rules on key and diff; rules-digest prints the digest alone, and clp shape-cluster prints the same value as RULES_DIGEST=, from the same shared helper). A ruled template takes its category from the rule and is never embedded or shown to the classifier, so the rules are part of the classification: changing a rule, adding or dropping one, or re-running fields --propose-rules with another --free-text-ratio re-keys the cache instead of silently reusing categories no rule would give now. The digest is canonical — sorted by field path, only each rule's field and category, so re-ordering the file or dropping the ratios and counts --propose-rules annotates it with changes nothing — and having no rules at all is itself a rule set, with a stable digest of its own.

The cache is one SQLite database, cache.sqlite in the cache directory. Each entry holds the schema, taxonomy and query plan, and one row per template: its hash (sha256 of the full template), its prefix_hash (sha256 of the first --max-chars characters) and its category. Templates are stored by hash, never by text: some apps log templates of hundreds of KB (CockroachDB's Pebble stats tables average ~184 KB), and an entry that held their text grew to 2.28 GB, which every lookup parsed in full. That entry is 2.2 MB as rows, and looking it up, storing it or merging into it each takes well under a second. The text stays in the archive's own dictionary dump; clp extract joins it on the hash. The shared hashing lives in bin/lib/log_shapes.py.

The same database stores what each analyzed archive's dictionary holds, in the archives and archive_shapes tables: per template its hash, its count, its full length and its first --max-chars characters, which is all the fingerprint, the prefix hashes and a report's truncated templates read. ingest renders a stats.log_shapes dump once, in one streaming pass, and stores it; the bootstrap does this the first time it sees an archive. An archive never changes, so a later run on the same archive reads the rows back (freqs --archive-ids, diff --archive-ids) instead of dumping the dictionary again. On the 357 MiB CockroachDB archive, 11,558 templates totalling 2.4 GB take 14 MB of rows; the first bootstrap takes 63 s and a later one 15 s (the dump and two rendering passes took 190 s before). diff --archive-ids answers UPTODATE only: when templates need classifying, their full text is not stored, so it exits 3 and the bootstrap dumps the dictionary. An archive nobody has analyzed for 30 days is dropped on the next ingest.

CLP=./plugins/clp/bin/clp
# Dump the dictionary and render it to canonical log shape NDJSON in one pipe:
"$CLP" search /tmp/archive 'stats.log_shapes' 2>/dev/null \
  | "$CLP" shape-cache normalize > /tmp/log-shapes.ndjson
"$CLP" shape-cache count --log-shapes-file /tmp/log-shapes.ndjson   # distinct templates
"$CLP" search /tmp/archive 'stats.log_shapes' 2>/dev/null \
  | "$CLP" shape-cache freqs             # {"count":N,"log_shape":...}, most frequent first
"$CLP" shape-cache diff  --log-shapes-file /tmp/log-shapes.ndjson \
      --field-rules /tmp/log-shape-field-rules.json                 # UPTODATE | GROWTH | NEW + a reason
"$CLP" shape-cache rules-digest --field-rules /tmp/log-shape-field-rules.json  # the digest the key folds in
# Render a dump once and store its archives; then read them back with no dump:
"$CLP" search /tmp/archive 'stats.log_shapes' 2>/dev/null \
  | "$CLP" shape-cache ingest --log-shapes-out /tmp/log-shapes.ndjson  # ARCHIVE_IDS=, LOG_SHAPE_COUNT=, COUNTS=
"$CLP" shape-cache stored --archive-ids <ID>         # exit 0 when stored with counts
"$CLP" shape-cache freqs  --archive-ids <ID>         # {"count","hash","length","log_shape":<prefix>}
"$CLP" shape-cache diff   --archive-ids <ID>         # UPTODATE, or exit 3 when the full text is needed
"$CLP" shape-cache forget <APP_KEY> --archive-ids <ID>   # delete a classification and/or stored archives
"$CLP" shape-cache list                              # cached entries + lineage
"$CLP" shape-cache show <APP_KEY>

(key, count, and diff also accept a raw stats.log_shapes dump directly — lines carrying a shape field are rendered on the fly — but store and pass around the normalized form so every tool sees identical strings.)

diff prints one tab-separated header line, followed by NDJSON {"log_shape":"…"} lines for the templates that still need classifying:

Header Meaning
UPTODATE\t<app_key>\t<count>\tup-to-date Fingerprint unchanged — reuse the cached classification, nothing to classify. A template that differs from a cached one only past the character limit takes its category through the shared prefix hash.
GROWTH\t<app_key>\t<base_key>\t<count>\t<new_count>\ttemplates-grown Archive grew from <base_key> — only the <new_count> new templates follow and need classifying.
NEW\t<app_key>\t<count>\tfirst-run No compatible base — all <count> templates follow.
NEW\t<app_key>\t<count>\trules-changed The same, but an entry does hold these templates under other field rules: the rule set changed, so the app is classified again. Not a first run — the bootstrap passes this through as CACHE_REASON= so the skill can say which.

The last field of every header is that reason. <count> is always the TRUE full template count, not the de-duplicated one. The GROWTH subset test is one indexed query over the prefix hashes, so a change that only affects bytes past the character limit reports UPTODATE rather than GROWTH.

On GROWTH the new classification is merged into the base entry with merge --base-key BK (templates by hash, taxonomy and query plan unioned; grown_from records the lineage), so a growing archive only ever costs the classification of its newly-added templates. merge prints the result without writing anything, so the insight pass reads it at once; put --key NK stores it, in one transaction, and can run in the background because readers see either the old entry or the new one.

Every stored classification is ranked: each taxonomy category carries a priority (high, medium, low) and a one-line why, and each query plan entry a structured match filter rather than a KQL string (see clp kql), its category, a priority, and a stage — core runs on every analysis, drill only when the user focuses on its category. merge and put refuse a classification that fails this check and write nothing. An entry stored before classifications were ranked is never reused: diff warns and treats it as absent, the app is classified again, and put replaces it.

Files of the old format (one <APP_KEY>.json file per entry, with template text) are ignored, and diff and list say so; they hold no ranking, so delete them.

The database carries a format version, and entries written in an older one are discarded, not migrated: an entry keyed before the field rules were part of the key cannot be told apart from one classified with no rules, and nothing records which rules produced it. The first command to open such a cache says how many classifications it dropped, in which database, and that those apps are classified again on their next run, and leaves a snapshot of the old database beside it (cache.sqlite.format1.<timestamp>). The stored archives survive, so nothing is dumped again for it.

Cache location: ~/.config/yscope-clp-plugin/log-shape-cache/, overridable with $CLP_LOG_SHAPE_CACHE_DIR, or per-command with --cache-dir on the subcommands that read or write the cache (freqs, ingest, stored, diff, get, merge, put, forget, list, show). normalize, count, key and rules-digest only transform/hash the input and do not accept it. --max-chars (default 500, or $CLP_LOG_SHAPE_MAX_CHARS) is accepted by the subcommands that compute or stamp the fingerprint or cut the stored prefixes (key, ingest, stored, diff, put); it must match the limit given to clp shape-cluster, or embedding and cache fingerprints diverge. --field-rules is accepted by key, diff and rules-digest, and must be the rules file clp shape-cluster is given, for the same reason.

Shape Cluster

Classification cost scales with the number of templates the LLM must label. clp shape-cluster shrinks that two ways: it truncates each template to a character limit and de-duplicates the results, so identical prefixes are only embedded once, then embeds the distinct texts through the semantic server's /v1/embeddings endpoint and greedily groups them at a cosine-similarity threshold, so the LLM classifies one representative per cluster (by cluster id) and expand propagates the category to every member mechanically, writing each member as its hashes — exact, because the LLM never echoes log shape strings. Representatives and members are always the FULL templates; truncation applies only to what is embedded.

Embeddings come from the same already-running server that powers semantic search. Nothing is installed, downloaded, or started locally — the clustering is pure Python standard library, with no third-party dependency.

CLP=./plugins/clp/bin/clp
"$CLP" shape-cluster cluster --max-chars 500 --input /tmp/log-shapes-to-classify.ndjson
"$CLP" shape-cluster expand --clusters /tmp/log-shape-clusters.json \
  --classification /tmp/log-shape-class.json     # id-based assignments from the LLM

cluster prints CLUSTERS=, TEMPLATES= (full count), EMBEDDED= (distinct truncated texts actually sent), and MAX_CHARS=.

Field rules cut the work further when the archive counts log shapes per field (a node_counts dump; see clp shape-cache fields). fields summarizes each text field, and a rules file gives whole fields one category, such as source code, diffs or agent reasoning:

"$CLP" shape-cluster fields --template-fields /tmp/log-shape-template-fields.ndjson \
  --freqs-file /tmp/log-shape-freqs.ndjson        # one line per text field
"$CLP" shape-cluster cluster --max-chars 500 --input /tmp/log-shapes-to-classify.ndjson \
  --template-fields /tmp/log-shape-template-fields.ndjson \
  --field-rules /tmp/log-shape-field-rules.json   # {"field_rules": [{"field", "category"}]}

A template whose values sit in ruled fields (at least 90% of them; --rule-share) takes the category of the ruled field holding most of them and is neither embedded nor clustered (FIELD_RULED=); expand adds it back by hash and refuses a rule whose category is missing from the classifier's taxonomy (on GROWTH, --categories-from names the base classification, whose categories count too, since the classifier lists only the ones it adds). On one 44,818-record Claude Code session, rules on 24 fields assigned 17,205 of 25,483 templates, and the remaining 8,278 formed 1,543 clusters instead of 4,182.

fields --propose-rules FILE derives those rules instead of reading them off a summary, from the one mechanical test that decides it: a field whose template count approaches its value count is free text, where every value became its own template, so clustering groups nothing and embedding buys nothing. It proposes a rule for each field whose ratio is at least --free-text-ratio (0.8), or that has at least --free-text-templates (500) templates at a ratio of at least --free-text-floor (0.5) — the second test catching diff lines, which repeat enough to sit at 0.68 while plainly being free text — and never for a field with fewer than --free-text-min-templates (5) templates, since a rule saves nothing there and JSON that uses data as keys makes a one-template field per datum. Each rule carries proposed_by, the ratio and the counts, so the analyst sees what is about to be ruled and can drop or rename it; the category is derived from the field and has to be added to the taxonomy like any other. Nothing is applied — the file is written and a {"summary": …} line measures what it would rule. On that same session it proposed 39 rules covering 24,604 of 25,483 templates, leaving 879 to embed in 388 clusters, against the 8,278 the hand-picked rules left.

  • Endpoint: --semantic-endpoint, then $CLP_SEMANTIC_ENDPOINT, then the semantic-endpoint config file, then the built-in remote endpoint — the same chain as Semantic search. The launcher resolves and health-checks it, then passes it down.
  • Model contract: BAAI/bge-base-en-v1.5, int8[768] — matching what clp-s advertises, so both hit the same server-side cache. Threshold: cosine 0.80 (override with --threshold or $CLP_LOG_CLUSTER_THRESHOLD; raise to 0.85–0.90 to split more, lower to merge more). --batch-size sets texts per request (default 256); --max-request-bytes ($CLP_LOG_SHAPE_MAX_REQUEST_BYTES, default 100 000 000) caps the encoded size of any single request body.
  • Truncation: --max-chars ($CLP_LOG_SHAPE_MAX_CHARS, default 500) caps each template at that many UTF-8 characters before embedding; it must match the value used by clp shape-cache, which fingerprints the same truncated set (both use lib/log_shapes.py).
  • expand is stdlib-only and fully offline — it needs no endpoint — and validates that every cluster id is assigned exactly once before writing anything (exit 2 otherwise), which protects the log shape cache from partial classifications.
  • Exit codes for cluster: 0 ok, 1 input problem, 2 the embedding server is unreachable/rejected, 3 usage error. --help and expand never touch the network.
  • setup has been removed; it now exits 2 with a pointer to the endpoint settings. Existing venvs under ~/.config/yscope-clp-plugin/venvs/log-shape-cluster are no longer used and can be deleted.

Query Starters

For session-log analysis (which tools fired, what failed, how long a turn took, what context was used), see the per-use-case trajectory skills:

  • Claude Code: analyze-logs skill, specialised route (in the installed plugin)
  • Codex: codex-trajectory skill (in the installed plugin)

For harness/test/patch failures and Docker/resource issues, see the Trajectory sections in those skills — both have a "Query Starters" table covering SWE-bench runs, test failures, patch failures, and Docker issues.

For semantic search suggestions, see the Semantic Search section in the search skill.

For broad trajectory debugging, suggest a subagent when available. Ask it to run the query sequence and return only the archive path, queries, top findings, and next useful queries.