Skip to content

feat(openclaw): add event_t JSONL emission for pilot integration#47

Open
fangxiu-wf wants to merge 1 commit into
alibaba:mainfrom
fangxiu-wf:fangxiu/openclaw-add-jsonl-emission
Open

feat(openclaw): add event_t JSONL emission for pilot integration#47
fangxiu-wf wants to merge 1 commit into
alibaba:mainfrom
fangxiu-wf:fangxiu/openclaw-add-jsonl-emission

Conversation

@fangxiu-wf

Copy link
Copy Markdown
Collaborator

Summary

Add an event-level JSONL output channel to opentelemetry-instrumentation-openclaw so loongsuite-pilot can ingest each LLM/tool event as one line of event_t schema and fan it out to SLS / JSONL / HTTP sinks. The two paths (OTLP trace + JSONL) are independent.

Companion CR: pilot side at sls/loongsuite-pilot CR #82 — both should be merged together.

Modes

endpoint log_enabled Behavior
set unset / false OTLP-only (existing default)
unset true JSONL-only (no OTLP)
set true Dual-mode (both run)
both unset both unset Plugin refuses to activate

Shared config: ~/.openclaw/otel-config.json

New file (separate from ~/.openclaw/openclaw.json), used as the integration contract with pilot:

{
  "log_enabled": true,
  "log_dir": "~/.loongsuite-pilot/logs/openclaw",
  "log_filename_format": "hook",
  "captureMessageContent": false
}

Plugin-config priority for any field: openclaw.json plugins.entries.config > ~/.openclaw/otel-config.json > env (OPENCLAW_LOG_ENABLED / OPENCLAW_LOG_DIR / OPENCLAW_CAPTURE_MESSAGE_CONTENT / OPENCLAW_TELEMETRY_DEBUG) > default.

What changes

File Change
src/jsonl-emitter.ts new — JSONL writer + 4 event_t record builders
src/jsonl-hooks.ts new — self-contained hook listeners, decoupled from OTLP trace path
src/types.ts Extend ArmsTraceConfig (4 new optional fields)
src/index.ts Activate-time config merge from shared otel-config.json; allow JSONL-only mode without OTLP endpoint; register JSONL hooks alongside (dual-mode) or instead of (JSONL-only) the existing trace path
test/jsonl-emitter.test.ts new — 14 unit cases (builders, emitter, write failure, shared config)
test/integration-jsonl.test.ts new — 4 E2E cases (one turn with tool call, double-fire dedup, captureMessageContent toggle, multi-turn step.id)
README.md New "Event-level JSONL Output" section
CHANGELOG.md 0.1.4-beta entry
package.json Bump version to 0.1.4-beta

Output schema

<log_dir>/openclaw-YYYY-MM-DD.jsonl, one JSON object per line. Required fields per record: time_unix_nano, event.id, event.name{llm.request, llm.response, tool.call, tool.result}, session.id, user.id, agent.type (always "openclaw"), turn.id, step.id. Optional fields per event type carry provider/model/usage/tool/error info. Filename matches pilot BaseHookInput's hook filename format.

Why a separate hook module?

The OTLP trace path (activate() body in src/index.ts) is large and stateful (per-channel TraceContext, span lifecycles, propagation). Adding JSONL emission inline would either entangle the two paths or require null-guarding every OTLP call site. Instead, registerJsonlHooks() registers its own listeners with minimal per-runId state. In dual-mode, both listener sets fire on the same hook events independently. Write failures are logged at warn-level and never propagate to the gateway.

Test plan

  • npm test — all 129 existing tests still pass + 18 new tests pass
  • npm run build — typecheck clean
  • V1 schema validation: 4 valid event_t records produced from a one-turn synthetic run, all required fields present, agent.type=openclaw, usage tokens preserved
  • Reviewer: spot-check transformRecord field mapping correctness in pilot CR
  • Reviewer: Run on a real OpenClaw gateway with both OTLP + JSONL enabled and confirm trace path behavior unchanged

Backward compat

  • OTLP-only mode (existing default) is unchanged: if log_enabled is unset, JSONL hooks are not registered, and the OTLP path runs identically to before.
  • Existing config files / env vars are read in the same priority order; only the new fields are added.

🤖 Generated with Claude Code

Add an event-level JSONL output channel alongside the existing OTLP trace
exporter so loongsuite-pilot can ingest each LLM/tool event as one line of
event_t schema and fan it out to SLS / JSONL / HTTP sinks.

The two paths are independent:
- endpoint set, log_enabled unset/false  -> OTLP-only (existing default)
- endpoint unset, log_enabled true       -> JSONL-only (no OTLP)
- endpoint set, log_enabled true         -> dual-mode (both paths run)
- both unset                             -> plugin refuses to activate

Plugin reads ~/.openclaw/otel-config.json (new shared config file, separate
from openclaw.json) for log_enabled / log_dir / log_filename_format /
captureMessageContent. Plugin-config priority: openclaw.json plugins.entries
> otel-config.json > env (OPENCLAW_LOG_ENABLED / OPENCLAW_LOG_DIR /
OPENCLAW_CAPTURE_MESSAGE_CONTENT / OPENCLAW_TELEMETRY_DEBUG) > default.

Output: <log_dir>/openclaw-YYYY-MM-DD.jsonl, appended via fs.appendFileSync,
matching pilot BaseHookInput's hook filename format. Each line carries
required event_t fields: time_unix_nano, event.id, event.name, session.id,
user.id, agent.type ("openclaw"), turn.id, step.id; plus event-specific
fields (gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.*_tokens,
gen_ai.tool.*, response.finish_reasons, error.*).

JSONL listeners are registered in a self-contained module (src/jsonl-hooks.ts)
that maintains its own minimal per-runId state, so the OTLP trace path is
untouched. Write failures are logged at warn level and never propagate.

Tests: 14 unit cases for JsonlEmitter + builders + readSharedOtelConfig;
4 end-to-end cases simulating one LLM turn with tool call, double-fire dedup,
captureMessageContent toggle, and step.id increment across turns. All 129
existing tests still pass.

Bumps version to 0.1.4-beta.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant