Skip to content

fix: optimize stdout#2562

Open
StartE wants to merge 22 commits into
alibaba:mainfrom
StartE:yili/fix_stdout
Open

fix: optimize stdout#2562
StartE wants to merge 22 commits into
alibaba:mainfrom
StartE:yili/fix_stdout

Conversation

@StartE

@StartE StartE commented May 20, 2026

Copy link
Copy Markdown
Collaborator

fix: 修复容器标准输出 Docker JSON 拆片丢失与 JSON 多行模式失效问题

背景

修复新版标准输出采集中 Docker JSON 场景的两个核心缺陷:

  1. Docker 16KB 拆片无法合并(2026.04.15 截断问题):ParseDockerJsonLogLine 未检测 log 字段末尾是否有 \n,导致拆片中间段没有 Part 标记,MergeByFlag 不生效
  2. JSON 多行模式完全不可用(2025.12 多行 JSON 解析失败):InputContainerStdio 在 JSON 多行场景错误地设置 RequiringJsonReader=true,导致 JsonLogFileReader 在容器格式剥离前提前做 {} 配对,破坏了 docker/containerd 的 runtime 解析

改动

1. Docker 拆片识别与 Part 标记(Step 1)

ProcessorParseContainerLogNative.cpp:

  • ParseDockerJsonLogLine 新增拆片检测逻辑:log 解码后不以 \n 结尾 → 标记为 Partial(设置 PartLogFlag + HAS_PART_LOG
  • \n 结尾 → Full line,去掉末尾 \n 后正常输出
  • 复用现有 MergeType=flag 合并路径,与 containerd P/F 行为统一

2. JSON 多行从 Reader 层下沉到 Processor 层(Step 2)

InputContainerStdio.cpp:

  • 移除 mContext->SetRequiringJsonReaderFlag(true),确保容器 stdio 始终使用普通 LogFileReader
  • JSON 多行模式改为创建 ProcessorMergeMultilineLogNative(MergeType=json),而非 ProcessorSplitLogStringNative(SplitChar='\0')

ProcessorMergeMultilineLogNative.cpp/h:

  • 新增 MergeType::BY_JSON 枚举和 MergeLogsByJson() 方法
  • 基于 brace depth 状态机实现 JSON 块识别与合并(正确处理引号内 {}、转义引号 \"
  • 超限保护:累积大小超过 maxSize(默认 BUFFER_SIZE)时强制输出并告警
  • 批次末尾未闭合 JSON 块不丢弃,合并后输出

架构

修复后的 inner processor 链路:

Split(\n) → Parse(docker/containerd) → MergeByFlag(拆片还原) → [可选] MergeByJson / MergeByRegex(业务多行)

关键设计原则:

  • 先还原物理拆片,再做业务多行:保证 docker 16KB 拆片和 containerd P/F 在业务多行之前完成合并
  • JSON 块识别在 processor 层完成:不再依赖 reader 层的 JsonLogFileReader,避免在容器格式剥离前破坏物理行边界

线下测试报告

https://alidocs.dingtalk.com/i/nodes/QG53mjyd800agdlKHwY7ePYe86zbX04v

后续 Step

项目 状态 说明
stdout/stderr 流隔离 待实现 MergeByFlag 按 _source_ 分桶合并,防止跨流拼接
Reader 回退优化(单行,Step 3) 待实现 StdioFileReader 基于 docker log 字段末尾 \n 做稳定切分点
Reader 回退优化(多行,Step 4) 待实现 正序扫描 {}/转义正则匹配回退
跨 read buffer / 跨 flush 拆片合并 已知风险 当前 MergeByFlag 仅在同一 PipelineEventGroup 内合并

@StartE StartE changed the title optimize stdout fix: optimize stdout May 20, 2026
@StartE
StartE requested a review from Takuka0311 May 20, 2026 07:55
StartE and others added 17 commits May 20, 2026 09:20
The e2e test asserts all logs match `.*name.*test_multiline.*`, but the
single-line JSON `{"single_line": true}` did not contain those fields,
causing the assertion to fail even though multiline merge worked correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…n partial log handling

- Move inQuote out of per-event loop so quoted braces spanning multiple
  stdout lines no longer corrupt braceDepth, preventing premature JSON
  block splitting
- Guard SetMetadata(HAS_PART_LOG) with HasMetadata check and use
  SetMetadataNoCopy to avoid redundant arena allocations per partial event
- Add TestMergeJsonBraceInStringCrossEvent covering cross-event quote
  state with embedded braces
- Add TestDockerJsonPartialLogAllPartial covering all-partial batch
  (no trailing full line) through Split→Parse→MergeByFlag pipeline
- Update container_stdio_ut_spec.md with TC-JSON-05b and TC-DOCKER-PARTIAL-04

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…LogsByJson

- Add pendingEscape flag to carry backslash-escape state across events,
  preventing incorrect inQuote flip when \" is split at a line boundary
- Guard WHOLE_FILE mode in CreateInnerProcessors to avoid null pointer
  dereference when unsupported multiline mode is configured
- Add IsLowLevelAlarmValid() rate limiting to oversized JSON block
  warning, consistent with HandleUnmatchLogs
- Log debug message when braceDepth goes negative on malformed input
- Tighten json_multiline E2E assertion to require nested object fields,
  ensuring the test fails if multi-line merge is broken
- Add TestMergeJsonEscapedQuoteCrossEvent covering the pendingEscape fix

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add else branch in CreateInnerProcessors to return error for
  unsupported Multiline.Mode values, preventing null pointer dereference
  if new enum values are added in the future
- Add TestMergeJsonConsecutiveBackslashes verifying \\\\ pairs up
  correctly without leaving pendingEscape set
- Add TestMergeJsonEmptyEventInterleaved verifying empty events are
  skipped without breaking merge state
- Add TestMergeJsonSingleCharEvents verifying single-char boundary
  conditions for inQuote and braceDepth tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update section 5 to reflect all 18 test cases including Round 2/3
additions: EscapedQuoteCrossEvent, ConsecutiveBackslashes,
EmptyEventInterleaved, and SingleCharEvents.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The regex pattern for matching multi-line JSON content used `.*` which
does not cross newline boundaries in Go's regexp package. Adding `(?s)`
enables dotall mode so `.` matches newlines in the pretty-printed JSON.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LogFieldKV validates ALL collected logs against the regex. The test
generates both a multi-line and a single-line JSON log, but the previous
regex required `nested` and `key` which only exist in the merged
multi-line log. Use a regex that validates both logs are complete JSON
objects (start with `{`) containing `name` and `test_multiline`, while
still catching merge failures (unmerged individual lines lack these).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Takuka0311

Copy link
Copy Markdown
Collaborator

containerd有测试512K的例子吗?

Comment thread core/unittest/processor/container_stdio_ut_spec.md
Comment thread core/plugin/input/InputContainerStdio.cpp
…ssorMergeMultilineLogNative

MergeLogsByJson already forced a split with an alarm when a merged block
exceeded max_read_buffer_size (LogFileReader::BUFFER_SIZE). MergeLogsByFlag
and MergeLogsByRegex had no such bound, so an oversized merged log could be
emitted with no split and no alarm.

- Generalize mMaxJsonBlockSize -> mMaxMergeBlockSize, shared by all three
  merge branches.
- Extract HandleMergeSizeExceeded() for the unified SPLIT_LOG_FAIL_ALARM +
  rate-limited first-1KB warning; reuse it in MergeLogsByJson.
- MergeLogsByFlag: force-split accumulated partials once accumulatedSize
  exceeds maxSize (no separator, so concatenation still reconstructs the
  original line; data preserved).
- MergeLogsByRegex: track accumulatedSize via pushEvent(), force-split via
  forceSplit() at the continue path and at end of loop using cut-and-continue
  semantics (keep isPartialLog, begin=cur+1).
- Add TestMergeFlagOversized and TestMergeRegexOversized; rename the member
  reference in the existing json oversized test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@StartE

StartE commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator Author

containerd有测试512K的例子吗?

docker/contianerd的单行cri标准输出都是16k就截断,报告中有截断的测试;

接下来补充的是,是处理当多行场景,例如多行json时,超出块max read max size 512k的时候,合并及丢弃场景

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants