fix: optimize stdout#2562
Open
StartE wants to merge 22 commits into
Open
Conversation
… being merged as Partial
The e2e test asserts all logs match `.*name.*test_multiline.*`, but the
single-line JSON `{"single_line": true}` did not contain those fields,
causing the assertion to fail even though multiline merge worked correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…n partial log handling - Move inQuote out of per-event loop so quoted braces spanning multiple stdout lines no longer corrupt braceDepth, preventing premature JSON block splitting - Guard SetMetadata(HAS_PART_LOG) with HasMetadata check and use SetMetadataNoCopy to avoid redundant arena allocations per partial event - Add TestMergeJsonBraceInStringCrossEvent covering cross-event quote state with embedded braces - Add TestDockerJsonPartialLogAllPartial covering all-partial batch (no trailing full line) through Split→Parse→MergeByFlag pipeline - Update container_stdio_ut_spec.md with TC-JSON-05b and TC-DOCKER-PARTIAL-04 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…LogsByJson - Add pendingEscape flag to carry backslash-escape state across events, preventing incorrect inQuote flip when \" is split at a line boundary - Guard WHOLE_FILE mode in CreateInnerProcessors to avoid null pointer dereference when unsupported multiline mode is configured - Add IsLowLevelAlarmValid() rate limiting to oversized JSON block warning, consistent with HandleUnmatchLogs - Log debug message when braceDepth goes negative on malformed input - Tighten json_multiline E2E assertion to require nested object fields, ensuring the test fails if multi-line merge is broken - Add TestMergeJsonEscapedQuoteCrossEvent covering the pendingEscape fix Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add else branch in CreateInnerProcessors to return error for unsupported Multiline.Mode values, preventing null pointer dereference if new enum values are added in the future - Add TestMergeJsonConsecutiveBackslashes verifying \\\\ pairs up correctly without leaving pendingEscape set - Add TestMergeJsonEmptyEventInterleaved verifying empty events are skipped without breaking merge state - Add TestMergeJsonSingleCharEvents verifying single-char boundary conditions for inQuote and braceDepth tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update section 5 to reflect all 18 test cases including Round 2/3 additions: EscapedQuoteCrossEvent, ConsecutiveBackslashes, EmptyEventInterleaved, and SingleCharEvents. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The regex pattern for matching multi-line JSON content used `.*` which does not cross newline boundaries in Go's regexp package. Adding `(?s)` enables dotall mode so `.` matches newlines in the pretty-printed JSON. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LogFieldKV validates ALL collected logs against the regex. The test
generates both a multi-line and a single-line JSON log, but the previous
regex required `nested` and `key` which only exist in the merged
multi-line log. Use a regex that validates both logs are complete JSON
objects (start with `{`) containing `name` and `test_multiline`, while
still catching merge failures (unmerged individual lines lack these).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Collaborator
|
containerd有测试512K的例子吗? |
Takuka0311
reviewed
Jul 1, 2026
…ssorMergeMultilineLogNative MergeLogsByJson already forced a split with an alarm when a merged block exceeded max_read_buffer_size (LogFileReader::BUFFER_SIZE). MergeLogsByFlag and MergeLogsByRegex had no such bound, so an oversized merged log could be emitted with no split and no alarm. - Generalize mMaxJsonBlockSize -> mMaxMergeBlockSize, shared by all three merge branches. - Extract HandleMergeSizeExceeded() for the unified SPLIT_LOG_FAIL_ALARM + rate-limited first-1KB warning; reuse it in MergeLogsByJson. - MergeLogsByFlag: force-split accumulated partials once accumulatedSize exceeds maxSize (no separator, so concatenation still reconstructs the original line; data preserved). - MergeLogsByRegex: track accumulatedSize via pushEvent(), force-split via forceSplit() at the continue path and at end of loop using cut-and-continue semantics (keep isPartialLog, begin=cur+1). - Add TestMergeFlagOversized and TestMergeRegexOversized; rename the member reference in the existing json oversized test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
docker/contianerd的单行cri标准输出都是16k就截断,报告中有截断的测试; 接下来补充的是,是处理当多行场景,例如多行json时,超出块max read max size 512k的时候,合并及丢弃场景 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: 修复容器标准输出 Docker JSON 拆片丢失与 JSON 多行模式失效问题
背景
修复新版标准输出采集中 Docker JSON 场景的两个核心缺陷:
ParseDockerJsonLogLine未检测log字段末尾是否有\n,导致拆片中间段没有 Part 标记,MergeByFlag不生效InputContainerStdio在 JSON 多行场景错误地设置RequiringJsonReader=true,导致JsonLogFileReader在容器格式剥离前提前做{}配对,破坏了 docker/containerd 的 runtime 解析改动
1. Docker 拆片识别与 Part 标记(Step 1)
ProcessorParseContainerLogNative.cpp:ParseDockerJsonLogLine新增拆片检测逻辑:log解码后不以\n结尾 → 标记为 Partial(设置PartLogFlag+HAS_PART_LOG)\n结尾 → Full line,去掉末尾\n后正常输出MergeType=flag合并路径,与 containerd P/F 行为统一2. JSON 多行从 Reader 层下沉到 Processor 层(Step 2)
InputContainerStdio.cpp:mContext->SetRequiringJsonReaderFlag(true),确保容器 stdio 始终使用普通LogFileReaderProcessorMergeMultilineLogNative(MergeType=json),而非ProcessorSplitLogStringNative(SplitChar='\0')ProcessorMergeMultilineLogNative.cpp/h:MergeType::BY_JSON枚举和MergeLogsByJson()方法{}、转义引号\")maxSize(默认BUFFER_SIZE)时强制输出并告警架构
修复后的 inner processor 链路:
关键设计原则:
JsonLogFileReader,避免在容器格式剥离前破坏物理行边界线下测试报告
https://alidocs.dingtalk.com/i/nodes/QG53mjyd800agdlKHwY7ePYe86zbX04v
后续 Step
_source_分桶合并,防止跨流拼接StdioFileReader基于 docker log 字段末尾\n做稳定切分点{}/转义正则匹配回退