Skip to content

feat: Add json-file log driver - #392

Merged
aaithal merged 4 commits into
aws:mainfrom
aaithal:add-json-file-driver
May 28, 2026
Merged

feat: Add json-file log driver#392
aaithal merged 4 commits into
aws:mainfrom
aaithal:add-json-file-driver

Conversation

@aaithal

@aaithal aaithal commented May 27, 2026

Copy link
Copy Markdown
Contributor

feat: Add json-file log driver

Overview

This PR adds a json-file log driver to the shim-logger, wrapping moby's
upstream jsonfilelog
behind the same driver pattern that awslogs, fluentd, and splunk use today.

The driver writes container stdout/stderr to a host file in the
Docker JSON-file format — one JSON object per line, e.g.:

{"log":"hello\n","stream":"stdout","time":"2026-05-27T21:07:48.123456789Z"}

It supports moby's max-size, max-file, compress, tag, labels,
labels-regex, env, and env-regex options unchanged, plus a required
--log-path for the per-container output file.

Usage

shim-loggers-for-containerd \
  --log-driver json-file \
  --log-path <abs-path-to-output-file> \
  [--max-size 10m] \
  [--max-file 5] \
  [--compress true] \
  [--json-file-tag '{{.ImageName}}/{{.ID}}'] \
  [--json-file-labels foo,bar] \
  [--json-file-labels-regex '^app\..*'] \
  [--json-file-env KEY1,KEY2] \
  [--json-file-env-regex '^APP_.*']

Changes Made

  • New driver package logger/jsonfile — wraps moby's jsonfilelog.New,
    honoring the existing NonBlockingMode wrapping pattern when buffered mode
    is requested. Optional fields are only added to the moby config map when
    non-empty so moby's ValidateLogOpts does not reject stray empty values.

  • New common option logger.WithLogPath — sets info.LogPath on the
    logger.Info struct moby's jsonfilelog.New reads to derive the output
    filename. The shim does not create the directory; the caller
    (containerd / agent) is responsible for precreating it with the right
    ownership and mode.

  • Wiring through init.go / args.go / main.go — registers all 9
    flags, adds getJSONFileArgs() (returns an error if --log-path is unset),
    and dispatches case jsonfile.DriverName to runJSONFileDriver. Adds
    json-file to the --log-driver help text.

  • Prefixed input flag names for the 5 options that collide with other
    drivers: --json-file-tag, --json-file-labels, --json-file-labels-regex,
    --json-file-env, --json-file-env-regex. Same convention splunk and
    fluentd already use — shim-logger's pflag namespace is flat across drivers.
    The flags are forwarded to moby under their bare names (tag, labels,
    labels-regex, env, env-regex).

  • compress defaults to "do not set" — moby rejects compress=true
    unless max-file >= 2 and max-size is set, so making it opt-in keeps a
    partial config from becoming a hard failure.

Drive-by: relax heap threshold for TestMemoryScenario_LargeLines_NonBlocking_DefaultBuffer

Commit 3 of this PR (26c2185) is unrelated to the json-file work. The
test added in #391 uses a 210 MiB threshold derived from a 160 MiB baseline
plus a 30% race-instrumentation buffer; in practice the OS allocator and Go
version each add their own jitter on top of -race, so Windows + Go 1.24

  • race observes ~225 MiB and trips the check on this PR's CI run (Linux +
    Go 1.24 + race observes 188.8 MiB). Bumped to 256 MiB (1.6× the 160 MiB
    baseline) — a real regression doubling the working set would still trip.

Happy to drop this commit and split into a separate PR if maintainers
prefer; calling it out so it's visible.

Testing

Unit testsmake test-unit passes across all packages:

  • logger/jsonfile/logger_test.go covers config translation, empty-args →
    empty config map, TagSpecified=false suppression, and the
    validate-keys-not-values contract of moby's ValidateLogOpts.
  • args_test.go adds TestGetJSONFileArgs covering the --log-path-required
    error path, log-path-only defaults, and the full options pass-through.

E2E testsmake test-e2e-for-json-file runs 6 specs against a
local containerd, all pass:

Spec Verifies
envelope format every line is {"log":...,"stream":"stdout","time":"<RFC3339Nano>"}
--max-size rotation enough output forces 2–3 rotated files
--max-file cap over-rotation is bounded by the cap
no-rotation below --max-size a single small printf produces 1 file
file mode active file has mode 0640 (moby's hardcoded value)
--log-path required missing flag fails the container task fast

Two e2e infra additions: e2e/common.go grows a SendCommandByContainerd
helper (the existing SendTestLogByContainerd delegates to it) so specs
can run arbitrary shell commands when they need to control payload shape;
--log-path is resolved to an absolute path at suite-build time because
the shim-logger inherits its CWD from containerd, not from the test
runner. Comment in the test file explains the latter.


By submitting this pull request, I confirm that you can use, modify, copy,
and redistribute this contribution, under the terms of your choice.

@aaithal
aaithal requested a review from a team as a code owner May 27, 2026 21:48
@aaithal aaithal changed the title Add json file driver feat: Add json-file log driver May 27, 2026
aaithal added 2 commits May 27, 2026 22:33
Wraps moby's jsonfilelog (github.com/docker/docker/daemon/logger/jsonfilelog)
behind the shim-logger driver pattern used by awslogs / fluentd / splunk:

  --log-driver json-file --log-path <abs-path> [--max-size <n>]
                                               [--max-file <n>]
                                               [--compress true|false]
                                               [--json-file-tag <tmpl>]
                                               [--json-file-labels <csv>]
                                               [--json-file-labels-regex <re>]
                                               [--json-file-env <csv>]
                                               [--json-file-env-regex <re>]

Notes:
  * --log-path is required and is forwarded to moby via the new
    logger.WithLogPath InfoOpt; the directory is the caller's responsibility
    (containerd / agent), the shim does not mkdir.
  * --json-file-{tag,labels,labels-regex,env,env-regex} are renamed at the
    pflag boundary to avoid colliding with splunk/fluentd flags in the flat
    pflag namespace; they are passed to moby under their bare names
    (tag, labels, labels-regex, env, env-regex).
  * Optional fields are only added to the moby config map when non-empty so
    moby's ValidateLogOpts does not reject stray empty values.
  * compress defaults to off (not set in config) — moby rejects
    compress=true unless max-file>=2 and max-size is set, so making it
    opt-in keeps a partial config from becoming a hard failure.
  * Buffered (non-blocking) mode is honored via the same NonBlockingMode
    wrapping pattern used by the existing drivers.

Unit tests:
  * logger/jsonfile/logger_test.go — config translation, optional-field
    omission, tag suppression when not specified, validate-keys-only
    contract of moby's ValidateLogOpts.
  * args_test.go — log-path required, log-path-only defaults, full
    option pass-through via viper.
Six Ginkgo specs run against a local containerd, mirroring the
fluentd e2e style:

  * envelope format — every line is {"log":...,"stream":...,"time":...}
  * --max-size rotation — forces 2-3 rotated files, asserts cap
  * --max-file cap — over-rotation is bounded by the cap
  * no-rotation lower bound — single small printf produces 1 file
  * file mode 0640 — moby's hardcoded mode is preserved end-to-end
  * --log-path required — missing flag fails the container task fast

Two infra additions:

  * e2e/common.go grows SendCommandByContainerd, a sibling of
    SendTestLogByContainerd that runs an arbitrary shell command in the
    test container. Needed when a spec has to control payload size or
    shape (e.g., to force log rotation).
  * --log-path is resolved to an absolute path at suite-build time via
    filepath.Abs. The shim-logger inherits its CWD from containerd, not
    from the test runner, so a relative log-path would fail to resolve
    in the shim's context.

Wires the new spec into e2e/main_test.go and adds a test-e2e-for-json-file
target to the Makefile mirroring test-e2e-for-fluentd / -splunk.
@aaithal
aaithal force-pushed the add-json-file-driver branch from 1765b78 to 9f4204b Compare May 27, 2026 22:35
TestMemoryScenario_LargeLines_NonBlocking_DefaultBuffer's threshold of
210 MiB is too close to the actual peak under -race instrumentation.
Observed values:

  * Linux + Go 1.24 + race  → 188.8 MiB
  * Windows + Go 1.24 + race → 224.1 MiB (CI failure on PR aws#392)

The pre-existing comment said "with -race instrumentation, add ~30%
headroom," pinning the threshold at 1.30 × the 160 MiB baseline = 208,
rounded to 210. In practice the OS allocator and Go version each add
their own jitter on top of -race, which puts Windows + Go 1.24 ~6.7%
over the original budget.

Bump the threshold to 256 MiB (1.6 × baseline). The test still serves
as a regression detector — a real leak would double the working set,
which would still trip the check.
JoseVillalta
JoseVillalta previously approved these changes May 28, 2026

@JoseVillalta JoseVillalta left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-blocking minor comments.

Comment thread args.go Outdated
Comment thread e2e/jsonfile_test.go Outdated
Comment thread e2e/jsonfile_test.go
xxx0624
xxx0624 previously approved these changes May 28, 2026
Comment thread e2e/jsonfile_test.go
* args.go: trim the inline GetString-everywhere rationale on
  getJSONFileArgs (~30 lines) to a 3-line summary; move the full
  rationale to a package-level doc comment, since it applies equally
  to splunk/fluentd's existing GetString patterns and isn't
  json-file-specific. (per JoseVillalta@)

* e2e/jsonfile_test.go: drop the leading './' on jsonFileLogDir;
  '../jsonfile-logs' reads cleaner and the './' was a no-op.
  (per JoseVillalta@)

* e2e/jsonfile_test.go: drop the redundant '<base>.gz' clause in
  listLogFiles — '<base>.<anything>' already matches '<base>.2.gz'
  via HasPrefix(name, baseName+'.'). (per JoseVillalta@)

* e2e/jsonfile_test.go: add a non-blocking-mode buffer-pressure
  spec (aws#7). Configures --mode=non-blocking --max-buffer-size=64k,
  produces ~200 KiB of output, asserts the binary exits cleanly and
  every line that lands in the file is a valid Docker envelope. The
  spec exercises the NonBlockingMode wrapping in jsonfile.RunLogDriver
  introduced by this PR. Doesn't assert specific drop counts (timing-
  dependent). (per xxx0624@)
@aaithal
aaithal dismissed stale reviews from xxx0624 and JoseVillalta via 2063ed7 May 28, 2026 17:53

@JoseVillalta JoseVillalta left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aaithal
aaithal merged commit e8076cb into aws:main May 28, 2026
15 checks passed
aaithal added a commit that referenced this pull request May 28, 2026
TestMemoryScenario_LargeLines_NonBlocking_DefaultBuffer's threshold of
210 MiB is too close to the actual peak under -race instrumentation.
Observed values:

  * Linux + Go 1.24 + race  → 188.8 MiB
  * Windows + Go 1.24 + race → 224.1 MiB (CI failure on PR #392)

The pre-existing comment said "with -race instrumentation, add ~30%
headroom," pinning the threshold at 1.30 × the 160 MiB baseline = 208,
rounded to 210. In practice the OS allocator and Go version each add
their own jitter on top of -race, which puts Windows + Go 1.24 ~6.7%
over the original budget.

Bump the threshold to 256 MiB (1.6 × baseline). The test still serves
as a regression detector — a real leak would double the working set,
which would still trip the check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants