build: proper linuxkit dependency tracking via content-hash propagation

[rucoder]

Problem: linuxkit dependency tracking gap

Linuxkit uses the git tree hash of a package's source directory as its cache key. This breaks when packages depend on other packages whose content changes:

pkg/zfs  ──────────────────────►  pkg/pillar
                                  pkg/dom0-ztools
                                  pkg/vtpm

When pkg/zfs is rebuilt (e.g. after a ZFS version bump), the consumers retain a stale cache hit because their own git trees did not change. Linuxkit has no way to know that the image it cached for pkg/pillar was built against the old pkg/zfs.

The problem is structural: consumer packages reference their dependencies via Dockerfile.in templates that are rendered by parse-pkgs.sh at build time. The rendered Dockerfile is gitignored, so it is invisible to linuxkit's tree-hash computation.

This is addressed in linuxkit/linuxkit#4210

Design

1. Declare dependencies in build.yml

Each package that depends on others declares it via a linuxkit build arg wildcard in build.yml:

# pkg/pillar/build.yml
buildArgs:
  - REL_HASH_%=@lkt:pkgs:../*

@lkt:pkgs:../* tells linuxkit to glob all sibling packages and resolve each to its content tag. Linuxkit filters the results against the ARG declarations in the package's Dockerfile, so only actually-used deps are included. The resolved tag values are folded into the package's own hash via SHA1 — effectively hashing the rendered dependency references.

Result: when pkg/zfs gets a new content tag, pkg/pillar's computed tag also changes, even though pkg/pillar's git tree is unchanged.

2. Two-stage Makefile build

Stage 1 — update-hashes (runs once per make invocation):

linuxkit pkg update-hashes --hash-dir .gen-deps \
    pkg/alpine:build.yml  pkg/zfs:build-2.4.yml  pkg/pillar:build.yml  ...

update-hashes processes all packages in topological dependency order (deps before consumers) and writes a YAML manifest per package to .gen-deps/:

# .gen-deps/pillar.hash
tag: lfedge/eve-pillar:a3f9b2c1...(hash-of-sources+zfs-tag+alpine-tag)
build-yml: build.yml
deps:
  - path: pkg/zfs
    tag: lfedge/eve-zfs:8d4e1f09...-2.4
  - path: pkg/alpine
    tag: lfedge/eve-alpine:c7a2b3d5...

Hash files are written with write-if-changed semantics: the mtime only advances when the tag actually changes, so unchanged packages do not trigger downstream rebuilds.

Stage 2 — eve-% (PHONY, per-package):

linuxkit pkg build --hash-dir .gen-deps pkg/pillar

Linuxkit reads the tag from .gen-deps/pillar.hash and checks whether that exact image exists in the Docker cache. If it does, no build occurs. If not, it builds. update-hashes runs exactly once per make invocation regardless of how many packages are targeted.

3. tools/gen-hash-deps — make-level ordering

tools/gen-hash-deps reads .gen-deps/*.hash and emits hash-deps.mk (included by the root Makefile) with:

# section 1: hash-file dep rules — topological ordering for future file-target builds
.gen-deps/pillar.hash: .gen-deps/zfs.hash .gen-deps/alpine.hash

# section 2: cache-export ordering
pillar-cache-export-docker-load: zfs-cache-export-docker-load

End-to-end: ZFS version bump

ZFS_VERSION=2.4  →  build-2.4.yml selected  →  pkg/zfs gets new tag
                 →  update-hashes recomputes pkg/pillar tag (includes new zfs tag)
                 →  pkg/pillar tag changed  →  docker cache miss  →  rebuild
                 →  pkg/dom0-ztools, pkg/vtpm same

Next make run: nothing changed → tags unchanged → update-hashes writes no new mtimes → all docker cache hits → instant.

What this PR contains

1. pkg/zfs — the first package that exercises the new mechanism: a FROM scratch linuxkit package that builds OpenZFS userspace once, with versioned build-2.3.yml / build-2.4.yml variants. Replaces three redundant ZFS compilations that were scattered across the tree.

2. pkg/pillar, pkg/dom0-ztools, pkg/vtpm — converted to consume pkg/zfs via @lkt:pkgs:../*, with build.yml deps: entries that make the dependency explicit.

3. Makefile + tools/gen-hash-deps — two-stage build wiring: update-hashes target, eve-% simplified to a post-hash build, hash-deps.mk include.

How to test

# Full build
make pkgs

# ZFS version bump — pillar/dom0-ztools/vtpm must rebuild automatically
make pkg/zfs ZFS_VERSION=2.4.1
make pkg/pillar ZFS_VERSION=2.4.1   # new tag → cache miss → rebuild
make pkg/pillar ZFS_VERSION=2.4.1   # same tag → cache hit → instant

Changelog notes

None

PR Backports

• 16.0-stable: No.
• 14.5-stable: No.
• 13.4-stable: No.

Checklist

• I've provided a proper description
• I've added the proper documentation
• I've tested my PR on amd64 device
• I've tested my PR on arm64 device
• I've written the test verification instructions
• I've set the proper labels to this PR
• I've checked the boxes above, or I've provided a good reason why I didn't check them.

[rene]

@rucoder , TBH I think this is just introducing more (unwanted) complex to our build system to a problem that is not technically ours... this actually should be addressed by linuxkit, not by EVE build system. We had subtle bugs because of this feature that was introduced to help, but just caused problems... when we figured out the hashes were not being honored and package rebuilt accordingly, we just revert all commits that were introducing the automatic tags.... unless this is very well proven to work, I'm against moving forward....

[rucoder]

@rucoder , TBH I think this is just introducing more (unwanted) complex to our build system to a problem that is not technically ours... this actually should be addressed by linuxkit, not by EVE build system. We had subtle bugs because of this feature that was introduced to help, but just caused problems... when we figured out the hashes were not being honored and package rebuilt accordingly, we just revert all commits that were introducing the automatic tags.... unless this is very well proven to work, I'm against moving forward....

@rene I forgot to mention a PR to LK that fixes the feature, I just reintroduced it back linuxkit/linuxkit#4210 . PR is still in draft state becasue I'm trying to figure out how to reduce the complexity. As of now following things work reliably

1. Transitive dependencies: A->B->C if I change C - no matter code or parameter e.g. ZFS_VERSION A is rebuilt and gets UNIQUE hash. No more hash collisions

2. No need for <hash>-<custom tag> required because hash is unique depends on code AND parameters

3. we do not need get-deps anymore, I just temporary introduced new small and super fast tool to generate *.mk file from LK output

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.49%. Comparing base (2281599) to head (20aefeb).
⚠️ Report is 341 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5678      +/-   ##
==========================================
+ Coverage   19.52%   29.49%   +9.96%     
==========================================
  Files          19       18       -1     
  Lines        3021     2417     -604     
==========================================
+ Hits          590      713     +123     
+ Misses       2310     1552     -758     
- Partials      121      152      +31     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.