Skip to content

Commit 29ee99d

Browse files
committed
feat(fingerprint): reset cutover - projectV1 substrate + atomic v1:sha256 token (PR B)
The one-time substrate swap: ComputeIdentity hashes the canonical projectV1 projection via sha256 and stamps the atomic v1:sha256: content token; hashstructure is removed; lock format Version stays 1 with force-rehash reconciliation of pre-reset tokens. Includes the review-driven consolidations and fixups (squashed): - single FingerprintedStructTypes() source of truth for the decision test and emission probe, with a completeness/reachability guard (TestFingerprintedStructTypesIsComplete); - dropped the dead ComponentIdentity.Inputs breakdown (componentInputs is now the internal combiner input only); - SkipReason documented as deliberately unmeasured (render-only comment); - v1: token test-hardening (HasPrefix(v1:sha256:), meaningful bump placeholder); - config/fingerprinting guardrail instructions, the as-built RFC reconciliation (read-gate floor; parser-free reconciliation), and the component-identity doc pointer to the RFC.
1 parent 7ad6237 commit 29ee99d

19 files changed

Lines changed: 560 additions & 129 deletions

File tree

.github/copilot-instructions.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@ Documentation structure: `docs/user/reference/cli/` (auto-generated CLI docs, re
3030

3131
The TOML config files in `defaultconfigs/` are loaded via `internal/projectconfig/`.
3232

33+
**CRITICAL when editing config structs or fingerprinting:** Adding a field to `ComponentConfig` or its component structs (or touching `projectV1` / fingerprint tags / golden vectors) has silent traps and required manual steps. You MUST read `.github/instructions/projectconfig-fingerprint.instructions.md` before such changes. This also applies to CONSUMERS of fingerprint data (e.g., checking changed components between two commits via lockfile data).
34+
3335
**IMPORTANT**: Code generation runs automatically with build/test commands. `mage generate` (runs `go generate` for each package in parallel) is a prerequisite for building and runs automatically with `mage build` and `mage unit`. `mage docs` rebuilds the binary and updates the JSON schema (`schemas/azldev.schema.json`) and CLI docs (`docs/user/reference/cli/`). Run `mage docs` explicitly after changing config structs or Cobra command descriptions so that checked-in generated files stay current (checked by PR gates).
3436

3537
**CRITICAL**: Run `mage scenarioUpdate` when test expectations change (updates snapshots).

.github/instructions/go.instructions.md

Lines changed: 8 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -98,20 +98,16 @@ description: "Instructions for working on the azldev Go codebase. IMPORTANT: Alw
9898
}
9999
```
100100

101-
## Component Fingerprinting: version-set tags + `projectV1`
101+
## Component fingerprinting & config structs
102102

103-
Structs in `internal/projectconfig/` feed the component fingerprint, and **every field must carry an explicit `fingerprint` tag** - an absent tag fails `TestAllFingerprintedFieldsHaveDecision`. (The live hash is still `hashstructure.Hash()`; the hand-written `projectV1` in `internal/fingerprint/project.go` is wired in parallel and replaces it at the reset.)
103+
Editing structs in `internal/projectconfig/` or the fingerprint substrate in
104+
`internal/fingerprint/` - adding a field, touching `projectV1`, fingerprint tags,
105+
or golden vectors - has silent traps where a field vanishes or a frozen byte moves
106+
with the compiler still green. **Those rules live in
107+
[`projectconfig-fingerprint.instructions.md`](./projectconfig-fingerprint.instructions.md)**,
108+
read it before such changes.
104109

105-
When adding a field to a fingerprinted struct, ask: **"Does changing this field change the build output?"**
106-
- **No** (human docs, scheduling hints, CI policy, metadata, back-references) → tag it `fingerprint:"-"` and register the exclusion in `expectedExclusions` in `internal/projectconfig/fingerprint_test.go`.
107-
- **Yes** (build flags, spec source, defines, etc.) → tag it `fingerprint:"v1..*"` (omit-if-zero) and wire it into `projectV1`: add its `emit`/sub-projector line, set it in `emissionProbeConfig`, then append a `<toml-key>-set` golden vector with `go test ./internal/fingerprint -run TestGoldenVectors -update-golden`. The golden table is append-only - never edit an existing digest. The full contract lives in `goldenConfigs`'s doc comment.
108-
- **Build-meaningful zero?** A field whose *zero value* changes the build (e.g. a `bool` where `false` is a deliberate setting) is tagged `fingerprint:"!v1..*"` (always-emit) and wired with `builder.emitAlways(...)`, **not** `builder.emit(...)`; it MUST get a golden vector at its zero value (the differ-from-`minimal` guard in `golden_internal_test.go` then proves it actually emits).
109-
110-
If a parent field is already excluded (e.g. `Failure ComponentBuildFailureConfig ... fingerprint:"-"`), do **not** tag the inner struct's fields - the excluded subtree is pruned from both the walk and `projectV1`.
111-
112-
Run `mage unit` to verify: the decision test catches a missing tag, the emission probe catches a measured-but-unemitted field, and the golden vectors catch a moved digest.
113-
114-
### Cmdline Returns
110+
## Cmdline Returns
115111

116112
CLI commands should return meaningful structured results. azldev has output formatting helpers to facilitate this (for example, `RunFunc*` wrappers handle formatting, so callers typically should not call `reflectable.FormatValue` directly).
117113

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
---
2+
applyTo: "internal/projectconfig/**/*.go,internal/fingerprint/**/*.go"
3+
description: "Rules for interacting with project config structs and the fingerprint substrate. IMPORTANT: read before adding or changing any field on a config struct, or touching projectV1 / golden vectors / fingerprint tags."
4+
---
5+
6+
# Config structs & fingerprinting
7+
8+
Editing config structs (`internal/projectconfig/`) and the fingerprint substrate
9+
(`internal/fingerprint/`) has several **silent traps** - the compiler stays green
10+
while a field quietly vanishes or a frozen byte moves. Follow these rules.
11+
12+
## `ComponentConfig.WithAbsolutePaths` hand-lists its fields
13+
14+
`(*ComponentConfig).WithAbsolutePaths` in `internal/projectconfig/component.go`
15+
builds its result with an **explicit field-by-field struct literal**, not
16+
`deep.MustCopy`. It is called for *every* component at load time (via
17+
`mergeComponents` in `loader.go`), so any field missing from that literal is
18+
**silently zeroed on load** - it parses from TOML fine, then disappears before
19+
resolution, fingerprinting, or rendering. No error, no panic, just lost data.
20+
21+
> It is hand-written (unlike every sibling `WithAbsolutePaths`, which uses
22+
> `deep.MustCopy`) for one reason: it must **alias** the cyclic, project-wide
23+
> `SourceConfigFile` pointer rather than deep-copy it. Do not "simplify" it to
24+
> a blanket `deep.MustCopy` - that reintroduces the cycle.
25+
26+
**Every new field on `ComponentConfig` MUST be added to this literal.**
27+
28+
## Adding a field to a config struct
29+
30+
1. **Add the field** with a `toml:` key and a **mandatory** `fingerprint:` tag.
31+
An absent tag fails `TestAllFingerprintedFieldsHaveDecision`
32+
(`internal/projectconfig/fingerprint_test.go`).
33+
- Build input → `fingerprint:"v1..*"` (measured).
34+
- Not a build input → `fingerprint:"-"` **and** register the field in
35+
`expectedExclusions` in that same test file. If the field's **parent struct is
36+
already excluded** (e.g. `ComponentBuildConfig.Failure`), do *not* tag the
37+
inner struct's fields - that subtree is pruned from both the decision walk and
38+
`projectV1`, so its fields are neither policed nor measured.
39+
2. **If the field is on `ComponentConfig` directly → add it to
40+
`WithAbsolutePaths`.** Skip this and it never loads.
41+
3. **If you added a new struct *type* to the fingerprinted graph**, add it to
42+
`FingerprintedStructTypes()` in `internal/fingerprint/project.go` - the single
43+
source of truth that **both** the mandatory-tag decision test
44+
(`internal/projectconfig/fingerprint_test.go`) and the emission probe
45+
(`measuredLeafEmitKeys` in `internal/fingerprint/project_internal_test.go`) walk.
46+
Add it there and nowhere else.
47+
4. **If measured, emit it in `projectV1`** (`internal/fingerprint/project.go`,
48+
hand-written as of v1) inside the correct sub-projector, under its **frozen `toml`
49+
emit-key** (or an explicit `key=` in the tag - never the Go field name).
50+
- Removing a measured field won't compile (good); a Go rename is byte-neutral
51+
**only if** the `toml` key / `key=` is unchanged.
52+
5. **Set the field non-zero in `emissionProbeConfig()`**
53+
(`internal/fingerprint/project_internal_test.go`). The emission probe FAILS
54+
until you do - that is the guard against "measured but forgot to emit".
55+
6. **Append a `<toml-key>-set` golden vector** via
56+
`go test ./internal/fingerprint -run TestGoldenVectors -update-golden`.
57+
7. **Regenerate the schema in all three places** if the field is user-facing (has
58+
a `toml` key / `jsonschema` tag): `mage docs` updates
59+
`schemas/azldev.schema.json` **and** the CLI reference docs, while
60+
**`mage scenarioUpdate`** updates the `config generate-schema` scenario
61+
snapshots (`TestSnapshots*config_generate-schema*`), which embed the schema
62+
again. Running only `mage docs` leaves those snapshots red. Also hand-update the
63+
relevant `docs/user/reference/config/` page (e.g. `components.md`).
64+
65+
**Build-meaningful zero value?** A field whose *zero value* is itself a deliberate
66+
build setting (e.g. a `bool` where `false` carries meaning) is tagged
67+
`fingerprint:"!v1..*"` (always-emit) and wired with `builder.emitAlways(...)`, **not**
68+
`builder.emit(...)` - the omit-if-zero `emit` would drop it at its zero value and lose
69+
that signal. It MUST get a golden vector **at its zero value**; the
70+
differ-from-`minimal` guard in `golden_internal_test.go` then proves it actually emits.
71+
72+
## Golden vectors: append-only, and the frozen/growing split
73+
74+
- `maximalConfig()` is **FROZEN** - it is a golden-corpus config at the time of
75+
the initial implementation of that fingerprint algorithm, so its digest is
76+
pinned. **Never add a new field to `maximalConfig()`** (it would move the frozen
77+
`maximal` digest, a hard CI failure). Field growth goes in
78+
`emissionProbeConfig()` only.
79+
- Existing golden digests are **append-only**: `-update-golden` may add a new
80+
`(name -> digest)` entry, but a CI check FATALs any commit that *modifies* an
81+
existing digest. A moved frozen byte is a hard failure, not something a reviewer
82+
must eyeball.
83+
- A new **omit-if-zero measured field that no config sets is drift-neutral**: it
84+
emits no bytes, so no existing lock should move and no component version bump
85+
is needed. Only a component that actually sets the field drifts.
86+
- **One isolation vector per new field is enough.** Add a single `<toml-key>-set`
87+
vector that sets *only* the new field (like `single-overlay` /
88+
`defines-empty-value`). Projection slots are length-prefixed and independent, so
89+
a field's encoding is fully pinned in isolation - do **not** add a "maximal + new
90+
field" or "previous + new field" combinatorial vector (and you cannot touch the
91+
frozen `maximal`). Add *extra* vectors only for genuine internal discrimination
92+
cases the field introduces (empty-vs-absent, map-key membership, etc.).
93+
94+
## Other config struct / fingerprinting pitfalls
95+
96+
- **Strict parsing.** The loader runs `DisallowUnknownFields()`
97+
(`internal/projectconfig/loader.go`). Renaming or removing a `toml` key breaks
98+
every existing on-disk config that still uses it. Load-time config migration is
99+
deferred (lazy schema migration RFC PR E); until it exists, a `toml`-key rename
100+
needs a coordinated, version-pinned rollout, not a bare rename.
101+
(`fingerprint:"...,key=<old>"` keeps the *fingerprint* byte-neutral, but does
102+
**not** keep old TOML parseable.)
103+
- **`omitempty` + go-toml/v2 (`v2.3.1`).** Emptiness is decided by reflecting the
104+
**exported** fields *before* any `TextMarshaler` runs. A struct that marshals via
105+
`TextMarshaler` but holds its real value in an **unexported** field is treated as
106+
empty and **dropped even when set**. Keep round-tripped state in an exported field
107+
or a plain string.
108+
- **`MergeUpdatesFrom` uses mergo with `WithOverride` + `WithAppendSlice`.** Slices
109+
*append* across merge layers, and the same intent can resolve to `nil` *or* `[]`
110+
depending on merge order. That is exactly why the projection's omit predicate
111+
treats a nil-or-empty scalar slice as a zero value (both collapse to no emitted
112+
bytes) at the hash boundary - do not rely on raw slice identity for a
113+
fingerprinted field.
114+
- **A nested-struct field surfaces asymmetrically in output.** `encoding/json`
115+
`omitempty` does **not** drop an empty struct, so `component list -O json` shows
116+
`"foo":{}` on every component, while `config dump` (go-toml) *does* honor struct
117+
`omitempty` and omits it. Neither is fingerprint drift (the projector omits on
118+
projected emptiness) - don't mistake the JSON `{}` for churn.
119+
120+
## Standing invariants (do not break)
121+
122+
- **No reader recomputes a historical fingerprint.** Synthetic history and
123+
historic-overlay application read **stored** lock strings only
124+
(`synthistory.go`); only `computeCurrentFingerprint` (current tree) and the
125+
`update` re-stamp call `ComputeIdentity`.
126+
- **Stored fingerprint is the atomic `v1:sha256:<digest>` token**; lock *format*
127+
`version` stays `1` (the content version lives in the token prefix). A pre-`v1`
128+
(prefix-less) token reconciles by **force-rehash** - the existing string
129+
inequality re-stamps it; do not make that compare version-aware before the replay
130+
registry (RFC PR C) exists.
131+
- **`projectV1` output is frozen.** A new byte encoding is a new content version,
132+
never an edit to `projectV1`'s output. The golden vectors enforce this. The only
133+
allowed changes to `projectV1` are purely additive.

docs/developer/reference/component-identity-and-locking.md

Lines changed: 23 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -6,29 +6,41 @@
66
- **Build optimization**: only rebuild changed components and their dependents, skipping unchanged ones.
77
- **Automatic release bumping**: increment the release tag of changed packages automatically, and include the commits that caused the change in the changelog.
88

9+
> **Design & how-to.** The substrate design (version-set tags, the `v1:sha256:` content token, force-rehash reconciliation, and the future lazy schema migration) is specified in the [Lock-File Fingerprint Reset RFC](../rfc/lazy-schema-migration.md). The step-by-step rules for adding a fingerprinted field live in [`projectconfig-fingerprint.instructions.md`](../../../.github/instructions/projectconfig-fingerprint.instructions.md).
10+
911
## Fingerprint Inputs
1012

1113
A component's fingerprint is a SHA256 combining:
1214

13-
1. **Config hash**`hashstructure.Hash()` of the resolved `ComponentConfig` (after all merging). Fields tagged `fingerprint:"-"` are excluded.
14-
2. **Source identity** — content hash for local specs (all files in the spec directory), commit hash for upstream.
15-
3. **Overlay file hashes** — SHA256 of each file referenced by overlay `Source` fields.
16-
4. **Distro name + version**
17-
5. **Manual release bump counter** — increments with each manual release bump, ensuring a new fingerprint even if there are no config or source changes.
15+
1. **Config projection digest** - `sha256` of the canonical `projectV1` projection of the resolved `ComponentConfig` (after all merging). Only fields whose `fingerprint` tag measures them at v1 are emitted; `fingerprint:"-"` fields are excluded. A nil-or-empty scalar slice is treated as zero and omitted by the projection's omit predicate, so a merge-order nil-vs-`[]` difference never moves the digest.
16+
2. **Source identity** - content hash for local specs (all files in the spec directory), commit hash for upstream.
17+
3. **Overlay file hashes** - SHA256 of each file referenced by overlay `Source` fields.
18+
4. **Release version** - the distro's formal `releasever` (e.g. `4.0`), which feeds RPM macros like `%{dist}`; different release versions produce different package NEVRAs even with identical specs.
19+
5. **Manual release bump counter** - increments with each manual release bump, ensuring a new fingerprint even if there are no config or source changes.
20+
21+
The combined digest is stored as a single self-describing **content-version token**: `input-fingerprint = "v1:sha256:<hex>"`. The `v1` prefix is the *content* version (the projection/encoding generation), written together with the digest so the two can never desync. It is independent of the lock *format* `version` field, which stays `1`. A pre-reset (prefix-less) token reads as sub-floor and is force-rehashed to `v1` on the next `component update`.
1822

1923
Global change propagation works automatically: the fingerprint operates on the fully-merged config, so a change to a distro or group default changes the resolved config of every inheriting component.
2024

21-
## `fingerprint:"-"` Tag System
25+
## `fingerprint` Version-Set Tags
26+
27+
Every fingerprinted field carries an explicit `fingerprint` tag declaring **which content versions measure it**. There is no "included by default": an absent tag on a fingerprinted struct field fails the decision test (`TestAllFingerprintedFieldsHaveDecision`), so adding a field forces a conscious measure/exclude decision.
28+
29+
Grammar (`internal/fingerprint/versiontag.go`):
2230

23-
The `hashstructure` library uses `TagName: "fingerprint"`. Untagged fields are **included by default** (safe default: false positive > false negative).
31+
- `v1..*` - measured from v1 onward (the common case for a build input).
32+
- `vN..vM`, `vN..*`, comma-separated sets (`v1..v1,v3..*`) - measured only in those ranges.
33+
- `-` - never measured (excluded); register it in `expectedExclusions` in `internal/projectconfig/fingerprint_test.go`.
34+
- `!` prefix on a range (`!v1..*`) - always emit even at the zero value (scalar leaves only; an always-emit composite is rejected at v1).
35+
- `key=<name>` - override the emit-key (see below).
2436

25-
A guard test (`TestAllFingerprintedFieldsHaveDecision`) reflects over all fingerprinted structs and maintains a bi-directional allowlist of exclusions. It fails if a `fingerprint:"-"` tag is added without registering it, or if a registered exclusion's tag is removed.
37+
**Emit-key is the frozen `toml:` key, never the Go field name.** `projectV1` emits each measured field under its `toml:` key (or an explicit `key=`), sorted by that key. So a cosmetic Go rename is byte-neutral, and a TOML-key rename stays byte-neutral by pinning `key=<old>`. Renaming the emit-key itself is an output change, and therefore a new content version.
2638

2739
### Adding a New Config Field
2840

2941
1. Add the field to the struct in `internal/projectconfig/`.
30-
2. **If NOT a build input**: add `fingerprint:"-"` to the struct tag and register it in `expectedExclusions` in `internal/projectconfig/fingerprint_test.go`.
31-
3. **If a build input**: do nothing — included by default.
42+
2. Tag it: `fingerprint:"v1..*"` if it is a build input, or `fingerprint:"-"` (plus an `expectedExclusions` entry) if it is not.
43+
3. A measured, omit-if-zero field is **drift-neutral**: components that do not set it emit identical bytes, so no existing lock moves - it needs no content-version bump.
3244
4. Run `mage unit`.
3345

3446
### Adding a New Source Type
@@ -39,4 +51,4 @@ A guard test (`TestAllFingerprintedFieldsHaveDecision`) reflects over all finger
3951

4052
## Known Limitations
4153

42-
- It is difficult to determine WHY a diff occurred (e.g., which specific field changed) since the fingerprint is a single opaque hash. The JSON output includes an `inputs` breakdown (`configHash`, `sourceIdentity`, `overlayFileHashes`, etc.) that can help narrow it down by comparing the two identity files manually.
54+
- It is difficult to determine WHY a diff occurred (e.g., which specific field changed) since the fingerprint is a single opaque `v1:sha256:` token: `ComponentIdentity` emits only the combined `fingerprint`, not a per-input breakdown.

0 commit comments

Comments
 (0)