Skip to content

Commit dfd08aa

Browse files
committed
feat(federation): certify graph profile and evaluation
Signed-off-by: Tom Ballard <tom@armytage.co>
1 parent b24be92 commit dfd08aa

20 files changed

Lines changed: 484 additions & 26 deletions

File tree

decisions/roadmaps/corpus-federation.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,12 +150,26 @@ isolation, YAML structural bombs, overlong paths, hard links, mount/reparse
150150
boundaries, unsupported filesystem identity, cold/warm/no-cache parity, MCP
151151
budgets/audit, exports/Portal, and Linux/macOS/Windows containment.
152152

153+
Implementation evidence now includes a committed track using the existing
154+
DecisionGrounding scorer and metric gate: three direct parents, nested v1 and
155+
v2 ancestry, a 40-artifact transitive standards source, a 32-inbound weak hard
156+
negative, combined source-neutral ranking, and transitive search/relationship
157+
retrieval. The broader certification matrix above remains open; this evidence
158+
does not change the Proposed implementation requirements or this roadmap's
159+
Planned status.
160+
153161
### Profile guidance
154162

155163
Update explicit `decided init --parent-corpus` guidance to show the v2 manifest
156164
and `decided corpus digest --version 2` without creating a manifest, fetching a
157165
source, or changing ordinary init/profile bytes.
158166

167+
The opt-in guidance now recommends manifest version 2, the bounded `parents`
168+
sequence, and the explicit version-2 digest command while retaining the
169+
version-1 command for existing single-parent manifests. Focused CLI contracts
170+
prove fresh, idempotent, profiled, and unrequested flows; no manifest or parent
171+
bytes are written. This narrow evidence does not complete the programme.
172+
159173
## Constraints
160174

161175
- Materialised, reviewed local bytes only. No clone, fetch, refresh, registry,

docs/cli.md

Lines changed: 40 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ repin the parent.
4242

4343
```bash
4444
decided corpus digest --root vendor/standards --corpus decisions
45+
decided corpus digest --version 2 --root vendor/standards --corpus decisions
4546
```
4647

4748
`--root` is the parent repository root and bounds configuration discovery to
@@ -70,6 +71,14 @@ escape, and traversed symlinks are rejected with stable `parent-corpus-*`
7071
errors. Exit `0` means the digest was calculated; exit `1` means the bounded
7172
materialisation could not be safely snapshotted.
7273

74+
Manifest version 2 uses the explicit `--version 2` command. Its
75+
`sha256-v2:<64 lowercase hex>` digest also commits to whether
76+
`.decided/corpus.md` is present and, when present, its exact bytes. Those bytes
77+
bind the parent's own outgoing edges and pins, so graph updates are repinned
78+
bottom-up. The owned Markdown snapshot excludes every direct materialisation
79+
subtree declared by that manifest. The default command, version-1 digest,
80+
`sha256:` prefix, and single-parent workflow remain unchanged.
81+
7382
---
7483

7584
## validate
@@ -1159,12 +1168,32 @@ for fallback and aggregation behaviour.
11591168
[Org Grounding](org-grounding.md).
11601169
- **`--parent-corpus`** prints deterministic setup guidance for the operational
11611170
`.decided/corpus.md` manifest, including the exact `## inherits` and
1162-
`## overrides` headings. It tells you to materialise the parent inside the
1163-
repository first, then calculate its pin with `decided corpus digest --root
1164-
<parent-root> --corpus <parent-corpus>`. The flag works on fresh and
1165-
already-initialized repositories, with or without a profile. It is guidance
1166-
only: it never creates the manifest, fetches a parent, or writes parent bytes.
1167-
Without the flag, init files and human/JSON output are unchanged.
1171+
`## overrides` headings. Its recommended version-2 flow tells you to
1172+
materialise every parent inside the repository first, calculate each pin with
1173+
`decided corpus digest --version 2 --root <parent-root> --corpus
1174+
<parent-corpus>`, and declare one to 32 records in the `parents` sequence.
1175+
It also retains the original version-1 digest command for existing
1176+
single-parent manifests. The flag works on fresh and already-initialized
1177+
repositories, with or without a profile. It is guidance only: it never
1178+
creates the manifest, fetches a parent, or writes parent bytes. Without the
1179+
flag, init files and human/JSON output are unchanged.
1180+
1181+
The version-2 manifest it describes has this shape (repeat the parent record
1182+
as needed; list order grants no precedence):
1183+
1184+
````markdown
1185+
## inherits
1186+
1187+
```yaml
1188+
version: 2
1189+
parents:
1190+
- alias: standards
1191+
source: acme/standards
1192+
root: vendor/standards
1193+
corpus: decisions
1194+
digest: sha256-v2:<64-lowercase-hex>
1195+
```
1196+
````
11681197
- **Exit codes:** `0` initialized, or already initialized with the same key
11691198
(idempotent) · `1` a different key is already established (never silently
11701199
rewritten), or a client config exists but cannot be merged into (malformed
@@ -1207,7 +1236,11 @@ With `--parent-corpus --json`, the response additionally includes:
12071236
"manifest": ".decided/corpus.md",
12081237
"inherits_heading": "## inherits",
12091238
"overrides_heading": "## overrides",
1210-
"digest_command": "decided corpus digest --root <parent-root> --corpus <parent-corpus>"
1239+
"digest_command": "decided corpus digest --root <parent-root> --corpus <parent-corpus>",
1240+
"recommended_manifest_version": 2,
1241+
"parents_field": "parents",
1242+
"multiple_parents": true,
1243+
"digest_command_v2": "decided corpus digest --version 2 --root <parent-root> --corpus <parent-corpus>"
12111244
}
12121245
}
12131246
```

rust/decided/tests/cli.rs

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -95,9 +95,12 @@ fn init_parent_corpus_emits_guidance_without_creating_a_manifest() {
9595
);
9696
let stdout = String::from_utf8_lossy(&output.stdout);
9797
for expected in [
98-
"Parent corpus setup:",
99-
"Materialise the parent inside this repository",
98+
"Parent corpus setup (manifest version 2):",
99+
"Materialise every parent inside this repository",
100+
"decided corpus digest --version 2 --root <parent-root> --corpus <parent-corpus>",
100101
"decided corpus digest --root <parent-root> --corpus <parent-corpus>",
102+
"one to 32 verified parents",
103+
"version: 2 parents sequence",
101104
".decided/corpus.md",
102105
"## inherits",
103106
"## overrides",
@@ -138,6 +141,12 @@ fn init_parent_corpus_is_profile_composable_and_idempotent() {
138141
let fresh_stdout = String::from_utf8_lossy(&fresh.stdout);
139142
assert!(fresh_stdout.contains("\"profile\": \"default\""));
140143
assert!(fresh_stdout.contains("\"parent_corpus_guidance\": {"));
144+
assert!(fresh_stdout.contains("\"recommended_manifest_version\": 2"));
145+
assert!(fresh_stdout.contains("\"parents_field\": \"parents\""));
146+
assert!(fresh_stdout.contains("\"multiple_parents\": true"));
147+
assert!(fresh_stdout.contains(
148+
"\"digest_command_v2\": \"decided corpus digest --version 2 --root <parent-root> --corpus <parent-corpus>\""
149+
));
141150
assert!(root.join(".mcp.json").is_file());
142151
assert!(root.join(".cursor/mcp.json").is_file());
143152
assert!(!root.join(".decided/corpus.md").exists());
@@ -152,7 +161,7 @@ fn init_parent_corpus_is_profile_composable_and_idempotent() {
152161
);
153162
let idempotent_stdout = String::from_utf8_lossy(&idempotent.stdout);
154163
assert!(idempotent_stdout.starts_with("Already initialized: repository key RAC\n"));
155-
assert!(idempotent_stdout.contains("Parent corpus setup:"));
164+
assert!(idempotent_stdout.contains("Parent corpus setup (manifest version 2):"));
156165
assert_eq!(
157166
fs::read(root.join(".decided/config.yaml")).expect("read config after re-init"),
158167
before
@@ -162,6 +171,26 @@ fn init_parent_corpus_is_profile_composable_and_idempotent() {
162171
fs::remove_dir_all(root).expect("remove parent-profile scratch repository");
163172
}
164173

174+
#[test]
175+
fn init_profile_without_parent_request_keeps_guidance_absent() {
176+
let root = empty_scratch_root("profile-without-parent-guidance");
177+
let root_text = root.to_string_lossy().into_owned();
178+
let output = run(&["init", &root_text, "--profile", "enterprise", "--json"]);
179+
assert!(
180+
output.status.success(),
181+
"stdout={}, stderr={}",
182+
String::from_utf8_lossy(&output.stdout),
183+
String::from_utf8_lossy(&output.stderr)
184+
);
185+
let payload: serde_json::Value =
186+
serde_json::from_slice(&output.stdout).expect("profile init JSON");
187+
assert_eq!(payload["profile"], "enterprise");
188+
assert!(payload.get("parent_corpus_guidance").is_none());
189+
assert!(!root.join(".decided/corpus.md").exists());
190+
191+
fs::remove_dir_all(root).expect("remove profile-without-guidance scratch repository");
192+
}
193+
165194
#[test]
166195
fn init_parent_corpus_preserves_an_existing_non_default_key() {
167196
let root = empty_scratch_root("parent-existing-key");

rust/decided/tests/eval_graph.rs

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
//! DecisionGrounding certification for the activated version-2 corpus graph.
2+
3+
use serde_json::Value;
4+
use std::path::{Path, PathBuf};
5+
use std::process::{Command, Output};
6+
7+
fn fixtures() -> PathBuf {
8+
Path::new(env!("CARGO_MANIFEST_DIR")).join("../fixtures/eval")
9+
}
10+
11+
fn run(args: &[String]) -> Output {
12+
Command::new(env!("CARGO_BIN_EXE_decided"))
13+
.args(args)
14+
.output()
15+
.expect("run decided eval")
16+
}
17+
18+
fn graph_args(mode: &str) -> Vec<String> {
19+
let root = fixtures().join("federation");
20+
vec![
21+
"eval".to_string(),
22+
mode.to_string(),
23+
"--root".to_string(),
24+
root.join("graph-decisions").to_string_lossy().into_owned(),
25+
"--queries".to_string(),
26+
root.join("graph-track/queries.json")
27+
.to_string_lossy()
28+
.into_owned(),
29+
"--baseline".to_string(),
30+
root.join("graph-track/baseline.json")
31+
.to_string_lossy()
32+
.into_owned(),
33+
"--config".to_string(),
34+
root.join("graph-track/eval-config.json")
35+
.to_string_lossy()
36+
.into_owned(),
37+
]
38+
}
39+
40+
#[test]
41+
fn graph_track_gates_combined_ranking_floor_and_transitive_decisions() {
42+
let checked = run(&graph_args("--check"));
43+
assert!(
44+
checked.status.success(),
45+
"stdout={}, stderr={}",
46+
String::from_utf8_lossy(&checked.stdout),
47+
String::from_utf8_lossy(&checked.stderr)
48+
);
49+
assert_eq!(
50+
String::from_utf8_lossy(&checked.stdout),
51+
"decided eval: gate PASS\n"
52+
);
53+
54+
let first = run(&graph_args("--json"));
55+
let second = run(&graph_args("--json"));
56+
assert!(first.status.success());
57+
assert!(second.status.success());
58+
let first: Value = serde_json::from_slice(&first.stdout).expect("first scorecard JSON");
59+
let second: Value = serde_json::from_slice(&second.stdout).expect("second scorecard JSON");
60+
assert_eq!(first["metrics"], second["metrics"]);
61+
assert_eq!(first["per_query"], second["per_query"]);
62+
assert_eq!(first["metadata"]["n_queries"], 4);
63+
assert_eq!(first["metrics"]["overall"]["negative_violations"], 0);
64+
65+
let queries = first["per_query"].as_array().expect("per-query rows");
66+
let gq01 = queries.iter().find(|row| row["id"] == "GQ01").unwrap();
67+
let large_parent = gq01["returned"].as_array().unwrap();
68+
assert_eq!(large_parent[0], "FEDEVAL-000000000001");
69+
assert!(
70+
large_parent
71+
.iter()
72+
.position(|id| id == "FEDEVAL-000000000002")
73+
.is_some_and(|position| position >= 5),
74+
"the high-inbound lexical hard negative entered the top-five window"
75+
);
76+
77+
let gq03 = queries.iter().find(|row| row["id"] == "GQ03").unwrap();
78+
assert_eq!(gq03["returned"][0], "FEDEVAL-000000000004");
79+
let gq04 = queries.iter().find(|row| row["id"] == "GQ04").unwrap();
80+
assert_eq!(gq04["returned"][0], "FEDEVAL-000000000005");
81+
}
82+
83+
#[test]
84+
fn no_manifest_track_retains_its_committed_metrics() {
85+
let root = fixtures();
86+
let output = run(&[
87+
"eval".to_string(),
88+
"--json".to_string(),
89+
"--root".to_string(),
90+
root.join("corpus").to_string_lossy().into_owned(),
91+
"--queries".to_string(),
92+
root.join("queries.json").to_string_lossy().into_owned(),
93+
]);
94+
assert!(
95+
output.status.success(),
96+
"{}",
97+
String::from_utf8_lossy(&output.stderr)
98+
);
99+
let scorecard: Value = serde_json::from_slice(&output.stdout).expect("scorecard JSON");
100+
let baseline: Value = serde_json::from_slice(
101+
&std::fs::read(root.join("baseline.json")).expect("read committed baseline"),
102+
)
103+
.expect("baseline JSON");
104+
assert_eq!(scorecard["metrics"], baseline);
105+
}

rust/fixtures/eval/README.md

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -27,11 +27,17 @@ is part of the repository's product-knowledge corpus.
2727
category's `p_at_1` / `r_at_5`. Per-tool figures are diagnostic.
2828
- `baseline.json` — the committed `metrics` baseline, written by
2929
`decided eval --update-baseline` (human-only; CI never rebaselines).
30-
- `federation/` — the ADR-139 DecisionGrounding track. Its child inherits a
31-
40-artifact standards parent containing a precise inherited match, six
32-
lexical near-matches, and a 32-inbound-edge hard negative. The hard negative
33-
has graph rank 1 but remains outside the top-five window because the v0.28
34-
lexical floor clamps its graph contribution.
30+
- `federation/child/` — the version-1 ADR-139 DecisionGrounding track. Its
31+
child inherits a 40-artifact standards parent containing a precise inherited
32+
match, six lexical near-matches, and a 32-inbound-edge hard negative. The hard
33+
negative has graph rank 1 but remains outside the top-five window because the
34+
v0.28 lexical floor clamps its graph contribution.
35+
- `federation/graph-decisions/` plus `federation/graph-track/` — the version-2
36+
extension of that same benchmark family. The root has three direct parents,
37+
reaches the 40-artifact standards corpus through a nested version-1 node, and
38+
reaches another Decision through a nested version-2 node. It measures the
39+
combined source-neutral BM25 index, the same lexical graph-floor hard
40+
negative, and transitive inherited search and relationship retrieval.
3541

3642
## Running
3743

@@ -47,6 +53,13 @@ decided eval --check \
4753
--queries rust/fixtures/eval/federation/queries.json \
4854
--baseline rust/fixtures/eval/federation/baseline.json \
4955
--config rust/fixtures/eval/federation/eval-config.json
56+
57+
# ADR-139 version-2 graph track (same scorer and metric family)
58+
decided eval --check \
59+
--root rust/fixtures/eval/federation/graph-decisions \
60+
--queries rust/fixtures/eval/federation/graph-track/queries.json \
61+
--baseline rust/fixtures/eval/federation/graph-track/baseline.json \
62+
--config rust/fixtures/eval/federation/graph-track/eval-config.json
5063
```
5164

5265
## Calibration
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
repository_key: GRAPH
2+
corpus:
3+
source: eval/graph-root
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# DecisionGrounding graph-federation fixture
2+
3+
## inherits
4+
5+
```yaml
6+
version: 2
7+
parents:
8+
- alias: application
9+
source: eval/child
10+
root: child
11+
corpus: decisions
12+
digest: sha256-v2:ac0003002cde387382ca5228bdf0f346ba3abbbd735c9349e54b587c0f319d5e
13+
- alias: policies
14+
source: eval/policy-tree
15+
root: policy-tree
16+
corpus: decisions
17+
digest: sha256-v2:de108483b57c703c901e0095f133036bf8c8622d18f228b428dd23a7d6bf10b8
18+
- alias: audit
19+
source: eval/audit
20+
root: audit
21+
corpus: decisions
22+
digest: sha256-v2:b6c3e3b33d73d0c281498cb55b6996a80080e4c6e2da3c16d547c481f3b0c1e4
23+
```
24+
25+
## overrides
26+
27+
```yaml
28+
version: 2
29+
items: []
30+
```
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
repository_key: AUDIT
2+
corpus:
3+
source: eval/audit
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
schema_version: 1
3+
id: FEDEVAL-000000000007
4+
type: decision
5+
---
6+
# Quorum Audit Archive
7+
8+
## Status
9+
10+
Accepted
11+
12+
## Context
13+
14+
Audit records mention quorum operations without defining rollback recovery.
15+
16+
## Decision
17+
18+
Retain the annual quorum archive for compliance sampling.
19+
20+
## Consequences
21+
22+
This direct-parent record is a lexical distractor, not the transitive standard.
23+
24+
## Category
25+
26+
Technical

0 commit comments

Comments
 (0)