diff --git a/docs/import-schema.md b/docs/import-schema.md index a3ce9dce..b867747f 100644 --- a/docs/import-schema.md +++ b/docs/import-schema.md @@ -56,7 +56,7 @@ Task-only optional fields: Task fields are valid only when `kind` is `task`. When a task import is promoted, Brigade preserves these fields on the local task ledger item and keeps source-specific details in `metadata`. -Durable non-task imports with kind `decision`, `preference`, `link`, `command`, `finding`, or `incident` can be promoted into a local Memory Handoff draft. Promotion writes to the configured handoff inbox, runs handoff lint, then marks the import `promoted` only after the draft is valid. The promoted import stores `handoff_path`, `handoff_target_document`, `promoted_at`, and `handoff_source_fingerprint`. Review those drafts with `brigade handoff list`, `brigade handoff show`, and `brigade handoff archive`; those commands do not run the canonical ingestor or edit memory. +Durable non-task imports with kind `decision`, `preference`, `link`, `command`, `finding`, or `incident` can be promoted into a local Memory Handoff draft. Promotion writes to the configured handoff inbox, runs handoff lint, then marks the import `promoted` only after the draft is valid. The promoted import stores `handoff_path`, `handoff_target_document`, `promoted_at`, and `handoff_source_fingerprint`. Review those drafts with `brigade handoff list`, `brigade handoff show`, and `brigade handoff archive`. Those commands do not run the canonical ingestor or edit memory. Recommended metadata keys: @@ -133,9 +133,9 @@ Each candidate object has this exact schema: } ``` -`pattern_key` is computed from the normalized sequence templates joined with newline characters. Normalization replaces volatile tokens with fixed placeholders before hashing: run-id shapes become ``, UUIDs and bare hex tokens of 12 or more characters become ``, ISO timestamps and dates become ``, and absolute paths under `/tmp` or the user home become ``. Repo-relative paths are kept as-is. The same workflow observed across runs that differ only by those tokens therefore produces the same `pattern_key`. The candidate `id` and `suggested_runbook_id` are both `workflow-`. `sequence` holds the templates; `example_commands` holds the concrete commands from the most recent observation. `review_risk` is `high` when any example command matches the advisory destructive deny-list used by runbook policy, otherwise `normal`. +`pattern_key` is computed from the normalized sequence templates joined with newline characters. Normalization replaces volatile tokens with fixed placeholders before hashing: run-id shapes become ``, UUIDs and bare hex tokens of 12 or more characters become ``, ISO timestamps and dates become ``, and absolute paths under `/tmp` or the user home become ``. Repo-relative paths are kept as-is. The same workflow observed across runs that differ only by those tokens therefore produces the same `pattern_key`. The candidate `id` and `suggested_runbook_id` are both `workflow-`. `sequence` holds the templates. `example_commands` holds the concrete commands from the most recent observation. `review_risk` is `high` when any example command matches the advisory destructive deny-list used by runbook policy, otherwise `normal`. -`workflow propose-runbook` resolves an exact candidate id or a unique prefix against `.brigade/workflow/latest.json`. It writes or prints a runbook whose id is `suggested_runbook_id`, whose description names the scan candidate and provenance, whose `approved` field is false, whose `pins` list is empty, whose `allowed_commands` are sorted command names from `example_commands`, and whose steps use concrete runnable commands from `example_commands` with `timeout_seconds` set to 600. The generated payload is validated against `brigade runbook plan` policy before anything is written; when validation fails, the command reports the policy failures, writes nothing, and exits 1. +`workflow propose-runbook` resolves an exact candidate id or a unique prefix against `.brigade/workflow/latest.json`. It writes or prints a runbook whose id is `suggested_runbook_id`, whose description names the scan candidate and provenance, whose `approved` field is false, whose `pins` list is empty, whose `allowed_commands` are sorted command names from `example_commands`, and whose steps use concrete runnable commands from `example_commands` with `timeout_seconds` set to 600. The generated payload is validated against `brigade runbook plan` policy before anything is written. When validation fails, the command reports the policy failures, writes nothing, and exits 1. Repo-fleet imports must use safe labels only. Do not copy full local paths, guidance file contents, private config values, raw logs, scanner output, owner names, exact private repo names, or raw evidence into import text or metadata. @@ -245,6 +245,51 @@ brigade chat sweep ingest discord-export brigade chat sweep import-issues discord-export ``` -Each export finding must provide `provider`, `surface_id`, `issue_id`, `issue_type`, `priority`, `confidence`, `safe_summary`, `evidence_summary`, `suggested_task_text`, and `acceptance_criteria`. Supported provider families are `discord-export`, `slack-export`, `telegram-export`, `clickclack-export`, and `generic-jsonl`; aliases such as `discord`, `slack-json`, `telegram`, `clickclack`, `generic`, and `jsonl` are normalized to those canonical families. +Each export finding must provide `provider`, `surface_id`, `issue_id`, `issue_type`, `priority`, `confidence`, `safe_summary`, `evidence_summary`, `suggested_task_text`, and `acceptance_criteria`. Supported provider families are `discord-export`, `slack-export`, `telegram-export`, `clickclack-export`, and `generic-jsonl`. Aliases such as `discord`, `slack-json`, `telegram`, `clickclack`, `generic`, and `jsonl` are normalized to those canonical families. `ingest` writes normalized sweep JSON under `.brigade/chat-memory-sweeps/`, and `import-issues` routes actionable items through the existing source `chat-memory-sweep` import path. Raw private chat fields such as `raw_text`, `raw_messages`, `message_text`, `messages`, and `transcript` are rejected by default. Use safe summaries, channel labels, message ranges, confidence, and local evidence paths instead. + +## Provenance Envelope + +New work import items will carry a `brigade.provenance-envelope.v1` envelope under `metadata.provenance`. The envelope records source, origin, trust, and an exact-byte SHA-256 content digest so consumers can derive entitlements from a shared trust policy. Slice 1 ships the schema, validator, and legacy read synthesis in `src/brigade/provenance.py`. Ingestion stamping and consumer enforcement land in later slices. + +### Location + +- Work imports: `metadata.provenance` on each inbox record. +- Fixture/policy files ship as package data: `src/brigade/fixtures/provenance-envelope.v1.golden.json` and `src/brigade/fixtures/trust-policy.v1.json`. + +### Field sets + +Closed sets enforced by `validate_envelope`: + +- `origin`: `operator-input`, `workspace`, `agent-session`, `external-service`, `external-web`, `unknown`. +- `modality`: `human-written`, `model-generated`, `tool-output`, `external-web`, `mixed`, `unknown`. +- `attribution`: `observed`, `declared`, `inferred`. +- `trust.label`: `unknown`, `untrusted`, `reviewed`, `verified`, `quarantined`. +- `trust.injection.status`: `clean`, `flagged`, `pending`, `error`. +- `locator.kind`: `repo-relative`, `uri`. +- `hashes.content_scope`: `item.text.utf8.v1` (evidence items) and `message.text.utf8.v1` (inter-seat messages). + +### Exact-byte scopes + +`hashes.content` is the bare lowercase 64-char hex SHA-256 of the exact UTF-8 bytes of the persisted item `text` field. No trimming, newline normalization, or Unicode normalization. `hashes.raw`, when present, is the SHA-256 of the exact retained source bytes with `raw_scope = exact_bytes`. When `raw` is absent, `raw`, `raw_algorithm`, and `raw_scope` are all null. Both algorithms must be `sha256`. + +### Trust policy entitlements and caps + +`trust.trust_policy` stores only `schema = brigade.trust-policy.v1` and `schema_version = 1`. Consumers load `src/brigade/fixtures/trust-policy.v1.json` to derive entitlements per label: `unknown` (search, show_metadata, forensic_content_reveal), `untrusted` (search, show, brief_wrapped with caps), `reviewed`/`verified` (search, show, brief, cite, promote), `quarantined` (search_metadata, show_metadata). `untrusted_caps` are `max_items = 2` and `max_fraction = 0.5`. + +### Size ceiling + +The canonical compact JSON encoding (`ensure_ascii = False`, UTF-8, sorted keys, no whitespace) must be no greater than 4096 bytes. `validate_envelope` rejects larger envelopes with a `size`/`4096` error. + +### Absolute-path ban + +`locator.value` must be repo-relative or a non-file URI. `validate_envelope` rejects POSIX absolute paths (`/etc/passwd`), Windows drive paths (`C:\\Users\\foo`), UNC paths (`\\host\share\file`), and `file:` URIs. + +### Authority rule + +An inbound adapter envelope claiming `trust.label = reviewed` or `verified` must pass `authority_proof = {"assigned_by": , "label": }` with exactly those two keys, where `assigned_by` matches `trust.assigned_by` and `label` matches `trust.label`. Otherwise the ingester downgrades or rejects the assertion. An inbound adapter envelope is data, not authority. + +### Legacy banner + +When an item carries no provenance, `synthesize_legacy_provenance()` returns a non-null envelope with `origin = unknown`, `modality = unknown`, `attribution = inferred`, `trust.label = unknown`, null `repository`/`session`/`collection_id`/`item_id`/`locator`, and a null content digest. The display string is `UNKNOWN PROVENANCE - legacy item` (`provenance.LEGACY_DISPLAY`). A missing envelope is never treated as trusted. diff --git a/engines/evidence-ledger/docs/SCHEMA.md b/engines/evidence-ledger/docs/SCHEMA.md index 65f67bac..4d3d14ce 100644 --- a/engines/evidence-ledger/docs/SCHEMA.md +++ b/engines/evidence-ledger/docs/SCHEMA.md @@ -18,3 +18,52 @@ The MVP uses one SQLite migration with these concepts: Raw adapter lines are preserved in `items.raw_json`. Raw source references are stored in `raw_hash`, `raw_path`, and `raw_ordinal`. The migration lives in `internal/archive/db.go`. + +## Provenance Envelope + +New MiseLedger `items` rows will carry a `brigade.provenance-envelope.v1` envelope under `metadata_json.provenance`. Slice 1 ships the typed Go mirror, validator, and legacy read synthesis in `internal/provenance/envelope.go`. Persistence, ingest stamping, and consumer enforcement land in later slices. + +### Location + +- MiseLedger: `items.metadata_json.provenance` (embedded sorted-key JSON, no document newline). +- Go mirror package: `internal/provenance` (`Envelope`, `Validate`, `SynthesizeLegacyProvenance`). + +### Field sets + +Closed sets enforced by `Validate`: + +- `origin`: `operator-input`, `workspace`, `agent-session`, `external-service`, `external-web`, `unknown`. +- `modality`: `human-written`, `model-generated`, `tool-output`, `external-web`, `mixed`, `unknown`. +- `attribution`: `observed`, `declared`, `inferred`. +- `trust.label`: `unknown`, `untrusted`, `reviewed`, `verified`, `quarantined`. +- `trust.injection.status`: `clean`, `flagged`, `pending`, `error`. +- `locator.kind`: `repo-relative`, `uri`. +- `hashes.content_scope`: `item.text.utf8.v1` (evidence items) and `message.text.utf8.v1` (inter-seat messages). + +### Exact-byte scopes + +`hashes.content` is the bare lowercase 64-char hex SHA-256 of the exact UTF-8 bytes of the persisted item `text` field. No trimming, newline normalization, or Unicode normalization. `hashes.raw`, when present, is the SHA-256 of the exact retained source bytes with `raw_scope = exact_bytes`. When `raw` is absent, `raw`, `raw_algorithm`, and `raw_scope` are all null. Both algorithms must be `sha256`. The envelope `hashes.content` is a separate contract from the legacy SQLite `items.content_hash` dedupe column. Verify each against its own scope and never compare them for equality. + +### Nullable fields + +The following string fields are nullable (JSON `null` ↔ Go `*string` nil): `collection_id`, `item_id`, `captured_at`, `ingested_at`, `trust.assigned_at`, `hashes.content`, `hashes.raw_algorithm`, `hashes.raw_scope`, and `hashes.raw`. `Validate` distinguishes a null pointer from a present empty string. For non-legacy envelopes, `collection_id`, `item_id`, and `hashes.content` must be present (non-null) and non-empty. For legacy envelopes, all nullable pointers may be null. When `hashes.raw` is null, `raw_algorithm` and `raw_scope` must also be null. When `hashes.raw` is present, it must be a valid 64-char hex digest and both `raw_algorithm` (`sha256`) and `raw_scope` (`exact_bytes`) must be present and non-null. + +### Trust policy entitlements and caps + +`trust.trust_policy` stores only `schema = brigade.trust-policy.v1` and `schema_version = 1`. Consumers load the shared `src/brigade/fixtures/trust-policy.v1.json` fixture to derive entitlements per label: `unknown` (search, show_metadata, forensic_content_reveal), `untrusted` (search, show, brief_wrapped with caps), `reviewed`/`verified` (search, show, brief, cite, promote), `quarantined` (search_metadata, show_metadata). `untrusted_caps` are `max_items = 2` and `max_fraction = 0.5`. + +### Size ceiling + +The canonical compact JSON encoding (UTF-8, no whitespace, HTML escaping disabled) must be no greater than 4096 bytes. `Validate` rejects larger envelopes with a `size`/`4096` error. Key order is not part of the contract: the Go validator measures the byte count of its compact non-HTML-escaped encoding, which is independent of object key ordering, matching Python's `ensure_ascii=False` compact byte count. + +### Absolute-path ban + +`locator.value` must be repo-relative or a non-file URI. `Validate` rejects POSIX absolute paths (`/etc/passwd`), Windows drive paths (`C:\\Users\\foo`), UNC paths (`\\host\share\file`), and `file:` URIs. + +### Authority rule + +An inbound adapter envelope claiming `trust.label = reviewed` or `verified` must pass `ValidationContext{InboundAdapter: true, AuthorityProof: &AuthorityProof{AssignedBy, Label}}` where `AssignedBy` matches `trust.assigned_by` and `Label` matches `trust.label`. Otherwise the ingester downgrades or rejects the assertion. An inbound adapter envelope is data, not authority. + +### Legacy banner + +When an item carries no provenance, `SynthesizeLegacyProvenance()` returns a non-null envelope with `origin = unknown`, `modality = unknown`, `attribution = inferred`, `trust.label = unknown`, nil `repository`/`session`/`locator`, null `collection_id`/`item_id`/`captured_at`/`ingested_at`/`trust.assigned_at`, and null `hashes.content`/`raw`/`raw_algorithm`/`raw_scope`. The display string is `UNKNOWN PROVENANCE - legacy item` (`provenance.LegacyDisplay`). A missing envelope is never treated as trusted. diff --git a/engines/evidence-ledger/internal/provenance/envelope.go b/engines/evidence-ledger/internal/provenance/envelope.go new file mode 100644 index 00000000..7648b8e9 --- /dev/null +++ b/engines/evidence-ledger/internal/provenance/envelope.go @@ -0,0 +1,393 @@ +// Package provenance implements the brigade.provenance-envelope.v1 schema: +// a versioned source/origin/trust record plus an exact-byte SHA-256 content +// digest stamped on every evidence item and inter-seat message. +// +// Standard library only. This is the Slice 1 shared schema layer; ingestion, +// consumers, and CLI enforcement land in later slices. See +// docs/proposals/provenance-envelope.md. +package provenance + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "strings" +) + +const ( + Schema = "brigade.provenance-envelope.v1" + SchemaVersion = 1 + TrustPolicySchema = "brigade.trust-policy.v1" + TrustPolicyVersion = 1 + LegacyDisplay = "UNKNOWN PROVENANCE - legacy item" + HashAlgorithm = "sha256" + RawScope = "exact_bytes" + MaxCompactBytes = 4096 +) + +var ( + origins = map[string]struct{}{"operator-input": {}, "workspace": {}, "agent-session": {}, "external-service": {}, "external-web": {}, "unknown": {}} + modalities = map[string]struct{}{"human-written": {}, "model-generated": {}, "tool-output": {}, "external-web": {}, "mixed": {}, "unknown": {}} + attributions = map[string]struct{}{"observed": {}, "declared": {}, "inferred": {}} + trustLabels = map[string]struct{}{"unknown": {}, "untrusted": {}, "reviewed": {}, "verified": {}, "quarantined": {}} + injectionStatuses = map[string]struct{}{"clean": {}, "flagged": {}, "pending": {}, "error": {}} + locatorKinds = map[string]struct{}{"repo-relative": {}, "uri": {}} + contentScopes = map[string]struct{}{"item.text.utf8.v1": {}, "message.text.utf8.v1": {}} +) + +// Envelope mirrors the brigade.provenance-envelope.v1 JSON object. +type Envelope struct { + Schema string `json:"schema"` + SchemaVersion int `json:"schema_version"` + Source Source `json:"source"` + Origin string `json:"origin"` + Repository *Repository `json:"repository"` + Session *Session `json:"session"` + CollectionID *string `json:"collection_id"` + ItemID *string `json:"item_id"` + Locator *Locator `json:"locator"` + Attribution string `json:"attribution"` + Modality string `json:"modality"` + Trust Trust `json:"trust"` + Hashes Hashes `json:"hashes"` + CapturedAt *string `json:"captured_at"` + IngestedAt *string `json:"ingested_at"` +} + +// Source is the producer system triple. +type Source struct { + System string `json:"system"` + Kind string `json:"kind"` + Producer string `json:"producer"` +} + +// Repository identifies the repo origin. +type Repository struct { + ID string `json:"id"` + Revision *string `json:"revision"` +} + +// Session identifies the harness session when known. +type Session struct { + ID *string `json:"id"` + Harness *string `json:"harness"` +} + +// Locator is a repo-relative path or non-file URI. +type Locator struct { + Kind string `json:"kind"` + Value string `json:"value"` +} + +// Trust carries the label, assignment, policy reference, and injection verdict. +type Trust struct { + Label string `json:"label"` + AssignedBy string `json:"assigned_by"` + AssignedAt *string `json:"assigned_at"` + TrustPolicy TrustPolicy `json:"trust_policy"` + Injection Injection `json:"injection"` +} + +// TrustPolicy references the shared entitlement policy schema and version. +type TrustPolicy struct { + Schema string `json:"schema"` + SchemaVersion int `json:"schema_version"` +} + +// Injection is the injection scan verdict triple. +type Injection struct { + Status string `json:"status"` + Count int `json:"count"` + Rules []string `json:"rules"` +} + +// Hashes carries the content and raw SHA-256 digests and their scopes. +type Hashes struct { + ContentAlgorithm string `json:"content_algorithm"` + ContentScope string `json:"content_scope"` + Content *string `json:"content"` + RawAlgorithm *string `json:"raw_algorithm"` + RawScope *string `json:"raw_scope"` + Raw *string `json:"raw"` +} + +// AuthorityProof is the operator/verifier attestation required for inbound +// adapter envelopes that claim reviewed or verified trust. +type AuthorityProof struct { + AssignedBy string `json:"assigned_by"` + Label string `json:"label"` +} + +// ValidationContext carries the optional authority gate inputs for Validate. +type ValidationContext struct { + InboundAdapter bool + AuthorityProof *AuthorityProof +} + +// SHA256Bytes returns the bare lowercase 64-char hex SHA-256 digest of data. +func SHA256Bytes(data []byte) string { + sum := sha256.Sum256(data) + return hex.EncodeToString(sum[:]) +} + +// ContentSHA256 returns the bare lowercase 64-char hex SHA-256 digest of the +// exact UTF-8 bytes of text. +func ContentSHA256(text string) string { + return SHA256Bytes([]byte(text)) +} + +func isLegacy(env Envelope) bool { + return env.Origin == "unknown" && env.Modality == "unknown" && + env.Attribution == "inferred" && env.Trust.Label == "unknown" +} + +func validDigest(s string) bool { + if len(s) != 64 { + return false + } + for i := 0; i < len(s); i++ { + c := s[i] + if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f')) { + return false + } + } + return true +} + +func isAbsoluteLocator(value string) bool { + if value == "" { + return false + } + if strings.HasPrefix(value, "/") { + return true + } + if len(value) >= 2 && value[1] == ':' && ((value[0] >= 'a' && value[0] <= 'z') || (value[0] >= 'A' && value[0] <= 'Z')) { + return true + } + if strings.HasPrefix(value, `\\`) { + return true + } + if len(value) >= 5 && strings.EqualFold(value[:5], "file:") { + return true + } + return false +} + +// Validate returns nil if env is a valid envelope, or an error whose message +// describes the first validation failure. The error message contains the +// offending field name so callers can match on it. +func Validate(env Envelope, ctx ValidationContext) error { + var errs []string + legacy := isLegacy(env) + + if env.Schema != Schema { + errs = append(errs, fmt.Sprintf("schema must be %q", Schema)) + } + if env.SchemaVersion != SchemaVersion { + errs = append(errs, fmt.Sprintf("schema_version must be %d", SchemaVersion)) + } + + if env.Source.System == "" { + errs = append(errs, "source.system must be a non-empty string") + } + if env.Source.Kind == "" { + errs = append(errs, "source.kind must be a non-empty string") + } + if env.Source.Producer == "" { + errs = append(errs, "source.producer must be a non-empty string") + } + + if _, ok := origins[env.Origin]; !ok { + errs = append(errs, fmt.Sprintf("origin %q is not in the closed set", env.Origin)) + } + if _, ok := modalities[env.Modality]; !ok { + errs = append(errs, fmt.Sprintf("modality %q is not in the closed set", env.Modality)) + } + if _, ok := attributions[env.Attribution]; !ok { + errs = append(errs, fmt.Sprintf("attribution %q is not in the closed set", env.Attribution)) + } + + if env.Repository == nil { + if !legacy { + errs = append(errs, "repository must be an object") + } + } else if env.Repository.ID == "" { + errs = append(errs, "repository.id must be a non-empty string") + } + + if env.Session == nil && !legacy { + errs = append(errs, "session must be an object") + } + + if env.CollectionID == nil { + if !legacy { + errs = append(errs, "collection_id must be a string") + } + } else if *env.CollectionID == "" && !legacy { + errs = append(errs, "collection_id must be a non-empty string") + } + if env.ItemID == nil { + if !legacy { + errs = append(errs, "item_id must be a string") + } + } else if *env.ItemID == "" && !legacy { + errs = append(errs, "item_id must be a non-empty string") + } + + if env.Locator == nil { + if !legacy { + errs = append(errs, "locator must be an object") + } + } else { + if _, ok := locatorKinds[env.Locator.Kind]; !ok { + errs = append(errs, fmt.Sprintf("locator.kind %q is not in the closed set", env.Locator.Kind)) + } + if env.Locator.Value == "" { + errs = append(errs, "locator.value must be a non-empty string") + } else if isAbsoluteLocator(env.Locator.Value) || + (env.Locator.Kind == "repo-relative" && hasParentSegment(env.Locator.Value)) { + errs = append(errs, fmt.Sprintf("locator.value %q is unsafe; locator must be repo-relative or a non-file URI", env.Locator.Value)) + } + } + + if _, ok := trustLabels[env.Trust.Label]; !ok { + errs = append(errs, fmt.Sprintf("trust.label %q is not in the closed set", env.Trust.Label)) + } + if env.Trust.AssignedBy == "" { + errs = append(errs, "trust.assigned_by must be a non-empty string") + } + if env.Trust.TrustPolicy.Schema != TrustPolicySchema { + errs = append(errs, fmt.Sprintf("trust.trust_policy.schema must be %q", TrustPolicySchema)) + } + if env.Trust.TrustPolicy.SchemaVersion != TrustPolicyVersion { + errs = append(errs, fmt.Sprintf("trust.trust_policy.schema_version must be %d", TrustPolicyVersion)) + } + if _, ok := injectionStatuses[env.Trust.Injection.Status]; !ok { + errs = append(errs, fmt.Sprintf("trust.injection.status %q is not in the closed set", env.Trust.Injection.Status)) + } + if env.Trust.Injection.Count < 0 { + errs = append(errs, "trust.injection.count must be a nonnegative integer") + } + for _, rule := range env.Trust.Injection.Rules { + if rule == "" { + errs = append(errs, "trust.injection.rules entries must be non-empty strings") + break + } + } + + if ctx.InboundAdapter && (env.Trust.Label == "reviewed" || env.Trust.Label == "verified") { + if ctx.AuthorityProof == nil { + errs = append(errs, fmt.Sprintf("inbound adapter trust.label %q requires authority_proof with assigned_by and label", env.Trust.Label)) + } else if ctx.AuthorityProof.AssignedBy != env.Trust.AssignedBy { + errs = append(errs, "authority_proof.assigned_by must match trust.assigned_by") + } else if ctx.AuthorityProof.Label != env.Trust.Label { + errs = append(errs, "authority_proof.label must match trust.label") + } + } + + if env.Hashes.ContentAlgorithm != HashAlgorithm { + errs = append(errs, fmt.Sprintf("hashes.content_algorithm must be %q", HashAlgorithm)) + } + if _, ok := contentScopes[env.Hashes.ContentScope]; !ok { + errs = append(errs, fmt.Sprintf("hashes.content_scope %q is not in the closed set", env.Hashes.ContentScope)) + } + if env.Hashes.Content == nil { + if !legacy { + errs = append(errs, "hashes.content digest must be a bare lowercase 64-char hex string") + } + } else if !validDigest(*env.Hashes.Content) { + errs = append(errs, "hashes.content digest must be a bare lowercase 64-char hex string") + } + if env.Hashes.Raw == nil { + if env.Hashes.RawAlgorithm != nil { + errs = append(errs, "hashes.raw_algorithm must be null when hashes.raw is null") + } + if env.Hashes.RawScope != nil { + errs = append(errs, "hashes.raw_scope must be null when hashes.raw is null") + } + } else { + if !validDigest(*env.Hashes.Raw) { + errs = append(errs, "hashes.raw digest must be a bare lowercase 64-char hex string") + } + if env.Hashes.RawAlgorithm == nil || *env.Hashes.RawAlgorithm != HashAlgorithm { + errs = append(errs, fmt.Sprintf("hashes.raw_algorithm must be %q", HashAlgorithm)) + } + if env.Hashes.RawScope == nil || *env.Hashes.RawScope != RawScope { + errs = append(errs, fmt.Sprintf("hashes.raw_scope must be %q", RawScope)) + } + } + + if len(errs) > 0 { + return errors.New(strings.Join(errs, "; ")) + } + + var compactBuf bytes.Buffer + encoder := json.NewEncoder(&compactBuf) + encoder.SetEscapeHTML(false) + if err := encoder.Encode(env); err != nil { + return fmt.Errorf("envelope compact JSON marshal failed: %w", err) + } + compact := bytes.TrimSuffix(compactBuf.Bytes(), []byte("\n")) + if len(compact) > MaxCompactBytes { + return fmt.Errorf("envelope compact JSON size exceeds %d bytes", MaxCompactBytes) + } + return nil +} + +func hasParentSegment(value string) bool { + for _, part := range strings.Split(strings.ReplaceAll(value, `\`, "/"), "/") { + if part == ".." { + return true + } + } + return false +} + +// SynthesizeLegacyProvenance returns the non-null envelope used on read when an +// item carries no provenance, plus the human-readable legacy display banner. +func SynthesizeLegacyProvenance() (Envelope, string) { + return Envelope{ + Schema: Schema, + SchemaVersion: SchemaVersion, + Source: Source{ + System: "legacy", + Kind: "legacy", + Producer: "legacy.read_synthesis", + }, + Origin: "unknown", + Repository: nil, + Session: nil, + CollectionID: nil, + ItemID: nil, + Locator: nil, + Attribution: "inferred", + Modality: "unknown", + Trust: Trust{ + Label: "unknown", + AssignedBy: "ingest:legacy.read_synthesis", + AssignedAt: nil, + TrustPolicy: TrustPolicy{ + Schema: TrustPolicySchema, + SchemaVersion: TrustPolicyVersion, + }, + Injection: Injection{ + Status: "clean", + Count: 0, + Rules: []string{}, + }, + }, + Hashes: Hashes{ + ContentAlgorithm: HashAlgorithm, + ContentScope: "item.text.utf8.v1", + Content: nil, + RawAlgorithm: nil, + RawScope: nil, + Raw: nil, + }, + CapturedAt: nil, + IngestedAt: nil, + }, LegacyDisplay +} diff --git a/engines/evidence-ledger/internal/provenance/envelope_test.go b/engines/evidence-ledger/internal/provenance/envelope_test.go new file mode 100644 index 00000000..9eb6a712 --- /dev/null +++ b/engines/evidence-ledger/internal/provenance/envelope_test.go @@ -0,0 +1,332 @@ +package provenance_test + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/escoffier-labs/miseledger/internal/provenance" +) + +func fixturesDir(t *testing.T) string { + t.Helper() + wd, err := os.Getwd() + if err != nil { + t.Fatalf("getwd: %v", err) + } + return filepath.Join(wd, "..", "..", "..", "..", "src", "brigade", "fixtures") +} + +// strPtr returns a pointer to s, used to set nullable envelope string fields +// in tests that mutate fixture-backed envelopes. +func strPtr(s string) *string { return &s } + +type goldenCase struct { + Name string `json:"name"` + Scope string `json:"scope"` + Text string `json:"text"` + Envelope json.RawMessage `json:"envelope"` +} + +func loadGoldenCases(t *testing.T) []goldenCase { + t.Helper() + b, err := os.ReadFile(filepath.Join(fixturesDir(t), "provenance-envelope.v1.golden.json")) + if err != nil { + t.Fatalf("read golden: %v", err) + } + var doc struct { + Schema string `json:"schema"` + Cases []goldenCase `json:"cases"` + } + if err := json.Unmarshal(b, &doc); err != nil { + t.Fatalf("unmarshal golden: %v", err) + } + if len(doc.Cases) != 2 { + t.Fatalf("expected 2 golden cases, got %d", len(doc.Cases)) + } + return doc.Cases +} + +func validEnvelope(t *testing.T) provenance.Envelope { + t.Helper() + for _, c := range loadGoldenCases(t) { + if c.Name != "item_unicode_trailing_newline" { + continue + } + var env provenance.Envelope + if err := json.Unmarshal(c.Envelope, &env); err != nil { + t.Fatalf("unmarshal item envelope: %v", err) + } + return env + } + t.Fatal("item golden case not found") + return provenance.Envelope{} +} + +func TestContentSHA256ParityWithGoldenFixtures(t *testing.T) { + for _, c := range loadGoldenCases(t) { + var env provenance.Envelope + if err := json.Unmarshal(c.Envelope, &env); err != nil { + t.Fatalf("unmarshal %s: %v", c.Name, err) + } + if got := provenance.ContentSHA256(c.Text); got != *env.Hashes.Content { + t.Errorf("%s: ContentSHA256=%q want %q", c.Name, got, *env.Hashes.Content) + } + if got := provenance.SHA256Bytes([]byte(c.Text)); got != *env.Hashes.Content { + t.Errorf("%s: SHA256Bytes=%q want %q", c.Name, got, *env.Hashes.Content) + } + if len(*env.Hashes.Content) != 64 || strings.ToLower(*env.Hashes.Content) != *env.Hashes.Content { + t.Errorf("%s: digest not bare lowercase 64: %q", c.Name, *env.Hashes.Content) + } + } +} + +func TestMessageScopeDistinctBytesFromItem(t *testing.T) { + digests := map[string]string{} + for _, c := range loadGoldenCases(t) { + var env provenance.Envelope + if err := json.Unmarshal(c.Envelope, &env); err != nil { + t.Fatalf("unmarshal %s: %v", c.Name, err) + } + digests[c.Name] = *env.Hashes.Content + } + if digests["item_unicode_trailing_newline"] == digests["message_distinct_bytes"] { + t.Fatal("item and message content digests must differ") + } +} + +func TestEnvelopeFixturesUnmarshalIntoTypedStructs(t *testing.T) { + for _, c := range loadGoldenCases(t) { + var env provenance.Envelope + if err := json.Unmarshal(c.Envelope, &env); err != nil { + t.Fatalf("unmarshal %s: %v", c.Name, err) + } + if env.Schema != "brigade.provenance-envelope.v1" { + t.Errorf("%s: schema=%q", c.Name, env.Schema) + } + if env.SchemaVersion != 1 { + t.Errorf("%s: schema_version=%d", c.Name, env.SchemaVersion) + } + if env.Hashes.ContentScope != c.Scope { + t.Errorf("%s: content_scope=%q want %q", c.Name, env.Hashes.ContentScope, c.Scope) + } + if env.Trust.TrustPolicy.Schema != "brigade.trust-policy.v1" || env.Trust.TrustPolicy.SchemaVersion != 1 { + t.Errorf("%s: trust_policy mismatch: %+v", c.Name, env.Trust.TrustPolicy) + } + } +} + +func TestValidateAcceptsGoldenEnvelopes(t *testing.T) { + for _, c := range loadGoldenCases(t) { + var env provenance.Envelope + if err := json.Unmarshal(c.Envelope, &env); err != nil { + t.Fatalf("unmarshal %s: %v", c.Name, err) + } + if err := provenance.Validate(env, provenance.ValidationContext{}); err != nil { + t.Errorf("%s: validate: %v", c.Name, err) + } + } +} + +func TestValidateRejectsInvalidClosedSetValues(t *testing.T) { + tests := []struct { + name string + mutate func(*provenance.Envelope) + want string + }{ + {"origin underscore", func(e *provenance.Envelope) { e.Origin = "operator_input" }, "origin"}, + {"origin wrong case", func(e *provenance.Envelope) { e.Origin = "AGENT-SESSION" }, "origin"}, + {"modality underscore", func(e *provenance.Envelope) { e.Modality = "model_generated" }, "modality"}, + {"attribution bogus", func(e *provenance.Envelope) { e.Attribution = "guess" }, "attribution"}, + {"trust label trusted", func(e *provenance.Envelope) { e.Trust.Label = "trusted" }, "label"}, + {"injection status ok", func(e *provenance.Envelope) { e.Trust.Injection.Status = "ok" }, "injection"}, + {"content scope v2", func(e *provenance.Envelope) { e.Hashes.ContentScope = "item.text.utf8.v2" }, "scope"}, + {"locator kind absolute", func(e *provenance.Envelope) { e.Locator.Kind = "absolute" }, "locator"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + env := validEnvelope(t) + tt.mutate(&env) + err := provenance.Validate(env, provenance.ValidationContext{}) + if err == nil { + t.Fatalf("expected error for %s", tt.name) + } + if !strings.Contains(err.Error(), tt.want) && !strings.Contains(err.Error(), "closed") && !strings.Contains(err.Error(), "enum") { + t.Fatalf("error %q does not mention %q", err.Error(), tt.want) + } + }) + } +} + +func TestValidateRejectsBadDigestForms(t *testing.T) { + tests := []struct { + name string + mutate func(*provenance.Envelope) + }{ + {"uppercase", func(e *provenance.Envelope) { e.Hashes.Content = strPtr(strings.ToUpper(*e.Hashes.Content)) }}, + {"too short", func(e *provenance.Envelope) { e.Hashes.Content = strPtr("abc") }}, + {"non hex", func(e *provenance.Envelope) { e.Hashes.Content = strPtr(strings.Repeat("z", 64)) }}, + {"sha256 prefix", func(e *provenance.Envelope) { e.Hashes.Content = strPtr("sha256:" + strings.Repeat("a", 64)) }}, + {"empty", func(e *provenance.Envelope) { e.Hashes.Content = strPtr("") }}, + {"raw uppercase", func(e *provenance.Envelope) { e.Hashes.Raw = strPtr(strings.ToUpper(*e.Hashes.Raw)) }}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + env := validEnvelope(t) + tt.mutate(&env) + err := provenance.Validate(env, provenance.ValidationContext{}) + if err == nil { + t.Fatalf("expected error for %s", tt.name) + } + if !strings.Contains(err.Error(), "digest") && !strings.Contains(err.Error(), "hash") { + t.Fatalf("error %q does not mention digest/hash", err.Error()) + } + }) + } +} + +func TestValidateRejectsUnsafeAbsoluteLocators(t *testing.T) { + tests := []struct { + name string + value string + }{ + {"posix absolute", "/etc/passwd"}, + {"home absolute", "/home/user/secret"}, + {"windows drive", "C:\\Users\\foo"}, + {"windows unc", "\\\\host\\share\\file"}, + {"file URI", "file:///home/user/secret"}, + {"parent traversal", "../secret"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + env := validEnvelope(t) + env.Locator.Value = tt.value + err := provenance.Validate(env, provenance.ValidationContext{}) + if err == nil { + t.Fatalf("expected error for %s", tt.name) + } + if !strings.Contains(err.Error(), "absolute") && !strings.Contains(err.Error(), "locator") { + t.Fatalf("error %q does not mention absolute/locator", err.Error()) + } + }) + } +} + +func TestValidateAuthorityForInboundAdapter(t *testing.T) { + tests := []struct { + name string + inbound bool + label string + proof *provenance.AuthorityProof + wantValid bool + }{ + {"inbound reviewed no proof", true, "reviewed", nil, false}, + {"inbound reviewed proof match", true, "reviewed", &provenance.AuthorityProof{AssignedBy: "verifier:demo", Label: "reviewed"}, true}, + {"inbound reviewed assigned_by mismatch", true, "reviewed", &provenance.AuthorityProof{AssignedBy: "verifier:other", Label: "reviewed"}, false}, + {"inbound reviewed label mismatch", true, "reviewed", &provenance.AuthorityProof{AssignedBy: "verifier:demo", Label: "verified"}, false}, + {"inbound verified proof match", true, "verified", &provenance.AuthorityProof{AssignedBy: "verifier:demo", Label: "verified"}, true}, + {"inbound untrusted no proof", true, "untrusted", nil, true}, + {"not inbound reviewed no proof", false, "reviewed", nil, true}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + env := validEnvelope(t) + env.Trust.Label = tt.label + env.Trust.AssignedBy = "verifier:demo" + err := provenance.Validate(env, provenance.ValidationContext{InboundAdapter: tt.inbound, AuthorityProof: tt.proof}) + if (err == nil) != tt.wantValid { + t.Fatalf("%s: wantValid=%v got err=%v", tt.name, tt.wantValid, err) + } + }) + } +} + +func TestLegacyDisplayConstantAndSynthesis(t *testing.T) { + if provenance.LegacyDisplay != "UNKNOWN PROVENANCE - legacy item" { + t.Fatalf("LegacyDisplay=%q", provenance.LegacyDisplay) + } + env, display := provenance.SynthesizeLegacyProvenance() + if display != provenance.LegacyDisplay { + t.Fatalf("display=%q want provenance.LegacyDisplay=%q", display, provenance.LegacyDisplay) + } + if env.Origin != "unknown" || env.Modality != "unknown" || env.Attribution != "inferred" { + t.Fatalf("legacy fields wrong: %+v", env) + } + if env.Trust.Label != "unknown" { + t.Fatalf("legacy trust label=%q", env.Trust.Label) + } + if env.Hashes.Content != nil { + t.Fatalf("legacy content digest must be null/empty, got %q", *env.Hashes.Content) + } + if err := provenance.Validate(env, provenance.ValidationContext{}); err != nil { + t.Fatalf("legacy envelope invalid: %v", err) + } +} + +func TestLegacyMarshalUsesNullForNullableFields(t *testing.T) { + env, _ := provenance.SynthesizeLegacyProvenance() + data, err := json.Marshal(env) + if err != nil { + t.Fatalf("marshal legacy envelope: %v", err) + } + var payload map[string]any + if err := json.Unmarshal(data, &payload); err != nil { + t.Fatalf("unmarshal legacy envelope: %v", err) + } + for _, field := range []string{"collection_id", "item_id", "captured_at", "ingested_at"} { + if payload[field] != nil { + t.Errorf("%s=%#v, want JSON null", field, payload[field]) + } + } + trust := payload["trust"].(map[string]any) + if trust["assigned_at"] != nil { + t.Errorf("trust.assigned_at=%#v, want JSON null", trust["assigned_at"]) + } + hashes := payload["hashes"].(map[string]any) + for _, field := range []string{"content", "raw_algorithm", "raw_scope", "raw"} { + if hashes[field] != nil { + t.Errorf("hashes.%s=%#v, want JSON null", field, hashes[field]) + } + } +} + +func TestValidateSizeUsesNonHTMLEscapedCompactJSON(t *testing.T) { + env := validEnvelope(t) + env.Source.Producer = strings.Repeat("&", 2800) + + var canonical bytes.Buffer + encoder := json.NewEncoder(&canonical) + encoder.SetEscapeHTML(false) + if err := encoder.Encode(env); err != nil { + t.Fatalf("encode non-HTML-escaped envelope: %v", err) + } + compactSize := len(bytes.TrimSuffix(canonical.Bytes(), []byte("\n"))) + if compactSize > provenance.MaxCompactBytes { + t.Fatalf("test envelope size=%d, want <=%d", compactSize, provenance.MaxCompactBytes) + } + if err := provenance.Validate(env, provenance.ValidationContext{}); err != nil { + t.Fatalf("validator counted HTML escaping against size ceiling: %v", err) + } +} + +func TestValidateRejectsEnvelopeAbove4096CompactBytes(t *testing.T) { + env := validEnvelope(t) + env.ItemID = strPtr(strings.Repeat("x", 5000)) + err := provenance.Validate(env, provenance.ValidationContext{}) + if err == nil { + t.Fatal("expected size error for >4096 compact bytes") + } + if !strings.Contains(err.Error(), "size") && !strings.Contains(err.Error(), "4096") { + t.Fatalf("error %q does not mention size/4096", err.Error()) + } +} + +func TestValidateAcceptsEnvelopeUnder4096CompactBytes(t *testing.T) { + env := validEnvelope(t) + if err := provenance.Validate(env, provenance.ValidationContext{}); err != nil { + t.Fatalf("compact envelope rejected: %v", err) + } +} diff --git a/pyproject.toml b/pyproject.toml index 62167e04..32e414a3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -54,7 +54,7 @@ brigade = "brigade.cli:main" where = ["src"] [tool.setuptools.package-data] -brigade = ["py.typed", "templates/**/*", "guard/policies/*.json", "guard/examples/**/*.json"] +brigade = ["py.typed", "templates/**/*", "guard/policies/*.json", "guard/examples/**/*.json", "fixtures/*.json"] [tool.pytest.ini_options] testpaths = ["tests"] diff --git a/src/brigade/fixtures/provenance-envelope.v1.golden.json b/src/brigade/fixtures/provenance-envelope.v1.golden.json new file mode 100644 index 00000000..d25eb807 --- /dev/null +++ b/src/brigade/fixtures/provenance-envelope.v1.golden.json @@ -0,0 +1,76 @@ +{ + "schema": "brigade.provenance-envelope-fixtures.v1", + "schema_version": 1, + "cases": [ + { + "name": "item_unicode_trailing_newline", + "scope": "item.text.utf8.v1", + "text": "café\n", + "envelope": { + "schema": "brigade.provenance-envelope.v1", + "schema_version": 1, + "source": {"system": "work-inbox", "kind": "context", "producer": "ledger._make_import"}, + "origin": "workspace", + "repository": {"id": "escoffier-labs/brigade", "revision": null}, + "session": {"id": "demo-session", "harness": "cursor"}, + "collection_id": "demo-collection", + "item_id": "demo:item:1", + "locator": {"kind": "repo-relative", "value": "demo/item.txt"}, + "attribution": "observed", + "modality": "tool-output", + "trust": { + "label": "untrusted", + "assigned_by": "ingest:ledger._make_import", + "assigned_at": "2026-07-26T21:31:37.123456+00:00", + "trust_policy": {"schema": "brigade.trust-policy.v1", "schema_version": 1}, + "injection": {"status": "clean", "count": 0, "rules": []} + }, + "hashes": { + "content_algorithm": "sha256", + "content_scope": "item.text.utf8.v1", + "content": "7b49b9e063bd91a4f9252b413261f5557b9c570aa61516989499f64a62dbcdd6", + "raw_algorithm": "sha256", + "raw_scope": "exact_bytes", + "raw": "7b49b9e063bd91a4f9252b413261f5557b9c570aa61516989499f64a62dbcdd6" + }, + "captured_at": "2026-07-26T21:31:37+00:00", + "ingested_at": "2026-07-26T21:31:38+00:00" + } + }, + { + "name": "message_distinct_bytes", + "scope": "message.text.utf8.v1", + "text": "worker-result: done", + "envelope": { + "schema": "brigade.provenance-envelope.v1", + "schema_version": 1, + "source": {"system": "receipts", "kind": "worker-result", "producer": "aboyeur._worker_prompt"}, + "origin": "agent-session", + "repository": {"id": "escoffier-labs/brigade", "revision": "abc123def456"}, + "session": {"id": "demo-run", "harness": "claude"}, + "collection_id": "brigade_work_runs", + "item_id": "worker-result:demo", + "locator": {"kind": "repo-relative", "value": ".brigade/work/runs/demo.json"}, + "attribution": "observed", + "modality": "model-generated", + "trust": { + "label": "untrusted", + "assigned_by": "ingest:aboyeur._worker_prompt", + "assigned_at": "2026-07-26T22:00:00+00:00", + "trust_policy": {"schema": "brigade.trust-policy.v1", "schema_version": 1}, + "injection": {"status": "clean", "count": 0, "rules": []} + }, + "hashes": { + "content_algorithm": "sha256", + "content_scope": "message.text.utf8.v1", + "content": "829cbcc2ac2c1ec278ecef0361e669309a1e18924bc17f100781580ef5fb8f08", + "raw_algorithm": "sha256", + "raw_scope": "exact_bytes", + "raw": "829cbcc2ac2c1ec278ecef0361e669309a1e18924bc17f100781580ef5fb8f08" + }, + "captured_at": "2026-07-26T22:00:00+00:00", + "ingested_at": "2026-07-26T22:00:01+00:00" + } + } + ] +} diff --git a/src/brigade/fixtures/trust-policy.v1.json b/src/brigade/fixtures/trust-policy.v1.json new file mode 100644 index 00000000..f6cd8fe2 --- /dev/null +++ b/src/brigade/fixtures/trust-policy.v1.json @@ -0,0 +1,12 @@ +{ + "schema": "brigade.trust-policy.v1", + "schema_version": 1, + "entitlements": { + "unknown": ["search", "show_metadata", "forensic_content_reveal"], + "untrusted": ["search", "show", "brief_wrapped"], + "reviewed": ["search", "show", "brief", "cite", "promote"], + "verified": ["search", "show", "brief", "cite", "promote"], + "quarantined": ["search_metadata", "show_metadata"] + }, + "untrusted_caps": {"max_items": 2, "max_fraction": 0.5} +} diff --git a/src/brigade/provenance.py b/src/brigade/provenance.py new file mode 100644 index 00000000..8ea1c727 --- /dev/null +++ b/src/brigade/provenance.py @@ -0,0 +1,388 @@ +"""Provenance envelope builder, validator, and legacy read synthesis. + +Implements ``brigade.provenance-envelope.v1`` (see +docs/proposals/provenance-envelope.md, Slice 1). The envelope stamps every +evidence item and inter-seat message with a versioned source/origin/trust +record plus an exact-byte SHA-256 content digest. This module is the shared +schema layer; ingestion, consumers, and CLI enforcement land in later slices. + +Standard library only. Brigade is zero-runtime-dependency. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from typing import Any, Mapping, Sequence + +SCHEMA = "brigade.provenance-envelope.v1" +SCHEMA_VERSION = 1 +TRUST_POLICY_SCHEMA = "brigade.trust-policy.v1" +TRUST_POLICY_VERSION = 1 +LEGACY_DISPLAY = "UNKNOWN PROVENANCE - legacy item" +MAX_COMPACT_BYTES = 4096 + +ORIGINS = frozenset({"operator-input", "workspace", "agent-session", "external-service", "external-web", "unknown"}) +MODALITIES = frozenset({"human-written", "model-generated", "tool-output", "external-web", "mixed", "unknown"}) +ATTRIBUTIONS = frozenset({"observed", "declared", "inferred"}) +TRUST_LABELS = frozenset({"unknown", "untrusted", "reviewed", "verified", "quarantined"}) +INJECTION_STATUSES = frozenset({"clean", "flagged", "pending", "error"}) +LOCATOR_KINDS = frozenset({"repo-relative", "uri"}) +CONTENT_SCOPES = frozenset({"item.text.utf8.v1", "message.text.utf8.v1"}) +RAW_SCOPE = "exact_bytes" +HASH_ALGORITHM = "sha256" + +_HEX64 = re.compile(r"^[0-9a-f]{64}$") + + +def _is_legacy(env: Mapping[str, Any]) -> bool: + return ( + env.get("origin") == "unknown" + and env.get("modality") == "unknown" + and env.get("attribution") == "inferred" + and isinstance(env.get("trust"), Mapping) + and env.get("trust", {}).get("label") == "unknown" + ) + + +def sha256_bytes(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def content_sha256(text: str) -> str: + return sha256_bytes(text.encode("utf-8")) + + +def _valid_digest(value: Any) -> bool: + return isinstance(value, str) and bool(_HEX64.match(value)) + + +def _is_absolute_locator(value: Any) -> bool: + if not isinstance(value, str) or not value: + return False + if value.startswith("/"): + return True + if len(value) >= 2 and value[1] == ":" and value[0].isalpha(): + return True + if value.startswith("\\\\"): + return True + if value.lower().startswith("file:"): + return True + return False + + +def validate_envelope( + env: Any, + *, + inbound_adapter: bool = False, + authority_proof: Mapping[str, Any] | None = None, +) -> list[str]: + errors: list[str] = [] + if not isinstance(env, Mapping): + return ["envelope must be a JSON object"] + + if env.get("schema") != SCHEMA: + errors.append(f"schema must be {SCHEMA!r}") + if env.get("schema_version") != SCHEMA_VERSION: + errors.append(f"schema_version must be {SCHEMA_VERSION}") + + legacy = _is_legacy(env) + + src = env.get("source") + if not isinstance(src, Mapping): + errors.append("source must be an object") + else: + for key in ("system", "kind", "producer"): + val = src.get(key) + if not isinstance(val, str) or not val: + errors.append(f"source.{key} must be a non-empty string") + + origin = env.get("origin") + if origin not in ORIGINS: + errors.append(f"origin {origin!r} is not in the closed set {sorted(ORIGINS)}") + + modality = env.get("modality") + if modality not in MODALITIES: + errors.append(f"modality {modality!r} is not in the closed set {sorted(MODALITIES)}") + + attribution = env.get("attribution") + if attribution not in ATTRIBUTIONS: + errors.append(f"attribution {attribution!r} is not in the closed set {sorted(ATTRIBUTIONS)}") + + repo = env.get("repository") + if repo is None: + if not legacy: + errors.append("repository must be an object") + elif not isinstance(repo, Mapping): + errors.append("repository must be an object or null") + else: + rid = repo.get("id") + if not isinstance(rid, str) or not rid: + errors.append("repository.id must be a non-empty string") + rev = repo.get("revision") + if rev is not None and not isinstance(rev, str): + errors.append("repository.revision must be a string or null") + + session = env.get("session") + if session is None: + if not legacy: + errors.append("session must be an object") + elif not isinstance(session, Mapping): + errors.append("session must be an object or null") + else: + sid = session.get("id") + if sid is not None and not isinstance(sid, str): + errors.append("session.id must be a string or null") + harness = session.get("harness") + if harness is not None and not isinstance(harness, str): + errors.append("session.harness must be a string or null") + + for key in ("collection_id", "item_id"): + val = env.get(key) + if val is None: + if not legacy: + errors.append(f"{key} must be a string") + elif not isinstance(val, str): + errors.append(f"{key} must be a string or null") + + locator = env.get("locator") + if locator is None: + if not legacy: + errors.append("locator must be an object") + elif not isinstance(locator, Mapping): + errors.append("locator must be an object or null") + else: + lkind = locator.get("kind") + if lkind not in LOCATOR_KINDS: + errors.append(f"locator.kind {lkind!r} is not in the closed set {sorted(LOCATOR_KINDS)}") + lvalue = locator.get("value") + if not isinstance(lvalue, str) or not lvalue: + errors.append("locator.value must be a non-empty string") + elif _is_absolute_locator(lvalue) or ( + lkind == "repo-relative" and ".." in lvalue.replace("\\", "/").split("/") + ): + errors.append(f"locator.value {lvalue!r} is unsafe; locator must be repo-relative or a non-file URI") + + trust = env.get("trust") + if not isinstance(trust, Mapping): + errors.append("trust must be an object") + else: + label = trust.get("label") + if label not in TRUST_LABELS: + errors.append(f"trust.label {label!r} is not in the closed set {sorted(TRUST_LABELS)}") + assigned_by = trust.get("assigned_by") + if not isinstance(assigned_by, str) or not assigned_by: + errors.append("trust.assigned_by must be a non-empty string") + assigned_at = trust.get("assigned_at") + if assigned_at is not None and not isinstance(assigned_at, str): + errors.append("trust.assigned_at must be a string or null") + + policy = trust.get("trust_policy") + if not isinstance(policy, Mapping): + errors.append("trust.trust_policy must be an object") + else: + if policy.get("schema") != TRUST_POLICY_SCHEMA: + errors.append(f"trust.trust_policy.schema must be {TRUST_POLICY_SCHEMA!r}") + if policy.get("schema_version") != TRUST_POLICY_VERSION: + errors.append(f"trust.trust_policy.schema_version must be {TRUST_POLICY_VERSION}") + + injection = trust.get("injection") + if not isinstance(injection, Mapping): + errors.append("trust.injection must be an object") + else: + status = injection.get("status") + if status not in INJECTION_STATUSES: + errors.append( + f"trust.injection.status {status!r} is not in the closed set {sorted(INJECTION_STATUSES)}" + ) + count = injection.get("count") + if not isinstance(count, int) or isinstance(count, bool) or count < 0: + errors.append("trust.injection.count must be a nonnegative integer") + rules = injection.get("rules") + if not isinstance(rules, Sequence) or isinstance(rules, (str, bytes)): + errors.append("trust.injection.rules must be a list") + else: + for rule in rules: + if not isinstance(rule, str) or not rule: + errors.append("trust.injection.rules entries must be non-empty strings") + + if inbound_adapter and label in ("reviewed", "verified"): + if not isinstance(authority_proof, Mapping): + errors.append( + f"inbound adapter trust.label {label!r} requires authority_proof with assigned_by and label" + ) + else: + proof_keys = set(authority_proof.keys()) + if proof_keys != {"assigned_by", "label"}: + errors.append("authority_proof must have exactly assigned_by and label keys") + else: + if authority_proof.get("assigned_by") != assigned_by: + errors.append("authority_proof.assigned_by must match trust.assigned_by") + if authority_proof.get("label") != label: + errors.append("authority_proof.label must match trust.label") + + hashes = env.get("hashes") + if not isinstance(hashes, Mapping): + errors.append("hashes must be an object") + else: + if hashes.get("content_algorithm") != HASH_ALGORITHM: + errors.append(f"hashes.content_algorithm must be {HASH_ALGORITHM!r}") + cscope = hashes.get("content_scope") + if cscope not in CONTENT_SCOPES: + errors.append(f"hashes.content_scope {cscope!r} is not in the closed set {sorted(CONTENT_SCOPES)}") + content = hashes.get("content") + if content is None: + if not legacy: + errors.append("hashes.content digest must be a bare lowercase 64-char hex string") + elif not _valid_digest(content): + errors.append("hashes.content digest must be a bare lowercase 64-char hex string") + + raw = hashes.get("raw") + if raw is None: + if hashes.get("raw_algorithm") is not None: + errors.append("hashes.raw_algorithm must be null when hashes.raw is null") + if hashes.get("raw_scope") is not None: + errors.append("hashes.raw_scope must be null when hashes.raw is null") + else: + if not _valid_digest(raw): + errors.append("hashes.raw digest must be a bare lowercase 64-char hex string") + if hashes.get("raw_algorithm") != HASH_ALGORITHM: + errors.append(f"hashes.raw_algorithm must be {HASH_ALGORITHM!r}") + if hashes.get("raw_scope") != RAW_SCOPE: + errors.append(f"hashes.raw_scope must be {RAW_SCOPE!r}") + + for key in ("captured_at", "ingested_at"): + val = env.get(key) + if val is not None and not isinstance(val, str): + errors.append(f"{key} must be a string or null") + + if errors: + return errors + + compact = json.dumps(env, separators=(",", ":"), sort_keys=True, ensure_ascii=False) + if len(compact.encode("utf-8")) > MAX_COMPACT_BYTES: + errors.append(f"envelope compact JSON size exceeds {MAX_COMPACT_BYTES} bytes") + return errors + + +def build_envelope( + *, + source_system: str, + source_kind: str, + source_producer: str, + origin: str, + repository_id: str, + repository_revision: str | None, + session_id: str | None, + session_harness: str | None, + collection_id: str, + item_id: str, + locator_kind: str, + locator_value: str, + attribution: str, + modality: str, + trust_label: str, + trust_assigned_by: str, + trust_assigned_at: str | None, + injection_status: str, + injection_count: int, + injection_rules: Sequence[str], + text: str, + raw_bytes: bytes | None, + content_scope: str, + captured_at: str | None, + ingested_at: str | None, +) -> dict[str, Any]: + content_digest = content_sha256(text) + if raw_bytes is None: + raw_digest: str | None = None + raw_algorithm: str | None = None + raw_scope: str | None = None + else: + raw_digest = sha256_bytes(raw_bytes) + raw_algorithm = HASH_ALGORITHM + raw_scope = RAW_SCOPE + + env: dict[str, Any] = { + "schema": SCHEMA, + "schema_version": SCHEMA_VERSION, + "source": { + "system": source_system, + "kind": source_kind, + "producer": source_producer, + }, + "origin": origin, + "repository": {"id": repository_id, "revision": repository_revision}, + "session": {"id": session_id, "harness": session_harness}, + "collection_id": collection_id, + "item_id": item_id, + "locator": {"kind": locator_kind, "value": locator_value}, + "attribution": attribution, + "modality": modality, + "trust": { + "label": trust_label, + "assigned_by": trust_assigned_by, + "assigned_at": trust_assigned_at, + "trust_policy": { + "schema": TRUST_POLICY_SCHEMA, + "schema_version": TRUST_POLICY_VERSION, + }, + "injection": { + "status": injection_status, + "count": injection_count, + "rules": list(injection_rules), + }, + }, + "hashes": { + "content_algorithm": HASH_ALGORITHM, + "content_scope": content_scope, + "content": content_digest, + "raw_algorithm": raw_algorithm, + "raw_scope": raw_scope, + "raw": raw_digest, + }, + "captured_at": captured_at, + "ingested_at": ingested_at, + } + errors = validate_envelope(env) + if errors: + raise ValueError("invalid provenance envelope: " + "; ".join(errors)) + return env + + +def synthesize_legacy_provenance() -> tuple[dict[str, Any], str]: + env: dict[str, Any] = { + "schema": SCHEMA, + "schema_version": SCHEMA_VERSION, + "source": {"system": "legacy", "kind": "legacy", "producer": "legacy.read_synthesis"}, + "origin": "unknown", + "repository": None, + "session": None, + "collection_id": None, + "item_id": None, + "locator": None, + "attribution": "inferred", + "modality": "unknown", + "trust": { + "label": "unknown", + "assigned_by": "ingest:legacy.read_synthesis", + "assigned_at": None, + "trust_policy": { + "schema": TRUST_POLICY_SCHEMA, + "schema_version": TRUST_POLICY_VERSION, + }, + "injection": {"status": "clean", "count": 0, "rules": []}, + }, + "hashes": { + "content_algorithm": HASH_ALGORITHM, + "content_scope": "item.text.utf8.v1", + "content": None, + "raw_algorithm": None, + "raw_scope": None, + "raw": None, + }, + "captured_at": None, + "ingested_at": None, + } + return env, LEGACY_DISPLAY diff --git a/tests/test_provenance_envelope.py b/tests/test_provenance_envelope.py new file mode 100644 index 00000000..dce1b879 --- /dev/null +++ b/tests/test_provenance_envelope.py @@ -0,0 +1,218 @@ +"""Tests for brigade.provenance envelope builder, validator, and legacy synthesis.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from brigade import provenance + +FIXTURES = Path(__file__).resolve().parents[1] / "src" / "brigade" / "fixtures" +GOLDEN_PATH = FIXTURES / "provenance-envelope.v1.golden.json" +POLICY_PATH = FIXTURES / "trust-policy.v1.json" +LEGACY_LABELS = ("unknown", "untrusted", "reviewed", "verified", "quarantined") + + +def _golden_cases(): + return json.loads(GOLDEN_PATH.read_text())["cases"] + + +def _case(name): + return next(c for c in _golden_cases() if c["name"] == name) + + +def _set(env, path, value): + cur = env + for key in path[:-1]: + cur = cur[key] + cur[path[-1]] = value + + +def _item_kwargs(): + env = _case("item_unicode_trailing_newline")["envelope"] + text = _case("item_unicode_trailing_newline")["text"] + return dict( + source_system=env["source"]["system"], + source_kind=env["source"]["kind"], + source_producer=env["source"]["producer"], + origin=env["origin"], + repository_id=env["repository"]["id"], + repository_revision=env["repository"]["revision"], + session_id=env["session"]["id"], + session_harness=env["session"]["harness"], + collection_id=env["collection_id"], + item_id=env["item_id"], + locator_kind=env["locator"]["kind"], + locator_value=env["locator"]["value"], + attribution=env["attribution"], + modality=env["modality"], + trust_label=env["trust"]["label"], + trust_assigned_by=env["trust"]["assigned_by"], + trust_assigned_at=env["trust"]["assigned_at"], + injection_status=env["trust"]["injection"]["status"], + injection_count=env["trust"]["injection"]["count"], + injection_rules=list(env["trust"]["injection"]["rules"]), + text=text, + raw_bytes=text.encode("utf-8"), + content_scope=env["hashes"]["content_scope"], + captured_at=env["captured_at"], + ingested_at=env["ingested_at"], + ) + + +def _valid_envelope(): + return provenance.build_envelope(**_item_kwargs()) + + +def test_sha256_bytes_and_content_sha256_match_golden(): + text = _case("item_unicode_trailing_newline")["text"] + digest = _case("item_unicode_trailing_newline")["envelope"]["hashes"]["content"] + assert provenance.content_sha256(text) == digest + assert provenance.sha256_bytes(text.encode("utf-8")) == digest + assert len(digest) == 64 + assert digest == digest.lower() + + +def test_build_envelope_matches_golden_item_case(): + assert _valid_envelope() == _case("item_unicode_trailing_newline")["envelope"] + + +def test_message_scope_distinct_bytes_from_item(): + item = _case("item_unicode_trailing_newline")["envelope"]["hashes"]["content"] + msg = _case("message_distinct_bytes")["envelope"]["hashes"]["content"] + assert provenance.content_sha256("worker-result: done") == msg + assert item != msg + + +def test_build_envelope_raw_none_when_no_raw_bytes(): + kwargs = _item_kwargs() + kwargs["raw_bytes"] = None + env = provenance.build_envelope(**kwargs) + assert env["hashes"]["raw"] is None + + +def test_validate_accepts_golden_envelopes(): + for case in _golden_cases(): + assert provenance.validate_envelope(case["envelope"]) == [] + + +def test_trust_policy_fixture_shape(): + policy = json.loads(POLICY_PATH.read_text()) + assert policy["schema"] == "brigade.trust-policy.v1" + assert policy["schema_version"] == 1 + assert set(policy["entitlements"]) == set(LEGACY_LABELS) + assert policy["untrusted_caps"] == {"max_items": 2, "max_fraction": 0.5} + + +@pytest.mark.parametrize( + ("name", "path", "bad"), + [ + ("origin underscore", ("origin",), "operator_input"), + ("origin wrong case", ("origin",), "AGENT-SESSION"), + ("modality underscore", ("modality",), "model_generated"), + ("attribution bogus", ("attribution",), "guess"), + ("trust label trusted", ("trust", "label"), "trusted"), + ("injection status ok", ("trust", "injection", "status"), "ok"), + ("content scope v2", ("hashes", "content_scope"), "item.text.utf8.v2"), + ("locator kind absolute", ("locator", "kind"), "absolute"), + ], +) +def test_validate_rejects_invalid_closed_set_values(name, path, bad): + env = _valid_envelope() + _set(env, path, bad) + errors = provenance.validate_envelope(env) + assert errors, f"expected error for {name}" + assert any("closed" in e or "enum" in e or path[-1] in e for e in errors) + + +@pytest.mark.parametrize( + ("name", "mutate"), + [ + ("uppercase", lambda e: _set(e, ("hashes", "content"), e["hashes"]["content"].upper())), + ("too short", lambda e: _set(e, ("hashes", "content"), "abc")), + ("non hex", lambda e: _set(e, ("hashes", "content"), "z" * 64)), + ("sha256 prefix", lambda e: _set(e, ("hashes", "content"), "sha256:" + "a" * 64)), + ("empty", lambda e: _set(e, ("hashes", "content"), "")), + ("raw uppercase", lambda e: _set(e, ("hashes", "raw"), e["hashes"]["raw"].upper())), + ], +) +def test_validate_rejects_bad_digest_forms(name, mutate): + env = _valid_envelope() + mutate(env) + errors = provenance.validate_envelope(env) + assert errors, f"expected error for {name}" + assert any("digest" in e or "hash" in e for e in errors) + + +@pytest.mark.parametrize( + ("name", "value"), + [ + ("posix absolute", "/etc/passwd"), + ("home absolute", "/home/user/secret"), + ("windows drive", "C:\\Users\\foo"), + ("windows backslash UNC", "\\\\host\\share\\file"), + ("file URI", "file:///home/user/secret"), + ("parent traversal", "../secret"), + ], +) +def test_validate_rejects_unsafe_absolute_locators(name, value): + env = _valid_envelope() + _set(env, ("locator", "value"), value) + errors = provenance.validate_envelope(env) + assert errors, f"expected error for {name}" + assert any("absolute" in e or "locator" in e for e in errors) + + +@pytest.mark.parametrize( + ("name", "inbound_adapter", "label", "proof_assigned_by", "proof_label", "want_valid"), + [ + ("inbound reviewed no proof", True, "reviewed", None, None, False), + ("inbound reviewed proof match", True, "reviewed", "verifier:demo", "reviewed", True), + ("inbound reviewed assigned_by mismatch", True, "reviewed", "verifier:other", "reviewed", False), + ("inbound reviewed label mismatch", True, "reviewed", "verifier:demo", "verified", False), + ("inbound verified proof match", True, "verified", "verifier:demo", "verified", True), + ("inbound untrusted no proof", True, "untrusted", None, None, True), + ("not inbound reviewed no proof", False, "reviewed", None, None, True), + ], +) +def test_validate_authority_for_inbound_adapter( + name, inbound_adapter, label, proof_assigned_by, proof_label, want_valid +): + env = _valid_envelope() + _set(env, ("trust", "label"), label) + _set(env, ("trust", "assigned_by"), "verifier:demo") + proof = None + if proof_assigned_by is not None: + proof = {"assigned_by": proof_assigned_by, "label": proof_label} + errors = provenance.validate_envelope(env, inbound_adapter=inbound_adapter, authority_proof=proof) + assert (not errors) == want_valid, f"{name}: errors={errors}" + + +def test_synthesize_legacy_provenance(): + env, display = provenance.synthesize_legacy_provenance() + assert display == provenance.LEGACY_DISPLAY == "UNKNOWN PROVENANCE - legacy item" + assert env["schema"] == "brigade.provenance-envelope.v1" + assert env["schema_version"] == 1 + assert env["origin"] == "unknown" + assert env["modality"] == "unknown" + assert env["attribution"] == "inferred" + assert env["trust"]["label"] == "unknown" + assert env["hashes"]["content"] is None + assert provenance.validate_envelope(env) == [] + + +def test_validate_rejects_envelope_above_4096_compact_bytes(): + env = _valid_envelope() + _set(env, ("item_id",), "x" * 5000) + errors = provenance.validate_envelope(env) + assert errors + assert any("size" in e or "4096" in e for e in errors) + + +def test_validate_accepts_envelope_under_4096_compact_bytes(): + env = _valid_envelope() + compact = len(json.dumps(env, separators=(",", ":"), sort_keys=True).encode("utf-8")) + assert compact < 4096 + assert provenance.validate_envelope(env) == []