Skip to content

README.md

Latest commit

 

History

History
103 lines (82 loc) · 4.7 KB

README.md

File metadata and controls

103 lines (82 loc) · 4.7 KB
Raw

bugbane-updater

Builds Bugbane's IOC update feed from upstream STIX sources: one versioned bundle, unified diffs to the previous 5 versions, and a signed metadata file.

Layout (dist/v1/)

The path prefix equals signed.schema. A breaking schema change publishes v2/ alongside v1/ during a transition window.

  • update.json: RFC 8785 canonical JSON, {"proofs": {<keyid>: <sigsum proof>}, "signed": {schema, version, build_date, sha256}}. keyid is the hex sha256 of the raw ed25519 submitter public key. The proof covers the RFC 8785 bytes of signed, including any field unknown to the client. Additional keys add entries under their keyid.
  • <sha256>.json: full STIX bundle, content-addressed. One object per line, sorted by id, fixed head/tail lines, trailing newline.
  • deltas/delta-<from>-<to>.diff: zero-context unified diff between fulls, patch-compatible.
  • index.html: human-browsable listing (metadata, full, deltas, source repo); not part of the protocol. Rendered from config/index.html.tmpl (string.Template $version $schema $build_date $sha256 $full $deltas); omitted if the template is absent.

Protocol

  1. GET v1/update.json. If signed.version <= local, stop.
  2. GET deltas/delta-<local>-<version>.diff. Apply hunks positionally, no context matching, no fuzz.
  3. Adopt the result iff sha256(result) == signed.sha256. On 404 or mismatch, GET <signed.sha256>.json and check the same hash.

Deltas are unsigned and unlisted; signed.sha256 is the only integrity anchor. Hunks ascend, so apply can stream: hold the delta in memory, stream the old full into a temp file with a rolling sha256, rename on match.

Client requirements:

  • Check that signed.schema matches the path prefix.
  • Scope local version state to the schema; after moving to a new schema, bootstrap from version 0.
  • Ignore unknown fields in signed.
  • When verifying, canonicalize the parsed signed subtree as received; do not rebuild it from known fields.
  • Unknown schema: keep last-good indicators, surface staleness.
  • signed.sunset (ISO date, optional): if now < sunset, surface "app update required before "; if now >= sunset, keep last-good indicators and surface "update the app to continue receiving updates".

Builder requirements:

  • Never reset or reuse a version number.
  • Serve the final sunset-bearing update.json and its full at v<schema>/ indefinitely; exclude retired trees from deploy deletion.

Sources

config/sources.yaml has two lists. indexes: points at upstream indicators.yaml-style files; every github: entry is fetched at build time, so upstream additions flow in with no manual change here (a non-github entry is skipped with a log line). sources: adds your own {repo, branch, paths} directly. Both are tracked by branch, fetched at the resolved commit SHA (every SHA, including each index, recorded in the manifest), then merged and deduped.

Canonicalization

At build time: created, modified, valid_from are set to a constant; ids are rewritten to type--uuid5(content); *_ref/*_refs are remapped to the new ids; objects with equal content dedup. Exclusion id: entries match canonical ids.

Build

CI runs on schedule, workflow_dispatch, or repository_dispatch: urgent-update. Signing selects by secret: SIGSUM_SUBMIT_KEY → -P sigsum-generic-2025-1 (prod); else SIGSUM_TEST_KEY → -P sigsum-test1-2025 (test log, no rate-limit token); else release unsigned. seal then embeds the proof; the public key is derived from the private key. Each release (v<schema>.<version>) carries the servable tree as dist-v<schema>-<version>.tar.gz (sealed update.json, <sha256>.json, deltas) plus manifest-<version>.json for audit. A pull-based deploy fetches the tarball, re-checks the full hash, and swaps the serving root atomically. The build's restore step reconstructs its archive from the last 5 release tarballs.

signed is generated (schema, sha256, version, build_date) plus the fields of config/signed.yaml merged verbatim (e.g. sunset). Any change to the signed body besides version/build_date forces a new version.

Releases are tagged v<schema>.<version>; the archive restore filters by schema prefix. Schema transition: freeze the old schema on a maintenance branch (schema-v<n>), bump SCHEMA on the default branch, build both from the default branch's workflow via a ref matrix (scheduled workflows run only on the default branch). Version counters are independent per schema. Deploys sync each v<n>/ subtree separately.

Local:

pip install -e .[dev] && pytest
bugbane-updater build --build-date "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
bugbane-updater seal --proof <proof> --key <pubkey>
bugbane-updater verify