Binary Transparency for Package Registries

This repo contains an implementation of a transparency log for binary transparency for package registries.

cmd/bt-log provides an HTTP server that accepts POST requests to an /add endpoint. The JSON request should contain a single string, a package identified by a pURL string:

{
    "purl": "pkg:pypi/my-package@1.2.3?checksum=sha256:3b9730808f265c6d174662668435c4cf1fc9ddcd369831a646fa84bff8594f0c"
}

The pURL must contain:

1. A pURL type that matches the name of the package registry, e.g. pypi, gem
2. The name of a package. Namespace is optional
3. The package version, e.g. 1.2.3, v1.2.3
4. A single qualifier containing the SHA 256 checksum

The JSON response will include the index of the entry, the inclusion proof, and the checkpoint as per the C2SP checkpoint spec:

{
    "index": 123,
    "checkpoint": "base64(checkpoint)",
    "inclusionProof": ["base64(hash)", "base64(hash)"]
}

The HTTP server also exposes endpoints per the C2SP tlog-tiles spec:

• /checkpoint, which is updated every second
• /tile, which serves the raw tile data and entry bundles

Log deployment

This will create a directory in the filesystem to store a log, and start the HTTP server that can add entries to this log.

First, generate private and public keys:

go run ./cmd/gen-key --origin=binarytransparency.log/example

This will output private and public keys in Go's signed note format:

cat private.key
PRIVATE+KEY+binarytransparency.log/example+5de0f997+AXNNv9racVtMynH7oHIogZ4xS5sAIHBl47hlrcf6vsfu

cat public.key
binarytransparency.log/example+5de0f997+AcPfp2roeTxqSqmPdDkA9rIAd0pe3C5Je6Rze2SqBDUp

Then, start the log:

go run ./cmd/bt-log --storage-dir=/tmp/bt-log --private-key=private.key --public-key=public.key --entry-type=purl --purl-type=pypi

Replace --purl-type with the name of the package registry.

Witnessing

To prevent split-view attacks, where a log serves different views to different callers, checkpoints must be witnessed, where an independent auditor verifies a consistency proof that the log remains append-only and returns a cosignature over the checkpoint.

This repository contains a lightweight witness that implements the C2SP tlog-witness spec.

To initialize the witness, create a SQLite database and a signing key. The database will store a log's verification key and origin, and the last size and tree hash the witness verified.

The commands below will initialize the log signing key, create the database, initialize the database, add a row with the log's verification key, generate a witness signing key, and start the witness.

go run ./cmd/gen-key --origin=binarytransparency.log/example

sqlite3 -line witness.db '.database'
go run ./cmd/witness-add-key --database-path witness.db --public-key public.key

go run ./cmd/gen-key --origin=witness.log/example --private-key-path witness-private.key --public-key-path witness-public.key
go run ./cmd/witness-server --database-path witness.db --private-key witness-private.key --public-key witness-public.key

Then, start the log. The log will verify cosigned checkpoints using the provided witness verification key.

go run ./cmd/bt-log --storage-dir=/tmp/bt-log --private-key=private.key --public-key=public.key --entry-type=purl --purl-type=pypi --witness-url="http://localhost:8081" --witness-public-key=witness-public.key

The checkpoint in the log's response will contain a co-signed checkpoint:

curl -XPOST http://localhost:8080/add -d "{\"purl\":\"pkg:pypi/pkgname@1.2.3?checksum=sha256:5141b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be92\"}" -o bundle

cat bundle | jq -r .checkpoint | base64 -d

The signed checkpoint will have two signatures, one from the log and one from the witness.

Docker Deployment

Using the provided Docker Compose file, you can initialize and deploy the log and witness.

For the log, only the Tessera POSIX backend is supported.

You'll need to pick a storage backend for the witness. SQLite, PostgreSQL and MySQL are supported.

SQLite

Run the following administrative jobs once to generate the log and witness keys and initialize the witness database:

docker compose --profile admin --profile sqlite build
docker compose run gen-key-log
docker compose run gen-key-witness

Run the log and witness:

docker compose --profile sqlite up --wait

To clean up containers and volumes:

docker compose --profile sqlite down --remove-orphans --volumes

PostgreSQL

Run the following administrative jobs once to generate the log and witness keys and initialize the witness database:

docker compose --profile admin --profile postgres build
docker compose run gen-key-log
docker compose run gen-key-witness

Run the log and witness:

docker compose --profile postgres up --wait

To clean up containers and volumes:

docker compose --profile postgres down --remove-orphans --volumes

MySQL

Run the following administrative jobs once to generate the log and witness keys and initialize the witness database:

docker compose --profile admin --profile mysql build
docker compose run gen-key-log
docker compose run gen-key-witness

Run the log and witness:

docker compose --profile mysql up --wait

To clean up containers and volumes:

docker compose --profile mysql down --remove-orphans --volumes

Upcoming Work

• Change pURL to a custom representation
• Lightweight monitor to demonstrate verifying ID-hash mapping is always 1-1 and alerting on publication
  • ID-hash mapping verification
  • Regex to match entries
  • Use slog for output
  • Transform pURL to entry, request entry from registry, compare hash
  • Add e2e to GHA script
• Add unit tests
• Containerize for e2e tests