Skip to content

sumdbindex

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

sumdbindex

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
main.go
main.go
 
 
main_test.go
main_test.go
 
 

README.md

Verifiable Index: Go Checksum Database (SumDB)

This is a demo of building a Verifiable Index for Go's Checksum Database. The index allows module maintainers to verifiably look up all non-pseudo versions of their module served by the Module Proxy.

There is a testing instance of this service running at https://vindex.gopherwatch.org.

Background

The sum.golang.org service, often referred to as Go Checksum DB or SumDB, serves as an auditable checksum database for Go modules. Its primary purpose is to enhance the security and integrity of the Go module ecosystem by providing a public, tamper-proof record of module checksums. This ensures that when developers fetch a module, they receive the exact version of the code that others are using and that hasn't been maliciously altered.

SumDB functions as a transparency log. This means that every module version and its corresponding checksums are added to the database is an immutable, append-only fashion. Each new entry is cryptographically committed to, making it impossible to modify or remove an entry without detection. This transparency allows anyone to independently verify the integrity of the database and confirm that no module has been altered or replaced. The auditable nature of the database is crucial for trust in the Go module supply chain.

The entries in the SumDB log match what is written in a go.sum file. For example, any project depending on the v1.0 release of Tessera will have the following in its go.sum file:

github.com/transparency-dev/tessera v1.0.0 h1:4OT1V9xJLa5NnYlFWWlCdZkCm18/o12rdd+bCTje7XE=
github.com/transparency-dev/tessera v1.0.0/go.mod h1:TLvfjlkbmsmKVEJUtzO2eb9Q2IBnK3EJ0dI4G0oxEOU=

This is the same content written to the SumDB at index 43930254 (query link).

In addition to sum.golang.org, modules can have all their versions enumerated via proxy.golang.org. For example, all versions of github.com/transparency-dev/tessera can be looked up using https://proxy.golang.org/github.com/transparency-dev/tessera/@v/list. This index provides a convenient way to enumerate versions of Go modules. However, it's important to note that these lookups in proxy.golang.org are not verifiable in the same way as sum.golang.org. While it helps with discovery, it does not offer the same cryptographic guarantees of integrity and immutability as the checksum database. The security and verification of module content rely solely on the auditable records within sum.golang.org.

This Verifiable Index addresses this lookup use-case, while preserving verifiability. The index is built from this log of modules by parsing the module name from every leaf. A pointer to the leaf is added to the index if it represents a non-pseudo version. The index can be queried using the client.

Status

The goal is to find an operator that will run this service to allow verifiable lookups. For now, users may run this themselves using the instructions below.

Running

The Verifiable Index and Output Log are managed by a single binary, which can be run using:

OUTPUT_LOG_PRIVATE_KEY=PRIVATE+KEY+SumDBIndex+a5ed0e81+AYT6tfHpqGaSoH0gYpM7fhj1tEkM3wwYR/IhtiYh1pnj \
go run ./vindex/cmd/sumdbindex \
  --storage_dir ~/vindex-sumdb/

This will take some time on the first run as it needs to download every entry from the SumDB log in order to build the index.

The command above starts a web server that hosts the following URLs:

  • /inputlog/ - the tlog-tiles base URL for a proxy of the SumDB API
  • /vindex/lookup - the provisional vindex lookup API
  • /outputlog/ - the tlog-tiles base URL for the output log
  • /metrics - Prometheus metrics endpoint

Note

This brings up a proxy server that makes SumDB available via the local server at /inputlog/. This allows the Verifiable Index and its clients to use standard tlog-tiles APIs to query the data, instead of requiring a custom client for SumDB.

Witnessing

The Output Log supports synchronous witnessing using the tlog-witness spec. Before a new Output Log checkpoint is published, this will request witness signatures and include them in the published checkpoint.

In order to do this, the --output_log_witness_policy flag should be used to provide a filepath containing a valid witness policy. An example policy using the public test witness network can be found at https://github.com/transparency-dev/tesseract/blob/main/deployment/live/gcp/static-ct-staging/logs/witness.policy.

Querying

Auditing Local Modules

The best tool to use is the SumDB-specific verification tool: sumdbverify.

General Query

The SumDB log is processed into a verifiable map which can be looked up using the client in ./vindex/cmd/client. Below is an example of querying the index to list all releases of github.com/transparency-dev/tessera. The output lists all of the indices where the module is logged, and the entry from the log at this index.

go run ./vindex/cmd/client \
  --vindex_base_url http://localhost:8088/vindex/ \
  --in_log_base_url http://localhost:8088/inputlog/ \
  --out_log_pub_key=SumDBIndex+a5ed0e81+AXEnbaKj+9gCH3f69vcQokgkcFocCl+GlaMXrAg8mRzd \
  --in_log_pub_key=sum.golang.org+033de0ae+Ac4zctda0e5eza+HJyk9SxEdh+s3Ux18htTTAD8OuAn8 \
  --in_log_origin="go.sum database tree" \
  --lookup=github.com/transparency-dev/tessera

37258746)
github.com/transparency-dev/tessera v0.1.2 h1:s8h0HQ5knhvCmQ2TdATw7FMTvdZY+RMcAcgsaPrKm1k=
github.com/transparency-dev/tessera v0.1.2/go.mod h1:jARcxCIWevuPy8XsxN7Q8waDV+M4rY9fNKIXJXBbzXo=


37258761)
github.com/transparency-dev/tessera v0.1.0 h1:TZuf5eLhDf9EWw5MaT9KLasKOgyzlW8mTu6gndTpK98=
github.com/transparency-dev/tessera v0.1.0/go.mod h1:cpk4hVzA5aXcaP6r5UD3EJBQWauI0hprgn27xF5a3ls=


37258762)
github.com/transparency-dev/tessera v0.1.1 h1:vk0XI98cEHZYXDZxa14RlzLh+hwmkVQ0mWrdiTRQHTU=
github.com/transparency-dev/tessera v0.1.1/go.mod h1:uvyZ7WGpaRDPY+4Lme+s1vEUOluYevTYzrDg9j05cYU=


38108519)
github.com/transparency-dev/tessera v0.2.0 h1:KZu0vt1nL6gSRJziDqnNlKMuzjeM+ZXANW2B4Oo/r9o=
github.com/transparency-dev/tessera v0.2.0/go.mod h1:lJCDw1om4T8H73MWQaZ2XBg5Ca0mKozvZZrtd6j5UZw=


41510961)
github.com/transparency-dev/tessera v1.0.0-rc1 h1:topftrvPcD6detUWoR/cdoPgkyI88wQuHasd6pMBjmM=
github.com/transparency-dev/tessera v1.0.0-rc1/go.mod h1:ilpKqGrwDD/6uop5nDj/X60o0qt33GK1uInjIcfZTP0=


42710781)
github.com/transparency-dev/tessera v1.0.0-rc2 h1:BKtDWr0nhL9dG66cS4DyKU9lpZFbUZrpHGh+BpqakcU=
github.com/transparency-dev/tessera v1.0.0-rc2/go.mod h1:aaLlvG/sEPMzT96iIF4hua6Z9pLzkfDtkbaUAR4IL8I=


43267373)
github.com/transparency-dev/tessera v1.0.0-rc3 h1:v385KqMekDUKI3ZVJHCHE5MAz8LBrWsEKa6OzYLrz0k=
github.com/transparency-dev/tessera v1.0.0-rc3/go.mod h1:aaLlvG/sEPMzT96iIF4hua6Z9pLzkfDtkbaUAR4IL8I=


43930254)
github.com/transparency-dev/tessera v1.0.0 h1:4OT1V9xJLa5NnYlFWWlCdZkCm18/o12rdd+bCTje7XE=
github.com/transparency-dev/tessera v1.0.0/go.mod h1:TLvfjlkbmsmKVEJUtzO2eb9Q2IBnK3EJ0dI4G0oxEOU=

Note that this matches the list of entries on the unverifiable proxy endpoint: https://proxy.golang.org/github.com/transparency-dev/tessera/@v/list.

When running this command on a regular basis, you can avoid seeing entries you have already processed by providing the --min_idx flag.

go run ./vindex/cmd/client \
  --vindex_base_url http://localhost:8088/vindex/ \
  --in_log_base_url http://localhost:8088/inputlog/ \
  --out_log_pub_key=SumDBIndex+a5ed0e81+AXEnbaKj+9gCH3f69vcQokgkcFocCl+GlaMXrAg8mRzd \
  --in_log_pub_key=sum.golang.org+033de0ae+Ac4zctda0e5eza+HJyk9SxEdh+s3Ux18htTTAD8OuAn8 \
  --in_log_origin="go.sum database tree" \
  --lookup=github.com/transparency-dev/tessera \
  --min_idx=40000000

I0924 16:14:28.105308 2932871 client.go:66] Dropping 4 pointers to index less than min_idx 40000000
41510961)
github.com/transparency-dev/tessera v1.0.0-rc1 h1:topftrvPcD6detUWoR/cdoPgkyI88wQuHasd6pMBjmM=
github.com/transparency-dev/tessera v1.0.0-rc1/go.mod h1:ilpKqGrwDD/6uop5nDj/X60o0qt33GK1uInjIcfZTP0=


42710781)
github.com/transparency-dev/tessera v1.0.0-rc2 h1:BKtDWr0nhL9dG66cS4DyKU9lpZFbUZrpHGh+BpqakcU=
github.com/transparency-dev/tessera v1.0.0-rc2/go.mod h1:aaLlvG/sEPMzT96iIF4hua6Z9pLzkfDtkbaUAR4IL8I=


43267373)
github.com/transparency-dev/tessera v1.0.0-rc3 h1:v385KqMekDUKI3ZVJHCHE5MAz8LBrWsEKa6OzYLrz0k=
github.com/transparency-dev/tessera v1.0.0-rc3/go.mod h1:aaLlvG/sEPMzT96iIF4hua6Z9pLzkfDtkbaUAR4IL8I=


43930254)
github.com/transparency-dev/tessera v1.0.0 h1:4OT1V9xJLa5NnYlFWWlCdZkCm18/o12rdd+bCTje7XE=
github.com/transparency-dev/tessera v1.0.0/go.mod h1:TLvfjlkbmsmKVEJUtzO2eb9Q2IBnK3EJ0dI4G0oxEOU=

To inspect either the input or output log, you can use the woodpecker tool (using the corresponding public key to the private key used above):

# To inspect the Output Log
go run github.com/mhutchinson/woodpecker@main \
  --custom_log_type=tiles \
  --custom_log_url=http://localhost:8088/outputlog/ \
  --custom_log_vkey=SumDBIndex+a5ed0e81+AXEnbaKj+9gCH3f69vcQokgkcFocCl+GlaMXrAg8mRzd

# To inspect the Input Log
go run github.com/mhutchinson/woodpecker@main \
  --custom_log_type=tiles \
  --custom_log_url=http://localhost:8088/inputlog/ \
  --custom_log_origin="go.sum database tree" \
  --custom_log_vkey=sum.golang.org+033de0ae+Ac4zctda0e5eza+HJyk9SxEdh+s3Ux18htTTAD8OuAn8

Use the left and right arrow keys to browse, g to jump to a specific index, and q to quit.

Performance

Setup

This creates a local clone of SumDB in a tlog-tiles format. Having a local copy removes a lot of variance when running performance checks.

# Run a tlog-proxy for sumdb
go run github.com/transparency-dev/incubator/sumdb/cmd@main --listen=":8089" &

# Clone it
go run github.com/transparency-dev/tessera/cmd/experimental/migrate/posix@main \
  --storage_dir ~/log-clones/sumdb \
  --source_url http://localhost:8089/ \
  --save_checkpoint

# Check it
echo sum.golang.org+033de0ae+Ac4zctda0e5eza+HJyk9SxEdh+s3Ux18htTTAD8OuAn8 > ~/.go.sum.vkey
go run github.com/transparency-dev/tessera/cmd/fsck@main \
  --storage_url file:///$HOME/log-clones/sumdb/ \
  --public_key ~/.go.sum.vkey \
  --origin "go.sum database tree"

Building the VIndex

For performance testing, we use:

  • The local copy of the log for performance reasons
  • --oneshot and time to get clear timing figures for initialization

Building the VIndex for the first time needs to consume everything in the log:

time OUTPUT_LOG_PRIVATE_KEY=PRIVATE+KEY+SumDBIndex+a5ed0e81+AYT6tfHpqGaSoH0gYpM7fhj1tEkM3wwYR/IhtiYh1pnj \
  go run ./vindex/cmd/sumdbindex \
  --storage_dir ~/vindex-sumdb/ \
  --oneshot \
  --input_override_url=file:///$HOME/log-clones/sumdb/ \
  --persist_index=true
...
OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir      164.35s user 105.55s system 147% cpu 3:03.23 total

Building the VIndex for the second time reloads state from file and is much faster:

time OUTPUT_LOG_PRIVATE_KEY=PRIVATE+KEY+SumDBIndex+a5ed0e81+AYT6tfHpqGaSoH0gYpM7fhj1tEkM3wwYR/IhtiYh1pnj \
  go run ./vindex/cmd/sumdbindex \
  --storage_dir ~/vindex-sumdb/ \
  --oneshot \
  --input_override_url=file:///$HOME/log-clones/sumdb/ \
  --persist_index=true
...
OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir      19.17s user 5.37s system 122% cpu 20.058 total

Monitoring

The server exports metrics via Prometheus on the /metrics endpoint.

Key areas covered by the metrics include:

  • Map Function Keys: Tracks the number of keys returned by the MapFn for each leaf (vindex_map_fn_keys).
  • Sync Performance: Histograms tracking the duration of fetching, mapping, and processing leaves during the synchronization phase (under vindex_sync_*).
  • Build Performance: Histograms tracking the duration of various steps in the build process, including WAL processing, MPT updates, publishing to the Output Log, and total build time (under vindex_build_*).