Proposal to simplify and reduce the scope of sigstore/scaffolding

[cmurphy]

Objectives

1. Reduce the number of binaries and images that need to be maintained.
2. Eliminate complex code that is never used in production.
3. Make the setup Github action (Kubernetes-based Sigstore) as lean as possible.

Proposed Changes

Deprecate and remove the createcerts command.

Why

1. This is not used in production (for the Sigstore organization).
2. Its objective - to automatically create the signing key and certificate for the Fulcio CA when using the fileca certificate authority mode - is easily accomplished with a few lines of shell using the openssl command and using the results to create Kubernetes Secrets.
3. Since we only use the fileca mode for development and testing, generating these secrets securely is not a concern for our purposes.
4. Creating the secret internally as a Kubernetes process necessitates that clients that require the root CA, such as the CT log, retrieve it from Fulcio’s rootCert HTTP endpoint. We don’t want to continue to encourage this practice, preferring instead to generate keys externally and distribute them in a secure manner, such as via TUF.

How

1. Update the hack/setup-scaffolding.sh and hack/setup-scaffolding-form-release.sh scripts to generate the private key and self-signed CA and inject them as Secrets in the cluster for Fulcio. Change CTLog to accept a root PEM directly instead of fetching it from the rootCerts API.
2. Change the default value of createcerts.enabled from true to false in the Fulcio Helm chart, and deprecate its usage.

Deprecate and remove rekor-createsecret and createcertchain

Why

For the same reasons as above - remove the dependency on a service's public endpoint and use an out-of-band mechanism to securely distribute key material instead.

Remove Trillian manifests

Why

There are Tessera-based replacements for the Trillian log server backend. The complexity of maintaining a relational database and multiple Trillian scripts for preparing and tracking the state of the tree is unnecessarily burdensome.

How

1. Change the ct_server image to TesseraCT (already part of Deployment of tile-based CT log rekor-tiles#73).
   a. This removes the use of createctconfig from Scaffolding and removes the use of createtree as a dependency of the CT log.
   b. Future: deprecate the ctlog Helm chart entirely once TesseraCT is well used in production.
2. Change the Rekor service to Rekor v2.
   a. This fully removes the use of createtree from the Scaffolding setup scripts.
   b. Enables removing all Trillian Kubernetes manifests.
   c. Future: deprecate the rekor Helm chart entirely.

Move utility commands to better locations

Why

Scaffolding has become a grab-bag of tools that have no better home. We need to slim down Scaffolding to fulfill its primary purpose, which is automating Kubernetes deployments with minimal manual intervention. The only Go commands that should live in Scaffolding should be those that are used as images in Jobs managed by helm-charts.

How

1. Move create-tink-keyset to sigstore/sigstore (public) or to sigstore/public-good-instance (private). This command is not used in helm charts, it is run externally before deployment.

2. Move prober to sigstore/sigstore-probers.

3. Deprecate (do not remove):

   • cloudsqlproxy (Trillian)
   • createcerts (Fulcio)
   • createctconfig (CTLog/ct_server)
   • rekor-createsecret (Rekor v1)
   • createcertchain (TSA)
   • createdb (Trillian)
   • createtree (Trillian)
   • updatetree (Trillian)

4. Simplify/rewrite or move test-only commands (exact work TBD)

   • managectroots
   • verifyfulcio
   • getoidctoken

Replace all ko-based Kubernetes manifests with helm-charts

Why

1. The Kubernetes manifests in Scaffolding are not used in production and diverge heavily from the configuration that is used in production.
2. The charts in helm-charts have no functional testing and are only ever “tested” in live environments.
3. The burden of maintaining two separate Kubernetes configuration repositories is unnecessarily high.

How

The exact manner of building images with ko and then injecting them into helm-charts still needs further exploration, I’m not sure if we’ll still be able to use ko apply or ko resolve for this.

Comments welcome

Is Scaffolding being used in a way beyond 1) CI testing and 2) glue jobs for helm-charts that I'm not aware of and that these changes would adversely affect?

[haydentherapper]

Have you reviewed the sharding guides for CT and Rekor?

Have you also reviewed where we reference binaries built in scaffolding in helm-charts? If we're deprecating the binaries, what's the plan for these references, to stop distributing the images in the charts?

For a few off the top of my head:

• createtree and updatetree are used for sharding Rekor v1 logs. I don't think createdb is, but I think it could be used to build a Trillian-based log from scratch.
• I don't see references to createcerts - IIRC, this was used as an alternative to ExternalSecrets. For any of the create-* commands, I'd double check they're not referenced in any playbooks,
• For cloudsqlproxy, I'd double check with Bob, but I believe this is still in use - https://github.com/sigstore/helm-charts/blob/5e1240a266fbb929acc8a031275a3a23645b7dc4/charts/trillian/Chart.yaml#L40.

Remove Trillian manifests
Change the Rekor service to Rekor v2.

We aren't turning down Rekor v1 yet, so losing the ability to test it isn't ideal especially since we'll likely be making more changes to it to support alternate backends, e.g. the move from Trillian to Tessera. How can we still support testing Rekor v1 while not overburdening ourselves with maintaining divergent testing infra?

Deprecate (do not remove)

Is being able to remove these predicated on the turndown of Trillian? For any others, will we state that we will remove these in X months?

Replace all ko-based Kubernetes manifests with helm-charts

+1, these are unmaintained and I see no reason to not recommend the Helm charts exclusively.

Is Scaffolding being used in a way beyond 1) CI testing and 2) glue jobs for helm-charts that I'm not aware of and that these changes would adversely affect?

It's definitely used for private deployments, but users can fork the repo if they need to continue to maintain the Kubernetes manifests or supporting binaries.

To add one more step at the end, we should publish a blog post and update the documentation on how to deploy Sigstore locally and within CI. There are too many guides, none of which are maintained under a Sigstore repo - 1, 2, 3, 4, 5 plus code

[cmurphy]

Have you reviewed the sharding guides for CT and Rekor?

The changes I'm proposing assume we're in a future where we're using TesseraCT and Rekor v2 only, and those sharding guides would not apply.

Have you also reviewed where we reference binaries built in scaffolding in helm-charts? If we're deprecating the binaries, what's the plan for these references, to stop distributing the images in the charts?

Yes, these are very much on my mind and I've mentioned some changes that the helm charts would need to move away from using these binaries. When we've fully migrated CT and Rekor to Tessera, most of the commands for managing Trillian become obsolete: createtree, updatetree, createdb, cloudsqlproxy are all for Trillian itself or its MySQL/Cloud SQL storage.

I don't see references to createcerts

This is for Fulcio when it is using the fileca CA type: https://github.com/sigstore/helm-charts/blob/5e1240a266fbb929acc8a031275a3a23645b7dc4/charts/fulcio/values.yaml#L117 - we don't use it in production.

Is being able to remove these predicated on the turndown of Trillian? For any others, will we state that we will remove these in X months?

It comes down to what is used in the helm charts. Even when we stop using Trillian in PGI, the rekor and ctlog helm charts will still be out in use in the wild. (Also the fulcio chart is probably used with createcerts enabled for private deployments.) None of them have major release versions, so we're not technically beholden to any guarantees on compatibility, but I would propose that we add deprecation warnings for the whole of the rekor and ctlog charts and for the createcerts setting in the fulcio chart, for X months before removing these binaries from scaffolding.

---

We aren't turning down Rekor v1 yet, so losing the ability to test it isn't ideal

Alright, well, that puts a bit of a damper on my enthusiasm because adding testing for Rekor v2 while still maintaining all the scripts to support Trillian really doesn't simplify this at all.

When it comes to testing Sigstore, what do we really want Scaffolding, and in particular the setup action, to be?

setup is only used in cosign and policy-controller. Rekor v1 and Rekor v2 have their own integration testing. The docker-based setup-sigstore-env action is used for Sigstore-go and cosign. Other clients either have their own custom e2e framework or rely purely on conformance testing. Is it a goal to use the setup action to test Rekor v1 on Tessera, when neither Rekor v1 nor most clients are even using it currently?

What I'd like is for this testing workflow to be so simple that clients aren't afraid of adopting it into their pipelines, and I care more about clients gaining compatibility with Rekor v2 than maintaining compatiblity with v1.

[haydentherapper]

(sorry, this is a little messy to respond to multiple things - we could move this to GH discussion threads if needed)

Have you reviewed the sharding guides for CT and Rekor?
The changes I'm proposing assume we're in a future where we're using TesseraCT and Rekor v2 only, and those sharding guides would not apply.

I agree with for the CT log, as we can assume we'll replace both the staging and production instances in O(weeks) to O(months), with the only external blocker being a GA release of TesseraCT for migrating the production instance. For Rekor, we should assume that Rekor v1 will be up for a few years, or at least for now until we have a better sense of if it is feasible to re-architect Rekor v1 to work with Tessera.

Supporting binaries can be deprecated without further updates, but not yet deleted if they're used for Rekor v1 sharding. We could disable dependabot updates on these as well to truly freeze them, these should require no maintenance.

This is for Fulcio when it is using the fileca CA type...

Ah, thanks. SGTM for removal

I would propose that we add deprecation warnings for the whole of the rekor and ctlog charts and for the createcerts setting in the fulcio chart, for X months before removing these binaries from scaffolding.

SGTM. This can be a part of the blog post as well, with recommendations on how to update to the latest charts.

---

I agree that Scaffolding should be nothing more than easy-to-use testing infrastructure, though maybe I misspoke when I mentioned Trillian manifests. I think we should work towards deleting setup and all Kubernetes manifests for testing. setup-sigstore-env is a superior solution imo, is simpler to maintain, and still supports Rekor v1 testing without needing additional binaries.

Like you said, setup is only used by Cosign and policy-controller. Building on sigstore/cosign#4548, we should be able to remove setup from Cosign - testing that Cosign works in a Kubernetes cluster doesn't require all infrastructure to be deployed to a cluster. Not sure yet what to do about policy-controller though - maybe they can take ownership of the K8s manifests?