Add support for AWS backend #573

kipz · 2025-10-30T12:20:38Z

Summary

This PR adds support for AWS as a storage backend for Rekor tiles

Key changes:

Implemented AWS storage backend with:
- AWS S3 for object storage (with support for S3-compatible services like MinIO)
- Aurora MySQL / RDS MySQL** for database operations
Refactored e2e tests to use table-driven pattern for testing multiple backend configurations
Updated Makefile/build/CI to create different binaries for GCP and AWS
- rekor-server-aws - AWS-only dependencies
- rekor-server-gcp - GCP-only dependencies

Testing:

For GCP backend:

docker compose up -d --build --wait
TEST_BACKENDS=gcp go test -v -tags=e2e ./tests/

For AWS backend:

docker compose -f docker-compose-aws.yml up -d --build --wait
TEST_BACKENDS=aws go test -v -tags=e2e ./tests/

For both backends:

TEST_BACKENDS=all go test -v -tags=e2e ./tests/

See tests/README.md for more details.

codecov · 2025-11-05T21:47:53Z

Codecov Report

❌ Patch coverage is 0.96154% with 309 lines in your changes missing coverage. Please review.
✅ Project coverage is 35.04%. Comparing base (6967ce7) to head (27af530).

Files with missing lines	Patch %	Lines
internal/rekor/app/common.go	0.00%	87 Missing ⚠️
cmd/freeze-checkpoint/app/root.go	0.00%	68 Missing ⚠️
cmd/rekor-server/app/serve.go	0.00%	55 Missing ⚠️
cmd/rekor-server-gcp/app/backend.go	0.00%	31 Missing ⚠️
internal/tessera/driver_aws.go	9.09%	30 Missing ⚠️
cmd/rekor-server-aws/app/backend.go	0.00%	26 Missing ⚠️
cmd/rekor-server/app/root.go	0.00%	7 Missing ⚠️
cmd/rekor-server-gcp/main.go	0.00%	2 Missing ⚠️
cmd/rekor-server/app/backend.go	0.00%	2 Missing ⚠️
cmd/rekor-server-aws/main.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #573      +/-   ##
==========================================
- Coverage   36.09%   35.04%   -1.05%     
==========================================
  Files          43       48       +5     
  Lines        2696     2785      +89     
==========================================
+ Hits          973      976       +3     
- Misses       1618     1704      +86     
  Partials      105      105

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

haydentherapper

Thanks for working on this! Just a few comments around testing.

haydentherapper · 2025-11-05T22:44:15Z

tests/test_assertions.go

+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+


Can we add //go:build e2e for this and the helpers?

haydentherapper · 2025-11-05T22:47:36Z

tests/test_assertions.go

Is there a benefit to having the assertions and helpers in dedicated files? I'd leave these in e2e_test unless they'll be used elsewhere.

Actually, I wonder if it would be cleaner to have two separate e2e_test files, one for GCP and one for AWS. It'd be nice to be able to run an e2e test for one backend and not the other.

@haydentherapper I'm happy either way, but my goal here was to ensure functional parity across the different providers. I'll improve the test selection though so that it's easy to select which provider(s) to run the tests for.

haydentherapper · 2025-11-05T22:49:55Z

tests/e2e_test.go

 	if err != nil {
 		t.Skip("skipping persistent deduplication test because docker is not installed")
 	}
 	output, err := exec.Command(path, "compose", "ps", "rekor").Output()


Can this test be updated for both backends as well? Should just need to pass the docker compose file in.

haydentherapper · 2025-11-05T22:51:19Z

tests/e2e_test.go

Also will need to update where the test is invoked -

rekor-tiles/.github/workflows/test.yml

Line 115 in eb8f925

run: docker compose -f compose.yml up -d --build --wait --wait-timeout 60

haydentherapper · 2025-11-05T22:52:32Z

tests/e2e_test.go

Can you also update the README? https://github.com/sigstore/rekor-tiles/blob/eb8f925cee2c118b98d7dcbeca54123fc700908b/tests/README.md

AlCutter · 2025-11-06T12:14:35Z

Drive-by comment: Have you considered having infra-specific binaries (e.g. /cmd/rekor-server/gcp, /cmd/rekor-server/aws etc.) as opposed to bundling support for everything into one super large binary?

For reference, this multi-binary approach is what TesseraCT does in order to avoid binary bloat from linking libs which are not actually used at runtime: https://github.com/transparency-dev/tesseract/tree/main/cmd/tesseract

kipz · 2025-11-06T17:47:35Z

Drive-by comment: Have you considered having infra-specific binaries (e.g. /cmd/rekor-server/gcp, /cmd/rekor-server/aws etc.) as opposed to bundling support for everything into one super large binary?

For reference, this multi-binary approach is what TesseraCT does in order to avoid binary bloat from linking libs which are not actually used at runtime: https://github.com/transparency-dev/tesseract/tree/main/cmd/tesseract

I hadn't, but it seems like a good idea to me, thanks. What do others think? @haydentherapper ?

haydentherapper · 2025-11-07T17:21:24Z

Drive-by comment: Have you considered having infra-specific binaries (e.g. /cmd/rekor-server/gcp, /cmd/rekor-server/aws etc.) as opposed to bundling support for everything into one super large binary?
For reference, this multi-binary approach is what TesseraCT does in order to avoid binary bloat from linking libs which are not actually used at runtime: https://github.com/transparency-dev/tesseract/tree/main/cmd/tesseract

I hadn't, but it seems like a good idea to me, thanks. What do others think? @haydentherapper ?

Thanks @AlCutter, that's a great suggestion. @kipz, can you take a look at doing this?

kipz · 2025-11-10T14:25:57Z

Drive-by comment: Have you considered having infra-specific binaries (e.g. /cmd/rekor-server/gcp, /cmd/rekor-server/aws etc.) as opposed to bundling support for everything into one super large binary?
For reference, this multi-binary approach is what TesseraCT does in order to avoid binary bloat from linking libs which are not actually used at runtime: https://github.com/transparency-dev/tesseract/tree/main/cmd/tesseract

I hadn't, but it seems like a good idea to me, thanks. What do others think? @haydentherapper ?

Thanks @AlCutter, that's a great suggestion. @kipz, can you take a look at doing this?

Done - it's quite a big change, but hopefully heading in the right direction 😬

haydentherapper · 2025-11-13T21:07:42Z

Just wanted to say that this is on my radar to review, this will just take a moment to do so! I'm happy with the overall direction though.

kipz · 2025-11-14T13:18:48Z

Just wanted to say that this is on my radar to review, this will just take a moment to do so! I'm happy with the overall direction though.

Thanks. I'll take look at those failures too :)

This change enables users to deploy Rekor v2 on AWS infrastructure. The implementation uses AWS S3 for object storage and Aurora MySQL (or RDS MySQL) for database operations. The AWS backend includes: - S3 storage with support for S3-compatible services (e.g., MinIO) - Aurora MySQL/RDS MySQL for sequencing and deduplication - Docker Compose configuration for local development - Table-driven e2e tests supporting multiple backend configurations - Updated freeze-checkpoint tool to work with S3 All existing GCP backend functionality remains unchanged. Resolves: sigstore#572 Signed-off-by: James Carnegie <[email protected]>

Signed-off-by: James Carnegie <[email protected]>

- Add `//go:build aws || gcp` constraint to cmd/rekor-server/app/* files - Configure golangci-lint to use the `gcp` build tag by default - Update ko-local Makefile target to build each backend separately with GOFLAGS - Format import statements in main.go files Signed-off-by: James Carnegie <[email protected]>

Signed-off-by: James Carnegie <[email protected]>

kipz requested review from a team as code owners October 30, 2025 12:20

haydentherapper reviewed Nov 5, 2025

View reviewed changes

kipz force-pushed the kipz/aws-support branch from dc2e73d to ed31616 Compare November 7, 2025 10:42

kipz force-pushed the kipz/aws-support branch 2 times, most recently from c955894 to 088ff91 Compare November 10, 2025 13:41

kipz force-pushed the kipz/aws-support branch from 73ae2ba to d99f602 Compare November 10, 2025 14:26

kipz force-pushed the kipz/aws-support branch 2 times, most recently from 4687e33 to 83c9fbc Compare November 21, 2025 13:50

kipz added 6 commits December 9, 2025 10:45

Cleanup AWS testing after review

d00a2e8

Signed-off-by: James Carnegie <[email protected]>

Create separate binaries for each backend

646e55b

Signed-off-by: James Carnegie <[email protected]>

Fix CI and buid after binary split

d8bf512

Signed-off-by: James Carnegie <[email protected]>

Fix Docker builds and compose args for different providers

57d50cc

Signed-off-by: James Carnegie <[email protected]>

kipz force-pushed the kipz/aws-support branch from 83c9fbc to 57d50cc Compare December 9, 2025 10:46

kipz requested a review from haydentherapper December 9, 2025 10:50

Add support for AWS backend #573

Are you sure you want to change the base?

Add support for AWS backend #573

Uh oh!

Conversation

kipz commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

codecov bot commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haydentherapper left a comment

Choose a reason for hiding this comment

Uh oh!

haydentherapper Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

haydentherapper Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

haydentherapper Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

kipz Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

haydentherapper Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

haydentherapper Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

haydentherapper Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

AlCutter commented Nov 6, 2025

Uh oh!

kipz commented Nov 6, 2025

Uh oh!

haydentherapper commented Nov 7, 2025

Uh oh!

kipz commented Nov 10, 2025

Uh oh!

haydentherapper commented Nov 13, 2025

Uh oh!

kipz commented Nov 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kipz commented Oct 30, 2025 •

edited

Loading

codecov bot commented Nov 5, 2025 •

edited

Loading