ingest: push runs straight to main, carry s3 provenance - #32
Merged
Merged
Conversation
The campaign pipeline is S3-end-to-end now, and runs no longer arrive as reviewable PRs: stellar-rpc's bench-campaign.yml ingests a passing campaign's bundle itself and needs a direct path to the live site. - converter: --source-uri records campaign.source_uri (s3:// or gs://); --source-gcs stays for old runs. - viewer: the Source data / Raw results links render an S3 or GCS console URL by scheme, preferring source_uri. - ingest.sh: --push-main converts, gates on make test + make smoke, commits on HEAD (must be at origin/main), pushes to main, and prints the viewer URL. --local becomes the default mode; the push+PR mode and the never-provisioned GCS WIF workflow (ingest.yml) are gone. - s3:// bundles now carry provenance; the smoke test accepts both schemes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
There was a problem hiding this comment.
Pull request overview
This pull request updates the benchmark ingest pipeline to support S3 end-to-end provenance and enables CI to publish successful campaign runs directly to main (deploying Pages immediately), while keeping local ingestion as the default safe workflow.
Changes:
- Add
campaign.source_uri(S3 or GCS) as preferred provenance, with viewer links rendering to the appropriate cloud console URL by scheme. - Introduce
scripts/ingest.sh --push-mainto gate onmake test+make smoke, commit onmain, and push/deploy directly; remove the old push+PR mode and delete the unused ingest workflow. - Update docs/tests/Makefile to reflect the new URI/provenance and ingest flow.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/smoke/smoke.mjs | Updates smoke expectations to accept s3:// and gs:// provenance in raw-results text. |
| scripts/ingest.sh | Defaults to --local; adds --push-main mode with origin/main safety rails, test gating, and direct push to main. |
| SCHEMA.md | Documents new optional campaign.source_uri while keeping legacy source_gcs. |
| README.md | Updates ingest documentation for S3 provenance and the automated --push-main flow. |
| Makefile | Adds URI= support for --source-uri and updates ingest help text to match new modes. |
| docs/summary.js | Renders Source/Raw links for either scheme via a new bucketLink helper. |
| docs/app.js | Renders provenance link in the viewer masthead via scheme-aware bucketLink. |
| converter/tests/test_phase.py | Extends test harness args to include source_uri. |
| converter/tests/test_golden.py | Extends test harness args to include source_uri. |
| converter/tests/test_campaign.py | Adds coverage ensuring source_uri is recorded and doesn’t implicitly set source_gcs. |
| converter/convert.py | Adds --source-uri CLI flag and writes campaign.source_uri in output JSON. |
| .github/workflows/ingest.yml | Removes the never-provisioned GCS/WIF ingest workflow. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The S3 console URL interpolated the bucket and key prefix straight into the query string, so a key holding #, ?, & or a space would have changed what the link points at (esc() escapes HTML, not URLs). Encode both, and the gs:// path too, keeping the "/" separators literal so every real bundle URI still produces the same link it did before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
stellar-rpc's bench-campaign workflow (stellar/stellar-rpc#921) now ingests a passing campaign's bundle itself, and the pipeline is S3 end-to-end. Runs should land on the live site directly instead of arriving as reviewable
run/<id>PRs.What changed
--source-urirecordscampaign.source_uri(s3://orgs://; additive, schema stays v1).--source-gcsremains for old runs.source_uri.--push-mainmode — requires HEAD at origin/main, converts, gates onmake test+make smoke, commits on main, pushes (the push is the Pages deploy), printsviewer: <url>.--localis now the default mode; the push+PR mode is removed.ingest.yml; README/SCHEMA/Makefile updated (URI=var).Testing
78 converter tests + 330 viewer smoke checks green; shellcheck clean; dry-runs exercised for a remote s3 URI and a fixture bundle.
Notes
BENCHMARKS_DEPLOY_KEYsecret on stellar/stellar-rpc, to be provisioned).packs-gs, test defaults, docs) is a separate follow-up.🤖 Generated with Claude Code