Skip to content

ci: add nightly job to sync source mirror with pkgs in builtin - #6069

Open
alecbcs wants to merge 1 commit into
spack:developfrom
alecbcs:ci/add-scheduled-mirror-update
Open

ci: add nightly job to sync source mirror with pkgs in builtin#6069
alecbcs wants to merge 1 commit into
spack:developfrom
alecbcs:ci/add-scheduled-mirror-update

Conversation

@alecbcs

@alecbcs alecbcs commented Aug 12, 2026

Copy link
Copy Markdown
Member

Add a minimal nightly GitHub Actions CI/CD job that compares the state of the package repository to the mirrored source cache and mirrors any artifacts that are missing from the cache. (As of the time of posting there are 439 missing package versions that would be uploaded by this workflow.)

Will need to work with @kwryankrattiger and @jjnesbitt before merging to update the OIDC rules in AWS to allow this new workflow.

Signed-off-by: Alec Scott <alec@llnl.gov>
@alecbcs
alecbcs requested a review from scheibelp August 12, 2026 22:22
@spackbot-triage spackbot-triage Bot added the ci Related to CI/CD workflows and pipelines label Aug 12, 2026

@scheibelp scheibelp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other points (not requests for this PR):

  • (request for later) If a patch/resource is dropped, this doesn't catch that: if we already placed the "main" version tarball in a mirror, and a resource was missed, then this job will also miss that (I see though that patches/resources generally are included, just not in this rare case). I think this can be addressed later.
  • (not a request) What about wrong checksums (i.e. if someone specifies the wrong checksum for a version)? I think the behavior would be OK: this script will collect every wrong checksum, download the version (which should fail, but spack mirror generally proceeds on in this case). We'd want to clean those up though.


# output specs one per line for use by `spack mirror create`
for spec in specs_to_output:
print(spec)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request: limit this to something like 100-500 and print a warning if we go over that (tty.warn? something that prints to stderr)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Related to CI/CD workflows and pipelines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants