Migrate to cff concourse by aramprice · Pull Request #1616 · cloudfoundry/bosh-backup-and-restore

aramprice · 2026-05-30T00:45:33Z

New pipeline: https://bosh.ci.cloudfoundry.org/teams/main/pipelines/bbr-cli

Currently configured to pull https://bosh.ci.cloudfoundry.org/teams/main/pipelines/bbr-cli/resources/bosh-backup-and-restore-ci migrate-to-cff-concourse branch.

S3 Buckets have been renamed. Existing buckets have not been modified but do exist in an AWS account named "PCF Backup Restore".

Pipeline ✅

Copilot

Pull request overview

This PR migrates the project’s Concourse pipeline to a CFF-style setup by replacing Shepherd-based environment provisioning with BBL-managed bosh-lite environments and updating CI tasks/images accordingly.

Changes:

Reworks ci/pipeline.yml to run system tests and DRATs against BBL-provisioned environments (with new bbl-up, extract-bbl-env, and bbl-down tasks).
Updates DRATs and B-DRATs integration config generation tasks to source env credentials from bbl print-env rather than pipeline-passed CredHub/BOSH params.
Adjusts pipeline Git auth (deploy key for main repo) and updates task images to cloudfoundry/cf-deployment-concourse-tasks.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
ci/tasks/setup-drats-integration-config/task.yml	Drops explicit CredHub/BOSH/jumpbox params; relies on BBL-sourced env vars plus SYSTEM_DOMAIN.
ci/tasks/setup-drats-integration-config/task.sh	Uses `bbl print-env` from the mapped env input; derives jumpbox info from BOSH proxy env.
ci/tasks/setup-b-drats-integration-config/task.yml	Switches task image to `cloudfoundry/cf-deployment-concourse-tasks`.
ci/tasks/setup-b-drats-integration-config/task.sh	Generates B-DRATs integration config from `bbl print-env`-provided director/jumpbox credentials.
ci/tasks/extract-bbl-env/task.yml	Adds a task to extract BBL env details into a `bosh-env/metadata.yml` format for downstream tasks.
ci/tasks/extract-bbl-env/task.sh	Implements extraction/parsing of jumpbox + BOSH creds from BBL env.
ci/tasks/bbl-up/task.yml	Adds a BBL “up” task to provision bosh-lite on GCP with shared params.
ci/tasks/bbl-up/task.sh	Runs `bbl plan`/`bbl up` with bosh-deployment and plan patches.
ci/tasks/bbl-down/task.yml	Adds a BBL “down” task to tear down the environment.
ci/tasks/bbl-down/task.sh	Runs `bbl down --no-confirm` in the state directory.
ci/pipeline.yml	Major pipeline refactor: introduces BBL provisioning/teardown, removes Shepherd resources, updates git auth, and rewires DRATs jobs.
ci/configure.sh	Refactors pipeline rendering + `fly set-pipeline`, adds a DEBUG output path.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

aramprice · 2026-05-30T05:30:31Z

@coderabbitai review

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

linux-foundation-easycla · 2026-06-02T21:39:49Z

The committers listed above are authorized under a signed CLA.

✅ login: aramprice / name: aram price (11fbce2, 13d7567, 318d0b0, 3194c64, 38086e1, 475a25d, 4eac096, 540fda1, 69afd9e, 7dfa66e, 8533517, 89fb6a6, 9da52d3, a350606, c1d41ad, d472c88, dba824c, dd6ba78, ee547d6)

aramprice · 2026-06-02T23:54:15Z

/easycla

Copilot

Pull request overview

Copilot reviewed 60 out of 60 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 69 out of 69 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 68 out of 68 changed files in this pull request and generated 3 comments.

Replace hard-coded stemcell OS names and file paths with environment variables (STEMCELL_OS, FIXTURES_DIR, AWS_REGION). Update all BOSH fixture manifests to use the ((stemcell-os)) variable, and update Go test suites across integration, system, and ssh packages to read fixture paths and stemcell configuration from the environment rather than hard-coded strings.

Copilot

Pull request overview

Copilot reviewed 69 out of 69 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 69 out of 69 changed files in this pull request and generated 4 comments.

Replace static pre-provisioned environments with dynamic bbl up/down provisioning for each pipeline job using GCP BOSH-lite directors. - Add bbl-up, bbl-down, and extract-bbl-env tasks for per-job environment lifecycle management - Switch run-drats CF deployment to ubuntu-noble warden containers with cf-deployment v57 compiled releases - Pin BPM to 1.4.33 to fix cgroup v2 job startup hang on noble hosts - Serialize CF instance group updates to prevent cgroup contention across simultaneous noble warden containers - Configure BOSH DNS recursors (8.8.8.8, 8.8.4.4) so warden containers can resolve external DNS (GCP metadata DNS is not reachable from the 10.244.x.x warden network) - Add BOSH DNS wildcard alias for *.bosh-lite.com to the CF router IP so noble containers resolve CF API and app routes via BOSH DNS - Add 8.8.8.8 to silk-cni dns_servers as external DNS fallback since 169.254.169.254 is not routable inside the warden network - Raise inotify limits on the director VM to prevent Envoy and systemd from aborting when the shared kernel watch limit is exhausted - Upgrade os-conf to v23 (bosh-lite.yml pins v18 which lacks pre-start-script); enables iptables FORWARD policy to be set at director startup so warden container IPs are reachable from jumpbox - Enable IP forwarding on the director VM via an os-conf pre-start-script ops file injected during bbl up; sets iptables FORWARD policy to ACCEPT so warden container IPs (10.244.x.x) are reachable from the jumpbox - Route DRATS acceptance tests through the jumpbox via sshuttle and dnsmasq since Concourse workers cannot reach 10.244.0.0/16 - Fix CredHub and UAA TLS trust by concatenating both the CredHub TLS CA and the BOSH director CA into CREDHUB_CA_CERT - Use `instances: 3` instead of `instances: 20` in fixtures/many-bbr-jobs.yml

aramprice force-pushed the migrate-to-cff-concourse branch from 0593906 to 1298a8e Compare May 30, 2026 00:48

aramprice marked this pull request as ready for review May 30, 2026 01:22

Copilot AI review requested due to automatic review settings May 30, 2026 01:22

Copilot started reviewing on behalf of aramprice May 30, 2026 01:23 View session

Copilot AI reviewed May 30, 2026

View reviewed changes

Comment thread ci/configure.sh

Comment thread ci/tasks/setup-drats-integration-config/task.sh Outdated

Comment thread ci/tasks/extract-bbl-env/task.sh

cf-foundation-community-automation Bot added this to Foundational Infrastructure Working Group May 30, 2026

cf-foundation-community-automation Bot moved this to Inbox in Foundational Infrastructure Working Group May 30, 2026

aramprice force-pushed the migrate-to-cff-concourse branch 2 times, most recently from 6934212 to 722c3ff Compare May 30, 2026 05:25

aramprice requested a review from Copilot May 30, 2026 05:25

Copilot started reviewing on behalf of aramprice May 30, 2026 05:25 View session

Copilot AI reviewed May 30, 2026

View reviewed changes

Comment thread ci/tasks/setup-drats-integration-config/task.sh

Comment thread ci/tasks/extract-bbl-env/task.sh Outdated

Comment thread ci/tasks/bbl-down/task.sh Outdated

aramprice force-pushed the migrate-to-cff-concourse branch 2 times, most recently from ed4fcc2 to 8c2b916 Compare May 31, 2026 00:41

aramprice requested a review from Copilot May 31, 2026 02:33

Copilot started reviewing on behalf of aramprice May 31, 2026 02:33 View session

Copilot AI reviewed May 31, 2026

View reviewed changes

Comment thread ci/tasks/setup-drats-integration-config/task.sh Outdated

Comment thread ci/tasks/setup-b-drats-integration-config/task.sh

aramprice force-pushed the migrate-to-cff-concourse branch 2 times, most recently from 2b06f7d to 16654a6 Compare June 2, 2026 21:32

aramprice force-pushed the migrate-to-cff-concourse branch from c7854b5 to 39140c0 Compare June 2, 2026 23:48

aramprice requested a review from Copilot June 2, 2026 23:54

Copilot started reviewing on behalf of aramprice June 2, 2026 23:54 View session

Copilot AI reviewed Jun 2, 2026

View reviewed changes

Comment thread system/deployment/system_suite_test.go

Comment thread system/deployment/restore_test.go

Comment thread system/deployment/restore_test.go Outdated

aramprice force-pushed the migrate-to-cff-concourse branch 3 times, most recently from 6249164 to dd6ba78 Compare June 3, 2026 18:05

aramprice requested a review from Copilot June 8, 2026 18:36

Copilot AI reviewed Jun 17, 2026

View reviewed changes

Comment thread system/util.go

Comment thread ci/tasks/setup-drats-integration-config/task.sh

Comment thread .github/workflows/go.yml Outdated

Comment thread ci/tasks/upload-stemcell/task.sh Outdated

mkocher approved these changes Jun 18, 2026

View reviewed changes

aramprice force-pushed the migrate-to-cff-concourse branch 5 times, most recently from e5c61f3 to 54cecfa Compare June 18, 2026 03:37

aramprice requested a review from Copilot June 18, 2026 03:38

Copilot started reviewing on behalf of aramprice June 18, 2026 03:39 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread ssh/ssh_suite_test.go

Comment thread integration/director/director_suite_test.go

Comment thread integration/deployment/deployment_suite_test.go

aramprice force-pushed the migrate-to-cff-concourse branch 2 times, most recently from b2ab96a to 5657ed6 Compare June 18, 2026 05:21

aramprice force-pushed the migrate-to-cff-concourse branch from 5657ed6 to 8259bf9 Compare June 18, 2026 06:40

aramprice requested review from a team, dlresende and mrosecrance and removed request for a team June 18, 2026 14:33

aramprice force-pushed the migrate-to-cff-concourse branch from 8259bf9 to 44a5991 Compare June 18, 2026 15:54

aramprice requested a review from Copilot June 18, 2026 15:55

Copilot started reviewing on behalf of aramprice June 18, 2026 15:56 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread system/util.go

Comment thread ssh/ssh_suite_test.go

Comment thread integration/director/director_suite_test.go

Comment thread integration/deployment/deployment_suite_test.go

aramprice requested a review from mkocher June 18, 2026 17:42

aramprice force-pushed the migrate-to-cff-concourse branch from 44a5991 to bf5b50b Compare June 18, 2026 18:31

aramprice requested a review from Copilot June 18, 2026 18:32

Copilot started reviewing on behalf of aramprice June 18, 2026 18:32 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread ci/pipeline.yml

Comment thread ssh/ssh_suite_test.go

Comment thread integration/director/director_suite_test.go

Comment thread integration/deployment/deployment_suite_test.go

mkocher approved these changes Jun 18, 2026

View reviewed changes

Conversation

aramprice commented May 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aramprice commented May 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

linux-foundation-easycla Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aramprice commented Jun 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aramprice commented May 30, 2026 •

edited

Loading

linux-foundation-easycla Bot commented Jun 2, 2026 •

edited

Loading