feat(carvel): add BOSH link support via job-spec-overlay and values-overlay sidecars by drich10 · Pull Request #663 · pivotal-cf/kiln

drich10 · 2026-06-16T20:09:51Z

Summary

Introduce two per-packageinstall sidecar conventions that kiln auto-detects in packageinstalls/:

Sidecar	Purpose	When it runs
`<entry>.job-spec-overlay.yml`	Declares additional BOSH link consumptions; kiln appends them to the generated `registry-data` job.MF	`kiln carvel bake` time
`<entry>.values-overlay.erb`	ERB fragment injected before `YAML.dump(values)`; mutates the values hash via BOSH link objects	BOSH deploy time (ERB)

Missing sidecar files are silently skipped (no-op). Both sidecars are co-located with the packageinstall YAML they extend.

Motivation

Carvel tiles need to consume BOSH links from co-deployed products (e.g. the binding_cache link provided by CF's loggr-syslog-binding-cache job) to inject environment-specific runtime values at BOSH deploy time. Previously only the hardcoded cluster-info link was available in the generated job spec. The binding-cache has no BOSH DNS alias, so its address can only be discovered via a BOSH link.

Approach — co-located sidecars (mirrors the values-overlay.erb pattern)

Tile authors declare BOSH link consumptions in a <name>.job-spec-overlay.yml sidecar alongside their packageinstall YAML. kiln reads all such files and aggregates their consumes: entries into the generated registry-data job spec. The schema mirrors the BOSH job spec consumes: block directly.

packageinstalls/
  tnz-ear-runtime-package-install.yml
  tnz-ear-runtime-package-install.job-spec-overlay.yml   ← NEW: BOSH link declarations
  tnz-ear-runtime-package-install.values-overlay.erb     ← NEW: values mutations at deploy time

Example job-spec-overlay.yml:

consumes:
- name: binding_cache
  type: binding_cache
  optional: false

Example values-overlay.erb:

<% addr = link("binding_cache").instances.first.address %>
<% values["syslog_agent"]["cache"]["url"] = "https://#{addr}:9000" %>

Kilnfile is not modified. It stays focused on dependency management (release sources, releases, stemcell). The BOSHLinkConsumer/BOSHLinks types and BoshLinks field from the initial commit have been removed.

Backward Compatibility

Tiles with no *.job-spec-overlay.yml files produce identical output to before (no-op)
Tiles with no *.values-overlay.erb files produce identical output to before (no-op)
All existing cluster-info link behaviour is preserved
Kilnfile schema unchanged

Test Plan

jobSpecOverlay — parses a consumes: list, handles empty list, handles missing key
buildRegistryDataSpec — includes user-declared links after cluster-info
buildRegistryDataSpec — no additional links when no sidecars present
buildRegistryDataSpec — optional flag rendered correctly
generateManifestTemplate with overlay — overlay content present and before YAML.dump
generateManifestTemplate with empty overlay — still produces valid template
Full suite: go test ./... — 0 failures (Docker integration test skipped due to no daemon)

Made with Cursor

Add BOSHLinkConsumer and BOSHLinks types and a BoshLinks field to the Kilnfile struct so that tile authors can declare additional BOSH link consumptions in their Kilnfile for carvel tiles. Co-authored-by: Cursor <cursoragent@cursor.com>

- Extract buildRegistryDataSpec() helper to build the registry-data job.MF; it appends any bosh_links.consumes entries from the Kilnfile after the hardcoded cluster-info link - Read Kilnfile in generateBoshReleaseDir() to discover additional BOSH link declarations (missing Kilnfile or missing field is a no-op) - Update generateManifestTemplate() to accept optional overlayContent injected before YAML.dump(values), allowing tile authors to mutate the values hash using BOSH link objects at deploy time - Read <entry>.values-overlay.erb alongside each packageinstall YAML and embed its content in the generated ERB template Co-authored-by: Cursor <cursoragent@cursor.com>

rizwanreza · 2026-06-17T15:47:39Z

I have a concern on bosh_links placement: should this live in Kilnfile?

Kilnfile is a dependency manifest. It tracks which BOSH releases to fetch and where from (think Gemfile). bosh_links is deploy-time BOSH job config. Those are different concerns, and mixing them here will confuse anyone reading the Kilnfile expecting only release sourcing info.

Three alternatives worth discussing:

Option A — Sidecar file (matches this PR's own pattern)

This PR already introduces *.values-overlay.erb as a co-located sidecar kiln auto-detects. Same idea works here: a registry-data-consumes.yml kiln reads and merges into the generated job spec. Keeps things co-located and Kilnfile clean.

Option B — base.yml with a kiln extension stanza
base.yml is already the tile's primary config, already processed by kiln. A carvel: block kiln reads and strips before handing off to OpsManager. Everything tile-level in one place.

Option C — Dedicated carvel.yml
Standalone file for carvel-specific build config. Cleanest separation, but adds a new convention.

Option A feels most natural given what's already in this PR. Open to other takes.

drich10 · 2026-06-18T16:08:10Z

@rizwanreza I like option A - any strong idea of how you'd describe the "overlay" section for the job spec vs the link definition section? I'll update the design/pr later today with this in mind

…ec-overlay sidecar - Remove BOSHLinkConsumer/BOSHLinks types and BoshLinks field from cargo.Kilnfile; Kilnfile stays focused on dependency management - Add local boshLinkConsumer and jobSpecOverlay types to carvel package - Detect <entry>.job-spec-overlay.yml alongside each packageinstall in packageinstalls/; parse and accumulate consumes entries across all entries; pass to buildRegistryDataSpec() - Update buildRegistryDataSpec signature to use local boshLinkConsumer - Tests: add jobSpecOverlay parse tests; update buildRegistryDataSpec tests to use local type; remove TestKilnfile_ParsesBoshLinks Co-authored-by: Cursor <cursoragent@cursor.com>

drich10 · 2026-06-19T15:50:57Z

@rizwanreza Updated — went with a variant of your Option A that mirrors the *.values-overlay.erb pattern more closely.

Instead of registry-data-consumes.yml at the tile root, the sidecar is named <entry>.job-spec-overlay.yml and lives alongside the packageinstall YAML in packageinstalls/. This keeps both sidecars co-located with the entry they extend and uses a consistent *-overlay.* naming convention:

Sidecar	Scope	Content
`<entry>.job-spec-overlay.yml`	Build time	`consumes:` list for the BOSH job spec
`<entry>.values-overlay.erb`	Deploy time	ERB mutations using BOSH link objects

bosh_links has been removed from Kilnfile entirely — the BOSHLinkConsumer/BOSHLinks types are gone from pkg/cargo and Kilnfile is back to being a pure dependency manifest.

…dateVariables Both branches added code after the progress() function: the PR branch added boshLinkConsumer/jobSpecOverlay types; main added validateVariables (from #664). Include both blocks. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

davewalter · 2026-06-22T21:40:54Z

+		})
 	})

 	Context("Bake", func() {


🤖 AI Review

🟡 Should Fix

The integration test (Bake context) exercises the full pipeline via testdata/sample-tile but that test data doesn't include any .job-spec-overlay.yml or .values-overlay.erb sidecars. The unit tests for the helper functions are good, but there's no end-to-end coverage of the new sidecar loading path through generateBoshReleaseDir. Consider adding a variant of testdata/sample-tile (or a fixture file) that includes at least one job-spec-overlay.yml and verifying it appears in the generated spec file.

- Surface OS errors other than not-found for both sidecar reads (values-overlay.erb and job-spec-overlay.yml) instead of silently swallowing them - Return an error when a job-spec-overlay.yml file exists but contains malformed YAML - Deduplicate BOSH link names before generating registry-data job spec; BOSH rejects duplicate consumes entries - Use yaml.Marshal to emit the additional consumes block in buildRegistryDataSpec, preventing YAML injection from link name/type values - Add test-install.job-spec-overlay.yml to sample-tile testdata; extend e2e spec assertion to verify binding_cache link appears in generated job spec - Add unit tests for injection safety, deduplication contract, and malformed YAML Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

rizwanreza

Just a minor thing but otherwise look great!

rizwanreza · 2026-06-24T16:31:05Z

+			seen[c.Name] = struct{}{}
+			deduped = append(deduped, c)
+		}
+	}


The deduplication keeps the first occurrence and discards the rest. If packageinstall A declares binding_cache with type: binding_cache and packageinstall B declares binding_cache with type: binding-cache-v2, B's type is silently dropped. The generated BOSH spec will use A's type with no indication that B's declaration was ignored. This would be a confusing bug to track down at deploy time.

good call - we log this now in the output since I think its a reasonable decision but puts the information in the tile author's view during build. WDYT?

When two packageinstalls declare a BOSH link with the same name but different type or optional, the second was previously silently dropped. Now emits a WARNING via progress output so the user knows which definition won and which was ignored, making deploy-time confusion much easier to debug. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add explicit assertion on which entry is kept in the optional-conflict case, a multi-conflict test verifying one warning per extra occurrence, and a nil-input edge case. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

drich10 and others added 2 commits June 16, 2026 16:05

drich10 changed the title ~~feat(carvel): add bosh_links.consumes support and values-overlay.erb~~ feat(carvel): add BOSH link support via job-spec-overlay and values-overlay sidecars Jun 19, 2026

drich10 marked this pull request as ready for review June 19, 2026 20:33

davewalter reviewed Jun 22, 2026

View reviewed changes

rizwanreza self-requested a review June 23, 2026 17:21

rizwanreza reviewed Jun 24, 2026

View reviewed changes

drich10 and others added 2 commits June 25, 2026 16:21

drich10 merged commit 6862b6f into main Jun 29, 2026
6 checks passed

drich10 deleted the feat/carvel-bosh-links-overlay branch June 29, 2026 19:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(carvel): add BOSH link support via job-spec-overlay and values-overlay sidecars - #663

feat(carvel): add BOSH link support via job-spec-overlay and values-overlay sidecars#663
drich10 merged 7 commits into
mainfrom
feat/carvel-bosh-links-overlay

drich10 commented Jun 16, 2026 •

edited

Loading

Uh oh!

rizwanreza commented Jun 17, 2026 •

edited

Loading

Uh oh!

drich10 commented Jun 18, 2026

Uh oh!

drich10 commented Jun 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davewalter Jun 22, 2026

Uh oh!

rizwanreza left a comment

Uh oh!

rizwanreza Jun 24, 2026

Uh oh!

drich10 Jun 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

drich10 commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Approach — co-located sidecars (mirrors the values-overlay.erb pattern)

Backward Compatibility

Test Plan

Uh oh!

rizwanreza commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drich10 commented Jun 18, 2026

Uh oh!

drich10 commented Jun 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davewalter Jun 22, 2026

Choose a reason for hiding this comment

🤖 AI Review

Uh oh!

rizwanreza left a comment

Choose a reason for hiding this comment

Uh oh!

rizwanreza Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

drich10 Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

drich10 commented Jun 16, 2026 •

edited

Loading

rizwanreza commented Jun 17, 2026 •

edited

Loading