Skip to content

Commit 9144f49

Browse files
rizwanrezarizwanreza
committed
GPP onboarding doc for Carvel based tiles
1 parent 9af3123 commit 9144f49

1 file changed

Lines changed: 232 additions & 0 deletions

File tree

‎gpp-onboarding-carvel.md‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
This playbook provides instructions for Carvel/Kubernetes tile teams to onboard their tiles onto the Golden Path to publish workflow.
2+
3+
For Carvel-based tiles, intermediary BOSH releases are generated automatically from imgpkg bundles (via ezbake) and do not need to be managed directly by your team. GPP handles BOSH release ingest and compilation behind the scenes. Your team only needs to manage final tile releases.
4+
5+
The result of this work will give you a re-bakable tile in [Artifactory](https://usw1.packages.broadcom.com/ui/repos/tree/General/tas-ecosystem-generic-prod-local/tile-releases) with compiled BOSH releases that is scanned by BlackDuck. For TVS integration please notify the Slingshots team when you're ready for it along with a link to your config please.
6+
7+
You may also optionally configure your Tile to generate RMT releases, and Open Source License Disclosure files from Blackduck.
8+
9+
## Pre-requisites for onboarding
10+
11+
### Your tile is built with Kiln
12+
13+
The Golden Path does not currently support tiles built with [tile-generator](https://github.com/cf-platform-eng/tile-generator).
14+
15+
Please consider using [kiln](https://github.com/pivotal-cf/kiln/blob/main/TILE_AUTHOR_GUIDE.md). Carvel tile workflows use the `kiln carvel` subcommand group (`bake`, `upload`, `publish`, `rebake`).
16+
17+
### Github repo access: tiles
18+
19+
Please provide write access to your private tile repositories to our bot account. Because BOSH releases are generated from the tile source (imgpkg bundles), separate BOSH release repositories are not required.
20+
21+
- For github enterprise (github.gwd.broadcom.com): [tanzu-tas-ecosystem](https://github.gwd.broadcom.net/tanzu-tas-ecosystem)
22+
- for github.com: [tas-ecosystem-bot](https://github.com/tas-ecosystem-bot)
23+
24+
### TNZ team membership
25+
26+
To create PRs against our configuration repo you need to be a member of the [`all`](https://github.gwd.broadcom.net/orgs/TNZ/teams/all) team in the [TNZ org](https://github.gwd.broadcom.net/orgs/TNZ).
27+
28+
### Broadcom artifactory access
29+
30+
Authentication is required for accessing repos and artifacts on the Broadcom Jfrog Artifactory service. To get access for your team to our artifact repos containing: bosh-releases, compiled-releases, tile-releases and tile-candidates, create a [1.Support Ticket](https://broadcomitsm.wolkenservicedesk.com/wolken-support/item_details?itemId=2422).
31+
Specify:
32+
33+
- Artifactory Server Name / URL: `https://usw1.packages.broadcom.com/ui`
34+
- Sample Business Justification:
35+
36+
```text
37+
Need read access to tas-ecosystem-* artifactory projects on https://usw1.packages.broadcom.com
38+
39+
For the following teammates / service accounts:
40+
- memberX
41+
- memberY
42+
- memberZ
43+
- bot / service account
44+
```
45+
46+
#### Credentials
47+
48+
Since artifactory is authenticated with Okta SSO, password authentication to the service it not allowed. Artifactory have `api_keys` and `identity_tokens` that are used as passwords.
49+
50+
Once access is granted and you are able to login to the artifactory ui via SSO, an `api_key` or `identity token` needs to be created for use with Kiln
51+
52+
1. Upper right click dropdown of: `Welcome, your_username`
53+
2. Click `Edit Profile`
54+
3. Create an `api_key` or `identity_token` here and use it as the password for `kiln` commands or the artifactory cli.
55+
56+
#### Network access
57+
58+
The `usw1.packages.broadcom.com` artifactory is also only available on the Broadcom network. If accessing remotely, full tunnel VPN is required.
59+
60+
If you CI is on the VMware / Broadcom Network and is blocked from accessing the artifactory, reach out to Google Chat Space: [#VMW-harbor-jfrog-migration](https://chat.google.com/room/AAAAcWIWWOA?cls=7)
61+
62+
## Golden Path Configuration
63+
64+
Configuration for the TAS Golden Path is stored in this repo and used as inputs to generate concourse pipelines.
65+
66+
For Carvel-based tiles, the onboarding is simpler than for traditional BOSH tiles because GPP manages BOSH release ingest and compilation behind the scenes. You do not need to add BOSH release config files to the `bosh/` folder. The existing [bosh-ingest](https://tpe-concourse-rock.acc.broadcom.net/teams/tas-ecosystem/pipelines/bosh-releases?group=ingest-releases) and [bosh-compile](https://tpe-concourse-rock.acc.broadcom.net/teams/tas-ecosystem/pipelines/bosh-releases?group=compile-releases) pipelines are available for inspection if needed but do not require configuration from your team.
67+
68+
Overall the following steps to complete are:
69+
70+
- Updating the `Kilnfile` in the git repository to use artifactory as a source for generated BOSH releases.
71+
- (optional) Creating a branch in the git repository of your tile for a pipeline to push Kilnfile.lock updates for your review.
72+
- Creating config files for your tile(s) in `tiles/` folder to generate pipeline that will:
73+
- Bake tile candidates with `kiln carvel bake`. Dev builds of tiles on your main / feature branch
74+
- [`kiln carvel rebake`](https://github.com/pivotal-cf/kiln) for versioned release tiles
75+
- Associate the BOSH releases consumed by the tile to the Blackduck tile project
76+
- (optional) Automatically creates RMT releases that are included in the next-available TPM managed Release Train to assist with publishing
77+
- If RMT is enabled, then your RMT release is eligible for automatic Open Source License Notice inclusion. Please see [creating open source license disclosures](./creating_open_source_license_disclosures.md).
78+
- (optional) TVS integration (notify the #tas-slingshots team with your tile config requesting this when ready)
79+
80+
### Tile repository updates
81+
82+
Set up your tile repository so that `kiln carvel` commands can fetch generated BOSH releases from Artifactory.
83+
84+
1. In the main / feature branch, update the `Kilnfile` to include artifactory as the remote source for BOSH releases.
85+
86+
```yaml
87+
release_sources:
88+
- type: artifactory
89+
id: artifactory_bosh_releases
90+
artifactory_host: $(variable "artifactory_host")
91+
repo: $(variable "artifactory_repo")
92+
username: $(variable "artifactory_username")
93+
password: $(variable "artifactory_password") # api_key or identity token
94+
publishable: true # if this repo contains releases that are suitable to ship to customers
95+
path_template: bosh-releases/{{.Name}}/{{.Name}}-{{.Version}}.tgz
96+
```
97+
98+
2. Use `kiln carvel upload` to generate the BOSH release from your imgpkg bundle, upload it to Artifactory, and update the `Kilnfile.lock` with the remote location and checksum.
99+
100+
Example `kiln carvel upload` command:
101+
102+
```bash
103+
$ kiln carvel upload \
104+
--artifactory-host https://usw1.packages.broadcom.com \
105+
--artifactory-repo tas-ecosystem-generic-prod-local \
106+
--artifactory-username <your_user_or_bot_account> \
107+
--artifactory-password <api_key-or-identity_token> \
108+
--output-file my-tile-1.0.0-dev.pivotal
109+
```
110+
111+
This uploads the generated BOSH release and writes a `Kilnfile.lock` referencing the remote artifact. Commit the updated `Kilnfile.lock` to your repository.
112+
113+
3. (Optional) Create a new update branch from the main / feature branch in your tile repository (eg: `autobump`). Our CI will force push commits to this branch.
114+
While this provides an auto update functionality, you are welcome to continue using your existing auto update tools (eg: dependabot).
115+
You can also specify your feature branch if you want our CI to push the `Kilnfile.lock` updates directly to your feature branch.
116+
If `branch` and `update_branch` are same, force push functionality is disabled. Ensure the branch specified in `update_branch` has [push access to our bot account](#github-repo-access-tiles).
117+
118+
### Tile config onboard
119+
120+
1. Clone [this](https://github.gwd.broadcom.net/TNZ/tas-ecosystem-configuration) git repository if you have not already and create a branch locally for your changes. You should have write access to the repo and not need to create a fork to create a PR. If not, please review [this pre-requisite](#tnz-team-membership)
121+
122+
2. Create a new file for each of your tiles under the [tiles](https://github.gwd.broadcom.net/TNZ/tas-ecosystem-configuration/tree/main/tiles) directory. Please note that `artifact_name` is especially significant because it determines the file name prefix in `artifactory`, project names prefixes in `blackduck`, and is the prefix used by the published release file in RMT / Broadcom Portal (when enabled).
123+
Hello Tile - `hello-tile.yml`
124+
125+
```yaml
126+
#@data/values
127+
---
128+
repo: https://github.gwd.broadcom.net/TNZ/hello-tile.git
129+
branch: main
130+
update_branch: auto-bump
131+
subpath: .
132+
artifact_name: crhntr-hello #! this is the prefix for the built tiles and must be consistent with blackduck too
133+
prerelease_format: build_increment_sha #! "sha" or "build_increment_sha" for versioning tile candidate builds. we recommend build_increment_sha
134+
team_members:
135+
- a@vmware.com
136+
- b@vmware.com
137+
team_google_chat_group: some-group #! required - google space / chat group for your team
138+
team_slack_channel: some-channel #! If your team has slack channel
139+
```
140+
141+
Scheduler Tile - `p-scheduler.yml` (auto update directly on feature branch)
142+
143+
```yaml
144+
#@data/values
145+
---
146+
repo: https://github.com/pivotal-cf/p-scheduler.git
147+
branch: master
148+
update_branch: master
149+
subpath: .
150+
artifact_name: p-scheduler #! this is the prefix for the built tiles and must be consistent with blackduck too
151+
prerelease_format: build_increment_sha #! "sha" or "build_increment_sha" for versioning tile candidate builds
152+
team_members:
153+
- a@vmware.com
154+
- b@vmware.com
155+
team_google_chat_group: some-group #! required - google space / chat group for your team
156+
team_slack_channel: some-channel #! If your team has slack channel
157+
```
158+
159+
3. (optional) Add fields for automatic RMT _**draft**_-release creation
160+
161+
>**Warning:** You will need to add upgrade specifiers (else Upgrade Planner will break!) and double check your release is ready to be set to GA. It defaults to a draft.
162+
163+
Release tiles can be used as the basis for automatic `RMT` draft release creation. As a draft this means further steps are required prior to publishing. These include manually setting your upgrade specifiers, double checking the version, GA/EOGs dates, and release type, etc we inferred for you or read from your tile configuration's `rmt` entry.
164+
165+
See [tile rmt release](https://github.gwd.broadcom.net/TNZ/tas-ecosystem-configuration/tree/main/docs/tile_rmt_release.md) for details.
166+
167+
4. (optional) Add a field to enable automatic Black Duck tile project associations. In order to begin updating your Black Duck tile project:
168+
>**Prerequisites:**
169+
> Follow the [BlackDuck Onboarding section](./creating_open_source_license_disclosures.md#blackduck-onboarding) for
170+
> your tile. BOSH release projects are managed by GPP for Carvel-based tiles.
171+
172+
1) Confirm your project exists @ https://broadcom-vmw.app.blackduck.com/ with the format `TNZ-CF-<artifact_name>-tile`, as `<artifact_name>` is found in your tile config.
173+
2) If not, submit a ticket to request it ([BlackDuck Onboarding section](./creating_open_source_license_disclosures.md#blackduck-onboarding)) or rename it yourself.
174+
> **Note:** Scanning is enabled by default. However, you may disable it by adding the following to your `./tiles/<tile>.yml` config:
175+
```yaml
176+
blackduck:
177+
enabled: false
178+
```
179+
180+
5. Create a PR to [this](https://github.gwd.broadcom.net/TNZ/tas-ecosystem-configuration) repository to add the newly created files that contain the tile information.
181+
An extensive PR Check job will verify your change and add a comment if anything needs to be addressed. When the job passes you can merge the PR.
182+
On merge, the respective golden path jobs will be created / updated for your tile. Please reach out to [#tas-slingshots on Google Chat](https://chat.google.com/room/AAAAZuDvKe0?cls=7) with any questions or issues getting your PR merged.
183+
184+
## Updating CI for your tile and automatic RMT _draft_ releases
185+
186+
### Use `kiln carvel publish --final`
187+
188+
If you are using CI to create new versions of tiles, the following updates can be made to take advantage of reproducible builds via `kiln carvel rebake`.
189+
190+
Update your CI to output final tile builds using `kiln carvel publish --final`. When passing the **_--final_** flag, Kiln creates a bake record file under the **_bake_records_** folder. As part of the final tile build CI job, the bake records file needs to be committed and pushed to the tile repository.
191+
192+
Golden Path to publish will then use this bake record to trigger `kiln carvel rebake`, producing a final tile from our [CI](https://runway-ci-srp.eng.vmware.com/teams/tas-ecosystem/) and upload it to [artifactory](https://build-artifactory.eng.vmware.com/ui/repos/tree/General/tas-ecosystem-generic-local/) repo under the sub-path: tile-releases.
193+
194+
In the case of a pre-release version the build will not result in a publish. This is useful to verify OSL triage status and test your candidate build. For example, you may create a
195+
bake record with version `2.4.41-dev.0`, rerun the OSL generation multiple times, then finally create a `2.41.1` to trigger the full publish.
196+
197+
Example `kiln carvel publish --final` command:
198+
199+
```bash
200+
$ kiln carvel publish --final --version 2.1.41 \
201+
--output-file my-tile-2.1.41.pivotal \
202+
--source-directory .
203+
```
204+
205+
_Example:_ Bake record that should be committed to the tile repo that is created by `kiln carvel publish --final` as file: `bake_records/2.1.41.json`:
206+
207+
```json
208+
{
209+
"source_revision": "1b19d8cb80e6cfdddd7be1c7a26c8210cbd4e4c5",
210+
"version": "2.1.41",
211+
"kiln_version": "0.90.0",
212+
"file_checksum": "7622143c54dc53087a6c2401f5030170515e14f466857564a980092d4c87a094",
213+
"tile_directory": "."
214+
}
215+
```
216+
217+
When executing `kiln carvel publish --final`, use the values for artifactory variables in your Kilnfile:
218+
219+
- artifactory_host: `https://usw1.packages.broadcom.com`
220+
- artifactory_repo: `tas-ecosystem-generic-prod-local`
221+
- artifactory_username: **_your account or service account for broadcom artifactory_**
222+
- artifactory_password: **_respective api_key or identity token_**
223+
224+
**_NOTE: https://usw1.packages.broadcom.com is accessible via Broadcom VPN with full tunnel gateway and TPE concourse workers_**
225+
226+
**_Commit the new bake record to the git repository of the tile_**
227+
228+
**_Warning: Your bake_records directory must only contain bake records_**
229+
230+
GPP will then automatically run `kiln carvel rebake` against the bake record to produce the final `.pivotal` file, validate the checksum, and upload it to Artifactory and optionally RMT.
231+
232+
Please refer to [Tile RMT Release](Publish-Tiles-to-RMT) to configure publishing your tile to RMT via Golden Path to Publish.

0 commit comments

Comments
 (0)