Skip to content

Commit 932aeef

Browse files
Add ci-cassandra Helmfile deployment with Let's Encrypt TLS
Adds .jenkins/k8s/ci-cassandra/ — a Helmfile-based deployment for ci-cassandra.infra.datastax.com that manages three Helm releases in dependency order: 1. cert-manager (jetstack/cert-manager) — issues and auto-renews the Let's Encrypt TLS certificate via HTTP-01 challenge on port 80. 2. cert-issuer (local chart) — ClusterIssuer + Certificate resources for ci-cassandra.infra.datastax.com. 3. cassius (jenkins/jenkins) — Jenkins controller and agents, layering the upstream jenkins-deployment.yaml with a site override that adds an nginx TLS-terminating sidecar on port 4443 and exposes port 443 on the NLB. TLS flow: Browser → NLB:443 (TCP passthrough) → nginx sidecar:4443 → Jenkins:8080 Port 80 stays open for Let's Encrypt HTTP-01 renewal challenges and Jenkins agent JNLP connections. No DNS change is required; the existing A record for ci-cassandra.infra.datastax.com (34.88.16.212) is used as-is. Also includes: - deploy.sh: install/upgrade script with --jenkins-only and --dry-run flags - nodeSelector fix: upstream jenkins-deployment.yaml uses node-selector (silently ignored by the chart); the override adds the correct key so the controller is pinned to the controller node pool - README.md documenting the layout, TLS flow, node pool requirements (including the missing agents-report pool), and operational runbook
1 parent 93b5aec commit 932aeef

10 files changed

Lines changed: 693 additions & 0 deletions

File tree

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# ci-cassandra Cluster Management
2+
<!--
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing, software
14+
distributed under the License is distributed on an "AS IS" BASIS,
15+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
See the License for the specific language governing permissions and
17+
limitations under the License.
18+
-->
19+
20+
This directory manages the Helm-based deployment of **ci-cassandra.infra.datastax.com**,
21+
the pre-commit Jenkins CI cluster for Apache Cassandra running on GKE in the
22+
`cassandra-oss` GCP project.
23+
24+
It extends the upstream `../jenkins-deployment.yaml` with:
25+
- **Let's Encrypt TLS** via cert-manager and an nginx sidecar
26+
- **Helmfile** so that cert-manager, the issuer, and Jenkins are all managed
27+
together in dependency order
28+
29+
## Directory layout
30+
31+
```
32+
ci-cassandra/
33+
├── helmfile.yaml # orchestrates all three releases in order
34+
├── cert-manager-values.yaml # jetstack/cert-manager Helm values
35+
├── cert-issuer-values.yaml # hostname, email, secret name
36+
├── jenkins-tls-overrides.yaml # nginx sidecar + port 443 + nodeSelector fix
37+
├── nginx-configmap.yaml # nginx config (applied by deploy.sh, not Helm)
38+
├── deploy.sh # install / upgrade script
39+
├── charts/
40+
│ └── cert-issuer/ # minimal local chart: ClusterIssuer + Certificate
41+
└── README.md # this file
42+
```
43+
44+
## Prerequisites
45+
46+
```bash
47+
brew install helm helmfile
48+
helm plugin install https://github.com/databus23/helm-diff # helm 3
49+
# helm 4: helm plugin install --verify=false https://github.com/databus23/helm-diff
50+
51+
gcloud auth login
52+
gcloud auth application-default login
53+
gcloud container clusters get-credentials mck-cassandra-jenkins-hamina-2 \
54+
--zone europe-north1-a --project cassandra-oss
55+
```
56+
57+
## How TLS works
58+
59+
```
60+
Browser → NLB:443 (TCP passthrough, 34.88.16.212)
61+
→ Jenkins pod:4443 (nginx sidecar — TLS termination)
62+
→ localhost:8080 (Jenkins)
63+
64+
Let's Encrypt renewal:
65+
LE server → NLB:80 → Jenkins pod:8080 → ACME solver pod
66+
```
67+
68+
The NLB on port 80 stays open. cert-manager spins up a temporary pod to
69+
answer the HTTP-01 challenge; the traffic reaches it through the existing
70+
port-80 path. After the challenge succeeds, cert-manager writes the
71+
certificate into the `cassius-jenkins-tls` Secret. The nginx sidecar mounts
72+
that Secret and serves HTTPS.
73+
74+
Certificates renew automatically ~30 days before expiry. No manual
75+
intervention is required after the initial deploy.
76+
77+
## First-time install
78+
79+
```bash
80+
cd .jenkins/k8s/ci-cassandra
81+
./deploy.sh
82+
```
83+
84+
The script:
85+
1. Checks prerequisites (helm, helmfile, helm-diff, kubectl, gcloud)
86+
2. Adds and updates Helm repos
87+
3. Applies the nginx ConfigMap
88+
4. Runs `helmfile apply`, which installs in order:
89+
- `cert-manager` (in the `cert-manager` namespace)
90+
- `cert-issuer` (ClusterIssuer + Certificate in `default`)
91+
- `cassius` (Jenkins in `default`)
92+
93+
Certificate issuance takes 1–3 minutes after deploy completes. Watch it:
94+
```bash
95+
kubectl get certificate cassius-jenkins-tls -n default -w
96+
```
97+
98+
## Upgrading Jenkins
99+
100+
```bash
101+
./deploy.sh --jenkins-only
102+
```
103+
104+
Upgrades only the `cassius` release (skips cert-manager and cert-issuer).
105+
106+
## Dry run
107+
108+
```bash
109+
./deploy.sh --dry-run
110+
```
111+
112+
Shows what would change without applying anything.
113+
114+
## Checking certificate status
115+
116+
```bash
117+
kubectl describe certificate cassius-jenkins-tls -n default
118+
kubectl get certificaterequest -n default
119+
kubectl get challenges -n default # only present during issuance
120+
```
121+
122+
## Rolling back a bad upgrade
123+
124+
```bash
125+
helm history cassius -n default
126+
helm rollback cassius <last-good-revision> -n default --wait --timeout 15m
127+
```
128+
129+
## Customising for a different site
130+
131+
Copy `cert-issuer-values.yaml` and `jenkins-tls-overrides.yaml` and adjust:
132+
- `acmeEmail` and `hostname` in cert-issuer-values.yaml
133+
- `storageClass` in jenkins-tls-overrides.yaml (e.g. `gp2` for AWS)
134+
- Any site-specific annotations (static IP, internal LB, etc.)
135+
136+
Pass your copy as additional `-f` arguments to `helmfile apply` or add it to
137+
the `values:` list in `helmfile.yaml`.
138+
139+
## Node pools
140+
141+
The GKE cluster requires these node pools (created once; see `../README.md`):
142+
143+
| Pool | Label | Machine type | Max nodes |
144+
|------|-------|--------------|-----------|
145+
| default-pool | `cassandra.jenkins.controller=true` | e2-standard-8 | 1 |
146+
| agents-small | `cassandra.jenkins.agent.small=true` | e2-highcpu-8 | 20 |
147+
| agents-report | `cassandra.jenkins.agent.report=true` | n2-standard-8 | 4 |
148+
| agents-medium | `cassandra.jenkins.agent.medium=true` | n2-highcpu-8 | 150 |
149+
| agents-large | `cassandra.jenkins.agent.large=true` | n2-standard-8 | 306 |
150+
151+
The `agents-report` pool is required for the `generateTestReports` pipeline
152+
stage. Without it, that stage will wait indefinitely.
153+
154+
For the current `ci-cassandra.infra.datastax.com` cluster, the `agents-report`
155+
pool does not yet exist and can be created with:
156+
157+
```bash
158+
gcloud container node-pools create agents-report \
159+
--cluster mck-cassandra-jenkins-hamina-2 \
160+
--zone europe-north1-a \
161+
--project cassandra-oss \
162+
--machine-type n2-standard-8 \
163+
--disk-type=pd-ssd \
164+
--disk-size=107 \
165+
--enable-autoscaling \
166+
--spot \
167+
--num-nodes=0 \
168+
--min-nodes=0 \
169+
--max-nodes=4 \
170+
--node-labels=cassandra.jenkins.agent=true,cassandra.jenkins.agent.report=true
171+
```
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
# Values for the local charts/cert-issuer chart.
18+
19+
# The email address Let's Encrypt uses for expiry warnings.
20+
# Change this to the operator's address before deploying.
21+
acmeEmail: "jeremiah1@ibm.com"
22+
23+
# The hostname the TLS certificate is issued for.
24+
# Must match the DNS name resolving to this cluster's load balancer.
25+
hostname: "ci-cassandra.infra.datastax.com"
26+
27+
# Name of the Kubernetes Secret that cert-manager writes the certificate into.
28+
# Referenced by the nginx sidecar in jenkins-tls-overrides.yaml.
29+
secretName: "cassius-jenkins-tls"
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
# Helm values for the jetstack/cert-manager release.
18+
# https://artifacthub.io/packages/helm/cert-manager/cert-manager
19+
20+
# Install the cert-manager CRDs as part of this Helm release so that
21+
# `helm upgrade` keeps them in sync with the chart version.
22+
crds:
23+
enabled: true
24+
25+
# Pin cert-manager to the controller node so the agent pools (which scale to
26+
# zero between builds) do not evict it mid-renewal.
27+
nodeSelector:
28+
cassandra.jenkins.controller: "true"
29+
30+
# Minimal resource requests; cert-manager is idle except during issuance/renewal.
31+
resources:
32+
requests:
33+
cpu: 10m
34+
memory: 64Mi
35+
36+
webhook:
37+
nodeSelector:
38+
cassandra.jenkins.controller: "true"
39+
resources:
40+
requests:
41+
cpu: 10m
42+
memory: 32Mi
43+
44+
cainjector:
45+
nodeSelector:
46+
cassandra.jenkins.controller: "true"
47+
resources:
48+
requests:
49+
cpu: 10m
50+
memory: 64Mi
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
apiVersion: v2
2+
name: cert-issuer
3+
description: |
4+
Minimal chart that creates a Let's Encrypt ClusterIssuer and a Certificate
5+
resource for the ci-cassandra Jenkins instance. It is a thin wrapper so
6+
that the ClusterIssuer and Certificate are managed by Helmfile alongside
7+
cert-manager and Jenkins, rather than applied with ad-hoc kubectl commands.
8+
type: application
9+
version: 0.1.0
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
{{/*
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing, software
13+
distributed under the License is distributed on an "AS IS" BASIS,
14+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
See the License for the specific language governing permissions and
16+
limitations under the License.
17+
*/}}
18+
19+
# ClusterIssuer — tells cert-manager how to talk to Let's Encrypt.
20+
#
21+
# Uses the HTTP-01 challenge: Let's Encrypt GETs
22+
# http://<hostname>/.well-known/acme-challenge/<token>
23+
# The NLB already forwards port 80 to the Jenkins pod on port 8080, so the
24+
# challenge token is served by the temporary ACME solver pod that cert-manager
25+
# starts, reachable through the same path.
26+
#
27+
# The ClusterIssuer is cluster-scoped; the `namespace` field is ignored.
28+
apiVersion: cert-manager.io/v1
29+
kind: ClusterIssuer
30+
metadata:
31+
name: letsencrypt-prod
32+
labels:
33+
app.kubernetes.io/managed-by: Helm
34+
annotations:
35+
meta.helm.sh/release-name: cert-issuer
36+
meta.helm.sh/release-namespace: default
37+
spec:
38+
acme:
39+
server: https://acme-v02.api.letsencrypt.org/directory
40+
email: {{ .Values.acmeEmail | quote }}
41+
# Stores the ACME account private key.
42+
privateKeySecretRef:
43+
name: letsencrypt-prod-account-key
44+
solvers:
45+
- http01:
46+
ingress:
47+
# serviceType ClusterIP keeps the solver pod off the public load
48+
# balancer; the NLB routes the challenge via the existing port-80
49+
# path to the Jenkins Service, which the Kubernetes plugin rewrites
50+
# temporarily during issuance.
51+
serviceType: ClusterIP
52+
---
53+
# Certificate — the actual TLS certificate for the Jenkins hostname.
54+
#
55+
# cert-manager watches this resource, drives the HTTP-01 challenge, and writes
56+
# the resulting certificate + private key into .Values.secretName. It renews
57+
# automatically ~30 days before expiry.
58+
apiVersion: cert-manager.io/v1
59+
kind: Certificate
60+
metadata:
61+
name: cassius-jenkins-tls
62+
namespace: default
63+
labels:
64+
app.kubernetes.io/managed-by: Helm
65+
annotations:
66+
meta.helm.sh/release-name: cert-issuer
67+
meta.helm.sh/release-namespace: default
68+
spec:
69+
secretName: {{ .Values.secretName | quote }}
70+
issuerRef:
71+
name: letsencrypt-prod
72+
kind: ClusterIssuer
73+
dnsNames:
74+
- {{ .Values.hostname | quote }}
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Default values — override via cert-issuer-values.yaml in the parent directory.
2+
acmeEmail: "change-me@example.com"
3+
hostname: "ci-cassandra.example.com"
4+
secretName: "cassius-jenkins-tls"

0 commit comments

Comments
 (0)