Skip to content
This repository was archived by the owner on Oct 6, 2023. It is now read-only.

Commit 26b1537

Browse files
gvidonMichał Sieczkowski
gvidon
and
Michał Sieczkowski
authored
Add last used cluster deployment config files (#316)
* New staging domain * Keep sensitive data in secrets * Run deploy-start sequence at container startup * Revert to stable image * Add production deployment manifests * Add chain spec deployment Job and Secret * Network policy for Hubble Commander service * Hubble Commander is not going to be scaled * Grab the env vars required for runnign chain spec job * Use the apiVersion which is up to date * Update chain spec file content * Apply the latest chain spec file content * Remove unneeded HUBBLE_BOOTSTRAP_NODE_URL from config maps * Allow payments service accessing hubble directly * Fix HUBBLE_ROLLUP_MIN_TXS_PER_COMMITMENT env var * Add the volume and affinity params * Move k8s deployment config files to cluster/ subdir Co-authored-by: Michał Sieczkowski <michal@ethworks.io>
1 parent ec31d7a commit 26b1537

24 files changed

Lines changed: 511 additions & 35 deletions

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,7 @@ config.yaml
99
e2e/geth-data/geth
1010
chain-spec.yaml
1111
backups/
12+
13+
# Kubernetes
14+
kustomization.yaml
15+
secrets.yaml

‎deployment/cluster/production/chain-spec/job.yaml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
apiVersion: batch/v1
2+
kind: Job
3+
metadata:
4+
name: chain-spec
5+
namespace: production
6+
spec:
7+
template:
8+
spec:
9+
imagePullSecrets:
10+
- name: regcred
11+
volumes:
12+
- name: chain-spec
13+
emptyDir:
14+
sizeLimit: 1Gi
15+
containers:
16+
- name: chain-spec
17+
image: "ghcr.io/worldcoin/hubble-commander@sha256:f65594ef5814288b2a3f188ca821edd113e3743bb0224c8f5386d25a4ecb1f72"
18+
command:
19+
- /bin/sh
20+
- -c
21+
- "/go/src/app/build/hubble deploy -file /chain-spec/chain-spec.yaml; cat /chain-spec/chain-spec.yaml | base64"
22+
volumeMounts:
23+
- name: chain-spec
24+
mountPath: /chain-spec
25+
envFrom:
26+
- configMapRef:
27+
name: hubble-commander
28+
- configMapRef:
29+
name: primary-hubble-commander
30+
- secretRef:
31+
name: hubble-commander
32+
- secretRef:
33+
name: primary-hubble-commander
34+
restartPolicy: Never
35+
backoffLimit: 0

‎deployment/cluster/production/chain-spec/secret.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
apiVersion: v1
2+
kind: Secret
3+
metadata:
4+
name: chain-spec
5+
namespace: production
6+
type: Opaque
7+
data:
8+
# Placeholder value
9+
chain-spec.yaml: |
10+
Y2hhaW5faWQ6ICIxMzM3IgphY2NvdW50X3JlZ2lzdHJ5OiAweGNjYTQzOTFjZjI1ZDk3NzNiNDgz
11+
MWUyNDBiMWRkZjkzMTdiMDk0Y2QKZGVwbG95bWVudF9ibG9jazogNDEyMgpyb2xsdXA6IDB4NzEw
12+
NzI1YzBkOWZjZWE2N2RlN2JhZWIwMzUzNzdkMDJlMTljNDJkYgpnZW5lc2lzX2FjY291bnRzOgot
13+
IHB1YmxpY19rZXk6IDB4MGRmNjhjYjg3ODU2MjI5YjBiYzNmMTU4ZmZmOGI4MmIwNGRlYjFhNGMy
14+
M2RhZGJmM2VkMmRhNGVjNmY2ZWZjYjFjMTY1YzZiNDdkOGM4OWFiMmRkYjA4MzFjMTgyMjM3YjI3
15+
YTRiM2Q5NzAxNzc1YWQ2YzE4MDMwM2Y4N2VmMjYwNTY2Y2IyZjBiY2M3Yjg5YzIyNjBkZTJmZWU4
16+
ZWMyOWQ3YjVlNTc1YTFlMzZlYjRiY2VhZDUyYTc0YTUxMWI3MTg4ZDdkZjdjOWQwOGY5NGI5ZGFh
17+
OWQ4OTEwNWZiZGYyMmJmMTRlMzBiODRmOGFkZWZiMzY5NWViZmYwMGU4OAogIHB1Yl9rZXlfaWQ6
18+
IDAKICBzdGF0ZV9pZDogMAogIGJhbGFuY2U6ICIxMDAwMDAwMDAwMDAwMDAwMDAwIgotIHB1Ymxp
19+
Y19rZXk6IDB4MDA5N2Y0NjVmZTgyN2NlNGRhZDc1MTk4OGY2Y2U1ZWM3NDc0NTgwNzU5OTIxODBj
20+
YTExYjA3NzZiOWVhM2E5MTBjM2VlNGRjYTRhMDNkMDZjMzg2Mzc3OGFmZmU5MWNlMzhkNTAyMTM4
21+
MzU2YTM1YWUxMjY5NWM1NjViMjRlYTYxNTFiODNlYWJkNDFhNjA5MGI4YWMzYmIyNWUxNzNjODRj
22+
M2IwODBhNTU0NTI2MGIxMzI3NDk1OTIwYzM0MmMwMmQ1MWNhYzQ0MTgyMjhkYjFhM2Q5OGFhMTJl
23+
NmZkN2IzMjY3YzcwMzQ3NWY1OTk5YjJlYzdhMTk3YWQ3ZDhiYwogIHB1Yl9rZXlfaWQ6IDEKICBz
24+
dGF0ZV9pZDogMQogIGJhbGFuY2U6ICIxMDAwMDAwMDAwMDAwMDAwMDAwIgotIHB1YmxpY19rZXk6
25+
IDB4MWNjZjE5ODcxMzIwYjdlODUwNDc1ODQ1ZDg3OWE5Zjk3MTdhNmM5Njk0ZmFiMTk0OThlNDI2
26+
MWI0NDJkZTRlMDExNDA2YmRjOTY3OTg0NzcxNTA4YTJlNTBkNzc0ZjQ5ZGIzNmJmNWIwNGIxNWY5
27+
ZjQxMWI4Yzg3MzNmZTBkOGUzMDFmOGYyZTlhYTk4ZjdkZGU3ZGUzNjM1YmFhMjE2ZmRjOTY5ZTc1
28+
MmY0ZWY2NDZmZDVmODFkODllNDZkMzk4MDRjMGFjOTJjN2VhNGNjNTk1N2I0MjE0ZWY0MWEwYWE0
29+
ZjFhNmYzNDNjZWJmYjU3N2U5ZGNhZjhmZjI1NTFkNQogIHB1Yl9rZXlfaWQ6IDIKICBzdGF0ZV9p
30+
ZDogMgogIGJhbGFuY2U6ICIxMDAwMDAwMDAwMDAwMDAwMDAwIgotIHB1YmxpY19rZXk6IDB4MDA5
31+
N2Y0NjVmZTgyN2NlNGRhZDc1MTk4OGY2Y2U1ZWM3NDc0NTgwNzU5OTIxODBjYTExYjA3NzZiOWVh
32+
M2E5MTBjM2VlNGRjYTRhMDNkMDZjMzg2Mzc3OGFmZmU5MWNlMzhkNTAyMTM4MzU2YTM1YWUxMjY5
33+
NWM1NjViMjRlYTYxNTFiODNlYWJkNDFhNjA5MGI4YWMzYmIyNWUxNzNjODRjM2IwODBhNTU0NTI2
34+
MGIxMzI3NDk1OTIwYzM0MmMwMmQ1MWNhYzQ0MTgyMjhkYjFhM2Q5OGFhMTJlNmZkN2IzMjY3Yzcw
35+
MzQ3NWY1OTk5YjJlYzdhMTk3YWQ3ZDhiYwogIHB1Yl9rZXlfaWQ6IDMKICBzdGF0ZV9pZDogMwog
36+
IGJhbGFuY2U6ICIxMDAwMDAwMDAwMDAwMDAwMDAwIgotIHB1YmxpY19rZXk6IDB4MjAwNzAwMzE3
37+
MmI1NDUzYzU0NjhkYzQxODFjMWZiMDU4ZjFhZGVjMjJmNWU5NTVmNTliMjVhY2RjMTgyMDdjNDA2
38+
ZjYxNWVjNjkzYWExYzkzYzdjNmNkMGRlMGZiM2E5YTRlZTY4Y2Q0YzA3OTFkZDdiMWMyMWFhNjE2
39+
MThiZjcxNTljNjEyYmRiZDA0YThmMjhlNDQwMDI2ZjgwYzFmYjdlZmM3ZWRhYWQ0NDNiYWU2NDNi
40+
YjcwMTU0ZTExZjBmMWRmOTI5ZWVmY2NmYmVkZGI0MzI5MDY2M2YxYTJiMzY3Y2ZiMGJjODZlOWRm
41+
MWU4NDlhNWYxOWY5NzA5ZjhiNwogIHB1Yl9rZXlfaWQ6IDQKICBzdGF0ZV9pZDogNAogIGJhbGFu
42+
Y2U6ICIxMDAwMDAwMDAwMDAwMDAwMDAwIgotIHB1YmxpY19rZXk6IDB4MDIyNjk5ZjAzYzFjOWZk
43+
ZGQ3Y2MyOWU1ZTNjODM3YzFkY2ZjYjQwMmVkOWI0N2Q0M2JkOTcwMjMyMTY2MmQ4NjBjYTIwODFk
44+
MDQ2YTAxZTg1MmI2ZTRlY2Q4NjBlZGY4NWUwZTBjMWFlZjVlNjJhZjM0Yzg1ZjdjZTIyMWJjMzEx
45+
NzlkZTk1YzQxNjIxMTFhNjA1ZDA5YWFhNjZhNjNhN2E1MDJjOTA0ZDQyZDc2NzVkYWQ1YjQ4MzI4
46+
YjhjY2RjMWExMGM2YmNmNzc0ZTdmNWExMmRlYWIxMWM0ODhlMTlmZDNmMzk5NWY3ZDljMjA5MGZm
47+
YWI4OGRlYjhiNzFmNwogIHB1Yl9rZXlfaWQ6IDUKICBzdGF0ZV9pZDogNQogIGJhbGFuY2U6ICIx
48+
MDAwMDAwMDAwMDAwMDAwMDAwIgo=

‎deployment/cluster/production/geth/deployment.yaml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
labels:
5+
app: geth
6+
name: geth
7+
namespace: production
8+
spec:
9+
replicas: 1
10+
selector:
11+
matchLabels:
12+
app: geth
13+
template:
14+
metadata:
15+
labels:
16+
app: geth
17+
admission.datadoghq.com/enabled: "true"
18+
annotations:
19+
ad.datadoghq.com/geth.logs: '[{"source":"go","service":"geth"}]'
20+
ad.datadoghq.com/tags: '{"env":"production"}'
21+
spec:
22+
containers:
23+
- name: geth
24+
image: "ethereum/client-go"
25+
command: ["/bin/sh"]
26+
27+
args: [
28+
"-c",
29+
30+
"echo \"Creating the cipher file...\" &&
31+
mkdir -p /root/ethereum/keystore &&
32+
echo '{\"address\":\"9f758331b439c1b664e86f2050f2360370f06849\",\"crypto\":{\"cipher\":\"aes-128-ctr\",\"ciphertext\":\"e1860e27080f7c6e7353cd132ae498b889d045849aa3f224d470d5d97ea09fc4\",\"cipherparams\":{\"iv\":\"3626816e04320eaa4bd4a7626787e839\"},\"kdf\":\"scrypt\",\"kdfparams\":{\"dklen\":32,\"n\":262144,\"p\":1,\"r\":8,\"salt\":\"34636cdd63c05f88c20632b5bdc312896125ec35e1b572a9c3276f75cab2408e\"},\"mac\":\"0a7edb666e831f892b06791da54c8d9ee61cbf0f99ab2ec03b6d812ee0ef334e\"},\"id\":\"db483d8e-2d98-490b-997e-26ed52ee6365\",\"version\":3}' | cat > /root/ethereum/keystore/UTC--2021-06-07T12-34-01.339648000Z--9f758331b439c1b664e86f2050f2360370f06849 &&
33+
geth --datadir=/root/ethereum --dev --dev.period=5 --http --http.addr=0.0.0.0 --ws --ws.addr=0.0.0.0",
34+
]
35+
36+
imagePullPolicy: Always
37+
ports:
38+
- containerPort: 8545
39+
- containerPort: 8546
40+
resources:
41+
requests:
42+
memory: "124Mi"
43+
cpu: "250m"
44+
limits:
45+
memory: "6144Mi"
46+
cpu: "1500m"

‎deployment/cluster/production/geth/service.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
kind: Service
2+
apiVersion: v1
3+
metadata:
4+
labels:
5+
app: geth
6+
name: geth
7+
namespace: production
8+
spec:
9+
type: ClusterIP
10+
selector:
11+
app: geth
12+
ports:
13+
- port: 8546
14+
targetPort: 8546

‎deployment/cluster/production/hubble-commander/config-map.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
labels:
5+
app: hubble-commander
6+
name: hubble-commander
7+
namespace: production
8+
data:
9+
HUBBLE_ETHEREUM_RPC_URL: ws://geth:8546
10+
HUBBLE_ETHEREUM_CHAIN_ID: "1337"
11+
HUBBLE_POSTGRES_HOST: consumer-app-crypto-db-production.cluster-c1amkrvabwaw.us-east-1.rds.amazonaws.com
12+
HUBBLE_POSTGRES_PORT: "5432"
13+
HUBBLE_BOOTSTRAP_CHAIN_SPEC_PATH: "/chain-spec/chain-spec.yaml"
14+
HUBBLE_ROLLUP_MIN_TXS_PER_COMMITMENT: "1"
15+
HUBBLE_ROLLUP_MIN_COMMITMENTS_PER_BATCH: "1"
16+
HUBBLE_LOG_LEVEL: "debug"
17+
HUBBLE_LOG_FORMAT: "json"

deployment/staging/hubble-commander/deployment.yaml renamed to deployment/cluster/production/hubble-commander/deployment.yaml

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ metadata:
44
labels:
55
app: hubble-commander
66
name: hubble-commander
7-
namespace: staging
7+
namespace: production
88
spec:
99
replicas: 1
1010
selector:
@@ -17,21 +17,35 @@ spec:
1717
admission.datadoghq.com/enabled: "true"
1818
annotations:
1919
ad.datadoghq.com/hubble-commander.logs: '[{"source":"go","service":"hubble-commander"}]'
20-
ad.datadoghq.com/tags: '{"env":"staging"}'
20+
ad.datadoghq.com/tags: '{"env":"production"}'
2121
spec:
2222
imagePullSecrets:
2323
- name: regcred
24+
volumes:
25+
- name: chain-spec-yaml
26+
secret:
27+
secretName: chain-spec
28+
items:
29+
- key: chain-spec.yaml
30+
path: chain-spec.yaml
2431
containers:
2532
- name: primary-hubble-commander
26-
image: "ghcr.io/worldcoin/hubble-commander:stable"
33+
image: "ghcr.io/worldcoin/hubble-commander@sha256:f65594ef5814288b2a3f188ca821edd113e3743bb0224c8f5386d25a4ecb1f72"
2734
imagePullPolicy: Always
35+
volumeMounts:
36+
- name: chain-spec-yaml
37+
mountPath: /chain-spec
2838
ports:
2939
- containerPort: 8080
3040
envFrom:
3141
- configMapRef:
3242
name: hubble-commander
3343
- configMapRef:
3444
name: primary-hubble-commander
45+
- secretRef:
46+
name: hubble-commander
47+
- secretRef:
48+
name: primary-hubble-commander
3549
livenessProbe:
3650
exec:
3751
command:
@@ -48,13 +62,20 @@ spec:
4862
memory: "6144Mi"
4963
cpu: "1500m"
5064
- name: secondary-hubble-commander
51-
image: "ghcr.io/worldcoin/hubble-commander:stable"
65+
image: "ghcr.io/worldcoin/hubble-commander@sha256:f65594ef5814288b2a3f188ca821edd113e3743bb0224c8f5386d25a4ecb1f72"
5266
imagePullPolicy: Always
67+
volumeMounts:
68+
- name: chain-spec-yaml
69+
mountPath: /chain-spec
5370
envFrom:
5471
- configMapRef:
5572
name: hubble-commander
5673
- configMapRef:
5774
name: secondary-hubble-commander
75+
- secretRef:
76+
name: hubble-commander
77+
- secretRef:
78+
name: secondary-hubble-commander
5879
livenessProbe:
5980
exec:
6081
command:

‎deployment/cluster/production/hubble-commander/ingress.yaml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
apiVersion: networking.k8s.io/v1beta1
2+
kind: Ingress
3+
metadata:
4+
name: hubble-commander
5+
labels:
6+
app: hubble-commander
7+
namespace: production
8+
annotations:
9+
cert-manager.io/cluster-issuer: "letsencrypt-production"
10+
nginx.ingress.kubernetes.io/ssl-redirect: "true"
11+
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
12+
nginx.ingress.kubernetes.io/rewrite-target: /
13+
spec:
14+
tls:
15+
- hosts:
16+
- production.api.worldcoin-distributors.com
17+
secretName: hubble-commander-secret
18+
rules:
19+
- host: production.api.worldcoin-distributors.com
20+
http:
21+
paths:
22+
- path: /hubble/primary(/|$)
23+
pathType: Exact
24+
backend:
25+
serviceName: hubble-commander
26+
servicePort: 80
27+
- path: /hubble/secondary(/|$)
28+
pathType: Exact
29+
backend:
30+
serviceName: hubble-commander
31+
servicePort: 81

‎deployment/cluster/production/hubble-commander/network-policy.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
apiVersion: networking.k8s.io/v1
2+
kind: NetworkPolicy
3+
metadata:
4+
labels:
5+
app: hubble-commander
6+
name: hubble-commander
7+
namespace: production
8+
spec:
9+
podSelector:
10+
matchLabels:
11+
app: hubble-commander
12+
policyTypes:
13+
- Ingress
14+
ingress:
15+
- from:
16+
- namespaceSelector:
17+
matchLabels:
18+
app.kubernetes.io/name: ingress-nginx
19+
- from:
20+
- podSelector:
21+
matchLabels:
22+
app: crypto-sync-transactions
23+
- podSelector:
24+
matchLabels:
25+
app: payments
26+
ports:
27+
- protocol: TCP
28+
port: 8080

‎deployment/cluster/production/hubble-commander/primary-config-map.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
labels:
5+
app: hubble-commander
6+
name: primary-hubble-commander
7+
namespace: production
8+
data:
9+
HUBBLE_POSTGRES_NAME: "primary-hubble-commander"

0 commit comments

Comments
 (0)