Skip to content

Commit 40b9272

Browse files
d47853d47853
committed
Merge branch 'develop'
2 parents 2b69c92 + ac49ae4 commit 40b9272

26 files changed

+90
-42
lines changed

‎README.md

+12
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,22 @@ It also contains Helm Charts so that the following applications can be deployed
1414
* [Gaffer](kubernetes/gaffer/)
1515
* [Example Gaffer Graph containing Road Traffic Dataset](kubernetes/gaffer-road-traffic/)
1616

17+
These charts can be accessed by cloning our repository or by using our Helm repo hosted on our [Github Pages Site](https://gchq.github.io/gaffer-docker)
18+
19+
To use this repo, run:
20+
```bash
21+
helm repo add gaffer https://gchq.github.io/gaffer-docker
22+
```
23+
1724
There are guides on how to deploy the charts on:
1825
* a local Kubernetes cluster, [using kind (Kubernetes IN Docker)](kubernetes/kind-deployment.md)
1926
* an [AWS EKS cluster](kubernetes/aws-eks-deployment.md)
2027

28+
These guides assume you've cloned the repository and are using the Helm charts contained within. We do also host our Helm Charts index on our Github pages site. You can get access to them by running:
29+
```bash
30+
helm repo add gaffer https://gchq.github.io/gaffer-docker
31+
```
32+
2133
## Versioning
2234
Each of our images will be tagged in DockerHub with the version of the software they represent. Every release,
2335
we update the image for that tag and add a new release which has the corresponding git tag.

‎app_version

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
0.6.1
1+
0.6.2

‎cd/deploy.sh

+12-8
Original file line numberDiff line numberDiff line change
@@ -40,11 +40,11 @@ uploadChart() {
4040
chart=$1
4141
version=$2
4242
token=$3
43+
id=$4
4344

4445
helm dependency update "kubernetes/${chart}"
4546
helm package "kubernetes/${chart}"
46-
curl -v -H "Authorization: token $token" -H "Content-Type: application/zip" --data-binary @${chart}-${version}.tgz "https://api.github.com/repos/gchq/gaffer-docker/releases/tag/v${version}/assets"
47-
rm ${chart}-${version}.tgz
47+
curl -v -H "Authorization: token $token" -H "Content-Type: application/zip" --data-binary @${chart}-${version}.tgz "https://uploads.github.com/repos/gchq/gaffer-docker/releases/${id}/assets?name=${chart}-${version}.tgz"
4848
}
4949

5050
# If branch is not master or is pull request, exit
@@ -95,18 +95,22 @@ JSON_DATA="{
9595
\"draft\": false
9696
}"
9797
echo "${JSON_DATA}"
98-
curl -v --data "${JSON_DATA}" -H "Authorization: token ${GITHUB_TOKEN}" "https://api.github.com/repos/gchq/${REPO_NAME}/releases"
98+
curl -v -o output.json --data "${JSON_DATA}" -H "Authorization: token ${GITHUB_TOKEN}" "https://api.github.com/repos/gchq/${REPO_NAME}/releases"
99+
idLine="$(grep '^ "id"' output.json)"
100+
id="$(echo ${idLine} | sed -e 's|^"id": \([0-9]*\).*|\1|')"
101+
rm output.json
99102

100103
# Upload Charts to Github releases
101-
uploadChart hdfs "${APP_VERSION}" "${GITHUB_TOKEN}"
102-
uploadChart gaffer "${APP_VERSION}" "${GITHUB_TOKEN}"
103-
uploadChart gaffer-road-traffic "${APP_VERSION}" "${GITHUB_TOKEN}"
104+
uploadChart hdfs "${APP_VERSION}" "${GITHUB_TOKEN}" "${id}"
105+
uploadChart gaffer "${APP_VERSION}" "${GITHUB_TOKEN}" "${id}"
106+
uploadChart gaffer-road-traffic "${APP_VERSION}" "${GITHUB_TOKEN}" "${id}"
104107

105108
# Update gh-pages
106109
git checkout gh-pages
107-
git merge master -m "Updated docs to latest version"
108-
helm repo index . --url "https://github.com/gchq/gaffer-docker/releases/tag/${TAG_NAME}" --merge index.yaml
110+
helm repo index . --url "https://github.com/gchq/gaffer-docker/releases/download/${TAG_NAME}" --merge index.yaml
111+
rm *.tgz
109112
git commit -am "Updated index.yaml"
113+
git merge master -m "Updated docs to latest version"
110114
git push
111115

112116
# Update version on develop

‎cd/update_app_version.sh

+6-1
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,12 @@ createNewVersion() {
4343

4444
# Performs a Find and replace on the Helm charts and the app_version file
4545
findAndReplace() {
46-
find "$(getRootDirectory)" \( -iname Chart.y*ml -o -name app_version \) -exec sed -i'' -e "s:$1:$2:g" {} +
46+
# Replace the app version file
47+
echo $2 > "$(getRootDirectory)/app_version"
48+
# Replace version marked with # managed version in the Chart.yaml files
49+
managed_version_tag='# managed version'
50+
find "$(getRootDirectory)" -iname Chart.y*ml -exec sed -i'' -e "s:$1 ${managed_version_tag}:$2 ${managed_version_tag}:g" {} +
51+
4752
}
4853

4954
if [ $# -gt 1 ]; then

‎kubernetes/aws-eks-deployment.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ The following instructions will guide you through provisioning and configuring a
1616

1717
If the versions of the container images you would like to deploy are not available in [Docker Hub](https://hub.docker.com/u/gchq) then you will need to host them in a registry yourself.
1818

19-
The following instructions build all the container images and host them in AWS ECR:
19+
The following instructions build all the container images and host them in AWS ECR when run from the ./kubernetes folder:
20+
2021

2122
```bash
2223
export HADOOP_VERSION=${HADOOP_VERSION:-3.2.1}

‎kubernetes/gaffer-road-traffic/Chart.yaml

+3-3
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,13 @@ apiVersion: v2
1616
name: gaffer-road-traffic
1717
description: A Gaffer instance containing sample GB road traffic data from the Department of Transport
1818
type: application
19-
version: 0.6.1
20-
appVersion: 1.12.0
19+
version: 0.6.2 # managed version
20+
appVersion: 1.12.0
2121
home: https://github.com/gchq/Gaffer
2222
sources:
2323
- https://github.com/gchq/gaffer-docker
2424
- https://data.gov.uk/dataset/208c0e7b-353f-4e2d-8b7a-1a7118467acc/gb-road-traffic-counts
2525
dependencies:
2626
- name: gaffer
27-
version: ^0.6.1
27+
version: ^0.6.2 # managed version
2828
repository: file://../gaffer/

‎kubernetes/gaffer-road-traffic/docs/aws-eks-deployment.md

+7-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
# Deploying Road Traffic Gaffer Graph on AWS EKS
2+
All scripts listed here are intended to be run from the kubernetes/gaffer-road-traffic folder
23

34
First follow the [instructions here](../../aws-eks-deployment.md) to provision and configure an [AWS EKS](https://aws.amazon.com/eks/) cluster that the Gaffer Road Traffic Helm Chart can be deployed on.
45

6+
## Using ECR
57
If you are hosting the container images in your AWS account, using ECR, then run the following commands to configure the Helm Chart to use them:
68

79
```bash
@@ -23,7 +25,11 @@ else
2325
fi
2426
```
2527

26-
Deploy the Helm Chart with:
28+
## Deploy Helm Chart
29+
30+
The last thing before deploying is to set the passwords for the various accumulo users in the values.yaml file. These are found under `accumulo.config.userManagement`.
31+
32+
Finally, deploy the Helm Chart by running this from the kubernetes/gaffer-road-traffic folder:
2733

2834
```
2935
export HADOOP_VERSION=${HADOOP_VERSION:-3.2.1}

‎kubernetes/gaffer-road-traffic/docs/kind-deployment.md

+4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,11 @@
11
# Deploying Road Traffic Gaffer Graph using kind
2+
All scripts listed here are intended to be run from the kubernetes/gaffer-road-traffic folder
23

34
First follow the [instructions here](../../kind-deployment.md) to provision and configure a local Kubernetes cluster, using [kind](https://kind.sigs.k8s.io/) (Kubernetes IN Docker), that the Gaffer Road Traffic Helm Chart can be deployed on.
45

6+
After the cluster is provisioned, update the values.yaml with the passwords for the various accumulo users. These are found under `accumulo.config.userManagement`.
7+
8+
Once that's done, run this to deploy and test the Road Traffic Graph.
59
```bash
610
export HADOOP_VERSION=${HADOOP_VERSION:-3.2.1}
711
export GAFFER_VERSION=${GAFFER_VERSION:-1.12.0}

‎kubernetes/gaffer-road-traffic/templates/check-data-loaded-hook.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ metadata:
2525
app.kubernetes.io/component: hook
2626
annotations:
2727
helm.sh/hook: post-install,post-upgrade
28+
helm.sh/hook-delete-policy: hook-succeeded
2829
helm.sh/hook-weight: "40"
2930
spec:
3031
containers:

‎kubernetes/gaffer-road-traffic/templates/data-loader-job.yaml

+3-6
Original file line numberDiff line numberDiff line change
@@ -36,12 +36,9 @@ spec:
3636
- name: graph-config
3737
mountPath: /gaffer/graph/
3838
readOnly: true
39-
# Schema isn't mounted because of GAFFER-2262
40-
# Can be un-commented once it has been resolved and released
41-
# In the mean time use the schema built into the container
42-
# - name: schema
43-
# mountPath: /gaffer/schema/
44-
# readOnly: true
39+
- name: schema
40+
mountPath: /gaffer/schema/
41+
readOnly: true
4542
- name: store-properties
4643
mountPath: /gaffer/store/
4744
readOnly: true

‎kubernetes/gaffer-road-traffic/templates/tests/integration/hdfs/hdfs-test.yaml

+4-6
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ metadata:
2525
app.kubernetes.io/component: test
2626
annotations:
2727
helm.sh/hook: test-success
28+
helm.sh/hook-delete-policy: hook-succeeded
2829
spec:
2930
restartPolicy: Never
3031
initContainers:
@@ -87,12 +88,9 @@ spec:
8788
- name: store-properties
8889
mountPath: /gaffer/store/
8990
readOnly: true
90-
# Schema isn't mounted because of GAFFER-2262
91-
# Can be un-commented once it has been resolved and released
92-
# In the meantime use the schema built into the container
93-
# - name: schema
94-
# mountPath: /gaffer/schema/
95-
# readOnly: true
91+
- name: schema
92+
mountPath: /gaffer/schema/
93+
readOnly: true
9694
containers:
9795
- name: hdfs-test
9896
image: {{ .Values.gaffer.testImages.curl.repository }}:{{ .Values.gaffer.testImages.curl.tag }}

‎kubernetes/gaffer-road-traffic/templates/tests/test-queries.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ metadata:
2121
app.kubernetes.io/component: test
2222
annotations:
2323
helm.sh/hook: test-success
24+
helm.sh/hook-delete-policy: hook-succeeded
2425
spec:
2526
restartPolicy: Never
2627
containers:

‎kubernetes/gaffer/Chart.yaml

+2-2
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ apiVersion: v2
1616
name: gaffer
1717
description: A large-scale entity and relation database supporting aggregation of properties
1818
type: application
19-
version: 0.6.1
19+
version: 0.6.2 # managed version
2020
appVersion: 1.12.0
2121
home: https://github.com/gchq/Gaffer
2222
sources:
@@ -27,6 +27,6 @@ dependencies:
2727
repository: https://kubernetes-charts-incubator.storage.googleapis.com
2828
condition: zookeeper.enabled
2929
- name: hdfs
30-
version: ^0.6.1
30+
version: ^0.6.2 # managed version
3131
repository: file://../hdfs/
3232
condition: hdfs.enabled

‎kubernetes/gaffer/docs/aws-eks-deployment.md

+5-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
# Deploying Gaffer on AWS EKS
2+
All the scripts found here are designed to be run from the kubernetes/gaffer folder.
23

34
First follow the [instructions here](../../aws-eks-deployment.md) to provision and configure an [AWS EKS](https://aws.amazon.com/eks/) cluster that the Gaffer Helm Chart can be deployed on.
45

6+
## Using ECR
57
If you are hosting the container images in your AWS account, using ECR, then run the following commands to configure the Helm Chart to use them:
68

79
```bash
@@ -22,8 +24,10 @@ else
2224
fi
2325
```
2426

25-
Deploy the Helm Chart with:
27+
## Deploy Helm Chart
28+
Next you'll need to setup the passwords for the accumulo users in the values.yaml file. These are found under `accumulo.config.userManagement`.
2629

30+
Finally, deploy the Helm Chart:
2731
```bash
2832
export HADOOP_VERSION=${HADOOP_VERSION:-3.2.1}
2933
export GAFFER_VERSION=${GAFFER_VERSION:-1.12.0}

‎kubernetes/gaffer/docs/kind-deployment.md

+5
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,12 @@
11
# Deploying Gaffer using kind
2+
All the scripts found here are designed to be run from the kubernetes/gaffer folder.
23

34
First follow the [instructions here](../../kind-deployment.md) to provision and configure a local Kubernetes cluster, using [kind](https://kind.sigs.k8s.io/) (Kubernetes IN Docker), that the Gaffer Helm Chart can be deployed on.
45

6+
Then update the Accumulo users' passwords in the values.yaml file. These are found under `accumulo.config.userManagement`.
7+
8+
Gaffer is then ready be deployed:
9+
510
```bash
611
export HADOOP_VERSION=${HADOOP_VERSION:-3.2.1}
712
export GAFFER_VERSION=${GAFFER_VERSION:-1.12.0}

‎kubernetes/gaffer/templates/accumulo/post-install-cmds/hook.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ metadata:
2222
name: {{ include "gaffer.fullname" . }}-accumulo-post-install-cmds
2323
annotations:
2424
helm.sh/hook: post-install,post-upgrade
25+
helm.sh/hook-delete-policy: hook-succeeded
2526
helm.sh/hook-weight: "20"
2627
labels:
2728
{{- include "gaffer.labels" . | nindent 4 }}

‎kubernetes/gaffer/templates/graph-status-check-hook.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ metadata:
2525
app.kubernetes.io/component: hook
2626
annotations:
2727
helm.sh/hook: post-install,post-upgrade
28+
helm.sh/hook-delete-policy: hook-succeeded
2829
helm.sh/hook-weight: "30"
2930
spec:
3031
containers:

‎kubernetes/gaffer/templates/graph-upgrade-hook.yaml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ metadata:
2121
app.kubernetes.io/component: hook
2222
annotations:
2323
helm.sh/hook: post-upgrade
24+
helm.sh/hook-delete-policy: hook-succeeded
2425
helm.sh/hook-weight: "35"
2526
spec:
2627
restartPolicy: Never
@@ -31,8 +32,7 @@ spec:
3132
args:
3233
- uk.gov.gchq.gaffer.accumulostore.utils.AddUpdateTableIterator
3334
- {{ .Values.graph.config.graphId }}
34-
# Argument for schema directory should be /gaffer/schema/, but this is a workaround for GAFFER-2262
35-
- /gaffer/schema/..data/
35+
- /gaffer/schema/
3636
- /gaffer/store/store.properties
3737
- update
3838
env:

‎kubernetes/gaffer/templates/tests/test-graph-status.yaml

+2-1
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@ metadata:
2020
{{- include "gaffer.labels" . | nindent 4 }}
2121
app.kubernetes.io/component: test
2222
annotations:
23-
"helm.sh/hook": test-success
23+
helm.sh/hook: test-success
24+
helm.sh/hook-delete-policy: hook-succeeded
2425
spec:
2526
containers:
2627
- name: curl

‎kubernetes/hdfs/Chart.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,5 +16,5 @@ apiVersion: v2
1616
name: hdfs
1717
description: Deploys the Hadoop Distributed File System (HDFS)
1818
type: application
19-
version: 0.6.1
19+
version: 0.6.2 # managed version
2020
appVersion: 3.2.1

‎kubernetes/hdfs/docs/aws-eks-deployment.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
# Deploying HDFS on AWS EKS
2+
All scripts listed here are intended to be run from the kubernetes/hdfs folder
23

34
First follow the [instructions here](../../aws-eks-deployment.md) to provision and configure an [AWS EKS](https://aws.amazon.com/eks/) cluster that the HDFS Helm Chart can be deployed on.
45

6+
## Using ECR
57
If you are hosting the container images in your AWS account, using ECR, then run the following commands to configure the Helm Chart to use them:
68

79
```bash
@@ -20,7 +22,7 @@ else
2022
fi
2123
```
2224

23-
Deploy the Helm Chart with:
25+
## Deploy Helm chart
2426

2527
```bash
2628
export HADOOP_VERSION=${HADOOP_VERSION:-3.2.1}

‎kubernetes/hdfs/docs/kind-deployment.md

+6-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
# Deploying HDFS using kind
2+
All scripts listed here are intended to be run from the kubernetes/hdfs folder
23

34
First follow the [instructions here](../../kind-deployment.md) to provision and configure a local Kubernetes cluster, using [kind](https://kind.sigs.k8s.io/) (Kubernetes IN Docker), that the HDFS Helm Chart can be deployed on.
45

6+
## Deploying Helm charts
7+
58
```bash
69
export HADOOP_VERSION=${HADOOP_VERSION:-3.2.1}
710

@@ -13,7 +16,6 @@ helm install hdfs . \
1316
helm test hdfs
1417
```
1518

16-
1719
## Accessing Web UI (via `kubectl port-forward`)
1820

1921
```
@@ -26,17 +28,17 @@ Then browse to: http://localhost:9870
2628
## Accessing Web UI (via [Nginx Ingress Controller](https://github.com/kubernetes/ingress-nginx))
2729

2830
Register the FQDNs for each component in DNS e.g.
29-
```
31+
```bash
3032
echo "127.0.0.1 hdfs.k8s.local" | sudo tee -a /etc/hosts
3133
```
3234

3335
Update the HDFS deployment to route ingress based on FQDNs:
34-
```
36+
```bash
3537
helm upgrade hdfs . -f ./values-host-based-ingress.yaml
3638
```
3739

3840
Set up port forwarding to the nginx ingress controller:
39-
```
41+
```bash
4042
sudo KUBECONFIG=$HOME/.kube/config kubectl port-forward -n ingress-nginx svc/ingress-nginx 80:80
4143
```
4244

‎kubernetes/hdfs/templates/post-install-cmds.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ metadata:
2222
name: {{ template "hdfs.fullname" . }}-post-install-cmds
2323
annotations:
2424
helm.sh/hook: post-install,post-upgrade
25+
helm.sh/hook-delete-policy: hook-succeeded
2526
helm.sh/hook-weight: "10"
2627
labels:
2728
{{- include "hdfs.labels" . | nindent 4 }}

‎kubernetes/hdfs/templates/tests/auth-test.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ metadata:
1818
name: {{ template "hdfs.fullname" . }}-auth-test
1919
annotations:
2020
helm.sh/hook: test
21+
helm.sh/hook-delete-policy: hook-succeeded
2122
labels:
2223
{{- include "hdfs.labels" . | nindent 4 }}
2324
app.kubernetes.io/component: test

‎kubernetes/hdfs/templates/tests/webhdfs-test.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ metadata:
1818
name: {{ template "hdfs.fullname" . }}-webhdfs-test
1919
annotations:
2020
helm.sh/hook: test
21+
helm.sh/hook-delete-policy: hook-succeeded
2122
labels:
2223
{{- include "hdfs.labels" . | nindent 4 }}
2324
app.kubernetes.io/component: test

0 commit comments

Comments
 (0)