Project 4

This is how we deployed the current custos environment. We were tasked to deploy the custos system on Jetstream 2, after a long process of trying and a lot of painstaking networking issues, we came to the conclusion by talking to Isuru, that the whole system is tested to be deployed on Jetstream 1. So we switched to that. We were able to deploy CUSTOS.

The issues that we faced on Jetstream 1:

• Dependency related to Kubernetes version
• Dependency on internal Kubernetes network eg: Calico
• Dependency on ingress
• Configuration problems
• We tried 7 different clusters with the combination of Rancher and BareMetal Kubernetes.

Custos Deployment on Jetstream 1

• Create two instances on Jetstream 1 with medium size
• Install Rancher on one of the instances.

For Rancher installation, please refer another of the team's link

After that, you need to install K8s cluster using this link

Once Rancher and cluster is done, we also need to:

• Create namespaces: Custos, Keycloak, Vault
• Install helm

https://github.com/airavata-courses/CoDist.git

git checkout project-4

cd CUSTOS/custos_deploy/

For all the master and worker nodes, we need to:

sudo mkdir /bitnami sudo mkdir /bitnami/mysql sudo mkdir /bitnami/postgresql sudo mkdir /hashicorp sudo mkdir /hashicorp/consul sudo mkdir /hashicorp/consul/data

chmod 777 -R /hashicorp

For every service, we need to go into that particular service before executing the steps

Cert-Manager Deployment

• cd cert-manager

• kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

• kubectl get all -n cert-manager

• Create ClusterIssuer

  • kubectl apply -f issuer.yaml

Keycloak Deployment

• cd postgres

Postgresql Deployment

helm repo add bitnami https://charts.bitnami.com/bitnami

Create PVs

Create three PVs for each mount point /bitnami/postgresql

• kubectl apply -f pv.yaml,pv1.yaml,pv2.yaml

Then deploy postgresql

• helm install keycloak-db-postgresql bitnami/postgresql -f values.yaml -n keycloak --version 10.12.3

• cd ..

• kubectl create -f https://raw.githubusercontent.com/operator-framework/operator-lifecycle-manager/master/deploy/upstream/quickstart/crds.yaml

• kubectl create -f https://raw.githubusercontent.com/operator-framework/operator-lifecycle-manager/master/deploy/upstream/quickstart/olm.yaml

• git clone https://github.com/keycloak/keycloak-operator

• cp operator.yaml keycloak-operator/deploy/

• cd keycloak-operator

• make cluster/prepare

• kubectl apply -f deploy/operator.yaml -n keycloak

• cd ..

• kubectl apply -f keycloak-db-secret.yaml -n keycloak

• kubectl apply -f custos-keycloak.yaml -n keycloak

Replace hostname in ingress.yaml

• kubectl apply -f ingress.yaml -n keycloak

user: admin

Get admin password.

• kubectl get secret credential-custos-keycloak -o yaml -n keycloak

• echo "passwordhere" | base64 --decode

Consul Deployment

• helm repo add hashicorp https://helm.releases.hashicorp.com

Create directory /hashicorp/consul/data in each of your nodes,

• sudo chmod 777 -R hashichorp

kubectl apply -f pv.yaml,pv1.yaml
kubectl apply -f storage.yaml
helm install consul hashicorp/consul --version 0.31.1 -n vault --values config.yaml

Vault Deployment

helm install vault hashicorp/vault --namespace vault -f values.yaml --version 0.10.0

Change hostname in ingress.yaml

Deploy Ingress

kubectl apply -f ingress.yaml -n vault

Follow instructions in UI which is hosted on 443 to generate vault token.

MySQL Deployment

cd mysql
kubectl apply -f pv.yaml,pv1.yaml
helm install mysql bitnami/mysql -f values.yaml -n custos --version 8.8.8

Custos Deployment

These are the pods for the deployment.

The Master Domain: https://js-169-57.jetstream-cloud.org

You can send a POST request: https://js-169-57.jetstream-cloud.org:32649/tenant-management/v1.0.0/oauth2/tenant with the body

{
    "client_name":"test1",
    "requester_email":"[email protected]",
    "admin_username":"test",
    "admin_first_name":"test",
    "admin_last_name":"test",
    "admin_email":"[email protected]",
    "contacts":["[email protected]","[email protected]"],
    "redirect_uris":["http://localhost:8080/callback*",
    "https://js-169-57.jetstream-cloud.org/callback*"],
    "scope":"openid profile email org.cilogon.userinfo",
    "domain":"js-169-57.jetstream-cloud.org",
    "admin_password":"test",
    "client_uri":"https://js-169-57.jetstream-cloud.org/",
    "logo_uri":"https://js-169-57.jetstream-cloud.org/",
    "application_type":"web",
    "comment":"Custos super tenant for production"
}

You should get a response:

{
    "client_id": "custos-wf6j89o3av92g578bmnu-10000001",
    "client_secret": "ONm71cEMHzZ3N85KV0S1mJSAS8Ae4wNpUPIw0SWT",
    "is_activated": false,
    "client_id_issued_at": 1651858886000,
    "client_secret_expires_at": 0,
    "registration_client_uri": "https://custos.scigap.org/apiserver/tenant-management/v1.0.0/oauth2/tenant?client_id=custos-wf6j89o3av92g578bmnu-10000001",
    "token_endpoint_auth_method": "client_secret_basic",
    "msg": "Use Base64 encoded clientId:clientSecret as auth token for authorization, Credentials are activated after admin approval"
}