Overview

This repostiory is used to add Longhorn trait to an ACM-nabled Anthos cluster

⚠️ Verify there is NO longhorn-system namespace on the cluster BEFORE applying this Cluster Trait Repo

How to use

Must apply the following config-management-system namespace (no async way to do this yet):

apiVersion: configsync.gke.io/v1beta1
kind: RootSync
metadata:
  name: lh-trait-sync
  namespace: config-management-system
spec:
  sourceFormat: unstructured
  git:
    repo: "https://gitlab.com/gcp-solutions-public/retail-edge/available-cluster-traits/longhorn-anthos.git"
    branch: "main"
    dir: "/config"
    auth: "token"                             # Optionally, set as "none" and skip the ExternalSecret and secretRef
    secretRef:
      name: longhorn-git-creds                # matches the secret below

---

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: longhorn-git-creds-es
  namespace: config-management-system
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: gcp-secret-store
  target:                                       # K8s secret definition
    name: longhorn-git-creds                    ############# Matches the secretRef above
    creationPolicy: Owner
  data:
  - secretKey: username                         # K8s secret key name inside secret
    remoteRef:
      key: longhorn-access-token-creds          #  GCP Secret Name
      property: username                        # field inside GCP Secret
  - secretKey: token                            # K8s secret key name inside secret
    remoteRef:
      key: longhorn-access-token-creds          #  GCP Secret Name
      property: token                           # field inside GCP Secret

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: longhorn-default-setting
  namespace: longhorn-system
data:
  default-setting.yaml: |-
    ...
    ... (see setting up config below)
    ...

Create Required GCP Secret (secretRef above)

# One time only
gcloud secrets create longhorn-git-creds -n config-management-system --replication-policy="automatic"
export TOKEN="<token-name>"
export TOKEN_VALUE="<token-value>"
# this can be run multiple times, adds a new version each time
echo -n "{ \"username\":\"${TOKEN}\", \"token\":\"${TOKEN_VALUE}\" }" | gcloud secrets versions add longhorn-git-creds --data-file=-

Create/Update Config folder (update the code to latest version)

nomos hydrate --source-format=unstructured --output=config --no-api-server-check

Setup & Config for Longhorn

Your cluster needs to have some default configuration setup at the Filesystem level as well as a custom YAML file with configuration options.

Filesystem

1. Create a folder at /customer (this may be a second disk, in which you will want to create the partition, add the ext or other filesystem, then mount at /customer)
2. Helper example script in script/create-customer-partition.sh available as an example of creating an LVM on a /dev/<disk>, formatting ext4 and mounting at /customer. Please review and adjust as needed for your conditions.

Configuration

1. Create a GCS bucket to store backups

2. Create a GCP Secret Mangaer secret with the GSA json key containing permission to the GCS bucket (please not, this is not intended for a full production server, using Workload Identity would likely be a better approach for prod)

3. Copy the default-longhorn-config-template.yaml to default-longhorn-config.yaml (this file is ignored by git)

   # Example if using `envsubst`
   envsubst default-longhorn-config-template.yaml default-longhorn-config.yaml
   

4. Fill in any changes and add this to the RootSync Cluster Trait Repository configuration (see above)

Local Validation

Assuming nomos is installed (via gcloud components install nomos)

nomos vet --no-api-server-check --source-format=unstructured --path config/

Docker method

Using this link to find the version of nomos-docker: https://cloud.google.com/anthos-config-management/docs/how-to/updating-private-registry#expandable-1

docker pull gcr.io/config-management-release/nomos:latest
docker run -it -v $(pwd):/code/ gcr.io/config-management-release/nomos:latest nomos vet --no-api-server-check --source-format un structured --path /code/config/

ACM Overview

See our documentation for how to use each subdirectory.