Helmwave v1.0.0

[zhilyaev]

helmwave.yml

It's not a final! Just the current developed version.

project: example
version: 1.0.0

repositories:
  bitnami: https://charts.bitnami.com/bitnami

registries:
  ghcr.io: {}

defaults:
  # Default fields for all releases
  release:
    chart: bitnami/nginx
    namespace: default
    context: my-kube-context

  # Default fields for all registry
  registry:
    insecure: false

  # Default fields for all registry
  repository:
    insecure: false


releases:
  nginx@example@custom-context:
    version: 0.3.0

  nginx@hello: # = nginx@hello@my-kube-context
    version: 0.1.0

  nginx: # = nginx@default@my-kube-context
    version: 0.2.0

[zhilyaev]

Graph of Kubernetes structures dependencies

// todo: add url

Components

[zhilyaev]

Repositories vs Registries?

V1

2 different types.

repositories:
  bitnami: https://charts.bitnami.com/bitnami

registries:
  ghcr.io: {}

V2

Merge Repositories & Repositories into one spec.

repositories:
  bitnami: https://charts.bitnami.com/bitnami
  ghcr.io: oci://ghcr.io

[zhilyaev]

V1

[zhilyaev]

V2

[zhilyaev]

Tags vs Labels?

Labels!

releases:
  app@prod:
    labels:
      env: prod

  db@prod:
    labels:
      env: prod


  app@stage:
    labels:
      env: stage

  db@stage:
    labels:
      env: stage

$ helmwave build --labels env=prod
$ helmwave build --labels env=stage

Label Logic

!!! Looking for solutions

glob:

• $ helmwave build --labels env=pro?
• $ helmwave build --labels branch=feature/*

RE

• $ helmwave build --labels branch=~ /feature/i
• $ helmwave build -l env=(prod|stage)

expressions, = , !=, >, <, ?

• $ helmwave build --labels env!=stage
• $ helmwave build --labels lvl>0

[tony-engineering]

I like the ability to filter which releases with several tags. From my understanding it is currently not possible to perform AND operation with tags https://helmwave.readthedocs.io/en/latest/examples/tags/?h=tags . Why don't we just keep tags but enhance the possibilities ?

I am currently doing like that to select stages:

CI_ENVIRONMENT_NAME=prod helmwave build --yml --plandir .helmwave.prod --tags=yace-exporter

and in the code:

{{ if eq (requiredEnv "CI_ENVIRONMENT_NAME") "prod" }}
  - name: yace-exporter
    namespace: monitoring
    create_namespace: true
    wait: true
    timeout: 1m
    max_history: 3
    tags:
      - yace-exporter
      - non-critical
    chart:
      name: nerdswords/yet-another-cloudwatch-exporter
      version: {{ $versions | get "versions.yace" "" }}
    values:
      - values/yace/_.yaml
      - values/yace/{{ requiredEnv "CI_ENVIRONMENT_NAME" }}.yaml
{{ end }}

[zhilyaev]

@tony-engineering did you try HELMWAVE_MATCH_ALL_TAGS?

[zhilyaev]

Values

V1

Non-map.

values:
  - s3://mybucket/config/file.json
  - ../path/to/values.json
  - src: git+https://github.com/hairyhenderson/gomplate//docs-src/content/functions/values.yml
    alias: gitfile

V2

URL is a key.

values:
  "https://raw.githubusercontent.com/my.store.yml":
    alias: my-store
    render: sops

V3

Alias is a key.

values:
  my:
    src: https://raw.githubusercontent.com/my.yml
    render: sops

---

The third way looks good.

[zhilyaev]

V2

[zhilyaev]

V3

[tony-engineering]

does v3 means that this code

  - name: kube-prometheus
    namespace: monitoring
    create_namespace: true
    wait: true
    timeout: 5m
    max_history: 3
    tags:
      - prometheus
      - critical
    chart:
      name: bitnami/kube-prometheus # oci://registry-1.docker.io/bitnamicharts/kube-prometheus
      version: {{ $versions | get "versions.kube-prometheus" "" }}
    values:
      - values/kube-prometheus/_.yaml
      - src: values/kube-prometheus/{{ requiredEnv "CI_ENVIRONMENT_NAME" }}.yaml
        renderer: copy

would become

  - name: kube-prometheus
    namespace: monitoring
    create_namespace: true
    wait: true
    timeout: 5m
    max_history: 3
    tags:
      - prometheus
      - critical
    chart:
      name: bitnami/kube-prometheus # oci://registry-1.docker.io/bitnamicharts/kube-prometheus
      version: {{ $versions | get "versions.kube-prometheus" "" }}
    values:
      global:
        values/kube-prometheus/_.yaml
      env-specific:
        src: values/kube-prometheus/{{ requiredEnv "CI_ENVIRONMENT_NAME" }}.yaml
        renderer: copy

?

[zhilyaev]

@tony-engineering. Not at all. A little update. I will use the V5 spec for FileRef to explain.

values:
      global:
        values/kube-prometheus/_.yaml # it is not correct syntax.
      env-specific: # ok
        src: values/kube-prometheus/{{ requiredEnv "CI_ENVIRONMENT_NAME" }}.yaml
        renderer: copy

Pretty Example for V5:

values:
      # *.tpl –> run gomplate render
      global.tpl: values/kube-prometheus/_.yaml
      # copy
      env-specific: values/kube-prometheus/{{ requiredEnv "CI_ENVIRONMENT_NAME" }}.yaml

Full Example for V5:

values:
      # *.tpl –> run gomplate render
      global.tpl: 
        src: values/kube-prometheus/_.yaml
      # copy
      env-specific:  
        src: values/kube-prometheus/{{ requiredEnv "CI_ENVIRONMENT_NAME" }}.yaml

[r3nic1e]

In general yaml maps/dicts are not ordered but for helm values order really matter. So I prefer having an order-specific list here just in case YAML decoder will do something not expected here.

[zhilyaev]

StoreFiles

releases:
  app:
    storefiles:
      my: # <--- this is an alias
        src: https://raw.githubusercontent.com/my.store.yml
        render: sops
    values:
      a: # <--- this is an alias
        src: a.values.yml
        render: copy
      b: # <--- this is an alias
        src: b.values.yml
        render: copy

How to call the data from the store?

$ cat a.values.yml

podLabels:
  name: (ds "store/my").bla

Get any stores by datasource by store/$store_alias

How to call the data from another value?

!!! You can face a problem with cyclic dependencies.

$ cat b.values.yml

podLabels:
  name: (ds "values/a").bla

Get any values by datasource by values/$values_alias

[zhilyaev]

Defaults

#963, #924, #925, #266, #1104

Defaults can be passed by helmwave.yml or environment variables.
helmwave has a lot of features and settings that can interact with each other.
That's why we want to keep it once inside the helmwave.yml defaults section.

Any fields in this section can be passed by ENVIRONMENT VARIABLES.

Example

• export HELMWAVE_DEFAULTS_GOMLATE_RIGHT=']]'
• cat helmwave.yml

defaults:
  gomplate:
    leftDelim: '[['
    rightDelim: ']]'

It's equal to

cat helmwave.yml

defaults:
  gomplate:
    leftDelim: '[['

[zhilyaev]

Gomplate vs Sprig?

Gomplate!

Sprig must die.

[zhilyaev]

Slice or Map?

Map!

releases:
- name: foo
- name: bar

releases:
  foo: 
  bar:

It's prettier for yaml merge and finding the uniqname problems.

[tony-engineering]

From my perspective there is no big difference

[zhilyaev]

Magic Uniqname

Uniqname is $release_name@$namespace@kubecontext in one string. It helps helmwave to understand the destination of your stack.

V1

releases:
  nginx@example@custom-context:
    version: 0.3.0

  nginx@hello: # = nginx@hello@my-shell-kube-context
    version: 0.1.0

  nginx: # = nginx@default@my-shell-kube-context
    version: 0.2.0

• If kubecontext is empty, helmwave would use your current kubeconfig to understand it.
• If kubecontext is empty, helmwave would use your current kubeconfig to understand it.
• The release name is always required.

Issues:

• If Helmwave uses the kubeconfig default value. And a user changed their context after building a plan. Helmwave has a chance to fail.

V2

If you escalate to a new scope, you should also provide a new context for the uniqname.

Uniqname is only a name

releases:
  nginx: {} # ok
  redis: {} # ok

Uniqname escalates to namespace

If one of the releases has a namespace, it means everyone should have a namespace.

releases:
  nginx@prod: {} # prod is a namespace
  redis@stage: {} # stage is a namespace

  my: {} # err invalid, namespace not defined.

Uniqname escalates to kubecontext

If one of the releases has a kubecontext, it means everyone should have a kubecontext.

releases:
  nginx@default@stage: {} # stage is a kubecontext
  nginx@default@prod : {} # prod is a kubecontext

  my: {} # err invalid, namespace and kubecontext not defined.
  my@default: {} # err invalid, namespace not defined.

[zhilyaev]

V1

[zhilyaev]

V2

[tony-engineering]

I like the idea of making helmwave aware of the kubecontext. It happened to me sometilmes to deploy to the wrong cluster, or being confused because the helmwave build command is showing diff between non-prod and prod cluster...
its a pain to always need to verify kubecontext before building

[zhilyaev]

Deploy manifests step-by-step

Order by weight

#1100

apiVersion: v1
kind: ConfigMap
metadata:
  name: order-by-weight
  annotations:
    "helmwave.app/weight": "1"

flowchart LR
    subgraph weight_0
      configmap_A
      configmap_B
    end

    weight_0 --> weight_1

    subgraph weight_1
      deployment_A
      deployment_B
    end

Loading

Issues

• Helmwave should use Nelm. Like that: https://github.com/werf/nelm/tree/main#advanced-resource-ordering

Order by Needs

apiVersion: v1
kind: Deployment
metadata:
  name: order-by-needs
  annotations:
    "helmwave.app/needs": "configMap/order-by-weight,<$kind>/<$name>"

flowchart LR
      configmap_A --> deployment_A
      configmap_B --> deployment_B

Loading

Issues

• Helmwave must patch the original Helm package. Nelm doesn't have this feature yet.
• Cyclic dependencies
• Should helmwave check probes for workloads?
• Could you get the status resource ready?

[zhilyaev]

@w33dw0r7d
What should I do if I need deployment-a --> deployment-b . Suppose I have a frontend and a backend. Frontend (A) can't start without Backend (B)?

[w33dw0r7d]

Make two releases with depends_on.

[zhilyaev]

It's a good choice, but sometimes we have cases. Where engineers can't divide it into 2 releases.

[w33dw0r7d]

What cases?

[zhilyaev]

1. Existing charts that you do not own.
2. You don't have the resources to split it.
3. Specific logic for a release. Well, you can take a look at the gitlab/gitlab chart. Or let me continue with my previous example.

graph LR
deployment-a --> deployment-b
service-account & configmap --> deployment-a & deployment-b

Loading

Both deployments need the same resources. But we will split it into 2. It means one of them became an owner. That's a bad pattern. And then you also need to describe the names of existing resources in your second release. This is pain.

[zhilyaev]

Levels of abstraction

flowchart TD
    configmap_A & deployment_A & service_A & ingress_A --> helm_chart_A
    configmap_B & deployment_B & service_B & ingress_B --> helm_chart_B
    helm_chart_A --> release_A1 & release_A2 --> helmwave
    helm_chart_B --> release_B1 & release_B2 --> helmwave

Loading

[zhilyaev]

Deploy releases step-by-step

#1022 , #734

Strict needs

releases:
  app:
    needs:
       namespace:
         prod: {}
       name:
           db: {}
       labels:
           app: db

Issues

• How to manage OR & AND operators?
• How to manage app: prod-*, etc?

Smart needs

releases:
  app:
    needs:
       my-rule-name:
         - namespace = "prod" and name = "db" and label.app = "db"

Issues

• SQLite is too much.
• How to use Kubernetes labels for specific usage?

[zhilyaev]

Client-Side Apply (CSA) vs Server-Side Apply (SSA)

Server-Side Apply!

#1100

[zhilyaev]

How to choose a render?

#1017 , #1018

V1

Create an additional field where users can set a render.

files:
    my:
      src: 
        url: https://raw.githubusercontent.com/my.yml
      render: sops

V2

KISS, it's the users' responsibility.

Users can use Overriding MIME Types

type=text/plain can force disable Gomplate

files:
    only-copy-file:
      src:  https://raw.githubusercontent.com/my.yml?type=text/plain
   render-and-copy:
      src: https://raw.githubusercontent.com/my.yml
   sops-and-copy:
      src: https://raw.githubusercontent.com/my.yml?type=text/sops # create custom MIME types for specific usage.

V3

Use Headers to pass options

files:
    only-copy-file:
      src: 
        url: https://raw.githubusercontent.com/my.yml
        header:
          x-bla-type: text/plain  # create custom header
   render-and-copy:
      src: 
        url: https://raw.githubusercontent.com/my.yml
   sops-and-copy:
      src: 
        url: https://raw.githubusercontent.com/my.yml
        header:
          x-bla-type: text/sops # create custom header with custom values

V4

Define options by extensions?

If a file extension is tpl, Gomplate will run the render.

If a file extension is sops.yml, Gomplate will run the sops.

The default behavior is just to copy.

files:
    only-copy-file:
      src:  https://raw.githubusercontent.com/my.yml
   render-and-copy:
      src: https://raw.githubusercontent.com/my.yml.tpl
   sops-and-copy:
      src: https://raw.githubusercontent.com/my.sops.yml

V5

like V4, but extensions defined in aliases.

files:
    my.yml: # <-- just copy
      src:  https://raw.githubusercontent.com/my.txt
   my.yml.tpl: # <-- render and copy
      src: https://raw.githubusercontent.com/my.txt
   my.yml.sops: # <-- sops and copy
      src: https://raw.githubusercontent.com/my.txt

---

The easiest way to understand is V1 or V4.

[zhilyaev]

V5
https://github.com/helmwave/fileref has implemented V5

[zhilyaev]

V1

[zhilyaev]

V2

[zhilyaev]

V3

[zhilyaev]

V4

[zhilyaev]

Pull or Push model CD?

Together!

#740, #739 , #736

Helmwave transform helmwave.yml into plandir.
Plandir contains CRD helmwave manifests. The most Important is a kind: Plan. Then helmwave reads kind: Plan to get information about what helmwave should do. You can push that plan locally. Or install helmwave-operator. Then the helmwave operator will listen to CRD helmwave manifests.

!!! Plan dir structure and specification in progress.

Release

api: helmwave.app/v1alpha1
kind: Release
metadata:
  name: nginx-a
  namespace: default
spec:
  chart: 
    name: oci://ghcr:io/nginx
    version: 1.0.0

Repository

api: helmwave.app/v1alpha1
kind: Repository
metadata:
  name: ghcr
  namespace: default
spec:
  insecure: false
  url: oci://ghcr.io

Plan

api: helmwave.app/v1alpha1
kind: Plan
spec: ...

[tony-engineering]

Version pinning ?

Nice suggestions, but I think we miss a very important feature: version pinning. Its a bad practice to always take the latest version of a chart, its safer to update charts manually. Or have a system where only patch versions are update automatically. Helm and Helmfile tool are supporting this https://helm.sh/docs/chart_best_practices/dependencies/#versions .
I worked around it by doing this:

charts.lock.yaml

# In this file you can pin versions
# Gitlab CI check-for-updates job will request manual
# change if new version is available
versions:
  grafana: "8.10.3"
  yace: "0.38.0"
  karpenter: "1.3.2"
  kube-prometheus: "9.6.3"
  metrics-server:  "3.12.2"
  alb-controller: "1.11.0"
  ingress-nginx: "4.12.0"
  aws-for-fluent-bit: "0.1.35"
  kube-janitor: "0.3.1"
  mlflow-postgres: "13.2.30"
  mlflow: "0.7.19"

always get version like this in helmwave.yml.tml

{{ $versions := "" }}
{{ $versions := readFile "./charts.lock.yaml" | fromYaml }}


releases:
  - name: kube-prometheus
    namespace: monitoring
    chart:
      name: bitnami/kube-prometheus # oci://registry-1.docker.io/bitnamicharts/kube-prometheus
      version: {{ $versions | get "versions.kube-prometheus" "" }}

if I want to programatilly check for available updates, I need to run a set of commands (I put it in a Gitlab CI job). Providing an empty chart.lock.yaml makes helmwave create a planfile with latest version of charts.

- echo "" > charts.lock.yaml
  - helmwave yml
  - helmwave build
  - cp .helmwave/planfile b
  - "versionsDiff=$(diff a b | grep 'name: .helmwave' || true)"
  - echo "Available updates:"
  - echo $versionsDiff
  - echo "To apply updates, increase versions in charts.lock.yaml"
  - |
    if [ "$versionsDiff" != "" ]; then
      exit 1
    else
      exit 0
    fi

as you can see its very home-made, it would be amazing to integrate such feature into helmwave

[zhilyaev]

@tony-engineering
Helmwave shows chart changes in logs. Do you wanna fail the Helmwave build process if the chart version has changed?

[zhilyaev]

Plandir

V1

.helmwave/

• planfile.yml
• manifests/
  • nginx@default.yml
  • mongo@prod.yml
• values/
  • nginx@default/
    • a.yml
    • b.yml
• stores/
  • nginx@default/
    • vault.yml
    • creds.yml.tpl
• crds/
  • plan.yml
  • nginx@default.yml
  • mongo@prod.yml

Deploy with kubectl

kubectl -f .helmwave/manifests 

Run the pull model

kubectl -f .helmwave/crds 

V2

.helmwave/

• planfile.yml
• nginx@default/
  • manifests.yml
  • release.yml
  • values/
    • a.yml
    • b.yml
  • stores/
    • vault.yml
    • creds.yml.tpl
• mongo@prod/
  • manifests.yml
  • release.yml

[zhilyaev]

V1

[zhilyaev]

V2

[zhilyaev]

helmwave.yml.tpl

Self-render configuration.

V1

Left 2 steps for render. It's current behavior.

helmwave.yml.tpl -- helmwave yml --> helmwave.yml

V2

Only 1 file!

helmwave.yml can contain templating. For example werf.yml, (Pulumi.yaml)[https://www.pulumi.com/blog/pulumi-yaml/]

You will get a template error in helmwave build command.

[zhilyaev]

V1

[zhilyaev]

V2

[zhilyaev]

Plan behavior

V1

The current concept is that a plan is a subject on which you can perform actions. Such as deploy a plan, delete a plan, and rollback a plan.

It means that if you delete the release from the plan. It won't delete the release.

For example, you need to uninstall the release FooBar helmwave build -t FooBar. You must generate the plan with only FooBar. Then run helmwave down to uninstall FooBar. And then rebuild the plan with all your new stuff.

flowchart LR

subgraph Plan
   release_A
   release_B
end

generate((generate)) --> Plan --> up((up)) & down((down)) & rollback((rollback))

Loading

V2

Plan it's what we want to see in a k8s cluster. It means a plan is a list of actions for a deployer.
planfile.yaml will look like:

releases:
 a:
   plan: uninstall
 b:
   plan: install
 c:
   plan: rollback

$ helmwave apply -f planfile
....
Deployed 1/1
Uninstalled 1/1
Rollback 1/1

Everything is ok!

Thus, inconsistency of the plan may occur. When the command helmwave yml is called again.

flowchart LR
up((up))
 down((down))
  rollback((rollback))

subgraph Plan
   release_A --> down
   release_B --> up
   release_C --> rollback
end

generate((generate)) --> Plan --> apply((apply))

Loading

V3

It's like V1. Set the option behavior for helmwave cli.

• helmwave up --sync --prev-plandir .helmwave_prev --plandir .helmwave
• helmwave sync

Issues

• You have to keep 2 plans for sync.

V4

is coming...

[zhilyaev]

V1

[zhilyaev]

V2

[zhilyaev]

V3