Nelm is a Helm 4 alternative. It is a Kubernetes deployment tool that manages Helm Charts and deploys them to Kubernetes. It is also the deployment engine of werf. Nelm does everything that Helm does, but better, and even quite some on top of it. Nelm is based on an improved and partially rewritten Helm codebase, to introduce:
terraform plan-like capabilities;- improved CRD management;
- out-of-the-box secrets management;
- advanced resource ordering capabilities;
- advanced resource lifecycle capabilities;
- improved resource state/error tracking;
- continuous printing of logs, events, resource statuses, and errors during deployment;
- fixed hundreds of Helm bugs, e.g. "no matches for kind Deployment in version apps/v1beta1";
- performance and stability improvements and more.
The Nelm goal is to provide a modern alternative to Helm, with long-standing issues fixed and many new major features introduced. Nelm moves fast, but our focus remains on Helm Chart and Release compatibility, to ease the migration from Helm.
Nelm is production-ready: as the werf deployment engine, it was battle-tested across thousands of projects for years.
- Install
- Quickstart
- CLI overview
- Helm compatibility
- Key features
- Usage
- Reference
werf.io/weightannotationwerf.io/deploy-dependency-<id>annotation<id>.external-dependency.werf.io/resourceannotation<id>.external-dependency.werf.io/nameannotationwerf.io/ownershipannotationwerf.io/deploy-onannotationwerf.io/delete-policyannotationwerf.io/delete-propagationannotationwerf.io/track-termination-modeannotationwerf.io/fail-modeannotationwerf.io/failures-allowed-per-replicaannotationwerf.io/no-activity-timeoutannotationwerf.io/sensitiveannotationwerf.io/sensitive-pathsannotationwerf.io/log-regexannotationwerf.io/log-regex-for-<container_name>annotationwerf.io/log-regex-skipannotationwerf.io/log-regex-skip-for-<container_name>annotationwerf.io/skip-logsannotationwerf.io/skip-logs-for-containersannotationwerf.io/show-logs-only-for-number-of-replicasannotationwerf.io/show-logs-only-for-containersannotationwerf.io/show-service-messagesannotationwerf_secret_filefunctiondump_debugfunctionprintf_debugfunctioninclude_debugfunctiontpl_debugfunction
- Feature gates
NELM_FEAT_PREVIEW_V2environment variableNELM_FEAT_REMOTE_CHARTSenvironment variableNELM_FEAT_NATIVE_RELEASE_LISTenvironment variableNELM_FEAT_NATIVE_RELEASE_UNINSTALLenvironment variableNELM_FEAT_PERIODIC_STACK_TRACESenvironment variableNELM_FEAT_FIELD_SENSITIVEenvironment variableNELM_FEAT_CLEAN_NULL_FIELDSenvironment variable
- More documentation
- Future plans
- Limitations
Follow instructions on GitHub Releases.
-
Create a directory for a new chart:
mkdir mychart cd mychart -
Create
Chart.yamlwith the following content:apiVersion: v2 name: mychart version: 1.0.0 dependencies: - name: cert-manager version: 1.13.3 repository: https://charts.jetstack.io
-
Generate
Chart.lock:nelm chart dependency download
-
Create
values.yamlwith the following content:cert-manager: installCRDs: true startupapicheck: enabled: false
-
Deploy the first release:
nelm release install -n myproject -r myproject
-
Plan the second release with an increased number of replicas, where only a single field in a single Deployment will be updated:
nelm release plan install -n myproject -r myproject --set cert-manager.replicaCount=2
-
Deploy the second release, where only the Deployment will be updated:
nelm release install -n myproject -r myproject --set cert-manager.replicaCount=2
Release commands:
release install Deploy a chart to Kubernetes.
release rollback Rollback to a previously deployed release.
release plan install Plan a release install to Kubernetes.
release uninstall Uninstall a Helm Release from Kubernetes.
release list List all releases in a namespace.
release history Show release history.
release get Get information about a deployed release.
Chart commands:
chart lint Lint a chart.
chart render Render a chart.
chart download Download a chart from a repository.
chart upload Upload a chart to a repository.
chart pack Pack a chart into an archive to distribute via a repository.
Secret commands:
chart secret key create Create a new chart secret key.
chart secret key rotate Reencrypt secret files with a new secret key.
chart secret values-file edit Interactively edit encrypted values file.
chart secret values-file encrypt Encrypt values file and print result to stdout.
chart secret values-file decrypt Decrypt values file and print result to stdout.
chart secret file edit Interactively edit encrypted file.
chart secret file encrypt Encrypt file and print result to stdout.
chart secret file decrypt Decrypt file and print result to stdout.
Dependency commands:
chart dependency download Download chart dependencies from Chart.lock.
chart dependency update Update Chart.lock and download chart dependencies.
Repo commands:
repo add Set up a new chart repository.
repo remove Remove a chart repository.
repo update Update info about available charts for all chart repositories.
repo login Log in to an OCI registry with charts.
repo logout Log out from an OCI registry with charts.
Other commands:
completion bash Generate the autocompletion script for bash
completion fish Generate the autocompletion script for fish
completion powershell Generate the autocompletion script for powershell
completion zsh Generate the autocompletion script for zsh
version Show version.Nelm is built upon the Helm codebase with some parts of Helm reimplemented. It is backward-compatible with Helm Charts and Helm Releases.
Helm Charts can be deployed by Nelm with no changes. All the obscure Helm Chart features, such as lookup functions, are supported.
To store release information, Nelm uses Helm Releases. You can deploy the same release with Helm and Nelm interchangeably, and it will work just fine. No migration needed from/to Helm.
Nelm has a different CLI layout, flags and environment variables, but we largely support all the same features as Helm.
Helm plugins support is not planned due to technical difficulties with the Helm plugins API. Instead, we intend to implement functionality of the most useful plugins natively, like we already did with nelm release plan install and nelm chart secret.
Generally, the migration from Helm to Nelm should be as simple as changing Helm commands to Nelm commands in your CI, for example:
| Helm command | Nelm command equivalent |
|---|---|
helm upgrade --install --atomic --wait -n myns myrls ./chart |
nelm release install --auto-rollback -n myns -r myrls ./chart |
helm uninstall -n myns myrls |
nelm release uninstall -n myns -r myrls |
helm template ./chart |
nelm chart render ./chart |
helm dependency build |
nelm chart dependency download |
The resource deployment subsystem of Helm is rewritten from scratch in Nelm. During the deployment, Nelm builds the Directed Acyclic Graph (DAG) of all operations we want to perform in the cluster to do the release, then the DAG is executed. The DAG allowed us to implement advanced resource ordering capabilities, such as:
- The
werf.io/weightannotation: similar tohelm.sh/hook-weight, but also works for non-hook resources. Resources with the same weight deployed in parallel. - The
werf.io/deploy-dependency-<id>annotation: do not deploy the annotated resource until the dependency is present or ready. This is the most powerful and effective way to enforce deployment order in Nelm. - The
<id>.external-dependency.werf.io/resourceannotation: do not deploy the annotated resource until the dependency is ready. The dependency can be an external, non-release resource, e.g. a resource created by a third-party operator. - Helm Hooks and their weights are supported, too.
Helm doesn't offer any resource lifecycle capabilities, except helm.sh/resource-policy: keep and helm.sh/hook-delete-policy for Hooks. On top of these, Nelm offers the following:
- The
werf.io/delete-policyannotation. Inspired byhelm.sh/hook-delete-policy, but works for any resource. Setbefore-creationto always recreate the resource,before-creation-if-immutableto only recreate if the resource is immutable,succeededorfailedto delete the resource on success or failure. - The
werf.io/ownershipannotation.anyoneallows to get Hook-like behavior for regular resources: don't delete the resource if it is removed from the Chart or when the whole release is removed, and never check or apply release annotations. - The
werf.io/deploy-onannotation. Inspired byhelm.sh/hook. Render and deploy the resource only on install/upgrade/rollback/uninstall in a pre/main/post stage.
These annotations make Helm Hooks obsolete: regular resources can do all the same things now.
Nelm has powerful resource tracking built from the ground up, much more advanced than what Helm has:
- Reliable detection of resources readiness, presence, absence or failures.
- Standard Kubernetes Resources have their own smart status trackers.
- Popular Custom Resources have hand-crafted rules to detect their statuses.
- For unknown Custom Resources, we heuristically determine their readiness by analyzing their status fields. Works for most Custom Resources. No false positives.
- The table with statuses, errors, and other info about currently tracked resources is printed every few seconds during the deployment.
During the deployment, Nelm finds Pods of deploying resources and periodically prints their container logs. With annotation werf.io/show-service-messages: "true", resource events are also printed. Can be configured with CLI flags and annotations.
nelm release plan install shows exactly what's going to happen in the cluster on the next release. It shows 100% accurate diffs between current and to-be resource versions, utilizing robust dry-run Server-Side Apply instead of client-side trickery.
nelm chart secret commands manage encrypted values files such as secret-values.yaml or encrypted arbitrary files like secret/mysecret.txt. These files are decrypted in-memory during templating and can be used in templates as .Values.my.secret.value and {{ werf_secret_file "mysecret.txt" }}, respectively.
CRDs from the crds/ directory of the chart deployed not only on the very first release install, but also on release upgrades. Also, CRDs not only can be created, but can be updated as well.
Nelm-specific features are described below. For general documentation, see Helm docs and werf docs.
Values files can be encrypted and stored in a Helm chart or a git repo. Such values files are decrypted in-memory during templating.
Create a secret key:
export NELM_SECRET_KEY="$(nelm chart secret key create)"Create a new secret-values file:
nelm chart secret values-file edit secret-values.yaml... with the following content:
password: verysecurepassword123Reference encrypted value in Helm templates:
password: {{ .Values.password }}Render the chart:
nelm chart renderpassword: verysecurepassword123NOTE: $NELM_SECRET_KEY must be set for any command that encrypts/decrypts secrets, including nelm chart render.
Arbitrary files can be encrypted and stored in the secret/ directory of a Helm chart. Such files are decrypted in-memory during templating.
Create a secret key:
export NELM_SECRET_KEY="$(nelm chart secret key create)"Create a new secret file:
nelm chart secret file edit secret/config.yaml... with the following content:
user: john-doe
password: verysecurepassword123Reference encrypted secret in Helm templates:
config: {{ werf_secret_file "config.yaml" | nindent 4 }}Render the chart:
nelm chart renderconfig:
user: john-doe
password: verysecurepassword123Nelm-specific features are described below. For general documentation, see Helm docs and werf docs.
This annotation works the same as helm.sh/hook-weight, but can be used for both hooks and non-hook resources. Resources with the same weight are grouped together, then the groups deployed one after the other, from low to high weight. Resources in the same group are deployed in parallel. This annotation has higher priority than helm.sh/hook-weight, but lower than werf.io/deploy-dependency-<id>.
Example:
werf.io/weight: "10"
werf.io/weight: "-10"Format:
werf.io/weight: "<any number>"
Default:
0
The resource will deploy only after all of its dependencies are satisfied. It waits until the specified resource is just present or is also ready. It serves as a more powerful alternative to hooks and werf.io/weight. You can only point to resources in the release. This annotation has higher priority than werf.io/weight and helm.sh/hook-weight.
Example:
werf.io/deploy-dependency-db: state=ready,kind=StatefulSet,name=postgres
werf.io/deploy-dependency-app: state=present,kind=Deployment,group=apps,version=v1,name=app,namespace=appFormat:
werf.io/deploy-dependency-<anything>: state=ready|present[,name=<name>][,namespace=<namespace>][,kind=<kind>][,group=<group>][,version=<version>]
The resource will deploy only after all of its external dependencies are satisfied. It waits until the specified resource is present and ready. You can only point to resources outside the release.
Example:
secret.external-dependency.werf.io/resource: secret/config
someapp.external-dependency.werf.io/resource: deployments.v1.apps/appFormat:
<anything>.external-dependency.werf.io/resource: <kind>[.<version>.<group>]/<name>
Set the namespace of the external dependency defined by <id>.external-dependency.werf.io/resource. <id> must match on both annotations. If not specified, the release namespace is used.
Example:
someapp.external-dependency.werf.io/name: someapp-productionFormat:
<anything>.external-dependency.werf.io/name: <name>
Inspired by Helm hooks. Sets the ownership of the resource. release means that the resource is deleted if removed from the chart or when the release is uninstalled, and release annotations of the resource are applied/validated during deploy. anyone means the opposite: resource is never deleted on uninstall or when removed from the chart, and release annotations are not applied/validated during deploy.
Example:
werf.io/ownership: anyoneFormat:
werf.io/ownership: anyone|release
Default:
"release" for general resources, "anyone" for hooks and CRDs from "crds/" directory
Inspired by helm.sh/hook. Render the resource for deployment only on the specified deploy types and stages. Has precedence over helm.sh/hook.
Beware that with werf.io/ownership: release if the resource is rendered for install, but, for example, not for upgrade, then it is going to be deployed on install, but then deleted on upgrade, so you might want to consider werf.io/ownership: anyone.
Example:
werf.io/deploy-on: pre-install,upgradeFormat:
werf.io/deploy-on: [pre-install][,install][,post-install][,pre-upgrade][,upgrade][,post-upgrade][,pre-rollback][,rollback][,post-rollback][,pre-uninstall][,uninstall][,post-uninstall]
Default:
"install,upgrade,rollback" for general resources, populated from "helm.sh/hook" for hooks
Inspired by helm.sh/hook-delete-policy. Controls resource deletions during resource deployment. before-creation means always recreate the resource, before-creation-if-immutable means recreate the resource only when we got "field is immutable" error during its update, succeeded means delete the resource at the end of the current deployment stage if the resource was successfully deployed, failed means delete the resource if it's readiness check failed. Has precedence over helm.sh/hook-delete-policy.
Example:
werf.io/delete-policy: before-creation,succeededFormat:
werf.io/delete-policy: [before-creation][,before-creation-if-immutable][,succeeded][,failed]
Default:
nothing for general resources (unless Job, then "before-creation-if-immutable"), mapped from "helm.sh/hook-delete-policy" for hooks
Set the deletion propagation policy for the resource. Foreground means delete the resource after deleting all of its dependents, Background means delete the resource immediately, and delete all of its dependents in the background, and Orphan means delete the resource, but leave all of its dependents untouched.
Example:
werf.io/delete-propagation: BackgroundFormat:
werf.io/delete-propagation: Foreground|Background|Orphan
Default:
Foreground
Configure when to stop resource readiness tracking:
WaitUntilResourceReady: wait until the resource isready.NonBlocking: don't wait until the resource isready.
Example:
werf.io/track-termination-mode: NonBlockingFormat:
werf.io/track-termination-mode: WaitUntilResourceReady|NonBlocking
Default:
WaitUntilResourceReady
Configure what should happen when errors during tracking for the resource exceeded werf.io/failures-allowed-per-replica:
FailWholeDeployProcessImmediately: fail the release.IgnoreAndContinueDeployProcess: do nothing.
Example:
werf.io/fail-mode: IgnoreAndContinueDeployProcessFormat:
werf.io/fail-mode: FailWholeDeployProcessImmediately|IgnoreAndContinueDeployProcess
Default:
FailWholeDeployProcessImmediately
Set the number of allowed errors during resource tracking. When exceeded, act according to werf.io/fail-mode.
Example:
werf.io/failures-allowed-per-replica: "0"Format:
werf.io/failures-allowed-per-replica: "<any positive number or zero>"
Default:
1
Take it as a resource tracking error if no new events or resource updates are received during resource tracking for the specified time.
Example:
werf.io/no-activity-timeout: 8m30sFormat (more info):
werf.io/no-activity-timeout: <golang duration>
Default:
4m
DEPRECATED. Use werf.io/sensitive-paths instead.
Don't show diffs for the resource.
NELM_FEAT_FIELD_SENSITIVE feature gate alters behavior of this annotation.
Example:
werf.io/sensitive: "true"Format:
werf.io/sensitive: "true|false"
Default:
"false", but for "v1/Secret" — "true"
Don't show diffs for resource fields that match specified JSONPath expressions. Overrides the behavior of werf.io/sensitive.
Example:
werf.io/sensitive-paths: "$.spec.template.spec.containers[*].env[*].value,$.data.*"Format:
werf.io/sensitive-paths: <JSONPath>,<JSONPath>,...
Only show log lines that match the specified regex.
Example:
werf.io/log-regex: ".*ERR|err|WARN|warn.*"Format (more info):
werf.io/log-regex: <re2 regex>
For the specified container, only show log lines that match the specified regex.
Example:
werf.io/log-regex-for-backend: ".*ERR|err|WARN|warn.*"Format (more info):
werf.io/log-regex-for-backend: <re2 regex>
Don't show log lines that match the specified regex.
Example:
werf.io/log-regex-skip: ".*TRACE|trace|DEBUG|debug.*"Format (more info):
werf.io/log-regex-skip: <re2 regex>
For the specified container, exclude log lines that match the specified regex.
Example:
werf.io/log-regex-skip-for-backend: ".*ERR|err|WARN|warn.*"Format (more info):
werf.io/log-regex-skip-for-backend: <re2 regex>
Don't print container logs during resource tracking.
Example:
werf.io/skip-logs: "true"Format:
werf.io/skip-logs: "true|false"
Default:
false
Don't print logs for specified containers during resource tracking.
Example:
werf.io/skip-logs-for-containers: "backend,frontend"Format:
werf.io/skip-logs-for-containers: <container_name>[,<container_name>...]
Print logs only for the specified number of replicas during resource tracking. We print logs only for a single replica by default to avoid excessive log output and to optimize resource usage.
Example:
werf.io/show-logs-only-for-number-of-replicas: "999"Format:
werf.io/show-logs-only-for-number-of-replicas: "<any positive number or zero>"
Default:
1
Print logs only for specified containers during resource tracking.
Example:
werf.io/show-logs-only-for-containers: "backend,frontend"Format:
werf.io/show-logs-only-for-containers: <container_name>[,<container_name>...]
Show resource events during resource tracking.
Example:
werf.io/show-service-messages: "true"Format:
werf.io/show-service-messages: "true|false"
Default:
false
Read the specified secret file from the secret/ directory of the Helm chart.
Example:
config: {{ werf_secret_file "config.yaml" | nindent 4 }}
Format:
{{ werf_secret_file "<filename, relative to secret/ dir>" }}
If the log level is debug, then pretty-dumps the passed value to the logs. Handles just fine any kind of complex types, including .Values, or event root context. Never prints to the templating output.
Example:
{{ dump_debug $ }}
Format:
{{ dump_debug "<value of any type>" }}
If the log level is debug, then prints the result to the logs. Never prints to the templating output.
Example:
{{ printf_debug "myval: %s" .Values.myval }}
Format:
{{ printf_debug "<format string>" <args...> }}
Works exactly like the include function, but if the log level is debug, then also prints various include-related debug information to the logs. Useful for debugging complex includes/defines.
Example:
{{ include_debug "mytemplate" . }}
Format:
{{ include_debug "<template name>" <context> }}
Works exactly like the tpl function, but if the log level is debug, then also prints various tpl-related debug information to the logs. Useful for debugging complex tpl templates.
Example:
{{ tpl_debug "{{ .Values.myval }}" . }}
Format:
{{ tpl_debug "<template string>" <context> }}
Activates all feature gates that will be enabled by default in v2.
Example:
export NELM_FEAT_PREVIEW_V2=true
nelm release listAllows specifying not only local, but also remote charts as a command-line argument to commands such as nelm release install. Adds the --chart-version option as well.
Will be the default in the next major release.
Example:
export NELM_FEAT_REMOTE_CHARTS=true
nelm release install -n myproject -r myproject --chart-version 19.1.1 bitnami/nginxUse native Nelm implementation of the release list command instead of helm list exposed as release list. Implementations differ a bit, but serve the same purpose.
Will be the default in the next major release.
Example:
export NELM_FEAT_NATIVE_RELEASE_LIST=true
nelm release listUse a new native Nelm implementation of the release uninstall command. Not fully backwards compatible with previous implementation.
Will be the default in the next major release.
Example:
export NELM_FEAT_NATIVE_RELEASE_UNINSTALL=true
nelm release uninstall -n myproject -r myprojectEvery few seconds print stack traces of all goroutines. Useful for debugging purposes.
Example:
export NELM_FEAT_PERIODIC_STACK_TRACES=true
nelm release install -n myproject -r myprojectWhen showing diffs for Secrets or werf.io/sensitive: "true" annotated resources, instead of hiding the entire resource diff hide only the actual secret fields: $.data, $.stringData.
Will be the default in the next major release.
Example:
export NELM_FEAT_FIELD_SENSITIVE=true
nelm release plan install -n myproject -r myprojectImprove Helm chart compatibility. When rendering charts, remove keys with null values from the rendered resource manifests, before applying them. Otherwise, SSA often fail on null values, which didn't happen with 3WM.
Will be the default in the next major release.
Example:
export NELM_FEAT_CLEAN_NULL_FIELDS=true
nelm release install -n myproject -r myprojectFor documentation on regular Helm features, see Helm docs. A lot of useful documentation can be found in werf docs.
- Nelm CLI.
- Nelm v1.
- Refactor, stabilize.
- Advanced resource lifecycle management.
- Nelm v2.
- Migration to Helm v4.
- The Nelm operator, which can integrate with ArgoCD/Flux (#494).
- An alternative to Helm templating (#54).
- Resource patching support (#115).
- DRY values.yaml files (#495).
- Downloading charts directly from Git.
- Migrate the built-in secrets management to Mozilla SOPS (#62).
- Nelmfile.
- Nelm requires Server-Side Apply enabled in Kubernetes. It is enabled by default since Kubernetes 1.16. In Kubernetes 1.14-1.15 it can be enabled, but disabled by default. Kubernetes 1.13 and older doesn't have Server-Side Apply, thus Nelm won't work with it.
- Helm sometimes uses Values from the previous Helm release to deploy a new release. This is to make Helm easier to use without a proper CI/CD process. This is dangerous, goes against IaC and this is not what users expect. Nelm will never do this: what you explicitly pass via
--valuesand--setoptions will be merged with chart values files, then applied to the cluster, as expected.



