Skip to content

Commit d909741

Browse files
devantlerclaude
andauthored
feat(workload): render GitOps manifests in-process for validate & scan (#5344) (#5356)
* feat(workload): add in-process GitOps render package (#5344) Introduce pkg/svc/gitops/render, which expands a kustomize-built, Flux-substituted manifest stream into the resources Flux actually applies: each HelmRelease is resolved against the OCIRepository/HelmRepository sources in the same stream and rendered in-process via the embedded Helm client (helm.TemplateChart). Successfully rendered releases replace their CR; unrenderable ones (GitRepository sources, charts unreachable offline, missing sources) degrade gracefully — the CR is kept and a Degradation is recorded — so a run never hard-fails purely because a chart can't be resolved offline. Rendering is exposed behind a ChartResolver interface so callers can validate/scan the real applied manifests offline; wiring into the validate and scan commands follows in subsequent changes for #5344. Also fix a latent data race the renderer exposes: the Helm client mutated the process-global HELM_HTTP_TIMEOUT env var around each repository chart locate; concurrent renders now serialize that set/restore under a package-level mutex (covered by a new -race test). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(workload): render HelmReleases in 'validate' (#5344) ksail workload validate now renders HelmReleases in-process before validating, so the manifests Flux actually applies (Deployments, Services, …) are schema-checked — not just the opaque HelmRelease CR. For each kustomization, the build output is Flux-substituted, then HelmReleases are resolved against the OCIRepository/HelmRepository sources in the same directory and rendered via the embedded Helm client; the rendered children are validated with kubeconform. Rendering is on by default and controlled by the new spec.workload.validation.helmRender config field and the --skip-helm-render flag. HelmReleases that cannot be rendered offline (GitRepository sources, unreachable/private registries, missing sources) degrade gracefully: the HelmRelease CR is validated as before and a warning is emitted, so a run never hard-fails purely because a chart could not be resolved. Rendering applies to kustomization builds only; a single-file input remains a CR-schema check (keeping the gen-smoke CI canary green). Push-time validation keeps its existing CR-level behavior. Adds the shared gitopsRenderer/degradationSink helpers (also used by scan next), offline integration tests (valid render passes, invalid rendered manifest fails, unreachable chart degrades, --skip-helm-render bypasses, single-file canary), and regenerated schema/CRD/docs/chat artifacts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(workload): scan the rendered output in 'scan' (#5344) ksail workload scan now scans the manifests Flux actually applies instead of the raw files: when the target directory is a Kustomize root, it is rendered (Kustomize build + Flux substitution + in-process Helm templating) to a temp directory which kubescape then scans. Findings therefore reflect overlay patches that weaken a base and the output of rendered charts, not just the opaque HelmRelease/Kustomize sources. Rendering reuses the gitopsRenderer from the validate change and is on by default, honoring spec.workload.validation.helmRender. A new --raw flag restores the previous raw-file behavior for one release while consumers migrate (kept visible in --help so the escape hatch is discoverable). The rendered temp directory is canonicalized for kubescape and always cleaned up via defer. Single files and directories without a kustomization are scanned as-is, matching prior behavior. Removes the now-unused resolveValidatePathFromCmd helper (scan loads config directly, like validate). Adds offline tests for the render-vs-raw decision and temp-dir cleanup via an ExportResolveScanInput seam, and regenerates the scan flag + chat docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(workload): isolate Helm client per render; simplify degradation reporting (#5344) Code-review follow-up on the GitOps rendering work. The gitopsRenderer previously held one shared Helm client, but validate renders kustomizations in parallel — concurrent TemplateChart calls on a shared helm.action.Configuration are not safe. Construct a fresh template-only client per render instead (cheap, no cluster access); the stateless kustomize client is still shared. Adds a -race regression test that renders multiple kustomizations concurrently. Simplifications from the same review: extract a shared warnDegradations helper so scan's renderToTempDir no longer builds a throwaway degradationSink; newGitOpsRenderer no longer returns an error (client moved into expand), dropping the error plumbing in buildValidateRenderer and resolveScanInput; and expandHelmRelease reuses the already-parsed objectMeta instead of re-parsing the document. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 4571ab1 commit d909741

29 files changed

Lines changed: 2359 additions & 116 deletions

File tree

charts/ksail-operator/crds/ksail.io_clusters.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -938,6 +938,10 @@ spec:
938938
description: ValidationConfig defines configuration for the workload
939939
validate command.
940940
properties:
941+
helmRender:
942+
description: HelmRender controls whether HelmReleases are
943+
rendered before validation.
944+
type: boolean
941945
skipKinds:
942946
description: |-
943947
SkipKinds lists additional Kubernetes kinds to skip during validation.

docs/src/content/docs/cli-flags/workload/workload-scan.mdx

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,12 @@ Run security scans on Kubernetes manifests using Kubescape.
1111
This command scans manifests in the specified path against security frameworks
1212
such as NSA-CISA, MITRE ATT&CK, and CIS Benchmarks.
1313
14+
When the target directory is a Kustomize root, manifests are rendered before
15+
scanning (Kustomize build + Flux variable substitution + in-process Helm
16+
templating of HelmReleases), so findings reflect overlay patches and chart output
17+
rather than the raw files. HelmReleases that cannot be rendered offline are left
18+
as-is with a warning. Use --no-render to scan the raw files instead.
19+
1420
If no path is provided, the path is resolved in order:
1521
1. spec.workload.sourceDirectory from ksail.yaml (if a config file is found and the field is set)
1622
2. The default source directory when spec.workload.sourceDirectory is unset ("k8s" directory)
@@ -28,6 +34,7 @@ Flags:
2834
--compliance-threshold float32 Fail if compliance score is below this threshold (0-100)
2935
--format string Output format (pretty-printer, json, sarif, junit) (default "pretty-printer")
3036
--framework strings Security frameworks to scan against (e.g. nsa, mitre, cis, pss) (default [nsa])
37+
--no-render Scan the raw manifest files instead of the Kustomize + Helm rendered output (skip rendering entirely; restores the pre-rendering behavior)
3138
-o, --output string Output file path (stdout if empty)
3239
--verbose Show all resources in output, not just failed ones
3340

docs/src/content/docs/cli-flags/workload/workload-validate.mdx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ Usage:
4040
4141
Flags:
4242
--ignore-missing-schemas Ignore resources with missing schemas (default true)
43+
--skip-helm-render Skip rendering HelmReleases before validation (validate the HelmRelease CR as-is). By default, charts are rendered in-process and the rendered manifests are validated.
4344
--skip-kinds strings Additional Kubernetes kinds to skip during validation (merged with spec.workload.validation.skipKinds from ksail.yaml)
4445
--skip-secrets Skip validation of Kubernetes Secrets (default true)
4546
--strict Enable strict validation mode

pkg/apis/cluster/v1alpha1/types.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,9 @@ type ValidationConfig struct {
210210
// kubeconform would otherwise reject (e.g. valid newer fields flagged as
211211
// "additional properties not allowed").
212212
SkipKinds []string `json:"skipKinds,omitzero" jsonschema_description:"Additional Kubernetes kinds to skip during 'ksail workload validate' (Secrets are skipped by default via --skip-secrets). Use for CRDs whose CRDs-catalog schema is stale or missing, which kubeconform would otherwise reject."` //nolint:lll
213+
214+
// HelmRender controls whether HelmReleases are rendered before validation.
215+
HelmRender *bool `json:"helmRender,omitzero" jsonschema_description:"Render HelmReleases (Kustomize + Helm) before 'ksail workload validate' so the actually-applied manifests are validated rather than the opaque HelmRelease CR. Charts are resolved from the OCIRepository/HelmRepository sources in the same directory and rendered in-process; releases that cannot be rendered offline fall back to validating the CR. Defaults to true. Override per-run with --skip-helm-render."` //nolint:lll
213216
}
214217

215218
// WatchConfig defines configuration for the workload watch command.

pkg/apis/cluster/v1alpha1/zz_generated.deepcopy.go

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

pkg/cli/cmd/workload/__snapshots__/workload_test.snap

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,12 @@ Run security scans on Kubernetes manifests using Kubescape.
158158
This command scans manifests in the specified path against security frameworks
159159
such as NSA-CISA, MITRE ATT&CK, and CIS Benchmarks.
160160

161+
When the target directory is a Kustomize root, manifests are rendered before
162+
scanning (Kustomize build + Flux variable substitution + in-process Helm
163+
templating of HelmReleases), so findings reflect overlay patches and chart output
164+
rather than the raw files. HelmReleases that cannot be rendered offline are left
165+
as-is with a warning. Use --no-render to scan the raw files instead.
166+
161167
If no path is provided, the path is resolved in order:
162168
1. spec.workload.sourceDirectory from ksail.yaml (if a config file is found and the field is set)
163169
2. The default source directory when spec.workload.sourceDirectory is unset ("k8s" directory)
@@ -176,6 +182,7 @@ Flags:
176182
--format string Output format (pretty-printer, json, sarif, junit) (default "pretty-printer")
177183
--framework strings Security frameworks to scan against (e.g. nsa, mitre, cis, pss) (default [nsa])
178184
-h, --help help for scan
185+
--no-render Scan the raw manifest files instead of the Kustomize + Helm rendered output (skip rendering entirely; restores the pre-rendering behavior)
179186
-o, --output string Output file path (stdout if empty)
180187
--verbose Show all resources in output, not just failed ones
181188

@@ -217,6 +224,7 @@ Usage:
217224
Flags:
218225
-h, --help help for validate
219226
--ignore-missing-schemas Ignore resources with missing schemas (default true)
227+
--skip-helm-render Skip rendering HelmReleases before validation (validate the HelmRelease CR as-is). By default, charts are rendered in-process and the rendered manifests are validated.
220228
--skip-kinds strings Additional Kubernetes kinds to skip during validation (merged with spec.workload.validation.skipKinds from ksail.yaml)
221229
--skip-secrets Skip validation of Kubernetes Secrets (default true)
222230
--strict Enable strict validation mode
@@ -3123,6 +3131,12 @@ Run security scans on Kubernetes manifests using Kubescape.
31233131
This command scans manifests in the specified path against security frameworks
31243132
such as NSA-CISA, MITRE ATT&CK, and CIS Benchmarks.
31253133
3134+
When the target directory is a Kustomize root, manifests are rendered before
3135+
scanning (Kustomize build + Flux variable substitution + in-process Helm
3136+
templating of HelmReleases), so findings reflect overlay patches and chart output
3137+
rather than the raw files. HelmReleases that cannot be rendered offline are left
3138+
as-is with a warning. Use --no-render to scan the raw files instead.
3139+
31263140
If no path is provided, the path is resolved in order:
31273141
1. spec.workload.sourceDirectory from ksail.yaml (if a config file is found and the field is set)
31283142
2. The default source directory when spec.workload.sourceDirectory is unset ("k8s" directory)
@@ -3141,6 +3155,7 @@ Flags:
31413155
--format string Output format (pretty-printer, json, sarif, junit) (default "pretty-printer")
31423156
--framework strings Security frameworks to scan against (e.g. nsa, mitre, cis, pss) (default [nsa])
31433157
-h, --help help for scan
3158+
--no-render Scan the raw manifest files instead of the Kustomize + Helm rendered output (skip rendering entirely; restores the pre-rendering behavior)
31443159
-o, --output string Output file path (stdout if empty)
31453160
--verbose Show all resources in output, not just failed ones
31463161
@@ -3186,6 +3201,7 @@ Usage:
31863201
Flags:
31873202
-h, --help help for validate
31883203
--ignore-missing-schemas Ignore resources with missing schemas (default true)
3204+
--skip-helm-render Skip rendering HelmReleases before validation (validate the HelmRelease CR as-is). By default, charts are rendered in-process and the rendered manifests are validated.
31893205
--skip-kinds strings Additional Kubernetes kinds to skip during validation (merged with spec.workload.validation.skipKinds from ksail.yaml)
31903206
--skip-secrets Skip validation of Kubernetes Secrets (default true)
31913207
--strict Enable strict validation mode

pkg/cli/cmd/workload/export_test.go

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,25 @@ func ExportHasKustomizationFile(dir string) bool {
121121
return hasKustomizationFile(dir)
122122
}
123123

124+
// ExportResolveScanInput exposes resolveScanInput for testing the scan render
125+
// decision (rendered temp dir vs raw path) and temp-dir cleanup without
126+
// invoking kubescape.
127+
func ExportResolveScanInput(
128+
ctx context.Context,
129+
cmd *cobra.Command,
130+
path string,
131+
cfg *v1alpha1.Cluster,
132+
configFound, noRender bool,
133+
) (string, func(), error) {
134+
return resolveScanInput(ctx, cmd, path, cfg, configFound, noRender)
135+
}
136+
137+
// ExportResolveScanOutput exposes resolveScanOutput for testing the scan
138+
// --output directory creation and canonicalization.
139+
func ExportResolveScanOutput(output string) (string, error) {
140+
return resolveScanOutput(output)
141+
}
142+
124143
// ExportPollInterval exposes the engine's poll interval constant for testing.
125144
const ExportPollInterval = workloadwatch.PollInterval
126145

pkg/cli/cmd/workload/push.go

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -171,10 +171,16 @@ func validateManifests(cmd *cobra.Command, sourceDir string, outputTimer timer.T
171171
cmd.Context(),
172172
cmd,
173173
[]string{sourceDir},
174-
true, // skipSecrets
175-
true, // strict
176-
true, // ignoreMissingSchemas
177-
nil, // extra skip-kinds are read from ksail.yaml (configuredSkipKinds)
174+
validateFlags{
175+
skipSecrets: true,
176+
strict: true,
177+
ignoreMissingSchemas: true,
178+
// Push-time validation keeps the existing CR-level check (no Helm
179+
// render) so a push stays fast and offline; `ksail workload validate`
180+
// is where rendered-manifest validation applies.
181+
skipHelmRender: true,
182+
skipKinds: nil, // extra skip-kinds are read from ksail.yaml (configuredSkipKinds)
183+
},
178184
)
179185
if err != nil {
180186
return fmt.Errorf("validate manifests: %w", err)

pkg/cli/cmd/workload/render.go

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
package workload
2+
3+
import (
4+
"context"
5+
"fmt"
6+
"os"
7+
"path/filepath"
8+
"sync"
9+
10+
"github.com/devantler-tech/ksail/v7/pkg/client/helm"
11+
"github.com/devantler-tech/ksail/v7/pkg/client/kustomize"
12+
"github.com/devantler-tech/ksail/v7/pkg/notify"
13+
"github.com/devantler-tech/ksail/v7/pkg/svc/fluxsubst"
14+
"github.com/devantler-tech/ksail/v7/pkg/svc/gitops/render"
15+
"github.com/spf13/cobra"
16+
)
17+
18+
// renderedManifestPerm is the permission for the rendered manifests file written
19+
// to the scan temp directory.
20+
const renderedManifestPerm = 0o600
21+
22+
// gitopsRenderer expands a kustomization directory into the manifests Flux
23+
// actually applies: Kustomize build, Flux variable substitution, then in-process
24+
// Helm rendering of HelmReleases. It is shared by the validate and scan commands.
25+
type gitopsRenderer struct {
26+
kustomize *kustomize.Client
27+
}
28+
29+
// newGitOpsRenderer constructs a renderer. The kustomize client is stateless and
30+
// safe to share across goroutines; the Helm template client is created per
31+
// kustomization in expand (see below).
32+
func newGitOpsRenderer() *gitopsRenderer {
33+
return &gitopsRenderer{kustomize: kustomize.NewClient()}
34+
}
35+
36+
// expand builds, substitutes, and Helm-renders one kustomization directory. The
37+
// kustomize build error is returned unwrapped so the caller's simplifyBuildError
38+
// can strip the verbose "kustomize build <path>:" prefix.
39+
//
40+
// A fresh Helm template client is created per call: helm's action.Configuration
41+
// is not safe for concurrent use, and validate renders kustomizations in
42+
// parallel, so each render must be isolated. Construction needs no cluster
43+
// access and is cheap.
44+
func (g *gitopsRenderer) expand(ctx context.Context, kustDir string) (render.Result, error) {
45+
output, err := g.kustomize.Build(ctx, kustDir)
46+
if err != nil {
47+
return render.Result{}, err //nolint:wrapcheck // caller strips the kustomize prefix
48+
}
49+
50+
helmClient, err := helm.NewTemplateOnlyClient()
51+
if err != nil {
52+
return render.Result{}, fmt.Errorf("create helm template client: %w", err)
53+
}
54+
55+
expanded := fluxsubst.ExpandFluxSubstitutions(output.Bytes())
56+
57+
result, err := render.Expand(ctx, expanded, render.Options{
58+
Resolver: render.NewHelmChartResolver(helmClient),
59+
})
60+
if err != nil {
61+
return render.Result{}, fmt.Errorf("expand HelmReleases: %w", err)
62+
}
63+
64+
return result, nil
65+
}
66+
67+
// renderToTempDir renders one kustomization directory to a fresh temp directory
68+
// and returns it together with a cleanup func, so a file-based scanner
69+
// (kubescape) can read the actually-applied manifests. Non-silent render
70+
// degradations are warned to the user. The caller must invoke cleanup (e.g. via
71+
// defer) to remove the temp directory.
72+
func (g *gitopsRenderer) renderToTempDir(
73+
ctx context.Context,
74+
cmd *cobra.Command,
75+
kustDir string,
76+
) (string, func(), error) {
77+
result, err := g.expand(ctx, kustDir)
78+
if err != nil {
79+
return "", nil, fmt.Errorf("render %q: %w", kustDir, err)
80+
}
81+
82+
tmpDir, err := os.MkdirTemp("", "ksail-scan-*")
83+
if err != nil {
84+
return "", nil, fmt.Errorf("create scan temp dir: %w", err)
85+
}
86+
87+
cleanup := func() { _ = os.RemoveAll(tmpDir) }
88+
89+
manifestPath := filepath.Join(tmpDir, "manifests.yaml")
90+
91+
err = os.WriteFile(manifestPath, result.Bytes(), renderedManifestPerm)
92+
if err != nil {
93+
cleanup()
94+
95+
return "", nil, fmt.Errorf("write rendered manifests: %w", err)
96+
}
97+
98+
warnDegradations(cmd, result.Degradations)
99+
100+
return tmpDir, cleanup, nil
101+
}
102+
103+
// degradationSink collects render degradations across parallel validation tasks
104+
// so they can be reported once after the progress group completes (emitting
105+
// mid-group would interleave with the ANSI progress display).
106+
type degradationSink struct {
107+
mu sync.Mutex
108+
list []render.Degradation
109+
}
110+
111+
// add records degradations from one render result for later reporting.
112+
func (s *degradationSink) add(degradations []render.Degradation) {
113+
s.mu.Lock()
114+
defer s.mu.Unlock()
115+
116+
s.list = append(s.list, degradations...)
117+
}
118+
119+
// report warns about all collected degradations.
120+
func (s *degradationSink) report(cmd *cobra.Command) {
121+
s.mu.Lock()
122+
defer s.mu.Unlock()
123+
124+
warnDegradations(cmd, s.list)
125+
}
126+
127+
// warnDegradations emits a warning for each non-silent render degradation. Silent
128+
// degradations (e.g. a source object owned by a different kustomization) are
129+
// skipped to avoid noise on large repos.
130+
func warnDegradations(cmd *cobra.Command, degradations []render.Degradation) {
131+
for _, degradation := range degradations {
132+
if degradation.Silent {
133+
continue
134+
}
135+
136+
notify.WriteMessage(notify.Message{
137+
Type: notify.WarningType,
138+
Content: "skipped Helm render for HelmRelease %s (validating the resource as-is): %s",
139+
Args: []any{degradation.HelmRelease, degradation.Reason},
140+
Writer: cmd.ErrOrStderr(),
141+
})
142+
}
143+
}

0 commit comments

Comments
 (0)