New kubernetes backend #21796

grihabor · 2024-12-22T23:24:32Z

While helm backend already exists and works fine for kubernetes, creating a helm chart might be an overkill for a small thing like a single configmap or a secret. New kubernetes backend can deploy single object easily given a yaml file.

See docs for more details on usage.

I'm also planning to contribute our integration with python_format_string target (I will open a pr) that can handle simple python templating in text files.

cburroughs · 2025-01-07T20:08:56Z

cc @tgolsson who I know has done some k8s related work at https://github.com/tgolsson/pants-backends/tree/main/pants-plugins/k8s

Broad topic not intended to detail this whole PR: Something we have struggled with internally is that we would like to be able to generate the final reified k8s yaml for inspection or alternative deployments. (I have a PoC plugin that does this for helm) This is along the lines of 'package' but not not quite what helm means by 'package', and I'm unsure what to call it. It would be nice if all the Pant k8s generators could eventually both "spit out the yaml" and "deploy".

grihabor · 2025-01-08T11:16:04Z

Something we have struggled with internally is that we would like to be able to generate the final reified k8s yaml for inspection or alternative deployments.

Haha, we have exactly this problem, I've called the goal render and implemented it for helm_deployment and k8s_bundle

lilatomic

Looks good!
My only real sticking point is about the semantics of the requirement for context

lilatomic · 2025-01-08T14:03:22Z

src/python/pants/backend/k8s/goals/tailor.py

+    putative_targets = []
+
+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")


include "*.yml" too? or maybe make this a customisable option? Not necessary for a first pass

Personally, I want consistent .yaml or .yml in the whole repo, so probably it should be customisable, yes

I agree on this being initially set to find both *.yaml and *.yml files. Users of this could end up confused why tailor is not finding their .yml files.

lilatomic · 2025-01-08T14:17:02Z

src/python/pants/backend/k8s/goals/tailor.py

+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")
+        all_k8s_files = await Get(Paths, PathGlobs, all_k8s_files_globs)
+        unowned_k8s_files = set(all_k8s_files.files) - set(all_owned_sources)


I think that globbing every yaml file will be too broad, but I don't know that there's a standard way of detecting if a manifest is a k8s manifest (kubectl apply --validate=true --dry-run=client seems to need a server). Worst case people can turn off tailoring.

I'm thinking 2 most common use cases here:

Developer's repo with code will probably use some kind of templating to generate k8s_sources (I'm planning to open couple more prs for that), so tailor for yaml won't be that useful. In this case I expect devs to disable tailoring.

DevOps's repo with mostly k8s yaml files. In this case I expect tailor to work good

Why not just also scan the file for the apiVersion string as a heuristic? The go tailor rules use a similar pattern to search for package main to find potential putative go_binary targets.

pants/src/python/pants/backend/go/goals/tailor.py

Line 176 in 5f3f251

if has_package_main(file_content.content) and has_go_mod_ancestor(dirname, all_go_mod_dirs):

Also, given that YAML is used for many other uses, I'm thinking that any user with this backend enabled is probably going to become annoyed if tailor creates false positive k8s_source targets every time.

Also, would it make sense to have an option to restrict tailor usage to only certain directory trees?

Ok, it makes sense, but I don't want to do it here yet. Let me temporarily remove the whole tailor feature for now and I'll open a separate pr where we can continue the discussion. 12ad882

lilatomic · 2025-01-08T14:36:50Z

build-support/bin/external_tool_versions.py

+
+
+@dataclass(frozen=True)
+class VersionHash:


You could use pants.core.util_rules.external_tool.ExternalToolVersion

It's not quite the same thing, because platform here is not a predefined platform that pants uses, but a platform that needs to be mapped using url_platform_mapping. But I can make it work like this 6efda67

lilatomic · 2025-01-08T14:49:36Z

build-support/bin/external_tool_versions.py

+            )
+
+    backward_platform_mapping = {v: k for k, v in platform_mapping.items()}
+    for result in results:


you could have these output in semver order (instead of lexical order) by using from packaging.version import Version, by collecting the versions and then sorting with something like sorted(versions, key=lambda e: Version(e.version))

Sure 9d84c9e

build-support/bin/external_tool_versions.py

src/python/pants/backend/k8s/goals/deploy.py

lilatomic · 2025-01-08T15:27:06Z

src/python/pants/backend/k8s/goals/deploy.py

+    platform: Platform,
+) -> DeployProcess:
+    context = field_set.context.value
+    if context is None:


I think I've missed something in the logic here. Why do we force the field_set.context to have a value even when kubectl.pass_context is False?

The idea is that setting context on a target is required. This is because we had recurring issues because somebody forgot to set the correct context on the target and it got deployed to a different context by accident. However, we've got CI agents running in the cluster itself, so they can only deploy k8s objects to the specific cluster specified with KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, that's why we need to disable --context argument in pants.ci.toml.

I understand this is kinda custom setup, so people might need some other behavior. However, in general, I think context requirement is a good thing that can prevent misdeployments. I can probably move this context validation to a small linter and check context field there, wdyt?

It makes sense to have a failsafe to prevent deploying things in the wrong clusters. I was thinking of how the Helm backend does it. It has an optional HelmDeploymentNamespaceField and no field for context.

I think we can just add a line in the docs explaining the intended workflow. Something like

To prevent accidentally deploying kubernetes manifests to the wrong cluster, the context field is required on k8s_bundles for deployment.
For deploying the same k8s_bundle to multiple contexts, consider using parametrize like k8s_bundle(context=parametrize("stg", "prd"))
For CI agents which will only have access to a single context, set the kubectl.pass_context to false to have them use their default context.

I was thinking of how the Helm backend does it.

We have the exact same problem in helm backend. I had to vendor it and keep maintaining the patch that adds the context requirement

cburroughs · 2025-01-09T20:43:00Z

Haha, we have exactly this problem, I've called the goal render and implemented it for helm_deployment and k8s_bundle

I'd be excited to see that!

grihabor

@lilatomic Thank you for review! Ready for another round

grihabor · 2025-01-11T14:20:56Z

src/python/pants/backend/k8s/goals/tailor.py

+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")
+        all_k8s_files = await Get(Paths, PathGlobs, all_k8s_files_globs)
+        unowned_k8s_files = set(all_k8s_files.files) - set(all_owned_sources)


I'm thinking 2 most common use cases here:

Developer's repo with code will probably use some kind of templating to generate k8s_sources (I'm planning to open couple more prs for that), so tailor for yaml won't be that useful. In this case I expect devs to disable tailoring.

DevOps's repo with mostly k8s yaml files. In this case I expect tailor to work good

grihabor · 2025-01-11T14:22:18Z

src/python/pants/backend/k8s/goals/tailor.py

+    putative_targets = []
+
+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")


Personally, I want consistent .yaml or .yml in the whole repo, so probably it should be customisable, yes

grihabor · 2025-01-11T14:28:41Z

build-support/bin/external_tool_versions.py

+
+
+@dataclass(frozen=True)
+class VersionHash:


It's not quite the same thing, because platform here is not a predefined platform that pants uses, but a platform that needs to be mapped using url_platform_mapping. But I can make it work like this 6efda67

grihabor · 2025-01-11T21:04:17Z

build-support/bin/external_tool_versions.py

+            )
+
+    backward_platform_mapping = {v: k for k, v in platform_mapping.items()}
+    for result in results:


Sure 9d84c9e

src/python/pants/backend/k8s/goals/deploy.py

grihabor · 2025-01-11T21:14:27Z

src/python/pants/backend/k8s/goals/deploy.py

+    platform: Platform,
+) -> DeployProcess:
+    context = field_set.context.value
+    if context is None:


The idea is that setting context on a target is required. This is because we had recurring issues because somebody forgot to set the correct context on the target and it got deployed to a different context by accident. However, we've got CI agents running in the cluster itself, so they can only deploy k8s objects to the specific cluster specified with KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, that's why we need to disable --context argument in pants.ci.toml.

I understand this is kinda custom setup, so people might need some other behavior. However, in general, I think context requirement is a good thing that can prevent misdeployments. I can probably move this context validation to a small linter and check context field there, wdyt?

src/python/pants/backend/k8s/goals/deploy.py

grihabor · 2025-01-12T09:14:30Z

native_engine.IntrinsicError: Could not identify a process to backtrack to for: Missing digest: Was not present in the local store: Digest { hash: Fingerprint<91284dcbaa6e3a5ff00b04f4e5e1051626fc374e386a437aa899983a6eeaaf5a>, size_bytes: 1351 }, with workunit: Workunit { name: "process", level: Debug, span_id: SpanId(12402687231280246273), parent_ids: [SpanId(7266305093486847294)], state: Started { start_time: SystemTime { tv_sec: 1736631333, tv_nsec: 59791000 }, blocked: false }, metadata: Some(WorkunitMetadata { desc: Some("Scheduling: Resolving plugins: hdrhistogram"), message: None, stdout: None, stderr: None, artifacts: [], user_metadata: [] }) }

Looks like some flaky test

grihabor · 2025-01-13T10:00:56Z

Opened a pr for python_format_string grihabor#3

tdyas · 2025-01-25T14:53:31Z

3rdparty/python/requirements.txt

+# Only used in build-support/
+tqdm~=4.67.1
+types-tqdm


Why not put these into their own resolve if they are only used internally in the repository? (For example, the pbs-script resolve exists for this reason for the src/python/pants/backend/python/providers/python_build_standalone/scripts/generate_urls.py script.)

Do you want me to parametrize the whole src/python/pants like this?

**parametrize("python-default", resolve="python-default"), **parametrize("build-support", resolve="build-support"),

I'm importing ExternalTool-s in the pex, so the sources have to be present in the resolve.

I believe all the other not-really-requirements are in this requirements.txt exactly because of this problem

Anyway, I think it makes sense to move this build-support script to a separate pr f87cb21

tdyas · 2025-01-25T15:00:57Z

src/python/pants/backend/k8s/goals/tailor.py

+    putative_targets = []
+
+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")


I agree on this being initially set to find both *.yaml and *.yml files. Users of this could end up confused why tailor is not finding their .yml files.

tdyas · 2025-01-25T15:13:02Z

src/python/pants/backend/k8s/goals/tailor.py

+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")
+        all_k8s_files = await Get(Paths, PathGlobs, all_k8s_files_globs)
+        unowned_k8s_files = set(all_k8s_files.files) - set(all_owned_sources)


Why not just also scan the file for the apiVersion string as a heuristic? The go tailor rules use a similar pattern to search for package main to find potential putative go_binary targets.

pants/src/python/pants/backend/go/goals/tailor.py

Line 176 in 5f3f251

if has_package_main(file_content.content) and has_go_mod_ancestor(dirname, all_go_mod_dirs):

tdyas · 2025-01-25T15:14:22Z

src/python/pants/backend/k8s/goals/tailor.py

+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")
+        all_k8s_files = await Get(Paths, PathGlobs, all_k8s_files_globs)
+        unowned_k8s_files = set(all_k8s_files.files) - set(all_owned_sources)


Also, given that YAML is used for many other uses, I'm thinking that any user with this backend enabled is probably going to become annoyed if tailor creates false positive k8s_source targets every time.

tdyas · 2025-01-25T15:15:22Z

src/python/pants/backend/k8s/goals/tailor.py

+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")
+        all_k8s_files = await Get(Paths, PathGlobs, all_k8s_files_globs)
+        unowned_k8s_files = set(all_k8s_files.files) - set(all_owned_sources)


Also, would it make sense to have an option to restrict tailor usage to only certain directory trees?

huonw · 2025-02-06T03:20:02Z

If/when this PR becomes ready, we've just branched for 2.25, so merging this pull request now will come out in 2.26, please move the release notes updates to docs/notes/2.26.x.md. Thank you!

> fd _category_ docs/docs/ --max-depth 2 --exec rg position '{}' --with-filename --line-number | sort -k 3,3 -n docs/docs/introduction/_category_.json:3: "position": 1 docs/docs/getting-started/_category_.json:3: "position": 2 docs/docs/using-pants/_category_.json:3: "position": 3 docs/docs/python/_category_.json:3: "position": 4 docs/docs/go/_category_.json:3: "position": 5 docs/docs/jvm/_category_.json:3: "position": 6 docs/docs/shell/_category_.json:3: "position": 7 docs/docs/docker/_category_.json:3: "position": 8 docs/docs/kubernetes/_category_.json:3: "position": 9 docs/docs/helm/_category_.json:3: "position": 10 docs/docs/terraform/_category_.json:3: "position": 11 docs/docs/sql/_category_.json:3: "position": 12 docs/docs/ad-hoc-tools/_category_.json:3: "position": 13 docs/docs/javascript/_category_.json:3: "position": 13 docs/docs/writing-plugins/_category_.json:3: "position": 14 docs/docs/releases/_category_.json:3: "position": 15 docs/docs/contributions/_category_.json:3: "position": 16 docs/docs/tutorials/_category_.json:3: "position": 17

Co-authored-by: Daniel Goldman <[email protected]>

grihabor · 2025-02-08T19:59:57Z

@tdyas Thank you for review! I decided to remove some features for now and open separate prs for those. Ready for another round
cc @lilatomic @cburroughs

cburroughs · 2025-02-18T21:49:10Z

@grihabor are you looking for any design feedback at this point, or is this good to go from your point of view?

grihabor · 2025-02-18T22:01:36Z

@cburroughs We're heavily using the plugin with this design internally, so it would be nice to keep it roughly as is. But you can definitely share your thoughts if you have reasons to tweak something or you have some features you have in mind that would be impossible with this design. But yeah, I think we're good to go (I'm planning to open a couple more prs that generate k8s_sources and allow more flexible configuration)

grihabor · 2025-02-18T22:03:19Z

Looks like the check on macos failed because of some flaky behavior

native_engine.IntrinsicError: Could not identify a process to backtrack to for: Missing digest: Was not present in the local store: Digest { hash: Fingerprint<34532e8789e31738f75991de8002fee62b47d981c6511710e43500a85859298d>, size_bytes: 1309 }, with workunit: Workunit { name: "process", level: Debug, span_id: SpanId(2516749030027008830), parent_ids: [SpanId(8292622773906736883)], state: Started { start_time: SystemTime { tv_sec: 1739915114, tv_nsec: 284430000 }, blocked: false }, metadata: Some(WorkunitMetadata { desc: Some("Scheduling: Resolving plugins: hdrhistogram"), message: None, stdout: None, stderr: None, artifacts: [], user_metadata: [] }) }

cburroughs · 2025-02-19T15:52:56Z

This had sat for a while with and was trying to shepherd it along. I don't have any concerns with the design and am personally excited to see the other PRs you mentioned.

Thanks for the patches!

grihabor · 2025-03-05T11:56:48Z

Here is the pr with python_format_string: #22034

lilatomic reviewed Jan 8, 2025

View reviewed changes

grihabor commented Jan 11, 2025

View reviewed changes

grihabor mentioned this pull request Jan 12, 2025

New python_format_string codegen for k8s backend grihabor/pants#3

Draft

tdyas reviewed Jan 25, 2025

View reviewed changes

grihabor and others added 20 commits February 8, 2025 19:47

Copy internal plugin as is

6757da3

Fix linters

b0a61c8

Fix env var

c1aef66

Add a test

4fe0c5b

Fixing the test...

7a87934

Download kubectl via TemplatedExternalTool

c59e35e

Fix the tests

5b20a9d

Implement tailor goal

ce05f9f

Add a script to fetch kubectl versions

0eb4121

Add more known versions

4a6930a

Add more docs

bda87ec

Add notes

f915330

pants fix

6a8d143

Fix mypy

0af1af0

Reuse ExternalToolVersion

20882e0

Update build-support/bin/external_tool_versions.py

b9d298e

Co-authored-by: Daniel Goldman <[email protected]>

Sort versions, show progress via tqdm

7787fa3

Better error message

5cc12b8

Use immutable_input_digests for kubectl tool

fb136c6

grihabor added 5 commits February 8, 2025 20:28

Add types-tqdm

5e5a70d

Update k8s docs

d394df0

Sort versions, include only >=1.21

1a75443

Fix exe path

5c882d8

Update exe path

13b880f

grihabor force-pushed the k8s-backend branch from 1b32f76 to 13b880f Compare February 8, 2025 19:28

grihabor added 4 commits February 8, 2025 20:30

Move notes to 2.26

0ff22c1

Temporarily remove tailor goal implementation

12ad882

Add note about the required context field

6058ef9

Temporarily remove external_tool_versions script

f87cb21

grihabor requested review from tdyas and lilatomic February 8, 2025 20:00

grihabor added 2 commits February 9, 2025 15:43

Fix linters

83f5f7d

Merge branch 'main' into k8s-backend

45a786b

tdyas approved these changes Feb 18, 2025

View reviewed changes

tdyas merged commit 4c3418c into pantsbuild:main Feb 19, 2025
24 checks passed



		@dataclass(frozen=True)
		class VersionHash:



		@dataclass(frozen=True)
		class VersionHash:

New kubernetes backend #21796

New kubernetes backend #21796

Conversation

grihabor commented Dec 22, 2024 • edited Loading

cburroughs commented Jan 7, 2025

grihabor commented Jan 8, 2025

lilatomic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cburroughs commented Jan 9, 2025

grihabor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grihabor commented Jan 12, 2025

grihabor commented Jan 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huonw commented Feb 6, 2025

grihabor commented Feb 8, 2025

cburroughs commented Feb 18, 2025

grihabor commented Feb 18, 2025

grihabor commented Feb 18, 2025

cburroughs commented Feb 19, 2025

grihabor commented Mar 5, 2025

grihabor commented Dec 22, 2024 •

edited

Loading