kustomize feature of kubectl does not function

[darrenstarr]

Summary

microk8s kubectl apply -k

Has issues resolving paths.

What Should Happen Instead?

I discovered this problem while trying to deploy awx on microk8s. See the installation instructions here :
https://github.com/ansible/awx-operator#basic-install

The correct behavior is that when I execute

kubectl apply -k .

in a directory containing a properly formatted kustomization.yaml file, the awx deployment should take effect.

What is seen however is the following

root@microk8s:~# microk8s kubectl apply -k .
error: accumulating resources: accumulation err='accumulating resources from 'github.com/ansible/awx-operator/config/default?ref=2.2.1': evalsymlink failure on '/root/github.com/ansible/awx-operator/config/default?ref=2.2.1' : lstat /root/github.com: no such file or directory': git cmd = '/snap/microk8s/5219/usr/bin/git fetch --depth=1 origin 2.2.1': exit status 128

Reproduction Steps

1. Install microk8s on a fresh Ubuntu 22.04 LTS system using snap
2. Create a file named kustomization.yaml containing the following text

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.2.1
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.2.1
namespace: awx

3. execute microk8s kubectl apply -k .

Introspection Report

I don't have a fresh installation, the reproduction steps are consistent on every machine I've tried.

Can you suggest a fix?

I'm not sure about a fix, but this is a temporary work around.

Install kustomize on its own following the instructions on their website.

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  |  bash

Which installs kustomize in the current directory. Then use kustomize in "legacy form"

./kustomize build | microk8s kubectl apply -f -

Are you interested in contributing with a fix?

no, thought I am willing to test the fix if one is provided.

[neoaggelos]

Hi @darrenstarr thank you for raising this.

I am not able to reproduce on 1.26 or 1.27:

sudo snap install microk8s --classic
mkdir dir
cd dir
echo '
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.2.1
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.2.1
namespace: awx
' | tee kustomization.yaml
microk8s kubectl apply -k .

This works without issues:

namespace/awx created
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created
configmap/awx-operator-awx-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created

Which leads me to think that this is probably an environment issue. What OS is this on, and which MicroK8s version are you running?

[k1n6b0b]

I'm also having this issue, i believe its the git version with microk8s 1.27.1

# microk8s version
MicroK8s v1.27.1 revision 5250

This fails

# /snap/microk8s/5250/usr/bin/git --version
git version 2.25.1

This works

# /usr/bin/git --version
git version 2.34.1

If i try with a remote repository, git fails

# microk8s kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=main'             
error: failed to run '/snap/microk8s/5250/usr/bin/git fetch --depth=1 https://github.com/intel/intel-device-plugins-for-kubernetes main': fatal: couldn't find remote ref main
: exit status 128

If i use git from my local os bin (Ubuntu 22.04) and clone the repository, it reads the customization file but failes on the next remote repo it trys to grab

# microk8s kubectl apply -k .
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
error: accumulating resources: accumulation err='accumulating resources from 'base': '/home/[redacted]/intel-device-plugins-for-kubernetes/deployments/nfd/base' must resolve to a file': recursed accumulation of path '/home/[redacted]/intel-device-plugins-for-kubernetes/deployments/nfd/base': accumulating resources: accumulation err='accumulating resources from 'https://github.com/kubernetes-sigs/node-feature-discovery/deployment/overlays/default?ref=v0.13.1': URL is a git repository': failed to run '/snap/microk8s/5250/usr/bin/git fetch --depth=1 https://github.com/kubernetes-sigs/node-feature-discovery v0.13.1': fatal: couldn't find remote ref v0.13.1
: exit status 128

Is there any way to change the version or location for git used by microk8s? I tried but the common folder is read only? (I'm pretty novice with microk8s)

@darrenstarr do you think this is what you're seeing?

[k1n6b0b]

This workaround for snap allowed me to deploy via kustomize: https://askubuntu.com/questions/919091/why-cant-snap-files-be-modified-in-any-way

# sudo mount --bind -o nodev,ro /usr/bin /snap/microk8s/5250/usr/bin
# /snap/microk8s/5250/usr/bin/git --version
git version 2.34.1

then

# microk8s kubectl apply -k "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=main"
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
namespace/node-feature-discovery created
customresourcedefinition.apiextensions.k8s.io/nodefeaturerules.nfd.k8s-sigs.io created
customresourcedefinition.apiextensions.k8s.io/nodefeatures.nfd.k8s-sigs.io created
serviceaccount/nfd-master created
serviceaccount/nfd-worker created
role.rbac.authorization.k8s.io/nfd-worker created
clusterrole.rbac.authorization.k8s.io/nfd-master created
rolebinding.rbac.authorization.k8s.io/nfd-worker created
clusterrolebinding.rbac.authorization.k8s.io/nfd-master created
configmap/nfd-master-conf created
configmap/nfd-worker-conf created
service/nfd-master created
deployment.apps/nfd-master created
daemonset.apps/nfd-worker created

For me at least, it was a git version issue

[zdev-code]

Brand new install, only thing I did differently from @neoaggelos was to follow the official install guide and run microk8s enable dashboard dns registry istio

[neoaggelos]

@k1n6b0b OK, I was able to reproduce this, the git version difference was enlightening.

I have not dived deep into this, but it appears that the kustomization package is using some git features that are not available in 2.34.1 but not 2.25.1. I am not sure what the easiest way would be to bump this version, since microk8s is simply shipping the latest git from the ubuntu 20.04 archives.

One workaround for this would be to independently install kubectl on your host system and not rely on microk8s kubectl.

[kriegster108]

bumping, this is a problem and should be fixed if possible.

[smarthusker]

This workaround for snap allowed me to deploy via kustomize: https://askubuntu.com/questions/919091/why-cant-snap-files-be-modified-in-any-way

# sudo mount --bind -o nodev,ro /usr/bin /snap/microk8s/5250/usr/bin
# /snap/microk8s/5250/usr/bin/git --version
git version 2.34.1

then

# microk8s kubectl apply -k "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=main"
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
namespace/node-feature-discovery created
customresourcedefinition.apiextensions.k8s.io/nodefeaturerules.nfd.k8s-sigs.io created
customresourcedefinition.apiextensions.k8s.io/nodefeatures.nfd.k8s-sigs.io created
serviceaccount/nfd-master created
serviceaccount/nfd-worker created
role.rbac.authorization.k8s.io/nfd-worker created
clusterrole.rbac.authorization.k8s.io/nfd-master created
rolebinding.rbac.authorization.k8s.io/nfd-worker created
clusterrolebinding.rbac.authorization.k8s.io/nfd-master created
configmap/nfd-master-conf created
configmap/nfd-worker-conf created
service/nfd-master created
deployment.apps/nfd-master created
daemonset.apps/nfd-worker created

For me at least, it was a git version issue

Worked for me as well. Runnig microk8s v1.28.4 Rev 6227.

[rmpinchback]

This workaround for snap allowed me to deploy via kustomize: https://askubuntu.com/questions/919091/why-cant-snap-files-be-modified-in-any-way

# sudo mount --bind -o nodev,ro /usr/bin /snap/microk8s/5250/usr/bin
# /snap/microk8s/5250/usr/bin/git --version
git version 2.34.1

For me at least, it was a git version issue

Worked for me as well. Runnig microk8s v1.28.4 Rev 6227.

I discovered a cleaner solution to this that doesn't require mounting anything, at least for anybody that is writing scripts for microk8s operations anyways. Or if not cleaner, at least reflects where the breakage is taking place.

The issue is not the git version, quite.

The issue where a particular git version finds git-core. In other words, whatever is reported by: git --exec-path

The microk8s snap provides its own git-core under /snap/microk8s/current/usr/lib. Unfortunately, the git executable that the snap provides via /snap/microk8s/current/usr/bin does NOT resolve to that git-core, it instead resolves to /usr/lib/git-core. Then somehow (specifics unclear to me) the /usr/lib/git-core tries to use the dynamically linked libraries within the snap. It tries to use the pthread libraries within /snap/core20/current/usr/lib/x86_64-linux-gnu instead of the ones within ls /usr/lib/x86_64-linux-gnu.

For anybody who ends up hitting this issue either for the description above or because they were Googling on the pthreads complaint that you see in other issues, this is the problem. Yes the git versions are different, but it is the hip-check out to the wrong git-core, then back to the wrong libraries, that causes the failure.

A workaround and easy way to test a fix is to prefix any microk8s command with GIT_EXEC_PATH=/snap/microk8s/current/usr/lib/git-core

Example:
GIT_EXEC_PATH=/snap/microk8s/current/usr/lib/git-core microk8s helm plugin install https://github.com/HamzaZo/helm-adopt

Without that GIT_EXEC_PATH that command would fail on Ubuntu 22.04 on either a 1.29 or 1.30 channel. With it, the command works just fine.

The proper fix (not being somebody who ever dug into git source) is probably for Canonical to rebuilt git from source with some setting that fixes the exec path to point within the snap, instead of it falling through a default that calls back something outside the snap that was installed by apt.

Also, for anybody who tries the solution that was suggested by @smarthusker note that, maybe it worked before, but it doesn't work now. Using a mount to overlay the apt-installed git over the snap-installed git fails for multiple reasons: it mucks up some other microk8s operations, and you still run into the problem of where git-core is resolved vs the libraries linked to.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.