PV fails to mount NFS running on host.k3d.internal #892

nwithers-ecr · 2021-12-13T18:16:06Z

nwithers-ecr
Dec 13, 2021

What did you do

How was the cluster created?
- k3d cluster create -c k3d.yaml (runs with sudo, and without, fail the same way)

k3d.yaml

---
apiVersion: k3d.io/v1alpha3
kind: Simple
name: devops-local-nloop-00
image: +v1.20
servers: 1
agents: 3
registries:
  create:
    name: k3d-local
ports:
- port: 80:80
  nodeFilters:
    - loadbalancer
- port: 443:443
  nodeFilters:
    - loadbalancer
options:
  k3s:
    extraArgs:
      - arg: --no-deploy=traefik
        nodeFilters:
          - server:*

What did you do afterwards?
- skaffold dev --force=false

What did you expect to happen

Persistent volume mounts NFS in pod

Screenshots or terminal output

tail /etc/exports -n 2

nfs-common is installed via the Dockerfile

/mnt/devops-local-nloop-00-server 127.0.0.1(rw,sync,no_subtree_check)
/mnt/devops-local-nloop-00-server 172.19.0.0/24(rw,sync,no_subtree_check)

kubectl get node -o wide | awk '!/NAME/ {print $1FS$6}'

k3d-devops-local-nloop-00-agent-0 172.19.0.2
k3d-devops-local-nloop-00-agent-1 172.19.0.4
k3d-devops-local-nloop-00-server-0 172.19.0.3
k3d-devops-local-nloop-00-agent-2 172.19.0.5

kubectl describe pv document-generation-engine-datastore

Name:            document-generation-engine-datastore
Labels:          app.kubernetes.io/managed-by=Helm
                 skaffold.dev/run-id=ac0421ed-2d5f-4c57-8dc5-719c4cfab068
Annotations:     meta.helm.sh/release-name: nloop-overlay
                 meta.helm.sh/release-namespace: nloop-overlay
                 pv.kubernetes.io/bound-by-controller: yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:
Status:          Bound
Claim:           nloop-overlay/document-generation-engine-datastore
Reclaim Policy:  Retain
Access Modes:    RWX
VolumeMode:      Filesystem
Capacity:        10Gi
Node Affinity:   <none>
Message:
Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    host.k3d.internal
    Path:      /mnt/devops-local-nloop-00-server
    ReadOnly:  false
Events:        <none>

kubectl describe pvc document-generation-engine-datastore

Name:          document-generation-engine-datastore
Namespace:     nloop-overlay
StorageClass:
Status:        Bound
Volume:        document-generation-engine-datastore
Labels:        app.kubernetes.io/managed-by=Helm
               skaffold.dev/run-id=ac0421ed-2d5f-4c57-8dc5-719c4cfab068
Annotations:   meta.helm.sh/release-name: nloop-overlay
               meta.helm.sh/release-namespace: nloop-overlay
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       core-77786dbcc9-d2wbd
               document-generation-engine-5d8cfd54fd-2sgmh
Events:        <none>

kubectl desctibe pod core-7b65d64b9b-tp72k

Events:
  Type     Reason       Age    From               Message
  ----     ------       ----   ----               -------
  Normal   Scheduled    4m25s  default-scheduler  Successfully assigned nloop-overlay/core-77786dbcc9-d2wbd to k3d-devops-local-nloop-00-server-0
  Warning  FailedMount  84s    kubelet            MountVolume.SetUp failed for volume "document-generation-engine-datastore" : mount failed: exit status 255
Mounting command: mount
Mounting arguments: -t nfs -o nfsvers=4.2 host.k3d.internal:/mnt/devops-local-nloop-00-server /var/lib/kubelet/pods/8e09c42b-0377-4722-a7d8-7eac5abf0333/volumes/kubernetes.io~nfs/document-generation-engine-datastore
Output: mount: mounting host.k3d.internal:/mnt/devops-local-nloop-00-server on /var/lib/kubelet/pods/8e09c42b-0377-4722-a7d8-7eac5abf0333/volumes/kubernetes.io~nfs/document-generation-engine-datastore failed: Operation timed out
  Warning  FailedMount  3s (x2 over 2m22s)  kubelet  Unable to attach or mount volumes: unmounted volumes=[document-generation-engine-datastore], unattached volumes=[document-generation-engine-datastore default-token-v7fll]: timed out waiting for the condition

kubectl exec -it document-generation-engine-776bc48959-jbcsh -- dig host.k3d.internal

; <<>> DiG 9.16.22-Debian <<>> host.k3d.internal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 87
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 84ae7af9189edb93 (echoed)
;; QUESTION SECTION:
;host.k3d.internal.             IN      A

;; ANSWER SECTION:
host.k3d.internal.      14      IN      A       172.19.0.1

;; Query time: 0 msec
;; SERVER: 10.43.0.10#53(10.43.0.10)
;; WHEN: Mon Dec 13 15:33:07 UTC 2021
;; MSG SIZE  rcvd: 91

At this point I tried using a dynamic provisioner instead. however, even the provisioner fails to mount with the same error

helm upgrade -i nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --set nfs.server=host.k3d.internal --set nfs.path=/mnt/devops-local-nloop-00-server

Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    4m41s                default-scheduler  Successfully assigned nloop-overlay/nfs-subdir-external-provisioner-5f88856969-lwdfk to k3d-devops-local-nloop-00-agent-0
  Warning  FailedMount  34s (x4 over 3m40s)  kubelet            MountVolume.SetUp failed for volume "nfs-subdir-external-provisioner-root" : mount failed: exit status 255
Mounting command: mount
Mounting arguments: -t nfs host.k3d.internal:/mnt/devops-local-nloop-00-server /var/lib/kubelet/pods/f6218464-175e-469d-a892-9bdc41d1090a/volumes/kubernetes.io~nfs/nfs-subdir-external-provisioner-root
Output: mount: mounting host.k3d.internal:/mnt/devops-local-nloop-00-server on /var/lib/kubelet/pods/f6218464-175e-469d-a892-9bdc41d1090a/volumes/kubernetes.io~nfs/nfs-subdir-external-provisioner-root failed: Operation timed out
  Warning  FailedMount  20s (x2 over 2m38s)  kubelet  Unable to attach or mount volumes: unmounted volumes=[nfs-subdir-external-provisioner-root], unattached volumes=[nfs-subdir-external-provisioner-root nfs-subdir-external-provisioner-token-sfx7r]: timed out waiting for the condition

Which OS & Architecture

Linux DaringLion 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Which version of `k3d`

k3d version v5.1.0
k3s version v1.21.5-k3s2 (default)

docker

docker version

Client: Docker Engine - Community
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        dea9396
 Built:             Thu Nov 18 00:37:06 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       847da18
  Built:            Thu Nov 18 00:35:15 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.9.0)

Server:
 Containers: 21
  Running: 6
  Paused: 0
  Stopped: 15
 Images: 1888
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-91-generic
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 62.67GiB
 Name: DaringLion
 ID: N2MD:XACX:AT4A:ADPS:PHN6:6CUS:YR6H:VOWC:SN6M:YZT4:U6EW:X6AP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: nwithers
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  safe-registry:34443
  127.0.0.0/8
 Live Restore Enabled: false

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

PV fails to mount NFS running on host.k3d.internal #892

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

Uh oh!

PV fails to mount NFS running on host.k3d.internal #892

Uh oh!

Uh oh!

nwithers-ecr Dec 13, 2021

What did you do

k3d.yaml

What did you expect to happen

Screenshots or terminal output

tail /etc/exports -n 2

nfs-common is installed via the Dockerfile

kubectl get node -o wide | awk '!/NAME/ {print $1FS$6}'

At this point I tried using a dynamic provisioner instead. however, even the provisioner fails to mount with the same error

helm upgrade -i nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --set nfs.server=host.k3d.internal --set nfs.path=/mnt/devops-local-nloop-00-server

Which OS & Architecture

Which version of k3d

docker

Replies: 0 comments

nwithers-ecr
Dec 13, 2021

Which version of `k3d`