Backup partially fails: error taking snapshot of volume: error reading from server: EOF #182
Description
What steps did you take and what happened:
I followed the various steps from velero and this repo documentations to create a remote snapshot location to a minio instance running on a machine outside the cluster.
I deployed a couple of services and tried to do a backup, unfortunately I am facing an error. The backup reports as Partially Failed and the logs shows that certain volumes couldn't be backed up with the following error:
time="2022-10-13T12:42:00Z" level=warning msg="Epoll wait failed : interrupted system call" backup=velero/test3 cmd=/plugins/velero-blockstore-openebs logSource="/go/src/github.com/openebs/velero-plugin/pkg/clouduploader/server.go
:302" pluginName=velero-blockstore-openebs
<... can be 50+ similar log entries and then ...>
time="2022-10-13T12:42:00Z" level=info msg="1 errors encountered backup up item" backup=velero/test3 logSource="pkg/backup/backup.go:413" name=registry-server-7d5466494d-84fj5
time="2022-10-13T12:42:00Z" level=error msg="Error backing up item" backup=velero/test3 error="error taking snapshot of volume: rpc error: code = Unavailable desc = error reading from server: EOF" logSource="pkg/backup/backup.go:4
17" name=registry-server-7d5466494d-84fj5
What did you expect to happen:
I expect the backup to be successful
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)
kubectl logs deployment/velero -n velero-> herekubectl logs deployment/maya-apiserver -n openebs-> N/Avelero backup describe <backupname>Name: test3 Namespace: velero Labels: velero.io/storage-location=local-backup Annotations: velero.io/source-cluster-k8s-gitversion=v1.25.2 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=25 Phase: PartiallyFailed (run `velero backup logs test3` for more information) Errors: 4 Warnings: 165 Namespaces: Included: * Excluded: <none> Resources: Included: * Excluded: <none> Cluster-scoped: auto Label selector: <none> Storage Location: local-backup Velero-Native Snapshot PVs: auto TTL: 720h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2022-10-13 13:41:29 +0100 BST Completed: 2022-10-13 13:42:38 +0100 BST Expiration: 2022-11-12 12:41:28 +0000 GMT Total items to be backed up: 767 Items backed up: 767 Velero-Native Snapshots: 1 of 5 snapshots completed successfully (specify --details for more information) CSI Volume Snapshots: <none included>velero backup logs <backupname>-> here
Anything else you would like to add:
Currently migrating a 25 nodes cluster to k8s. This is the initial setup/test before definitive migration with just 5 nodes, self-hosted microk8s cluster running cStor
Environment:
- Velero version (use
velero version):Client: Version: v1.9.2 Git commit: - Server: Version: v1.9.2 - Velero features (use
velero client config get features):features: EnableCSI - Velero-plugin version:
3.3.0 - OpenEBS version:
3.3.0 - Kubernetes version (use
kubectl version):1.25 - Kubernetes installer & version:
- Cloud provider or hardware configuration: 5 nodes, home made from consumer products, bought over the years so nodes are slightly different
- Intel i7 or i9 from 8th to 10th Gen
- 16GB to 32GB
- 2 nodes have 3x 1TB SSD for cStor
- OS (e.g. from
/etc/os-release):
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy