|
| 1 | +# Gathering data with must-gather |
| 2 | + |
| 3 | +`must-gather` is an embedded tool in Scylla Operator that helps collecting all the necessary info when something goes wrong. |
| 4 | + |
| 5 | +The tool talks to the Kubernetes API, retrieves a predefined set of resources and saves them into a folder in your current directory. |
| 6 | +By default, all collected Secrets are censored to avoid sending sensitive data. |
| 7 | +That said, you can always review the archive before you attach it to an issue or your support request. |
| 8 | + |
| 9 | +Given it needs to talk to the Kubernetes API, at the very least, you need to supply the `--kubeconfig` flag with a path to the kubeconfig file for your Kubernetes cluster, or set the `KUBECONFIG` environment variable. |
| 10 | + |
| 11 | +## Running must-gather |
| 12 | + |
| 13 | +There is more than one way to run `must-gather`. |
| 14 | +Here are some examples of how you can run the tool. |
| 15 | + |
| 16 | +### Prerequisites |
| 17 | + |
| 18 | +All examples assume you have exported `KUBECONFIG` environment variable that points to a kubeconfig file on your machine. |
| 19 | +If not, you can run this command to export the common default location. |
| 20 | +Please make sure such a file exists. |
| 21 | + |
| 22 | +```bash |
| 23 | +export KUBECONFIG=~/.kube/config |
| 24 | +ls -l "${KUBECONFIG}" |
| 25 | +``` |
| 26 | + |
| 27 | +```note:: |
| 28 | + There can be slight deviations in the arguments for your container tool, depending on the container runtime, whether you use SELinux or similar factors. |
| 29 | +
|
| 30 | + As an example, the need for the `Z` option on volume mounts depends on whether you use SELinux and what context is applied on your file or directory. |
| 31 | + If you get an error mentioning `Error: lsetxattr <path>: operation not supported`, try it without the `Z` option. |
| 32 | +``` |
| 33 | + |
| 34 | +Let's also check whether your kubeconfig uses [external authentication plugin](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins). |
| 35 | +You can determine that by running |
| 36 | +```bash |
| 37 | +kubectl config view --minify |
| 38 | +``` |
| 39 | +and checking whether it uses an external exec plugin by looking for this pattern (containing the `exec` key) |
| 40 | +```yaml |
| 41 | +users: |
| 42 | +- name: <user_name> |
| 43 | + user: |
| 44 | + exec: |
| 45 | +``` |
| 46 | +If not, you can skip the rest of this section. |
| 47 | +
|
| 48 | +In case your kubeconfig depends on external binaries, you have to take a few extra steps because the external binary won't be available within our container to authenticate the requests. |
| 49 | +
|
| 50 | +Similarly to how Pods are run within Kubernetes, we'll create a dedicated ServiceAccount for must-gather and use it to run the tool. |
| 51 | +(When you are done using it, feel free to remove the Kubernetes resources created for that purpose.) |
| 52 | +
|
| 53 | +```bash |
| 54 | +kubectl create namespace must-gather |
| 55 | +kubectl -n must-gather create serviceaccount must-gather |
| 56 | +kubectl create clusterrolebinding must-gather --clusterrole=cluster-admin --serviceaccount=must-gather:must-gather |
| 57 | +export MUST_GATHER_TOKEN |
| 58 | +MUST_GATHER_TOKEN=$( kubectl -n must-gather create token must-gather --duration=1h ) |
| 59 | +kubeconfig=$( mktemp ) |
| 60 | +# Create a copy of the existing kubeconfig and |
| 61 | +# replace user authentication using yq, or by adjusting the fields manually. |
| 62 | +kubectl config view --minify --raw -o yaml | yq -e '.users[0].user = {"token": env(MUST_GATHER_TOKEN)}' > "${kubeconfig}" |
| 63 | +KUBECONFIG="${kubeconfig}" |
| 64 | +``` |
| 65 | + |
| 66 | +```note:: |
| 67 | + If you don't have `yq` installed, you can get it at https://github.com/mikefarah/yq/#install or you can replace the user authentication settings manually. |
| 68 | +``` |
| 69 | + |
| 70 | +### Podman |
| 71 | +```bash |
| 72 | +podman run -it --pull=always --rm -v="${KUBECONFIG}:/kubeconfig:ro,Z" -v="$( pwd ):/workspace:Z" --workdir=/workspace docker.io/scylladb/scylla-operator:latest must-gather --kubeconfig=/kubeconfig |
| 73 | +``` |
| 74 | + |
| 75 | +### Docker |
| 76 | +```bash |
| 77 | +docker run -it --pull=always --rm -v="${KUBECONFIG}:/kubeconfig:ro" -v="$( pwd ):/workspace" --workdir=/workspace docker.io/scylladb/scylla-operator:latest must-gather --kubeconfig=/kubeconfig |
| 78 | +``` |
| 79 | + |
| 80 | +## Limiting must-gather to a particular namespace |
| 81 | + |
| 82 | +If you are running a large Kubernetes cluster with many ScyllaClusters, it may be useful to limit the collection of ScyllaClusters to a particular namespace. |
| 83 | +Unless you hit scale issues, we advise not to use this mode, as sometimes the ScyllaClusters affect other collected resources, like the manager or they form a multi-datacenter. |
| 84 | + |
| 85 | +```bash |
| 86 | +scylla-operator must-gather --namespace="<namespace_with_broken_scyllacluster>" |
| 87 | +``` |
| 88 | + |
| 89 | +```note:: |
| 90 | + The `--namespace` flag affects only `ScyllaClusters`. |
| 91 | + Other resources related to the operator installation or cluster state will still be collected from other namespaces. |
| 92 | +``` |
| 93 | + |
| 94 | +### Collecting every resource in the cluster |
| 95 | + |
| 96 | +By default, `must-gather` collects only a predefined subset of resources. |
| 97 | +You can also request collecting every resource in the Kubernetes API, if the default set wouldn't be enough to debug an issue. |
| 98 | + |
| 99 | +```bash |
| 100 | +scylla-operator must-gather --all-resources |
| 101 | +``` |
![scylla-operator-bot[bot]](https://avatars.githubusercontent.com/in/112385?v=4&size=40){kind=avatar}
0 commit comments