|
9 | 9 |
|
10 | 10 | * DNSPerf 2.14.0
|
11 | 11 |
|
12 |
| -## Preflight checklist |
| 12 | +## 🔥 Stress Test 🔥 Benchmark with Kubernetes Pods |
13 | 13 |
|
14 |
| -### Does CoreDNS running expected version? |
| 14 | +Apply pre-configured testing deployment/pods |
15 | 15 |
|
16 |
| - $ kubectl get deployments coredns -n kube-system -o jsonpath='{$.spec.template.spec.containers[0].image}' |
| 16 | +```bash |
| 17 | +kubectl apply -f https://raw.githubusercontent.com/guessi/dnsperf-bench/main/k8s-dnsperf-bench.yaml |
| 18 | +``` |
17 | 19 |
|
18 |
| -### Does CoreDNS running with expected Corefile? |
| 20 | +Make sure the deployment is running as expected |
19 | 21 |
|
20 |
| - $ kubectl get configmap coredns -n kube-system -o jsonpath='{$.data.Corefile}' |
21 | 22 |
|
22 |
| -### Does CoreDNS running with correct resources configuration? |
| 23 | +```bash |
| 24 | +kubectl get pods -l app=dnsperf |
| 25 | +NAME READY STATUS RESTARTS AGE |
| 26 | +dnsperf-7b9cc5b497-d5nfs 1/1 Running 0 1m16s |
| 27 | +``` |
23 | 28 |
|
24 |
| - $ kubectl get deployments coredns -n kube-system -o jsonpath='{$.spec.template.spec.containers[0].resources}' |
| 29 | +Check benchmark results |
25 | 30 |
|
26 |
| -## Benchmark with Kubernetes Pods |
| 31 | +```bash |
| 32 | +kubectl logs -f deployments/dnsperf |
| 33 | +... |
| 34 | +Statistics: |
| 35 | + ... |
| 36 | + Queries per second: 17241.513774 |
| 37 | + ... |
| 38 | +``` |
27 | 39 |
|
28 |
| -### Apply pre-configured testing deployment/pods |
| 40 | +Even more stress 🔥🔥🔥 |
29 | 41 |
|
30 |
| - $ kubectl apply -f https://raw.githubusercontent.com/guessi/dnsperf-bench/master/k8s-dnsperf-bench.yaml |
31 |
| - configmap/dns-records-config created |
32 |
| - deployment.apps/dnsperf created |
| 42 | +```bash |
| 43 | +kubectl scale deployments/dnsperf --replicas 10 |
| 44 | +``` |
33 | 45 |
|
34 |
| -### Find out your Kubernetes DNS service IP address |
| 46 | +## 🤔 Evaluate why there would have performance issue |
35 | 47 |
|
36 |
| - $ kubectl get service kube-dns -n kube-system -o jsonpath='{$.spec.clusterIP}' |
| 48 | +Does CoreDNS running expected version? |
37 | 49 |
|
38 |
| -### If your DNS service address is not "10.100.0.10", you will need to change the value of predefined "DNS_SERVER_ADDR" |
| 50 | +```bash |
| 51 | +kubectl describe deployments coredns -n kube-system | grep 'Image:' |
| 52 | +``` |
39 | 53 |
|
40 |
| - $ kubectl edit deployment dnsperf |
| 54 | +Does CoreDNS running with expected Corefile? |
41 | 55 |
|
42 |
| -### Make sure the deployment is running as expected |
| 56 | +```bash |
| 57 | +kubectl describe configmap coredns -n kube-system |
| 58 | +``` |
43 | 59 |
|
44 |
| - $ kubectl get deploy dnsperf |
| 60 | +Does CoreDNS running with correct resources configuration? |
45 | 61 |
|
46 |
| - NAME READY UP-TO-DATE AVAILABLE AGE |
47 |
| - dnsperf 1/1 1 1 81s |
| 62 | +```bash |
| 63 | +kubectl get deployments coredns -n kube-system -o json | jq -r '.spec.template.spec.containers[0].resources' |
| 64 | +``` |
48 | 65 |
|
49 |
| - $ kubectl get pods -l app=dnsperf |
| 66 | +Does CoreDNS need more CPU/Memory resources? |
50 | 67 |
|
51 |
| - NAME READY STATUS RESTARTS AGE |
52 |
| - dnsperf-7b9cc5b497-d5nfs 1/1 Running 0 1m16s |
| 68 | +```bash |
| 69 | +kubectl top pods -n kube-system -l k8s-app=kube-dns |
| 70 | +``` |
53 | 71 |
|
54 |
| -### To check benchmark results |
| 72 | +How many CoreDNS Pods running? Have you enabled CoreDNS AutoScaler? |
55 | 73 |
|
56 |
| - $ kubectl logs -f deployments/dnsperf |
| 74 | +```bash |
| 75 | +kubectl get deployments coredns -n kube-system |
| 76 | +``` |
57 | 77 |
|
58 |
| - DNS Performance Testing Tool |
59 |
| - Version 2.14.0 |
| 78 | +> DO NOT report bug without trying to do performance tuning. If you try to gain too much stress without tuning CoreDNS configureation, it is expected to have some TIMEOUT or packet losts. It's expected if you don't change. |
60 | 79 |
|
61 |
| - [Status] Command line: dnsperf -f any -m udp -s 10.100.0.10 -p 53 -d /opt/records.txt -c 1 -T 1 -l 30 -t 5 -Q 100000 |
62 |
| - [Status] Sending queries (to 10.100.0.10:53) |
63 |
| - [Status] Started at: Thu Apr 11 16:47:46 2024 |
64 |
| - [Status] Stopping after 30.000000 seconds |
65 |
| - [Status] Testing complete (time limit) |
66 |
| - |
67 |
| - Statistics: |
68 |
| - |
69 |
| - Queries sent: 517317 |
70 |
| - Queries completed: 517317 (100.00%) |
71 |
| - Queries lost: 0 (0.00%) |
72 |
| - |
73 |
| - Response codes: NOERROR 517317 (100.00%) |
74 |
| - Average packet size: request 43, response 160 |
75 |
| - Run time (s): 30.004152 |
76 |
| - Queries per second: 17241.513774 |
77 |
| - |
78 |
| - Average Latency (s): 0.004805 (min 0.000101, max 0.029327) |
79 |
| - Latency StdDev (s): 0.002537 |
80 |
| - |
81 |
| - |
82 |
| -### Check resources utilization of the CoreDNS deployment |
83 |
| - |
84 |
| - $ kubectl top pods -n kube-system -l k8s-app=kube-dns |
85 |
| - |
86 |
| - NAME CPU(cores) MEMORY(bytes) |
87 |
| - coredns-79989457d9-fn2hc 1644m 15Mi |
88 |
| - coredns-79989457d9-xss5l 925m 16Mi |
89 |
| - |
90 |
| -## Stress Test |
91 |
| - |
92 |
| -### You may gain more replicas to stress your CoreDNS even harder |
93 |
| - |
94 |
| - $ kubectl scale deployments/dnsperf --replicas 4 |
95 |
| - |
96 |
| - deployment.apps/dnsperf scaled |
97 |
| - |
98 |
| -### After gaining workload, you should find it's CPU utilization even higher |
99 |
| - |
100 |
| - $ kubectl top pods -n kube-system -l k8s-app=kube-dns |
101 |
| - |
102 |
| - NAME CPU(cores) MEMORY(bytes) |
103 |
| - coredns-79989457d9-fn2hc 1839m 17Mi |
104 |
| - coredns-79989457d9-xss5l 1344m 16Mi |
105 |
| - |
106 |
| -> If you try to gain too much stress without tuning CoreDNS configureation (e.g. CPU, Memory, Replicas of CoreDNS), you should find some TIMEOUT, packet losts. That's expected... don't report it as bug! you should give CoreDNS more resources to handle that stress. |
107 |
| -
|
108 |
| -## Reference |
| 80 | +## 📑 Reference |
109 | 81 |
|
110 | 82 | - https://www.dnsperf.com/
|
0 commit comments