Skip to content

Improve benchmark report output #3550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Conversation

manno
Copy link
Member

@manno manno commented Apr 11, 2025

Improve the output by adding descriptions.
Hide the detailed view behind the --verbose flag, move dump output to --debug.

Example Output:

# Description of Setup

> This section contains information about the setup used for the current sample. Like the k8s version, the node resources and the images available to the node.

Node Resources
==============
CPU, Memory, Pods
2, 5033432Ki, 110

Images
======
docker.io/rancher/fleet-agent:dev, docker.io/rancher/fleet:dev, docker.io/rancher/klipper-helm:v0.9.3-build20241008, docker.io/rancher/klipper-lb:v0.4.9, docker.io/rancher/local-path-provisioner:v0.0.30, docker.io/rancher/mirrored-coredns-coredns:1.12.0, docker.io/rancher/mirrored-library-traefik:2.11.18, docker.io/rancher/mirrored-metrics-server:v0.7.2, docker.io/rancher/mirrored-pause:3.6

Kubernetes Version
==================
Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.31.5+k3s1


Population Size
===============
Reports in "benchmarks/db": 3

# Results for Current Sample

> These measurements were taken before and after the experiments.

Measurement            | Value     | Unit
===============================================
CPU                    | 132.5     | seconds
-----------------------------------------------
CRDCount               | 35        | CRDs
-----------------------------------------------
ClusterCount           | 1         | clusters
-----------------------------------------------
GCDuration             | 0.6       | seconds
-----------------------------------------------
Mem                    | -3        | MB
-----------------------------------------------
NetworkRX              | 239743271 | bytes
-----------------------------------------------
NetworkTX              | 100644938 | bytes
-----------------------------------------------
NodeCount              | 1         | nodes
-----------------------------------------------
RESTClientDELETE200    | 117       | requests
-----------------------------------------------
RESTClientDELETE404    | 2         | requests
-----------------------------------------------
RESTClientGET200       | 245       | requests
-----------------------------------------------
RESTClientGET404       | 1         | requests
-----------------------------------------------
RESTClientGET429       | 1         | requests
-----------------------------------------------
RESTClientPATCH200     | 3338      | requests
-----------------------------------------------
RESTClientPOST201      | 84        | requests
-----------------------------------------------
RESTClientPOST409      | 36        | requests
-----------------------------------------------
RESTClientPUT200       | 10872     | requests
-----------------------------------------------
RESTClientPUT409       | 18        | requests
-----------------------------------------------
ReconcileErrors        | 2048      | reconciles
-----------------------------------------------
ReconcileRequeue       | 11549     | reconciles
-----------------------------------------------
ReconcileSuccess       | 11018     | reconciles
-----------------------------------------------
ReconcileTime          | 14907.8   | seconds
-----------------------------------------------
ResourceCount          | -175      | resources
-----------------------------------------------
SumCPU                 | 2         | cores
-----------------------------------------------
SumMem                 | 4915      | MB
-----------------------------------------------
SumPods                | 110       | pods
-----------------------------------------------
TotalDuration          | 200.007   |
-----------------------------------------------
WorkqueueAdds          | 24615     | items
-----------------------------------------------
WorkqueueQueueDuration | 11339.7   | seconds
-----------------------------------------------
WorkqueueRetries       | 13597     | items
-----------------------------------------------
WorkqueueWorkDuration  | 14908.5   | seconds

# Summary for each Experiment

> The duration of each experiment is compared to the population's data.


Experiment                               | Duration | Population Mean       | Population StdDev       | ZScore |
                                         |          | Duration              | Duration                |        |
=======================================================================================================================
create-1-bundle                          | 1.10s    | 1.09s                 | 0.01s                   | 1.06   | worse
-----------------------------------------------------------------------------------------------------------------------
create-1-bundledeployment-10-resources   | 14.60s   | 11.36s                | 7.39s                   | 0.44   | worse
-----------------------------------------------------------------------------------------------------------------------
create-1-gitrepo-1-big-bundle            | 5.13s    | 5.12s                 | 0.01s                   | 0.50   | worse
-----------------------------------------------------------------------------------------------------------------------
create-1-gitrepo-1-bundle                | 4.12s    | 4.44s                 | 0.58s                   | -0.56  | better
-----------------------------------------------------------------------------------------------------------------------
create-1-gitrepo-50-bundle               | 7.30s    | 6.90s                 | 0.58s                   | 0.68   | worse
-----------------------------------------------------------------------------------------------------------------------
create-50-bundle                         | 6.12s    | 6.25s                 | 0.24s                   | -0.53  | better
-----------------------------------------------------------------------------------------------------------------------
create-50-bundledeployment-500-resources | 128.03s  | 89.62s                | 17.00s                  | 2.26   | worse
-----------------------------------------------------------------------------------------------------------------------
create-50-gitrepo-50-bundle              | 33.62s   | 34.16s                | 2.27s                   | -0.24  | better

# Final Score

b-2025-04-11_14:25:18.json: -0.62

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant