Skip to content

Conversation

@Ivo-Yankov
Copy link
Contributor

@Ivo-Yankov Ivo-Yankov commented Dec 9, 2025

Description

This pull request refactors the performance e2e test suite to leverage the rapid-fire commands. It also collects metrics in a 5-second interval for the duration of the test runtime. Those metrics are saved in $SOLO_HOME/logs/$namespace/timeline-metrics.json

A separate tool for visualizing the data from timeline-metrics.json was developed, but it will be pushed to a separate repository. It generates charts for the CPU and Memory usage metrics. Ideally, those charts and the JSON file should be linked in a comment in every PR. However, the data size of the JSON exceeds the maximum character limit for GitHub comments, and the API does not support adding file attachments to comments. The best solution for this would be to upload the JSON file and the generated chart image to long-term storage and link them in a PR. This is described in this issue.

The PR also fixes the output of the rapid-fire commands.

Chart for the latest run of the Performance tests:

Screenshot 2025-12-23 at 9 46 37

Related Issues

Pull request (PR) checklist

  • This PR added tests (unit, integration, and/or end-to-end)
  • This PR updated documentation
  • This PR added no TODOs or commented out code
  • This PR has no breaking changes
  • Any technical debt has been documented as a separate issue and linked to this PR
  • Any package.json changes have been explained to and approved by a repository manager
  • All related issues have been linked to this PR
  • All changes in this PR are included in the description
  • When this PR merges the commits will be squashed and the title will be used as the commit message, the 'commit message guidelines' below have been followed

Testing

  • This PR added unit tests
  • This PR added integration/end-to-end tests
  • These changes required manual testing that is documented below
  • Anything not tested is documented

The following manual testing was done:

  • TBD

The following was not tested:

  • TBD
Commit message guidelines We use 'Conventional Commits' to ensure that our commit messages are easy to read, follow a consistent format, and for automated release note generation. Please follow the guidelines below when writing your commit messages:
  1. BREAKING CHANGE: a commit that has a footer BREAKING CHANGE:, or appends a ! after the type/scope, introduces a breaking API change (correlating with MAJOR in Semantic Versioning). A BREAKING CHANGE can be part of commits of any type. NOTE: currently breaking changes will only bump the MAJOR version.
  2. The title is prefixed with one of the following:
Prefix Description Semantic Version Update Captured in Release Notes
feat: a new feature MINOR Yes
fix: a bug fix PATCH Yes
perf: performance PATCH Yes
refactor: code change that isn't feature or fix none No
test: adding missing tests none No
docs: changes to documentation none Yes
build: changes to build process none No
ci: changes to CI configuration none No
style: formatting, missing semi-colons, etc none No
chore: updating grunt tasks etc; no production code change none No

@trunk-io
Copy link

trunk-io bot commented Dec 9, 2025

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

@Ivo-Yankov Ivo-Yankov marked this pull request as draft December 9, 2025 13:14
@Ivo-Yankov Ivo-Yankov self-assigned this Dec 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2025

Unit Test Results - Linux

32 tests  ±0   32 ✅ ±0   0s ⏱️ ±0s
16 suites ±0    0 💤 ±0 
 1 files   ±0    0 ❌ ±0 

Results for commit 34b91f7. ± Comparison against base commit eca0979.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2025

Unit Test Results - Windows

  1 files  ±0  187 suites  ±0   5s ⏱️ ±0s
510 tests ±0  510 ✅ ±0  0 💤 ±0  0 ❌ ±0 
514 runs  ±0  514 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 34b91f7. ± Comparison against base commit eca0979.

♻️ This comment has been updated with latest results.

Signed-off-by: Ivo Yankov <[email protected]>
Signed-off-by: Ivo Yankov <[email protected]>
Signed-off-by: Ivo Yankov <[email protected]>
Signed-off-by: Ivo Yankov <[email protected]>
Signed-off-by: Ivo Yankov <[email protected]>
@codacy-production
Copy link

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
Report missing for 50ed0301 96.20%
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (50ed030) Report Missing Report Missing Report Missing
Head commit (79f94c8) 51048 42773 83.79%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#3049) 79 76 96.20%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Footnotes

  1. Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct.

@codacy-production
Copy link

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
Report missing for 50ed0301 96.20%
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (50ed030) Report Missing Report Missing Report Missing
Head commit (9196616) 51048 42770 83.78%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#3049) 79 76 96.20%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Footnotes

  1. Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct.

@codacy-production
Copy link

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
+0.41% (target: -1.00%) 96.20%
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (eca0979) 50980 42509 83.38%
Head commit (0fc6a9a) 51048 (+68) 42773 (+264) 83.79% (+0.41%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#3049) 79 76 96.20%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

@github-actions
Copy link
Contributor

Log Metrics - One Shot Single - using Podman

📊 Runner Resource Usage

╔═══════════════════════════════════════════════════════════════╗
║          GitHub Runner Resource Usage                         ║
╚═══════════════════════════════════════════════════════════════╝

⏱️  Test Duration: 7.0 minutes (8 data points)

📉 CPU Usage
100.0% | + #    
 90.0% | # #    
 80.0% | # #    
 70.0% | # #    
 60.0% | # #+   
 50.0% | # ##   
 40.0% | # ##   
 30.0% | # ##   
 20.0% | # ##+  
 10.0% |+#+###++
  0.0% |########
       +--------

📉 Memory Usage
100.0% |        
 90.0% |        
 80.0% |        
 70.0% |        
 60.0% |        
 50.0% |     +  
 40.0% |  + +#  
 30.0% |  #+##++
 20.0% | +######
 10.0% |+#######
  0.0% |########
       +--------

⚠️  WARNING: Resource usage exceeded 95% threshold!
    CPU Peak: 100%  |  Memory Peak: 40.0%


JSON Log Metrics

{
  "snapshotName": "one-shot-single-minimal",
  "date": "2025-12-22T16:46:29.405Z",
  "gitHubSha": "0fcb1d11a08cb613a8c4c0af8459d0e5e594506a",
  "soloVersion": "0.52.0",
  "soloChartVersion": "0.58.1",
  "consensusNodeVersion": "0.67.2",
  "mirrorNodeVersion": "0.143.0",
  "blockNodeVersion": "0.23.2",
  "relayVersion": "0.73.0",
  "explorerVersion": "25.1.1",
  "cpuInMillicores": 517,
  "memoryInMebibytes": 3197,
  "runtimeInMinutes": 0,
  "transactionCount": 0,
  "events": [],
  "clusterMetrics": [
    {
      "context": "default",
      "cpuInMillicores": 517,
      "memoryInMebibytes": 3197,
      "podMetrics": [
        {
          "namespace": "kube-system",
          "podName": "coredns-66bc5c9577-hscz9",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 17
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-66bc5c9577-m2gsx",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 17
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-kind-control-plane",
          "cpuInMillicores": 41,
          "memoryInMebibytes": 77
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-qj528",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 14
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-kind-control-plane",
          "cpuInMillicores": 96,
          "memoryInMebibytes": 585
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-kind-control-plane",
          "cpuInMillicores": 15,
          "memoryInMebibytes": 71
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-b8lph",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 19
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-799f7ccf68-pdt7n",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 15
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-7b8c8ddbd6-xvvrg",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-765c495b75-p464k",
          "cpuInMillicores": 65,
          "memoryInMebibytes": 40
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "envoy-proxy-node1-84ff6fb6b5-5g5ml",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 19
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "haproxy-node1-68d96cf878-4z25x",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 121
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "mirror-1-grpc-5cfb8f5b8-2sm9h",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 318
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "mirror-1-importer-66b7c94597-vmd7k",
          "cpuInMillicores": 68,
          "memoryInMebibytes": 456
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "mirror-1-monitor-5d79cb4b47-bfsnw",
          "cpuInMillicores": 102,
          "memoryInMebibytes": 272
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "mirror-1-postgres-postgresql-0",
          "cpuInMillicores": 37,
          "memoryInMebibytes": 145
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "mirror-1-redis-node-0",
          "cpuInMillicores": 31,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "mirror-1-rest-65d4d8fd6f-d8xfs",
          "cpuInMillicores": 23,
          "memoryInMebibytes": 171
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "mirror-1-restjava-85c95665d5-z4bsw",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 345
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "mirror-1-web3-57ff59d96b-m9cmw",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 422
        },
        {
          "namespace": "solo-f8a85c02",
          "podName": "mirror-ingress-controller-6b747475f4-fslzc",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 31
        },
        {
          "namespace": "solo-setup",
          "podName": "minio-operator-9ff787b47-q5q46",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 16
        }
      ]
    }
  ]
}

@github-actions
Copy link
Contributor

Log Metrics - One Shot Single

📊 Runner Resource Usage

╔═══════════════════════════════════════════════════════════════╗
║          GitHub Runner Resource Usage                         ║
╚═══════════════════════════════════════════════════════════════╝

⏱️  Test Duration: 7.0 minutes (8 data points)

📉 CPU Usage
100.0% |        
 90.0% |        
 80.0% |        
 70.0% |        
 60.0% |        
 50.0% |        
 40.0% |        
 30.0% |  +     
 20.0% |  #++   
 10.0% |++###+++
  0.0% |########
       +--------

📉 Memory Usage
100.0% |        
 90.0% |        
 80.0% |        
 70.0% |        
 60.0% |        
 50.0% |        
 40.0% |        
 30.0% |        
 20.0% | +++++++
 10.0% |+#######
  0.0% |########
       +--------

✅ Resource usage within normal limits
    CPU Peak: 22.1%  |  Memory Peak: 18.4%


JSON Log Metrics

{
  "snapshotName": "one-shot-single",
  "date": "2025-12-22T16:47:00.942Z",
  "gitHubSha": "0fcb1d11a08cb613a8c4c0af8459d0e5e594506a",
  "soloVersion": "0.52.0",
  "soloChartVersion": "0.58.1",
  "consensusNodeVersion": "0.67.2",
  "mirrorNodeVersion": "0.143.0",
  "blockNodeVersion": "0.23.2",
  "relayVersion": "0.73.0",
  "explorerVersion": "25.1.1",
  "cpuInMillicores": 873,
  "memoryInMebibytes": 5596,
  "runtimeInMinutes": 4,
  "transactionCount": 1370,
  "events": [],
  "clusterMetrics": [
    {
      "context": "default",
      "cpuInMillicores": 873,
      "memoryInMebibytes": 5596,
      "podMetrics": [
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-chv2b",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-hppvt",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 15
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-e2e-one-shot-single-20438025580-1-c1-control-plane",
          "cpuInMillicores": 23,
          "memoryInMebibytes": 69
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-7qfq7",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 12
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-e2e-one-shot-single-20438025580-1-c1-control-plane",
          "cpuInMillicores": 41,
          "memoryInMebibytes": 505
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-e2e-one-shot-single-20438025580-1-c1-control-plane",
          "cpuInMillicores": 11,
          "memoryInMebibytes": 74
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-pbwn7",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "kube-system",
          "podName": "kube-scheduler-e2e-one-shot-single-20438025580-1-c1-control-plane",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 24
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-fbb76bc7f-t9wq7",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 23
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-6fdc965494-wxml8",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-676f677b6b-s44gt",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 38
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-speaker-9k7lq",
          "cpuInMillicores": 6,
          "memoryInMebibytes": 53
        },
        {
          "namespace": "solo-754643e9",
          "podName": "envoy-proxy-node1-74bcfb64f8-cpnb6",
          "cpuInMillicores": 11,
          "memoryInMebibytes": 26
        },
        {
          "namespace": "solo-754643e9",
          "podName": "haproxy-node1-7bb68675fc-kmfwx",
          "cpuInMillicores": 10,
          "memoryInMebibytes": 106
        },
        {
          "namespace": "solo-754643e9",
          "podName": "hiero-explorer-1-5585d47c6-rz85c",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 2
        },
        {
          "namespace": "solo-754643e9",
          "podName": "minio-pool-1-0",
          "cpuInMillicores": 20,
          "memoryInMebibytes": 199
        },
        {
          "namespace": "solo-754643e9",
          "podName": "mirror-1-grpc-5bbf9d7878-qhkp2",
          "cpuInMillicores": 7,
          "memoryInMebibytes": 320
        },
        {
          "namespace": "solo-754643e9",
          "podName": "mirror-1-importer-745795b7f5-kj4c5",
          "cpuInMillicores": 84,
          "memoryInMebibytes": 450
        },
        {
          "namespace": "solo-754643e9",
          "podName": "mirror-1-monitor-f9bfd6bc7-5mxht",
          "cpuInMillicores": 84,
          "memoryInMebibytes": 278
        },
        {
          "namespace": "solo-754643e9",
          "podName": "mirror-1-postgres-postgresql-0",
          "cpuInMillicores": 46,
          "memoryInMebibytes": 155
        },
        {
          "namespace": "solo-754643e9",
          "podName": "mirror-1-redis-node-0",
          "cpuInMillicores": 30,
          "memoryInMebibytes": 7
        },
        {
          "namespace": "solo-754643e9",
          "podName": "mirror-1-rest-554b5dfb5-c6wkv",
          "cpuInMillicores": 31,
          "memoryInMebibytes": 116
        },
        {
          "namespace": "solo-754643e9",
          "podName": "mirror-1-restjava-66d594f79-slqcb",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 354
        },
        {
          "namespace": "solo-754643e9",
          "podName": "mirror-1-web3-56d96644f5-w5k6f",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 419
        },
        {
          "namespace": "solo-754643e9",
          "podName": "mirror-ingress-controller-674fc9847f-ztc5t",
          "cpuInMillicores": 15,
          "memoryInMebibytes": 38
        },
        {
          "namespace": "solo-754643e9",
          "podName": "network-node1-0",
          "cpuInMillicores": 415,
          "memoryInMebibytes": 1835
        },
        {
          "namespace": "solo-754643e9",
          "podName": "relay-1-686fc69666-7qsnr",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 215
        },
        {
          "namespace": "solo-754643e9",
          "podName": "relay-1-ws-7887947bbc-sq8zd",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 205
        },
        {
          "namespace": "solo-setup",
          "podName": "minio-operator-5df5d4fc74-5qphj",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 16
        }
      ]
    }
  ]
}

@github-actions
Copy link
Contributor

Log Metrics - External database new

📊 Runner Resource Usage

╔═══════════════════════════════════════════════════════════════╗
║          GitHub Runner Resource Usage                         ║
╚═══════════════════════════════════════════════════════════════╝

⏱️  Test Duration: 12.0 minutes (13 data points)

📉 CPU Usage
100.0% |             
 90.0% |             
 80.0% |             
 70.0% |             
 60.0% |      +      
 50.0% |      #      
 40.0% |    ++#++++  
 30.0% |   +#######+ 
 20.0% |  +#########+
 10.0% |++###########
  0.0% |#############
       +-------------

📉 Memory Usage
100.0% |             
 90.0% |             
 80.0% |             
 70.0% |             
 60.0% |             
 50.0% |             
 40.0% |             
 30.0% |    +++++++++
 20.0% | +++#########
 10.0% |+############
  0.0% |#############
       +-------------

✅ Resource usage within normal limits
    CPU Peak: 58.1%  |  Memory Peak: 24%


JSON Log Metrics

{
  "snapshotName": "external-database-test",
  "date": "2025-12-22T16:55:21.716Z",
  "gitHubSha": "0fcb1d11a08cb613a8c4c0af8459d0e5e594506a",
  "soloVersion": "0.52.0",
  "soloChartVersion": "0.58.1",
  "consensusNodeVersion": "0.67.2",
  "mirrorNodeVersion": "0.143.0",
  "blockNodeVersion": "0.23.2",
  "relayVersion": "0.73.0",
  "explorerVersion": "25.1.1",
  "cpuInMillicores": 1647,
  "memoryInMebibytes": 10915,
  "runtimeInMinutes": 11,
  "transactionCount": 0,
  "events": [],
  "clusterMetrics": [
    {
      "context": "kind-e2e-external-database-20438025580-1-c1",
      "cpuInMillicores": 698,
      "memoryInMebibytes": 4037,
      "podMetrics": [
        {
          "namespace": "external-database-test",
          "podName": "envoy-proxy-node1-559fd7cf65-g742q",
          "cpuInMillicores": 8,
          "memoryInMebibytes": 27
        },
        {
          "namespace": "external-database-test",
          "podName": "haproxy-node1-85866454b-lcqt7",
          "cpuInMillicores": 6,
          "memoryInMebibytes": 108
        },
        {
          "namespace": "external-database-test",
          "podName": "minio-pool-1-0",
          "cpuInMillicores": 7,
          "memoryInMebibytes": 168
        },
        {
          "namespace": "external-database-test",
          "podName": "network-node1-0",
          "cpuInMillicores": 602,
          "memoryInMebibytes": 2989
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-5qc6p",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-q2rsb",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 15
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-e2e-external-database-20438025580-1-c1-control-plane",
          "cpuInMillicores": 16,
          "memoryInMebibytes": 72
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-kqbgj",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-e2e-external-database-20438025580-1-c1-control-plane",
          "cpuInMillicores": 27,
          "memoryInMebibytes": 402
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-e2e-external-database-20438025580-1-c1-control-plane",
          "cpuInMillicores": 10,
          "memoryInMebibytes": 58
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-jpvkl",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 14
        },
        {
          "namespace": "kube-system",
          "podName": "kube-scheduler-e2e-external-database-20438025580-1-c1-control-plane",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 20
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-fbb76bc7f-d8t5q",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 22
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-6fdc965494-s74ss",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-676f677b6b-zm2gn",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 37
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-speaker-kcc5f",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 54
        },
        {
          "namespace": "solo-setup",
          "podName": "minio-operator-5df5d4fc74-pf6fd",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 15
        }
      ]
    },
    {
      "context": "kind-e2e-external-database-20438025580-1-c2",
      "cpuInMillicores": 949,
      "memoryInMebibytes": 6878,
      "podMetrics": [
        {
          "namespace": "database",
          "podName": "my-postgresql-0",
          "cpuInMillicores": 24,
          "memoryInMebibytes": 147
        },
        {
          "namespace": "external-database-test",
          "podName": "envoy-proxy-node2-866544c45d-94wv4",
          "cpuInMillicores": 9,
          "memoryInMebibytes": 27
        },
        {
          "namespace": "external-database-test",
          "podName": "haproxy-node2-54fc8fd459-qdgn4",
          "cpuInMillicores": 7,
          "memoryInMebibytes": 107
        },
        {
          "namespace": "external-database-test",
          "podName": "hiero-explorer-1-5585d47c6-dk7gq",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 2
        },
        {
          "namespace": "external-database-test",
          "podName": "minio-pool-1-0",
          "cpuInMillicores": 18,
          "memoryInMebibytes": 284
        },
        {
          "namespace": "external-database-test",
          "podName": "mirror-grpc-5cf598f7d-thftl",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 355
        },
        {
          "namespace": "external-database-test",
          "podName": "mirror-importer-64b94f86fd-t9hhg",
          "cpuInMillicores": 37,
          "memoryInMebibytes": 490
        },
        {
          "namespace": "external-database-test",
          "podName": "mirror-ingress-controller-bb5dc897b-t4wfs",
          "cpuInMillicores": 11,
          "memoryInMebibytes": 39
        },
        {
          "namespace": "external-database-test",
          "podName": "mirror-monitor-7ffc5947b-mlzzp",
          "cpuInMillicores": 47,
          "memoryInMebibytes": 299
        },
        {
          "namespace": "external-database-test",
          "podName": "mirror-redis-node-0",
          "cpuInMillicores": 25,
          "memoryInMebibytes": 7
        },
        {
          "namespace": "external-database-test",
          "podName": "mirror-rest-66b749c966-496dp",
          "cpuInMillicores": 22,
          "memoryInMebibytes": 115
        },
        {
          "namespace": "external-database-test",
          "podName": "mirror-restjava-678f86fbb5-p7qw9",
          "cpuInMillicores": 20,
          "memoryInMebibytes": 374
        },
        {
          "namespace": "external-database-test",
          "podName": "mirror-web3-54d66fb686-s4ggq",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 447
        },
        {
          "namespace": "external-database-test",
          "podName": "network-node2-0",
          "cpuInMillicores": 621,
          "memoryInMebibytes": 2939
        },
        {
          "namespace": "external-database-test",
          "podName": "relay-1-686fc69666-t8bvc",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 218
        },
        {
          "namespace": "external-database-test",
          "podName": "relay-1-ws-7887947bbc-plssl",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 209
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-6ns26",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-mwv68",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-e2e-external-database-20438025580-1-c2-control-plane",
          "cpuInMillicores": 24,
          "memoryInMebibytes": 78
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-8tvv8",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-e2e-external-database-20438025580-1-c2-control-plane",
          "cpuInMillicores": 36,
          "memoryInMebibytes": 458
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-e2e-external-database-20438025580-1-c2-control-plane",
          "cpuInMillicores": 10,
          "memoryInMebibytes": 66
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-xzpnc",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 14
        },
        {
          "namespace": "kube-system",
          "podName": "kube-scheduler-e2e-external-database-20438025580-1-c2-control-plane",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 21
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-fbb76bc7f-fvcnm",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 24
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-6fdc965494-s9crs",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-676f677b6b-hj4v4",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 37
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-speaker-bx84l",
          "cpuInMillicores": 6,
          "memoryInMebibytes": 54
        },
        {
          "namespace": "solo-setup",
          "podName": "minio-operator-5df5d4fc74-kt9pg",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 15
        }
      ]
    }
  ]
}

@github-actions
Copy link
Contributor

Log Metrics - Block Node

📊 Runner Resource Usage

╔═══════════════════════════════════════════════════════════════╗
║          GitHub Runner Resource Usage                         ║
╚═══════════════════════════════════════════════════════════════╝

⏱️  Test Duration: 4.0 minutes (5 data points)

📉 CPU Usage
100.0% |     
 90.0% |     
 80.0% |     
 70.0% |     
 60.0% |     
 50.0% |     
 40.0% |     
 30.0% |     
 20.0% |   ++
 10.0% |+++##
  0.0% |#####
       +-----

📉 Memory Usage
100.0% |     
 90.0% |     
 80.0% |     
 70.0% |     
 60.0% |     
 50.0% |     
 40.0% |     
 30.0% |     
 20.0% | ++++
 10.0% |+####
  0.0% |#####
       +-----

✅ Resource usage within normal limits
    CPU Peak: 16.9%  |  Memory Peak: 17.6%


JSON Log Metrics

{
  "snapshotName": "block-node-test",
  "date": "2025-12-22T16:54:02.468Z",
  "gitHubSha": "0fcb1d11a08cb613a8c4c0af8459d0e5e594506a",
  "soloVersion": "undefined",
  "soloChartVersion": "undefined",
  "consensusNodeVersion": "undefined",
  "mirrorNodeVersion": "undefined",
  "blockNodeVersion": "undefined",
  "relayVersion": "undefined",
  "explorerVersion": "undefined",
  "cpuInMillicores": 86,
  "memoryInMebibytes": 559,
  "runtimeInMinutes": 0,
  "transactionCount": 0,
  "events": [],
  "clusterMetrics": [
    {
      "context": "kind-e2e-block-node-20438025580-1-c1",
      "cpuInMillicores": 86,
      "memoryInMebibytes": 559,
      "podMetrics": [
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-d2799",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 13
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-gq876",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 14
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-e2e-block-node-20438025580-1-c1-control-plane",
          "cpuInMillicores": 20,
          "memoryInMebibytes": 37
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-xd4zx",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-e2e-block-node-20438025580-1-c1-control-plane",
          "cpuInMillicores": 30,
          "memoryInMebibytes": 282
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-e2e-block-node-20438025580-1-c1-control-plane",
          "cpuInMillicores": 11,
          "memoryInMebibytes": 56
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-wb522",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "kube-system",
          "podName": "kube-scheduler-e2e-block-node-20438025580-1-c1-control-plane",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 20
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-fbb76bc7f-bhtj2",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 20
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-6fdc965494-zczgt",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 7
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-676f677b6b-fq5z8",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 19
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-speaker-zgdxk",
          "cpuInMillicores": 8,
          "memoryInMebibytes": 52
        },
        {
          "namespace": "solo-setup",
          "podName": "minio-operator-5df5d4fc74-pv6kq",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 13
        }
      ]
    }
  ]
}

@github-actions
Copy link
Contributor

Log Metrics - Integration

📊 Runner Resource Usage

╔═══════════════════════════════════════════════════════════════╗
║          GitHub Runner Resource Usage                         ║
╚═══════════════════════════════════════════════════════════════╝

⏱️  Test Duration: 6.0 minutes (7 data points)

📉 CPU Usage
100.0% |       
 90.0% |       
 80.0% |       
 70.0% |       
 60.0% |       
 50.0% |       
 40.0% |       
 30.0% |       
 20.0% |      +
 10.0% |++++++#
  0.0% |#######
       +-------

📉 Memory Usage
100.0% |       
 90.0% |       
 80.0% |       
 70.0% |       
 60.0% |       
 50.0% |       
 40.0% |       
 30.0% |       
 20.0% | +++   
 10.0% |+###+++
  0.0% |#######
       +-------

✅ Resource usage within normal limits
    CPU Peak: 27.2%  |  Memory Peak: 12.8%


JSON Log Metrics

{
  "snapshotName": "metric-server-test",
  "date": "2025-12-22T16:52:38.995Z",
  "gitHubSha": "0fcb1d11a08cb613a8c4c0af8459d0e5e594506a",
  "soloVersion": "undefined",
  "soloChartVersion": "undefined",
  "consensusNodeVersion": "undefined",
  "mirrorNodeVersion": "undefined",
  "blockNodeVersion": "undefined",
  "relayVersion": "undefined",
  "explorerVersion": "undefined",
  "cpuInMillicores": 152,
  "memoryInMebibytes": 526,
  "runtimeInMinutes": 0,
  "transactionCount": 0,
  "events": [],
  "clusterMetrics": [
    {
      "context": "default",
      "cpuInMillicores": 152,
      "memoryInMebibytes": 526,
      "podMetrics": [
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-cf57j",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 13
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-fgscr",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 13
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-e2e-integration-20438025580-1-c1-control-plane",
          "cpuInMillicores": 20,
          "memoryInMebibytes": 32
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-hffwb",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-e2e-integration-20438025580-1-c1-control-plane",
          "cpuInMillicores": 98,
          "memoryInMebibytes": 277
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-e2e-integration-20438025580-1-c1-control-plane",
          "cpuInMillicores": 10,
          "memoryInMebibytes": 54
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-8djr9",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 15
        },
        {
          "namespace": "kube-system",
          "podName": "kube-scheduler-e2e-integration-20438025580-1-c1-control-plane",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 20
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-fbb76bc7f-cjv7t",
          "cpuInMillicores": 6,
          "memoryInMebibytes": 18
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-6fdc965494-qbmkd",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 8
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-676f677b6b-54wsc",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-speaker-qz5zl",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 50
        }
      ]
    }
  ]
}

@github-actions
Copy link
Contributor

Log Metrics - Node Upgrade

📊 Runner Resource Usage

╔═══════════════════════════════════════════════════════════════╗
║          GitHub Runner Resource Usage                         ║
╚═══════════════════════════════════════════════════════════════╝

⏱️  Test Duration: 5.0 minutes (6 data points)

📉 CPU Usage
100.0% |      
 90.0% |      
 80.0% |      
 70.0% |      
 60.0% |      
 50.0% |    + 
 40.0% |    # 
 30.0% |  ++#+
 20.0% | +####
 10.0% |+#####
  0.0% |######
       +------

📉 Memory Usage
100.0% |      
 90.0% |      
 80.0% |      
 70.0% |      
 60.0% |      
 50.0% |      
 40.0% |      
 30.0% |  + + 
 20.0% | +#+#+
 10.0% |+#####
  0.0% |######
       +------

✅ Resource usage within normal limits
    CPU Peak: 43.9%  |  Memory Peak: 22.4%


JSON Log Metrics

{
  "snapshotName": "node-upgrade-test",
  "date": "2025-12-22T17:02:11.957Z",
  "gitHubSha": "0fcb1d11a08cb613a8c4c0af8459d0e5e594506a",
  "soloVersion": "undefined",
  "soloChartVersion": "undefined",
  "consensusNodeVersion": "undefined",
  "mirrorNodeVersion": "undefined",
  "blockNodeVersion": "undefined",
  "relayVersion": "undefined",
  "explorerVersion": "undefined",
  "cpuInMillicores": 113,
  "memoryInMebibytes": 518,
  "runtimeInMinutes": 0,
  "transactionCount": 0,
  "events": [],
  "clusterMetrics": [
    {
      "context": "kind-e2e-node-upgrade-20438025580-1-c1",
      "cpuInMillicores": 113,
      "memoryInMebibytes": 518,
      "podMetrics": [
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-4s686",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 12
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-794hd",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 13
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-e2e-node-upgrade-20438025580-1-c1-control-plane",
          "cpuInMillicores": 13,
          "memoryInMebibytes": 32
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-jxz2m",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-e2e-node-upgrade-20438025580-1-c1-control-plane",
          "cpuInMillicores": 69,
          "memoryInMebibytes": 271
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-e2e-node-upgrade-20438025580-1-c1-control-plane",
          "cpuInMillicores": 9,
          "memoryInMebibytes": 52
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-nsqgv",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 15
        },
        {
          "namespace": "kube-system",
          "podName": "kube-scheduler-e2e-node-upgrade-20438025580-1-c1-control-plane",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 20
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-fbb76bc7f-bkj7n",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 19
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-6fdc965494-jsfbm",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 7
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-676f677b6b-v6r9r",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 17
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-speaker-b5hb6",
          "cpuInMillicores": 6,
          "memoryInMebibytes": 50
        }
      ]
    }
  ]
}

@github-actions
Copy link
Contributor

Log Metrics - Node Local PTT

📊 Runner Resource Usage

╔═══════════════════════════════════════════════════════════════╗
║          GitHub Runner Resource Usage                         ║
╚═══════════════════════════════════════════════════════════════╝

⏱️  Test Duration: 2.0 minutes (3 data points)

📉 CPU Usage
100.0% |   
 90.0% |   
 80.0% |   
 70.0% |   
 60.0% |   
 50.0% |   
 40.0% | + 
 30.0% | # 
 20.0% | #+
 10.0% |+##
  0.0% |###
       +---

📉 Memory Usage
100.0% |   
 90.0% |   
 80.0% |   
 70.0% |   
 60.0% |   
 50.0% |   
 40.0% |   
 30.0% | ++
 20.0% | ##
 10.0% |+##
  0.0% |###
       +---

✅ Resource usage within normal limits
    CPU Peak: 33.3%  |  Memory Peak: 21.6%


@github-actions
Copy link
Contributor

Log Metrics - Performance Test

JSON Log Metrics

{
  "snapshotName": "performance-tests-490357",
  "date": "2025-12-22T16:55:17.964Z",
  "gitHubSha": "0fcb1d11a08cb613a8c4c0af8459d0e5e594506a",
  "soloVersion": "0.52.0",
  "soloChartVersion": "0.58.1",
  "consensusNodeVersion": "0.67.2",
  "mirrorNodeVersion": "0.143.0",
  "blockNodeVersion": "0.23.2",
  "relayVersion": "0.73.0",
  "explorerVersion": "25.1.1",
  "cpuInMillicores": 2351,
  "memoryInMebibytes": 7933,
  "runtimeInMinutes": 13,
  "transactionCount": 16051,
  "events": [],
  "clusterMetrics": [
    {
      "context": "default",
      "cpuInMillicores": 2351,
      "memoryInMebibytes": 7933,
      "podMetrics": [
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-j7sv6",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 17
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-sgt7f",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-solo-e2e-c1-control-plane",
          "cpuInMillicores": 22,
          "memoryInMebibytes": 76
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-nsg5d",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 11
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-solo-e2e-c1-control-plane",
          "cpuInMillicores": 40,
          "memoryInMebibytes": 499
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-solo-e2e-c1-control-plane",
          "cpuInMillicores": 12,
          "memoryInMebibytes": 65
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-ndk29",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 15
        },
        {
          "namespace": "kube-system",
          "podName": "kube-scheduler-solo-e2e-c1-control-plane",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 21
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-fbb76bc7f-s9bnm",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 24
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-6fdc965494-ds8ml",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-676f677b6b-vs4n7",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 36
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-speaker-6wwc7",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 54
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "block-node-1-0",
          "cpuInMillicores": 1000,
          "memoryInMebibytes": 417
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "envoy-proxy-node1-74bcfb64f8-6zst5",
          "cpuInMillicores": 11,
          "memoryInMebibytes": 27
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "haproxy-node1-7bb68675fc-tk6br",
          "cpuInMillicores": 21,
          "memoryInMebibytes": 110
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "hiero-explorer-1-5585d47c6-bt9md",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 2
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "minio-pool-1-0",
          "cpuInMillicores": 19,
          "memoryInMebibytes": 305
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "mirror-1-grpc-694f884498-9r76t",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 325
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "mirror-1-importer-cdb86dc7f-86bvd",
          "cpuInMillicores": 55,
          "memoryInMebibytes": 482
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "mirror-1-monitor-75d7c495d7-2lhfh",
          "cpuInMillicores": 44,
          "memoryInMebibytes": 301
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "mirror-1-postgres-postgresql-0",
          "cpuInMillicores": 70,
          "memoryInMebibytes": 208
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "mirror-1-redis-node-0",
          "cpuInMillicores": 31,
          "memoryInMebibytes": 7
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "mirror-1-rest-77477985bd-rtwmz",
          "cpuInMillicores": 42,
          "memoryInMebibytes": 106
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "mirror-1-restjava-6dcddb4f4b-9vb96",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 387
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "mirror-1-web3-868c56874d-tl7xd",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 431
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "mirror-ingress-controller-6fddc99948-drzp6",
          "cpuInMillicores": 13,
          "memoryInMebibytes": 38
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "network-node1-0",
          "cpuInMillicores": 922,
          "memoryInMebibytes": 3504
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "relay-1-686fc69666-cs6v5",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 213
        },
        {
          "namespace": "solo-60174cc2",
          "podName": "relay-1-ws-7887947bbc-njhg5",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 211
        },
        {
          "namespace": "solo-setup",
          "podName": "minio-operator-5df5d4fc74-8qrg2",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 15
        }
      ]
    }
  ]
}

@github-actions
Copy link
Contributor

Log Metrics - Dual Cluster Full

📊 Runner Resource Usage

╔═══════════════════════════════════════════════════════════════╗
║          GitHub Runner Resource Usage                         ║
╚═══════════════════════════════════════════════════════════════╝

⏱️  Test Duration: 24.0 minutes (25 data points)

📉 CPU Usage
100.0% |                         
 90.0% |                         
 80.0% |                         
 70.0% |                         
 60.0% |                         
 50.0% |             +          +
 40.0% |             #  +       #
 30.0% |    ++       #  #  +    #
 20.0% |  ++##++++++ #++#+ #++++#
 10.0% |++##########+#####+######
  0.0% |#########################
       +-------------------------

📉 Memory Usage
100.0% |                         
 90.0% |                         
 80.0% |                         
 70.0% |                         
 60.0% |                         
 50.0% |                         
 40.0% |                         
 30.0% |    ++++ ++++++++        
 20.0% | +++####+########++++++++
 10.0% |+########################
  0.0% |#########################
       +-------------------------

✅ Resource usage within normal limits
    CPU Peak: 49.8%  |  Memory Peak: 26.4%


JSON Log Metrics

{
  "snapshotName": "dual-cluster-full",
  "date": "2025-12-22T17:21:28.084Z",
  "gitHubSha": "0fcb1d11a08cb613a8c4c0af8459d0e5e594506a",
  "soloVersion": "0.52.0",
  "soloChartVersion": "0.58.1",
  "consensusNodeVersion": "0.67.2",
  "mirrorNodeVersion": "0.143.0",
  "blockNodeVersion": "0.23.2",
  "relayVersion": "0.73.0",
  "explorerVersion": "25.1.1",
  "cpuInMillicores": 2579,
  "memoryInMebibytes": 9331,
  "runtimeInMinutes": 3,
  "transactionCount": 0,
  "events": [],
  "clusterMetrics": [
    {
      "context": "kind-e2e-dual-cluster-full-20438025580-1-c1",
      "cpuInMillicores": 1590,
      "memoryInMebibytes": 3524,
      "podMetrics": [
        {
          "namespace": "dual-cluster-full",
          "podName": "block-node-1-0",
          "cpuInMillicores": 1001,
          "memoryInMebibytes": 385
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "envoy-proxy-node1-559fd7cf65-dqchw",
          "cpuInMillicores": 10,
          "memoryInMebibytes": 27
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "haproxy-node1-85866454b-nfgjg",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 109
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "minio-pool-1-0",
          "cpuInMillicores": 9,
          "memoryInMebibytes": 175
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "network-node1-0",
          "cpuInMillicores": 484,
          "memoryInMebibytes": 1934
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-79r8x",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 17
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-pxh7h",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-e2e-dual-cluster-full-20438025580-1-c1-control-plane",
          "cpuInMillicores": 19,
          "memoryInMebibytes": 86
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-g6v2k",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 11
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-e2e-dual-cluster-full-20438025580-1-c1-control-plane",
          "cpuInMillicores": 31,
          "memoryInMebibytes": 515
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-e2e-dual-cluster-full-20438025580-1-c1-control-plane",
          "cpuInMillicores": 10,
          "memoryInMebibytes": 73
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-ngdjd",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 14
        },
        {
          "namespace": "kube-system",
          "podName": "kube-scheduler-e2e-dual-cluster-full-20438025580-1-c1-control-plane",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 23
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-fbb76bc7f-vqkw8",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 22
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-6fdc965494-nk6s5",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 11
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-676f677b6b-dldrl",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 38
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-speaker-mtgq6",
          "cpuInMillicores": 6,
          "memoryInMebibytes": 53
        },
        {
          "namespace": "solo-setup",
          "podName": "minio-operator-5df5d4fc74-wf9vn",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 15
        }
      ]
    },
    {
      "context": "kind-e2e-dual-cluster-full-20438025580-1-c2",
      "cpuInMillicores": 989,
      "memoryInMebibytes": 5807,
      "podMetrics": [
        {
          "namespace": "dual-cluster-full",
          "podName": "envoy-proxy-node2-866544c45d-6xhpk",
          "cpuInMillicores": 10,
          "memoryInMebibytes": 27
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "haproxy-node2-54fc8fd459-6wzfr",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 108
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "hiero-explorer-1-5585d47c6-8rv5s",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 2
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "minio-pool-1-0",
          "cpuInMillicores": 101,
          "memoryInMebibytes": 420
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "mirror-1-grpc-69d6c769d8-nlhbl",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 324
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "mirror-1-importer-76c8456c95-s7wrn",
          "cpuInMillicores": 82,
          "memoryInMebibytes": 494
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "mirror-1-monitor-58d8dbf95f-ppw98",
          "cpuInMillicores": 16,
          "memoryInMebibytes": 303
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "mirror-1-postgres-postgresql-0",
          "cpuInMillicores": 13,
          "memoryInMebibytes": 134
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "mirror-1-redis-node-0",
          "cpuInMillicores": 28,
          "memoryInMebibytes": 7
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "mirror-1-rest-6c4fdcb7d5-fvsfn",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 97
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "mirror-1-restjava-5b95f9d895-5h5jd",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 342
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "mirror-1-web3-589fdcc6b7-84cg7",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 418
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "mirror-ingress-controller-85858dfdf7-fspgp",
          "cpuInMillicores": 11,
          "memoryInMebibytes": 40
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "network-node2-0",
          "cpuInMillicores": 568,
          "memoryInMebibytes": 1753
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "relay-1-686fc69666-tswd5",
          "cpuInMillicores": 30,
          "memoryInMebibytes": 212
        },
        {
          "namespace": "dual-cluster-full",
          "podName": "relay-1-ws-7887947bbc-tvtfq",
          "cpuInMillicores": 19,
          "memoryInMebibytes": 203
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-kn66s",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 17
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-v92gg",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-e2e-dual-cluster-full-20438025580-1-c2-control-plane",
          "cpuInMillicores": 21,
          "memoryInMebibytes": 87
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-2nnn6",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 11
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-e2e-dual-cluster-full-20438025580-1-c2-control-plane",
          "cpuInMillicores": 34,
          "memoryInMebibytes": 538
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-e2e-dual-cluster-full-20438025580-1-c2-control-plane",
          "cpuInMillicores": 16,
          "memoryInMebibytes": 75
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-ntj5v",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 15
        },
        {
          "namespace": "kube-system",
          "podName": "kube-scheduler-e2e-dual-cluster-full-20438025580-1-c2-control-plane",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 23
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-fbb76bc7f-lr8d4",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 24
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-6fdc965494-dwdr4",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 10
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-676f677b6b-zx88c",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 37
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-speaker-k6h8g",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 54
        },
        {
          "namespace": "solo-setup",
          "podName": "minio-operator-5df5d4fc74-kqfkt",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 16
        }
      ]
    }
  ]
}

@github-actions
Copy link
Contributor

Log Metrics - Node Add Local

📊 Runner Resource Usage

╔═══════════════════════════════════════════════════════════════╗
║          GitHub Runner Resource Usage                         ║
╚═══════════════════════════════════════════════════════════════╝

⏱️  Test Duration: 20.0 minutes (21 data points)

📉 CPU Usage
100.0% |                     
 90.0% |                     
 80.0% |                     
 70.0% |                     
 60.0% |                     
 50.0% |                     
 40.0% |            +    +   
 30.0% |   +        #+   #   
 20.0% |  +#++++  ++##+  #+++
 10.0% |++######++#####++####
  0.0% |#####################
       +---------------------

📉 Memory Usage
100.0% |                     
 90.0% |                     
 80.0% |                     
 70.0% |                     
 60.0% |                     
 50.0% |                     
 40.0% |                     
 30.0% |                     
 20.0% | +++++++++++++++++++ 
 10.0% |+###################+
  0.0% |#####################
       +---------------------

✅ Resource usage within normal limits
    CPU Peak: 37.4%  |  Memory Peak: 19.2%


@github-actions
Copy link
Contributor

Log Metrics - Standard

📊 Runner Resource Usage

╔═══════════════════════════════════════════════════════════════╗
║          GitHub Runner Resource Usage                         ║
╚═══════════════════════════════════════════════════════════════╝

⏱️  Test Duration: 40.0 minutes (41 data points)

📉 CPU Usage
100.0% |                                         
 90.0% |                                         
 80.0% |                                         
 70.0% |                                         
 60.0% |                                         
 50.0% |                      +         +      + 
 40.0% |                      #+ ++ ++ +# ++  +# 
 30.0% |    +  +  +      + + +##+##+##+##+##++## 
 20.0% |   +#  #++#++ +  #+#+################### 
 10.0% |+++##++######+#++#######################+
  0.0% |#########################################
       +-----------------------------------------

📉 Memory Usage
100.0% |                                         
 90.0% |                                         
 80.0% |                                         
 70.0% |                                         
 60.0% |                                         
 50.0% |                                         
 40.0% |                         ++++++++++++++  
 30.0% |                   ++++++##############++
 20.0% |  +++++++++++++++++######################
 10.0% |++#######################################
  0.0% |#########################################
       +-----------------------------------------

✅ Resource usage within normal limits
    CPU Peak: 46.7%  |  Memory Peak: 38.4%


JSON Log Metrics

{
  "snapshotName": "performance-tests-431723",
  "date": "2025-12-22T17:22:19.095Z",
  "gitHubSha": "0fcb1d11a08cb613a8c4c0af8459d0e5e594506a",
  "soloVersion": "0.52.0",
  "soloChartVersion": "0.58.1",
  "consensusNodeVersion": "0.67.2",
  "mirrorNodeVersion": "0.143.0",
  "blockNodeVersion": "0.23.2",
  "relayVersion": "0.73.0",
  "explorerVersion": "25.1.1",
  "cpuInMillicores": 1357,
  "memoryInMebibytes": 7400,
  "runtimeInMinutes": 11,
  "transactionCount": 11319,
  "events": [],
  "clusterMetrics": [
    {
      "context": "default",
      "cpuInMillicores": 1357,
      "memoryInMebibytes": 7400,
      "podMetrics": [
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-kqmxk",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 17
        },
        {
          "namespace": "kube-system",
          "podName": "coredns-7c65d6cfc9-wcztk",
          "cpuInMillicores": 3,
          "memoryInMebibytes": 16
        },
        {
          "namespace": "kube-system",
          "podName": "etcd-e2e-standard-20438025580-1-c1-control-plane",
          "cpuInMillicores": 34,
          "memoryInMebibytes": 83
        },
        {
          "namespace": "kube-system",
          "podName": "kindnet-lbt6x",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 11
        },
        {
          "namespace": "kube-system",
          "podName": "kube-apiserver-e2e-standard-20438025580-1-c1-control-plane",
          "cpuInMillicores": 63,
          "memoryInMebibytes": 520
        },
        {
          "namespace": "kube-system",
          "podName": "kube-controller-manager-e2e-standard-20438025580-1-c1-control-plane",
          "cpuInMillicores": 16,
          "memoryInMebibytes": 65
        },
        {
          "namespace": "kube-system",
          "podName": "kube-proxy-hhf46",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 15
        },
        {
          "namespace": "kube-system",
          "podName": "kube-scheduler-e2e-standard-20438025580-1-c1-control-plane",
          "cpuInMillicores": 6,
          "memoryInMebibytes": 21
        },
        {
          "namespace": "kube-system",
          "podName": "metrics-server-fbb76bc7f-wntjs",
          "cpuInMillicores": 5,
          "memoryInMebibytes": 24
        },
        {
          "namespace": "local-path-storage",
          "podName": "local-path-provisioner-6fdc965494-8xd4w",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 9
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-controller-676f677b6b-vf55k",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 37
        },
        {
          "namespace": "metallb-system",
          "podName": "metallb-speaker-2b6c5",
          "cpuInMillicores": 7,
          "memoryInMebibytes": 54
        },
        {
          "namespace": "solo-65d74600",
          "podName": "envoy-proxy-node1-74bcfb64f8-9zfjj",
          "cpuInMillicores": 18,
          "memoryInMebibytes": 27
        },
        {
          "namespace": "solo-65d74600",
          "podName": "haproxy-node1-7bb68675fc-vgfrv",
          "cpuInMillicores": 57,
          "memoryInMebibytes": 110
        },
        {
          "namespace": "solo-65d74600",
          "podName": "hiero-explorer-1-5585d47c6-qgrbf",
          "cpuInMillicores": 1,
          "memoryInMebibytes": 2
        },
        {
          "namespace": "solo-65d74600",
          "podName": "minio-pool-1-0",
          "cpuInMillicores": 25,
          "memoryInMebibytes": 266
        },
        {
          "namespace": "solo-65d74600",
          "podName": "mirror-1-grpc-7d76dbd75b-ggrmd",
          "cpuInMillicores": 8,
          "memoryInMebibytes": 319
        },
        {
          "namespace": "solo-65d74600",
          "podName": "mirror-1-importer-5565f96d8d-j9486",
          "cpuInMillicores": 77,
          "memoryInMebibytes": 489
        },
        {
          "namespace": "solo-65d74600",
          "podName": "mirror-1-monitor-7dfd46dc45-5h6sl",
          "cpuInMillicores": 60,
          "memoryInMebibytes": 300
        },
        {
          "namespace": "solo-65d74600",
          "podName": "mirror-1-postgres-postgresql-0",
          "cpuInMillicores": 82,
          "memoryInMebibytes": 207
        },
        {
          "namespace": "solo-65d74600",
          "podName": "mirror-1-redis-node-0",
          "cpuInMillicores": 47,
          "memoryInMebibytes": 7
        },
        {
          "namespace": "solo-65d74600",
          "podName": "mirror-1-rest-5c464f8d9b-k4nqr",
          "cpuInMillicores": 49,
          "memoryInMebibytes": 110
        },
        {
          "namespace": "solo-65d74600",
          "podName": "mirror-1-restjava-59599bdbdb-jrscp",
          "cpuInMillicores": 9,
          "memoryInMebibytes": 346
        },
        {
          "namespace": "solo-65d74600",
          "podName": "mirror-1-web3-8584464f49-j86mn",
          "cpuInMillicores": 4,
          "memoryInMebibytes": 416
        },
        {
          "namespace": "solo-65d74600",
          "podName": "mirror-ingress-controller-7c88bd549b-gz75l",
          "cpuInMillicores": 21,
          "memoryInMebibytes": 37
        },
        {
          "namespace": "solo-65d74600",
          "podName": "network-node1-0",
          "cpuInMillicores": 736,
          "memoryInMebibytes": 3460
        },
        {
          "namespace": "solo-65d74600",
          "podName": "relay-1-686fc69666-pswbh",
          "cpuInMillicores": 8,
          "memoryInMebibytes": 211
        },
        {
          "namespace": "solo-65d74600",
          "podName": "relay-1-ws-7887947bbc-gzsvh",
          "cpuInMillicores": 9,
          "memoryInMebibytes": 206
        },
        {
          "namespace": "solo-setup",
          "podName": "minio-operator-5df5d4fc74-9kv88",
          "cpuInMillicores": 2,
          "memoryInMebibytes": 15
        }
      ]
    }
  ]
}

@codacy-production
Copy link

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
+0.40% (target: -1.00%) 96.20%
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (eca0979) 50980 42509 83.38%
Head commit (34b91f7) 51048 (+68) 42771 (+262) 83.79% (+0.40%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#3049) 79 76 96.20%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

@Ivo-Yankov Ivo-Yankov added PR: Ready to Merge A pull request that is ready to merge. and removed PR: Needs Team Approval A pull request that needs review from a team member. labels Dec 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

PR: Ready to Merge A pull request that is ready to merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reduce-memory-footprint: integrate NLG to run a better load

5 participants