Feature/multi loader logs collection #598

nosnelmil · 2025-02-04T02:21:14Z

Summary

Extends multi-loader by collecting key logs from nodes in the cluster for the Knative platform. Users can optionally collect the following logs:

TOP (resource usage metrics)
Prometheus snapshots
Logs from the Activator Pod
Logs from the Autoscaler Pod

Implementation Notes ⚒️

Added an additional Metrics field in the multi-loader config, accepting an array with any of the following values: top, prometheus, activator, autoscaler.
Introduced optional fields: MasterNode, ActivatorNode, AutoscalerNode, and WorkerNodes to allow users to manually specify IPs instead of relying on multi-loader to determine them (mostly unnecessary in typical scenarios).
Uses kubectl to automatically determine node IPs and classify them based on their roles.
Resets TOP metrics for all nodes before starting any experiment.
Collects Activator Pod logs from:
/var/log/pods/knative-serving_activator-*/activator/*
Collects Autoscaler Pod logs from:
/var/log/pods/knative-serving_autoscaler-*/autoscaler/*
Copies Prometheus snapshots by first triggering a snapshot via the Prometheus API on the master node and then retrieving the generated snapshot.
Additionally, log collection logic runs during multi-loader dry run to:
- Validate that identified IPs are reachable.
- Ensure SSH access and necessary permissions.
- Execute log retrieval commands to detect potential errors.
- Delete any collected logs after validation, as no experiments have been executed yet.

External Dependencies 🍀

N/A

Breaking API Changes ⚠️

N/A

docs/multi_loader.md

tools/multi_loader/common/utils.go

tools/multi_loader/metric/metric_manager.go

tools/multi_loader/types/config_types.go

leokondrashov

Looks nice. I left some comments.

tools/multi_loader/metric/metric_manager.go

docs/multi_loader.md

tools/multi_loader/common/utils.go

scripts/setup/setup_kind_metrics.sh

scripts/setup/setup_kind_node.sh

scripts/setup/setup_kind_ssh.sh

nosnelmil · 2025-03-04T04:02:01Z

@leokondrashov as discussed, added the log consolidation logic in 0dc0950

leokondrashov

Looks great, thanks. Please fix couple minor comments

docs/multi_loader.md

tools/multi_loader/metric/metric_manager.go

tools/multi_loader/metric/metric_manager_test.go

Signed-off-by: Lenson <[email protected]> add node discovery validators Signed-off-by: Lenson <[email protected]> add collect TOP metric functions Signed-off-by: Lenson <[email protected]> add multi-loader metric_manager Signed-off-by: Lenson <[email protected]> add autoscaler log collection Signed-off-by: Lenson <[email protected]> add activator log collection Signed-off-by: Lenson <[email protected]> add prometh log collection Signed-off-by: Lenson <[email protected]> refactor metric manager contants Signed-off-by: Lenson <[email protected]> minor fix for node discovery Signed-off-by: Lenson <[email protected]> fix node discovery Signed-off-by: Lenson <[email protected]> minor fix Signed-off-by: Lenson <[email protected]> minor fix Signed-off-by: Lenson <[email protected]> add logs for prometh Signed-off-by: Lenson <[email protected]> add pause between prometh collection Signed-off-by: Lenson <[email protected]> update wait time Signed-off-by: Lenson <[email protected]> update condition for node discovery Signed-off-by: Lenson <[email protected]> update logging Signed-off-by: Lenson <[email protected]>

Signed-off-by: Lenson <[email protected]> update kind ssh update script Signed-off-by: Lenson <[email protected]> fix setup kind ssh Signed-off-by: Lenson <[email protected]>

Signed-off-by: Lenson <[email protected]> update setup metrics script Signed-off-by: Lenson <[email protected]>

Signed-off-by: Lenson <[email protected]>

Signed-off-by: Lenson <[email protected]> fix log collection test commit a05990d Author: Lenson <[email protected]> Date: Mon Feb 3 15:39:39 2025 +0800 update test trigger Signed-off-by: Lenson <[email protected]> commit 3edb3b4 Author: Lenson <[email protected]> Date: Mon Feb 3 15:33:06 2025 +0800 update test Signed-off-by: Lenson <[email protected]> commit 56a0f7d Author: Lenson <[email protected]> Date: Mon Feb 3 15:18:40 2025 +0800 fix Signed-off-by: Lenson <[email protected]> commit 67c520d Author: Lenson <[email protected]> Date: Mon Feb 3 15:06:20 2025 +0800 fix Signed-off-by: Lenson <[email protected]> commit 48ff845 Author: Lenson <[email protected]> Date: Mon Feb 3 14:46:29 2025 +0800 test' Signed-off-by: Lenson <[email protected]> commit 295c761 Author: Lenson <[email protected]> Date: Mon Feb 3 14:45:35 2025 +0800 add adv log collection tests Signed-off-by: Lenson <[email protected]> commit 8469bdb Author: Lenson <[email protected]> Date: Mon Feb 3 14:45:05 2025 +0800 update logging Signed-off-by: Lenson <[email protected]> commit 10e295a Author: Lenson <[email protected]> Date: Mon Feb 3 14:44:42 2025 +0800 update kind ssh update script Signed-off-by: Lenson <[email protected]> commit c56a9d8 Author: Lenson <[email protected]> Date: Mon Feb 3 13:19:27 2025 +0800 add KinD ssh setup script Signed-off-by: Lenson <[email protected]> commit bf9a804 Author: Lenson <[email protected]> Date: Mon Feb 3 10:31:55 2025 +0800 update condition for node discovery Signed-off-by: Lenson <[email protected]> commit b3f078b Author: Lenson <[email protected]> Date: Fri Jan 31 18:35:03 2025 +0800 add multi loader log collection Signed-off-by: Lenson <[email protected]> add node discovery validators Signed-off-by: Lenson <[email protected]> add collect TOP metric functions Signed-off-by: Lenson <[email protected]> add multi-loader metric_manager Signed-off-by: Lenson <[email protected]> add autoscaler log collection Signed-off-by: Lenson <[email protected]> add activator log collection Signed-off-by: Lenson <[email protected]> add prometh log collection Signed-off-by: Lenson <[email protected]> refactor metric manager contants Signed-off-by: Lenson <[email protected]> minor fix for node discovery Signed-off-by: Lenson <[email protected]> fix node discovery Signed-off-by: Lenson <[email protected]> minor fix Signed-off-by: Lenson <[email protected]> minor fix Signed-off-by: Lenson <[email protected]> add logs for prometh Signed-off-by: Lenson <[email protected]> add pause between prometh collection Signed-off-by: Lenson <[email protected]> update wait time Signed-off-by: Lenson <[email protected]> commit 9bac3c4 Author: Lenson <[email protected]> Date: Tue Jan 21 13:00:50 2025 +0800 update multi loader docs Signed-off-by: Lenson <[email protected]> update multi-loader docs Signed-off-by: Lenson <[email protected]> commit bfd17be Author: Lenson <[email protected]> Date: Mon Jan 20 16:30:13 2025 +0800 minor multi loader fix Signed-off-by: Lenson <[email protected]> fix incorrect retry logging Signed-off-by: Lenson <[email protected]> remove iat and generated cli args Signed-off-by: Lenson <[email protected]> remove make clean from clean up Signed-off-by: Lenson <[email protected]> commit 91042aa Author: Lenson <[email protected]> Date: Thu Jan 16 15:53:19 2025 +0800 update tests Signed-off-by: Lenson <[email protected]> update multi loader e2e tests Signed-off-by: Lenson <[email protected]> revert setup.cfg Signed-off-by: Lenson <[email protected]> chmod script Signed-off-by: Lenson <[email protected]> update unit tests Signed-off-by: Lenson <[email protected]> fix e2e test Signed-off-by: Lenson <[email protected]> update tests Signed-off-by: Lenson <[email protected]> commit 69c3c3a Author: Lenson <[email protected]> Date: Tue Dec 31 11:49:55 2024 +0800 add failfast flag Signed-off-by: Lenson <[email protected]> update failfast flag description Signed-off-by: Lenson <[email protected]> update comments Signed-off-by: Lenson <[email protected]> update wordlist with multiloader specific words Signed-off-by: Lenson <[email protected]> simplify run experiment logic Signed-off-by: Lenson <[email protected]> refactor partial experiment naming Signed-off-by: Lenson <[email protected]> fix wrong indexing Signed-off-by: Lenson <[email protected]> add progress in logging Signed-off-by: Lenson <[email protected]> commit fc3ad98 Author: Lenson <[email protected]> Date: Sun Nov 17 14:07:35 2024 +0800 refactor multi loader Signed-off-by: Lenson <[email protected]> add multi-loader tests Signed-off-by: Lenson <[email protected]> update test Signed-off-by: Lenson <[email protected]> refactor multi-loader tests Signed-off-by: Lenson <[email protected]> add loader experiment Signed-off-by: Lenson <[email protected]> update logs Signed-off-by: Lenson <[email protected]> update log verbosity Signed-off-by: Lenson <[email protected]> update logs Signed-off-by: Lenson <[email protected]> update logs Signed-off-by: Lenson <[email protected]> rename multiloader driver to runner Signed-off-by: Lenson <[email protected]> refactor common files to multiloader folder Signed-off-by: Lenson <[email protected]> refactor multiloader functions Signed-off-by: Lenson <[email protected]> rename createNewStudy function name Signed-off-by: Lenson <[email protected]> fix formatting Signed-off-by: Lenson <[email protected]> remove extra features Signed-off-by: Lenson <[email protected]> remove extra features Signed-off-by: Lenson <[email protected]> add validation for platform Signed-off-by: Lenson <[email protected]> commit ca5e2ad Author: Lenson <[email protected]> Date: Sat Nov 16 18:49:35 2024 +0800 add multi loader documentation Signed-off-by: Lenson <[email protected]> update docs Signed-off-by: Lenson <[email protected]> fix docs Signed-off-by: Lenson <[email protected]> update documentation Signed-off-by: Lenson <[email protected]> commit 3c7e6b5 Author: Lenson <[email protected]> Date: Sat Nov 16 12:36:43 2024 +0800 add multi-loader Signed-off-by: Lenson <[email protected]> add multi-loader config reader Signed-off-by: Lenson <[email protected]> add multi loader base Signed-off-by: Lenson <[email protected]> add multi loader base Signed-off-by: Lenson <[email protected]> add node group struct Signed-off-by: Lenson <[email protected]> add multi loader runner Signed-off-by: Lenson <[email protected]> refactor multi loader config Signed-off-by: Lenson <[email protected]> add multi loader config validators Signed-off-by: Lenson <[email protected]> add knative specific config enricher Signed-off-by: Lenson <[email protected]> add additional knative platform type Signed-off-by: Lenson <[email protected]> add base runner entry point Signed-off-by: Lenson <[email protected]> refactor multi loader config Signed-off-by: Lenson <[email protected]> update multi loader config struct Signed-off-by: Lenson <[email protected]> update unpack study doc Signed-off-by: Lenson <[email protected]> add unpack study Signed-off-by: Lenson <[email protected]> add prepare experiment Signed-off-by: Lenson <[email protected]> update experiment config temp path Signed-off-by: Lenson <[email protected]> add run loader function Signed-off-by: Lenson <[email protected]> update log parser Signed-off-by: Lenson <[email protected]> update log parser Signed-off-by: Lenson <[email protected]> update log parser Signed-off-by: Lenson <[email protected]> add clean up function Signed-off-by: Lenson <[email protected]> add logs to indicate run status Signed-off-by: Lenson <[email protected]> expose entry points for multi loader runner Signed-off-by: Lenson <[email protected]> add multi loader runner execution Signed-off-by: Lenson <[email protected]> update default multi loader config path Signed-off-by: Lenson <[email protected]> add cpu limit validator Signed-off-by: Lenson <[email protected]> remove extra knative feature Signed-off-by: Lenson <[email protected]> remove knative extra features Signed-off-by: Lenson <[email protected]> add multi loader tests Signed-off-by: Lenson <[email protected]> add basic config Signed-off-by: Lenson <[email protected]> update basic config Signed-off-by: Lenson <[email protected]> update basic config Signed-off-by: Lenson <[email protected]> add basic configs Signed-off-by: Lenson <[email protected]> update base config Signed-off-by: Lenson <[email protected]> Signed-off-by: Lenson <[email protected]> update e2e test Signed-off-by: Lenson <[email protected]>

Signed-off-by: Lenson <[email protected]>

Signed-off-by: Lenson <[email protected]> update metrics description in docs Signed-off-by: Lenson <[email protected]>

Signed-off-by: Lenson <[email protected]>

Signed-off-by: Lenson <[email protected]> add interval for prometh snapshot collection Signed-off-by: Lenson <[email protected]>

Signed-off-by: Lenson <[email protected]>

nosnelmil · 2025-03-12T02:56:25Z

Hi @cvetkovic, this PR extends the previously added multi-loader tool by introducing enhanced log collection capabilities. The new feature allows users to gather logs from the Activator and Autoscaler nodes, retrieve TOP metrics from all cluster nodes, and capture Prometheus snapshots. Users can also specify the exact metrics they want to collect using the newly introduced Metric field in the multi-loader configuration.

I would appreciate your review and if everything looks good, I will tidy up the commits and prepare for merging into main. Thank you!

cvetkovic · 2025-03-13T08:58:24Z

.github/workflows/e2e_multi_loader.yaml

+      KIND_VERSION: v0.22.0
+      K8S_VERSION: v1.29
+      YAML_DIR: workloads/container
+    runs-on: ubuntu-20.04


Switch to 24.04. The current one will be deprecated.

Resolved in 44e22ad

cvetkovic · 2025-03-13T09:01:09Z

tools/multi_loader/runner/multi_loader_runner.go

@@ -48,13 +50,23 @@ func NewMultiLoaderRunner(configPath string, verbosity string, failFast bool) (*
 	// Determine platform
 	platform := ml_common.DeterminePlatformFromConfig(multiLoaderConfig)

+	// Determine Node Group for Knative platform
+	if strings.HasPrefix(platform, "Knative") && len(multiLoaderConfig.Metrics) > 0 {


"Knative" has been replaced by a constant.

Resolved in 149b908

Signed-off-by: Lenson <[email protected]>

nosnelmil force-pushed the feature/multi-loader-adv-logs branch 2 times, most recently from 3dfaaaf to d5d74ac Compare February 4, 2025 03:16

nosnelmil marked this pull request as draft February 4, 2025 03:17

nosnelmil requested a review from leokondrashov February 4, 2025 03:17

nosnelmil force-pushed the feature/multi-loader-adv-logs branch 9 times, most recently from 44717ff to c4798ba Compare February 13, 2025 06:59

nosnelmil marked this pull request as ready for review February 13, 2025 07:08

leokondrashov reviewed Feb 14, 2025

View reviewed changes

nosnelmil force-pushed the feature/multi-loader-adv-logs branch 2 times, most recently from 535398d to 0e11bd8 Compare February 19, 2025 16:18

nosnelmil requested a review from leokondrashov February 19, 2025 16:28

leokondrashov reviewed Feb 28, 2025

View reviewed changes

nosnelmil force-pushed the feature/multi-loader-adv-logs branch 4 times, most recently from ba47317 to bfb51ab Compare March 4, 2025 03:59

nosnelmil requested a review from leokondrashov March 4, 2025 04:36

leokondrashov reviewed Mar 11, 2025

View reviewed changes

docs/multi_loader.md Outdated Show resolved Hide resolved

docs/multi_loader.md Outdated Show resolved Hide resolved

tools/multi_loader/metric/metric_manager.go Show resolved Hide resolved

tools/multi_loader/metric/metric_manager_test.go Outdated Show resolved Hide resolved

nosnelmil added 4 commits March 12, 2025 10:17

add KinD ssh setup script

99518f1

Signed-off-by: Lenson <[email protected]> update kind ssh update script Signed-off-by: Lenson <[email protected]> fix setup kind ssh Signed-off-by: Lenson <[email protected]>

add setup prometh script for kind

87314b7

Signed-off-by: Lenson <[email protected]> update setup metrics script Signed-off-by: Lenson <[email protected]>

add setup kind node script

1deb274

Signed-off-by: Lenson <[email protected]>

nosnelmil added 2 commits March 12, 2025 10:17

add adv logs documentation

2810612

Signed-off-by: Lenson <[email protected]>

nosnelmil force-pushed the feature/multi-loader-adv-logs branch from 67ea824 to 02c8260 Compare March 12, 2025 02:31

nosnelmil added 10 commits March 12, 2025 10:34

fix spelling lint error

e8e87a8

Signed-off-by: Lenson <[email protected]>

update metrics info in ml docs

3185c1d

Signed-off-by: Lenson <[email protected]> update metrics description in docs Signed-off-by: Lenson <[email protected]>

remove fatal calls in util functions

0091db3

Signed-off-by: Lenson <[email protected]>

refactor kind setup scripts

83ebbab

Signed-off-by: Lenson <[email protected]>

simplify storing ssh public key command

cad5081

Signed-off-by: Lenson <[email protected]>

fix prometh snapshot collection

702e4dc

Signed-off-by: Lenson <[email protected]> add interval for prometh snapshot collection Signed-off-by: Lenson <[email protected]>

add log consolidation logic

c0e4fea

Signed-off-by: Lenson <[email protected]>

update script

5e22ef7

Signed-off-by: Lenson <[email protected]>

add metric unit test to actions

df3ccbc

Signed-off-by: Lenson <[email protected]>

fix spelling errors

8580b19

Signed-off-by: Lenson <[email protected]>

nosnelmil force-pushed the feature/multi-loader-adv-logs branch from 02c8260 to 8580b19 Compare March 12, 2025 02:38

nosnelmil requested review from cvetkovic and leokondrashov March 12, 2025 02:49

cvetkovic reviewed Mar 13, 2025

View reviewed changes

nosnelmil added 2 commits March 13, 2025 23:03

update e2e test ubuntu version

44e22ad

Signed-off-by: Lenson <[email protected]>

update to use knative constant

149b908

Signed-off-by: Lenson <[email protected]>

nosnelmil force-pushed the feature/multi-loader-adv-logs branch from 9d2f932 to 149b908 Compare March 13, 2025 15:34

nosnelmil requested a review from cvetkovic March 13, 2025 15:52

Merge branch 'vhive-serverless:main' into feature/multi-loader-adv-logs

e134aa2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/multi loader logs collection #598

Feature/multi loader logs collection #598

nosnelmil commented Feb 4, 2025

leokondrashov left a comment

nosnelmil commented Mar 4, 2025

leokondrashov left a comment

nosnelmil commented Mar 12, 2025

cvetkovic Mar 13, 2025

nosnelmil Mar 13, 2025

cvetkovic Mar 13, 2025

nosnelmil Mar 13, 2025

Feature/multi loader logs collection #598

Are you sure you want to change the base?

Feature/multi loader logs collection #598

Conversation

nosnelmil commented Feb 4, 2025

Summary

Implementation Notes ⚒️

External Dependencies 🍀

Breaking API Changes ⚠️

leokondrashov left a comment

Choose a reason for hiding this comment

nosnelmil commented Mar 4, 2025

leokondrashov left a comment

Choose a reason for hiding this comment

nosnelmil commented Mar 12, 2025

cvetkovic Mar 13, 2025

Choose a reason for hiding this comment

nosnelmil Mar 13, 2025

Choose a reason for hiding this comment

cvetkovic Mar 13, 2025

Choose a reason for hiding this comment

nosnelmil Mar 13, 2025

Choose a reason for hiding this comment