Skip to content

🌱 Add alloy log collectors - #3610

Open
UncivilizedGhost wants to merge 1 commit into
metal3-io:mainfrom
Nordix:saad/alloy_logs
Open

🌱 Add alloy log collectors#3610
UncivilizedGhost wants to merge 1 commit into
metal3-io:mainfrom
Nordix:saad/alloy_logs

Conversation

@UncivilizedGhost

@UncivilizedGhost UncivilizedGhost commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

this is just to test them. will clean script later

What this PR does / why we need it:

deploy-alloy.sh is used to have Grafana Alloy runs as a DaemonSet inside
the clusters used by E2E tests.
The set runs one alloy pod per node
Discovers pods on the local node and gets theurlog files from /var/log/pods/...
It pushes them to Loki with external labels

setup-host-docker-logs.sh creates a standalone Alloy container
with the Docker socket mounted
It discovery.docker to find containers, filters to
only known Metal3 infra containers (ironic, dnsmasq, vbmc, etc.)
and gets their logs via the Docker API
Pushes to the same Loki endpoint with the same external labels

Create scripts to test log collection by creating a log and deploying
cluster to it

Fixes #

Checklist:

  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • E2E tests have been added, if necessary.
  • Integration tests have been added, if necessary.

@metal3-io-bot metal3-io-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 12, 2026
@metal3-io-bot

metal3-io-bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign adilghaffardev for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@metal3-io-bot metal3-io-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Aug 12, 2026
@UncivilizedGhost
UncivilizedGhost force-pushed the saad/alloy_logs branch 4 times, most recently from 35048e0 to 7ff128f Compare August 14, 2026 08:04
@UncivilizedGhost

UncivilizedGhost commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

/copilot-review

@metal3-io-bot
metal3-io-bot requested a balanced review from Copilot August 14, 2026 08:04
@metal3-io-bot

metal3-io-bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Copilot code review has been requested by @UncivilizedGhost. Please allow a few moments for the review to be added.

Copilot AI reviewed Aug 14, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds optional Grafana Alloy collectors for shipping Kubernetes and host Docker logs to Loki.

Changes:

  • Integrates Alloy deployment into kind and E2E workflows.
  • Adds standalone setup, verification, and cleanup tooling.
  • Configures Kubernetes and Docker log discovery and forwarding.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
Makefile Adds log-collection targets.
scripts/ci-e2e.sh Deploys Alloy during E2E setup.
hack/kind_with_registry.sh Adds Alloy to kind setup.
hack/log-collection/deploy-alloy.sh Deploys Alloy via Helm.
hack/log-collection/setup.sh Creates a standalone test cluster.
hack/log-collection/verify.sh Provides collector diagnostics.
hack/log-collection/cleanup.sh Deletes the test cluster.
hack/log-collection/kind-config.yaml Defines the three-node kind cluster.
hack/log-collection/alloy-values.yaml Configures Kubernetes log collection.
hack/log-collection/setup-host-docker-logs.sh Starts host Docker collection.
hack/log-collection/cleanup-host-docker-logs.sh Stops host Docker collection.
hack/log-collection/docker-compose.host-logs.yml Defines the host Alloy container.
hack/log-collection/alloy-docker-config.alloy Configures Docker log forwarding.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread hack/kind_with_registry.sh Outdated
Comment thread hack/log-collection/verify.sh Outdated
Comment thread hack/log-collection/docker-compose.host-logs.yml Outdated
Comment thread hack/log-collection/deploy-alloy.sh Outdated
Comment thread hack/log-collection/alloy-values.yaml Outdated
Comment thread hack/log-collection/alloy-docker-config.alloy Outdated
Comment thread hack/log-collection/alloy-docker-config.alloy
@UncivilizedGhost
UncivilizedGhost force-pushed the saad/alloy_logs branch 4 times, most recently from b7cdd8f to 88c32a5 Compare August 17, 2026 10:44
@UncivilizedGhost UncivilizedGhost mentioned this pull request Aug 17, 2026
Merged
@UncivilizedGhost

UncivilizedGhost commented Aug 17, 2026

Copy link
Copy Markdown
Member Author

/copilot-review

@metal3-io-bot
metal3-io-bot requested a balanced review from Copilot August 17, 2026 11:45
@metal3-io-bot

metal3-io-bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Copilot code review has been requested by @UncivilizedGhost. Please allow a few moments for the review to be added.

Copilot AI reviewed Aug 17, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

hack/log-collection/verify.sh:57

  • wc -l commonly outputs leading whitespace (e.g., \" 3\"). In [[ ... -ne ... ]] numeric comparisons, that can trigger integer expression expected errors on some shells/environments. Prefer counting without wc padding (e.g., awk 'END{print NR}', or strip whitespace with tr -d '[:space:]') so the numeric comparison is robust.
log "Verifying pod count matches node count..."
NODE_COUNT=$(kubectl --context "${CTX}" get nodes --no-headers | wc -l)
POD_COUNT=$(kubectl --context "${CTX}" -n "${NAMESPACE}" get pods -l app.kubernetes.io/name=alloy --no-headers | wc -l)

if [[ "${POD_COUNT}" -ne "${NODE_COUNT}" ]]; then
  err "Expected ${NODE_COUNT} Alloy pods (one per node), but found ${POD_COUNT}."

hack/kind_with_registry.sh:29

  • The Alloy deployment block is duplicated in this script (once in the 'cluster already exists' path, and again after cluster creation). Consider extracting this into a small helper function (or a single common block) to reduce duplication and avoid future drift between the two code paths.
if [[ "$(kind get clusters)" =~ .*"${KIND_CLUSTER_NAME}".* ]]; then
    echo "cluster already exists, moving on"
    # Deploy Alloy on existing clusters
    SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
    export ALLOY_KUBE_CONTEXT="kind-${KIND_CLUSTER_NAME}"
    "${SCRIPT_DIR}/log-collection/deploy-alloy.sh"
    exit 0

Comment thread hack/log-collection/deploy-alloy.sh
@UncivilizedGhost UncivilizedGhost changed the title WIP: 🌱 Add alloy log collectors 🌱 Add alloy log collectors Aug 17, 2026
@metal3-io-bot metal3-io-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 17, 2026
@UncivilizedGhost

UncivilizedGhost commented Aug 18, 2026

Copy link
Copy Markdown
Member Author

/test metal3-centos-e2e-integration-test-main
/test metal3-ubuntu-e2e-integration-test-main

@adilGhaffarDev

adilGhaffarDev commented Aug 20, 2026

Copy link
Copy Markdown
Member

/test metal3-centos-e2e-integration-test-main
/test metal3-ubuntu-e2e-integration-test-main
project infra pr is merged.

adilGhaffarDev
adilGhaffarDev reviewed Aug 20, 2026
View reviewed changes

@adilGhaffarDev adilGhaffarDev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is currently missing collection of libvirt/QEMU logs, we can do it separate PR. Just wanted to point out so we dont miss it.

Comment thread scripts/ci-e2e.sh Outdated
ALLOY_KUBE_CONTEXT="$(kubectl config current-context 2>/dev/null || echo "")"
export ALLOY_KUBE_CONTEXT
"${REPO_ROOT}/hack/log-collection/deploy-alloy.sh"

@adilGhaffarDev adilGhaffarDev Aug 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only deploys Alloy to the current bootstrap context, so logs from every target cluster are missed. Please move bootstrap deployment into the Go bootstrap lifecycle and deploy Alloy from CreateTargetCluster after each workload cluster is ready, ensuring all target clusters get the DaemonSet.

@UncivilizedGhost UncivilizedGhost Aug 20, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

@UncivilizedGhost

UncivilizedGhost commented Aug 20, 2026

Copy link
Copy Markdown
Member Author

/test metal3-centos-e2e-integration-test-main
/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost

UncivilizedGhost commented Aug 20, 2026

Copy link
Copy Markdown
Member Author

/test metal3-centos-e2e-integration-test-main
/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost

UncivilizedGhost commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost

UncivilizedGhost commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

/test metal3-centos-e2e-integration-test-main
/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost

UncivilizedGhost commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

/test metal3-ubuntu-e2e-integration-test-main

1 similar comment
@UncivilizedGhost

UncivilizedGhost commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost

UncivilizedGhost commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost

UncivilizedGhost commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost

UncivilizedGhost commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost

UncivilizedGhost commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost

UncivilizedGhost commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost

UncivilizedGhost commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

/test metal3-ubuntu-e2e-integration-test-main

@UncivilizedGhost
Add Alloy log collectors
f8298a7 
deploy-alloy.sh is used to have Grafana Alloy runs as a DaemonSet inside
the clusters used by E2E tests.
The set runs one alloy pod per node
Discovers pods on the local node and gets theurlog files from /var/log/pods/...
It pushes them to Loki with external labels

setup-host-docker-logs.sh creates a standalone Alloy container
with the Docker socket mounted
It discovery.docker to find containers, filters to
only known Metal3 infra containers (ironic, dnsmasq, vbmc, etc.)
and gets their logs via the Docker API
Pushes to the same Loki endpoint with the same external labels

Create scipts to test log collection by creating a log and deploying
cluster to it

Signed-off-by: Saad Zia <saad.zia@est.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@fmuyassarov fmuyassarov Awaiting requested review from fmuyassarov
@Rozzii Rozzii Awaiting requested review from Rozzii
@zhouhao3 zhouhao3 Awaiting requested review from zhouhao3
+1 more reviewer
@adilGhaffarDev adilGhaffarDev adilGhaffarDev left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@UncivilizedGhost @metal3-io-bot @adilGhaffarDev