Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🌱 Promtail to alloy migration #11945

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

🌱 Promtail to alloy migration #11945

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
27835ca
Promtail to alloy migration
irapandey Mar 6, 2025
83f1e1f
Update hack/observability/alloy/kustomization.yaml
irapandey Mar 29, 2025
e4bdf01
Added Configuration link
irapandey Mar 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions Tiltfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,9 +420,9 @@ def deploy_provider_crds():
)

def deploy_observability():
if "promtail" in settings.get("deploy_observability", []):
k8s_yaml(read_file("./.tiltbuild/yaml/promtail.observability.yaml"), allow_duplicates = True)
k8s_resource(workload = "promtail", extra_pod_selectors = [{"app": "promtail"}], labels = ["observability"], resource_deps = ["loki"], objects = ["promtail:serviceaccount"])
if "alloy" in settings.get("deploy_observability", []):
k8s_yaml(read_file("./.tiltbuild/yaml/alloy.observability.yaml"), allow_duplicates = True)
k8s_resource(workload = "alloy", extra_pod_selectors = [{"app": "alloy"}], labels = ["observability"], resource_deps = ["loki"], objects = ["alloy:serviceaccount"])

if "loki" in settings.get("deploy_observability", []):
k8s_yaml(read_file("./.tiltbuild/yaml/loki.observability.yaml"), allow_duplicates = True)
Expand Down
7 changes: 3 additions & 4 deletions docs/book/src/developer/core/logging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,19 +135,19 @@ thorny parts of code. Over time, based on feedback from SRE/developers, more log

## Developing and testing logs

Our [Tilt](tilt.md) setup offers a batteries-included log suite based on [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/), [Loki](https://grafana.com/docs/loki/latest/fundamentals/overview/) and [Grafana](https://grafana.com/docs/grafana/latest/explore/logs-integration/).
Our [Tilt](tilt.md) setup offers a batteries-included log suite based on [alloy](https://grafana.com/docs/loki/latest/send-data/alloy/), [Loki](https://grafana.com/docs/loki/latest/fundamentals/overview/) and [Grafana](https://grafana.com/docs/grafana/latest/explore/logs-integration/).

We are working to continuously improving this experience, allowing Cluster API developers to use logs and improve them as part of their development process.

For the best experience exploring the logs using Tilt:
1. Set `--logging-format=json`.
2. Set a high log verbosity, e.g. `v=5`.
3. Enable Promtail, Loki, and Grafana under `deploy_observability`.
3. Enable alloy, Loki, and Grafana under `deploy_observability`.

A minimal example of a tilt-settings.yaml file that deploys a ready-to-use logging suite looks like:
```yaml
deploy_observability:
- promtail
- alloy
- loki
- grafana
enable_providers:
Expand Down Expand Up @@ -208,7 +208,6 @@ Will return logs from the `capi-controller-manager`, associated with the Cluster
Will return the logs from four CAPI providers - the Core provider, Kubeadm Control Plane provider, Kubeadm Bootstrap provider and the Docker infrastructure provider. It filters by the cluster name and the machine name and then formats the log lines to show just the source controller and the message. This allows us to correlate logs and see actions taken by each of these four providers related to the machine `my-cluster-linux-worker-1`.

For more information on formatting and filtering logs using Grafana and Loki see:
- [json parsing](https://grafana.com/docs/loki/latest/clients/promtail/stages/json/)
- [log queries](https://grafana.com/docs/loki/latest/logql/log_queries/)

## What about providers
Expand Down
2 changes: 1 addition & 1 deletion docs/book/src/developer/core/testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ analyzing them via Grafana.

1. Start the development environment as described in [Developing Cluster API with Tilt](tilt.md).
* Make sure to deploy Loki and Grafana via `deploy_observability`.
* If you only want to see imported logs, don't deploy promtail (via `deploy_observability`).
* If you only want to see imported logs, don't deploy alloy (via `deploy_observability`).
* If you want to drop all logs from Loki, just delete the Loki Pod in the `observability` namespace.
2. You can then import logs via the `Import Logs` button on the top right of the [Loki resource page](http://localhost:10350/r/loki/overview).
Just click on the downwards arrow, enter either a ProwJob URL, a GCS path or a local folder and click on `Import Logs`.
Expand Down
2 changes: 1 addition & 1 deletion docs/book/src/developer/core/tilt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ Supported values are:
* `loki`: To receive and store logs.
* `metrics-server`: To enable `kubectl top node/pod`.
* `prometheus`*: For collecting metrics from Kubernetes.
* `promtail`: For providing pod logs to `loki`.
* `alloy`: For providing pod logs to `loki`.
* `parca`*: For visualizing profiling data.
* `tempo`: To store traces.
* `visualizer`*: Visualize Cluster API resources for each cluster, provide quick access to the specs and status of any resource.
Expand Down
2 changes: 1 addition & 1 deletion docs/book/src/developer/core/tuning.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ When tuning controllers, both for scalability, performance or for reducing their

Cluster API provides a full stack of tools for tuning its own controllers as well as controllers for all providers if developed using controller runtime. As a bonus, most of this tooling can be used with any other controller runtime based controllers.

With tilt, you can easily deploy a full observability stack with Grafana, Loki, promtail, Prometheus, kube-state-metrics, Parca and Tempo.
With tilt, you can easily deploy a full observability stack with Grafana, Loki, alloy, Prometheus, kube-state-metrics, Parca and Tempo.

All tools are preconfigured, and most notably kube-state-metrics already collects CAPI metrics and Grafana is configured with a set of dashboards that we used in previous rounds of CAPI tuning. Overall, the CAPI dev environment offers a considerable amount of expertise, free to use and to improve for the entire community. We highly recommend to invest time in looking into those tools, learn and provide feedback.

Expand Down
6 changes: 3 additions & 3 deletions ...observability/promtail/kustomization.yaml → hack/observability/alloy/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@ resources:
- ../namespace.yaml

helmCharts:
- name: promtail
- name: alloy
repo: https://grafana.github.io/helm-charts
releaseName: promtail
releaseName: alloy
namespace: observability
valuesFile: values.yaml
version: 6.16.6
version: 0.12.3

helmGlobals:
# Store chart in ".charts" folder instead of "charts".
Expand Down
3 changes: 1 addition & 2 deletions hack/observability/promtail/values.yaml → hack/observability/alloy/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# Configuration for promtail chart, see https://github.com/grafana/helm-charts/tree/main/charts/promtail
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add the right link for alloy?

Copy link
Member

@chrischdi chrischdi Mar 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it is this? https://github.com/grafana/alloy/blob/main/operations/helm/charts/alloy/values.yaml

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

❯ helm show chart grafana/alloy
apiVersion: v2
appVersion: v1.7.2
dependencies:
- condition: crds.create
  name: crds
  repository: ""
  version: 0.0.0
description: Grafana Alloy
icon: https://raw.githubusercontent.com/grafana/alloy/main/docs/sources/assets/alloy_icon_orange.svg
name: alloy
type: application
version: 0.12.3


# Configuration for alloy - https://github.com/grafana/alloy/blob/main/operations/helm/charts/alloy/values.yaml
config:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure we have to change things here as its a different helm chart.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find dedicated charts for alloy like for promtail - https://grafana.com/docs/alloy/latest/set-up/install/kubernetes/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

helm show values grafana/alloy does not even have a config field.

So all configuration below would be not used (and we want them to work).

So I'm curious if this even shows us logs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This page maybe helps: https://grafana.com/docs/alloy/latest/set-up/migrate/from-promtail/

Note: the values.yaml is not the promtail configuration.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should get the promtail.yaml via: kustomize build --enable-helm --load-restrictor LoadRestrictionsNone hack/observability/promtail | grep stringData -A 38 | yq '.stringData[]' > promtail.yaml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me take a look and get back - I kinda had the same feeling but needed some input
Thanks 😄

# publish data to loki
clients:
Expand Down
2 changes: 1 addition & 1 deletion hack/observability/grafana/chart/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ datasources:
isDefault: true
editable: true
# This header has to be set as we had to set an OrgID
# in promtail to be able to push the logs to Loki.
# in alloy to be able to push the logs to Loki.
jsonData:
maxLines: 1000
derivedFields:
Expand Down
2 changes: 1 addition & 1 deletion test/framework/deployment_helpers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@ func (eh *watchPodLogsEventHandler) streamPodLogs(pod *corev1.Pod) {
}

// logMetadata contains metadata about the logs.
// The format is very similar to the one used by promtail.
// The format is very similar to the one used by alloy.
type logMetadata struct {
Job string `json:"job"`
Namespace string `json:"namespace"`
Expand Down