Skip to content

Commit 7905837

Browse files
aleks-paleks-p
authored
Example for Java profiling using Grafana Alloy in Kubernetes (#3603)
* example for java profiling using alloy in kubernetes * Split config into files * Add linux capabilities to the example
1 parent a3b9a83 commit 7905837

File tree

12 files changed

+380
-3
lines changed

12 files changed

+380
-3
lines changed

‎examples/grafana-agent-auto-instrumentation/java/FastSlow.java renamed to ‎examples/grafana-agent-auto-instrumentation/java/docker/FastSlow.java

File renamed without changes.

‎examples/grafana-agent-auto-instrumentation/java/README.md renamed to ‎examples/grafana-agent-auto-instrumentation/java/docker/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Grafana Agent Java profiling via auto-instrumentation example
1+
# Grafana Agent Java profiling via auto-instrumentation example in Docker
22

33
This repository provides a practical demonstration of leveraging the Grafana Agent for continuous Java application profiling using Pyroscope in a dockerized environment. It illustrates a seamless approach to profiling Java processes, aiding in performance optimization.
44

‎examples/grafana-agent-auto-instrumentation/java/config.river renamed to ‎examples/grafana-agent-auto-instrumentation/java/docker/config.river

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ logging {
44
}
55

66
discovery.process "all" {
7-
// join kuberenetes targets with process targets on container_id to have k8s labels
7+
// join kubernetes targets with process targets on container_id to have k8s labels
88
// join = discovery.kubernetes.containers.targets
99
}
1010

@@ -53,4 +53,4 @@ pyroscope.write "example" {
5353
external_labels = {
5454
"env" = "example",
5555
}
56-
}
56+
}

‎examples/grafana-agent-auto-instrumentation/java/docker-compose.yml renamed to ‎examples/grafana-agent-auto-instrumentation/java/docker/docker-compose.yml

File renamed without changes.

‎examples/grafana-agent-auto-instrumentation/java/grafana-provisioning/datasources/pyroscope.yml renamed to ‎examples/grafana-agent-auto-instrumentation/java/docker/grafana-provisioning/datasources/pyroscope.yml

File renamed without changes.

‎examples/grafana-agent-auto-instrumentation/java/grafana-provisioning/plugins/explore-profiles.yml renamed to ‎examples/grafana-agent-auto-instrumentation/java/docker/grafana-provisioning/plugins/explore-profiles.yml

File renamed without changes.

‎examples/grafana-agent-auto-instrumentation/java/java.Dockerfile renamed to ‎examples/grafana-agent-auto-instrumentation/java/docker/java.Dockerfile

File renamed without changes.

‎examples/grafana-agent-auto-instrumentation/java/kubernetes/README.md

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Grafana Alloy Java profiling via auto-instrumentation in Kubernetes
2+
3+
This repository provides a practical demonstration of leveraging Grafana Alloy for continuous Java application profiling using Pyroscope in Kubernetes.
4+
It illustrates a seamless approach to profiling Java processes, aiding in performance optimization.
5+
6+
## Overview
7+
8+
Grafana Alloy automates Java process discovery for profiling, streamlining the setup for applications. It enables precise and targeted profiling configurations through the Grafana Alloy settings.
9+
10+
Java profiling via Grafana Alloy is based on a few components:
11+
- `discovery.process` for process discovery
12+
- `discovery.kubernetes` for adding Kubernetes labels (namespace, pod, and more)
13+
- `discovery.relabel` for detecting java processes and setting up labels
14+
- `pyroscope.java` for enabling profiling for specific applications
15+
- `pyroscope.write` for writing the profiles data to a remote endpoint
16+
17+
Refer to the [official documentation](https://grafana.com/docs/pyroscope/latest/configure-client/grafana-agent/java/) for an in-depth understanding and additional configuration options for Java profiling with Grafana Alloy.
18+
Also, check the [Grafana Alloy Components reference](https://grafana.com/docs/alloy/latest/reference/components/) for more details on each used component.
19+
20+
### async-profiler
21+
22+
The `pyroscope.java` agent component internally uses the [async-profiler](https://github.com/async-profiler/async-profiler) library.
23+
This approach offers a key advantage over other instrumentation mechanisms in that you can profile applications that are already running without interruptions (code changes, config changes or restarts).
24+
25+
Under the hood, this is achieved by attaching to the application at a process level and issuing commands to control profiling.
26+
27+
## Getting started
28+
29+
To use this example:
30+
31+
1. Set up a local kubernetes cluster using Kind or a similar tool.
32+
2. Clone this repository and navigate to this example's directory.
33+
3. Create a `pyroscope-java` namespace:
34+
```shell
35+
kubectl create namespace pyroscope-java
36+
```
37+
4. Deploy the manifests:
38+
```shell
39+
kubectl apply -n pyroscope-java -f .
40+
```
41+
42+
After the deployment is operational, the Grafana Alloy will profile the Java application using the defined configuration.
43+
The example will deploy a Grafana instance in the same cluster, available via the `grafana` service at port 3000.
44+
45+
## Documentation
46+
47+
Refer to the [official documentation](https://grafana.com/docs/pyroscope/latest/configure-client/grafana-agent/java/) for an in-depth understanding and additional configuration options for Java profiling with Grafana Alloy.

‎examples/grafana-agent-auto-instrumentation/java/kubernetes/grafana-alloy.yaml

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
---
2+
3+
apiVersion: rbac.authorization.k8s.io/v1
4+
kind: ClusterRole # needed for the discovery.kubernetes alloy component
5+
metadata:
6+
name: grafana-alloy-role
7+
rules:
8+
- apiGroups: [""]
9+
resources: ["pods"]
10+
verbs: ["list", "watch"]
11+
12+
---
13+
14+
apiVersion: v1
15+
kind: ServiceAccount
16+
metadata:
17+
name: grafana-alloy
18+
19+
---
20+
21+
apiVersion: rbac.authorization.k8s.io/v1
22+
kind: ClusterRoleBinding
23+
metadata:
24+
name: grafana-alloy-binding
25+
roleRef:
26+
apiGroup: rbac.authorization.k8s.io
27+
kind: ClusterRole
28+
name: grafana-alloy-role
29+
subjects:
30+
- kind: ServiceAccount
31+
name: grafana-alloy
32+
namespace: pyroscope-java
33+
34+
---
35+
36+
apiVersion: apps/v1
37+
kind: DaemonSet
38+
metadata:
39+
name: grafana-alloy
40+
spec:
41+
selector:
42+
matchLabels:
43+
app: grafana-alloy
44+
template:
45+
metadata:
46+
labels:
47+
app: grafana-alloy
48+
spec:
49+
serviceAccountName: grafana-alloy
50+
containers:
51+
- name: grafana-alloy
52+
image: grafana/alloy
53+
command:
54+
- /bin/alloy
55+
- run
56+
- /etc/agent-config/config.river
57+
- --server.http.listen-addr=0.0.0.0:12345
58+
env:
59+
- name: HOSTNAME
60+
valueFrom:
61+
fieldRef:
62+
fieldPath: spec.nodeName
63+
- name: AGENT_MODE
64+
value: flow
65+
ports:
66+
- containerPort: 12345
67+
volumeMounts:
68+
- name: agent-config
69+
mountPath: /etc/agent-config
70+
securityContext:
71+
privileged: true
72+
runAsGroup: 0
73+
runAsUser: 0
74+
capabilities:
75+
add:
76+
- PERFMON
77+
- SYS_PTRACE
78+
- SYS_RESOURCE
79+
- SYS_ADMIN
80+
volumes:
81+
- name: agent-config
82+
configMap:
83+
name: agent-config
84+
hostPID: true
85+
86+
---
87+
88+
apiVersion: v1
89+
kind: ConfigMap
90+
metadata:
91+
name: agent-config
92+
data:
93+
config.river: |
94+
logging {
95+
level = "debug"
96+
format = "logfmt"
97+
}
98+
99+
// Discovers all kubernetes pods.
100+
// Relies on serviceAccountName=grafana-alloy in the pod spec for permissions.
101+
discovery.kubernetes "pods" {
102+
role = "pod"
103+
}
104+
105+
// Discovers all processes running on the node.
106+
// Relies on a security context with elevated permissions for the alloy container (running as root).
107+
// Relies on hostPID=true on the pod spec, to be able to see processes from other pods.
108+
discovery.process "all" {
109+
// Merges kubernetes and process data (using container_id), to attach kubernetes labels to discovered processes.
110+
join = discovery.kubernetes.pods.targets
111+
}
112+
113+
// Drops non-java processes and adjusts labels.
114+
discovery.relabel "java" {
115+
targets = discovery.process.all.targets
116+
// Drops non-java processes.
117+
rule {
118+
source_labels = ["__meta_process_exe"]
119+
action = "keep"
120+
regex = ".*/java$"
121+
}
122+
// Sets up the service_name using the namespace and container names.
123+
rule {
124+
source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
125+
target_label = "service_name"
126+
separator = "/"
127+
}
128+
// Sets up kubernetes labels (labels with the __ prefix are ultimately dropped).
129+
rule {
130+
action = "replace"
131+
source_labels = ["__meta_kubernetes_pod_node_name"]
132+
target_label = "node"
133+
}
134+
rule {
135+
action = "replace"
136+
source_labels = ["__meta_kubernetes_namespace"]
137+
target_label = "namespace"
138+
}
139+
rule {
140+
action = "replace"
141+
source_labels = ["__meta_kubernetes_pod_name"]
142+
target_label = "pod"
143+
}
144+
rule {
145+
action = "replace"
146+
source_labels = ["__meta_kubernetes_pod_container_name"]
147+
target_label = "container"
148+
}
149+
// Sets up the cluster label.
150+
// Relies on a pod-level annotation with the "cluster_name" name.
151+
// Alternatively it can be set up using external_labels in pyroscope.write.
152+
rule {
153+
action = "replace"
154+
source_labels = ["__meta_kubernetes_pod_annotation_cluster_name"]
155+
target_label = "cluster"
156+
}
157+
}
158+
159+
// Attaches the Pyroscope profiler to the processes returned by the discovery.relabel component.
160+
// Relies on a security context with elevated permissions for the alloy container (running as root).
161+
// Relies on hostPID=true on the pod spec, to be able to access processes from other pods.
162+
pyroscope.java "java" {
163+
profiling_config {
164+
interval = "15s"
165+
alloc = "512k"
166+
cpu = true
167+
lock = "10ms"
168+
sample_rate = 100
169+
}
170+
forward_to = [pyroscope.write.local.receiver]
171+
targets = discovery.relabel.java.output
172+
}
173+
174+
pyroscope.write "local" {
175+
// Send metrics to the locally running Pyroscope instance.
176+
endpoint {
177+
url = "http://pyroscope:4040"
178+
}
179+
external_labels = {
180+
"static_label" = "static_label_value",
181+
}
182+
}
183+
---

‎examples/grafana-agent-auto-instrumentation/java/kubernetes/grafana.yaml

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: grafana
5+
spec:
6+
replicas: 1
7+
selector:
8+
matchLabels:
9+
app: grafana
10+
template:
11+
metadata:
12+
labels:
13+
app: grafana
14+
spec:
15+
containers:
16+
- name: grafana
17+
image: grafana/grafana:latest
18+
env:
19+
- name: GF_INSTALL_PLUGINS
20+
value: grafana-pyroscope-app
21+
- name: GF_AUTH_ANONYMOUS_ENABLED
22+
value: "true"
23+
- name: GF_AUTH_ANONYMOUS_ORG_ROLE
24+
value: Admin
25+
- name: GF_AUTH_DISABLE_LOGIN_FORM
26+
value: "true"
27+
ports:
28+
- containerPort: 3000
29+
volumeMounts:
30+
- name: grafana-provisioning
31+
mountPath: /etc/grafana/provisioning
32+
volumes:
33+
- name: grafana-provisioning
34+
configMap:
35+
name: grafana-provisioning
36+
items:
37+
- key: datasources
38+
path: datasources/datasources.yaml
39+
- key: plugins
40+
path: plugins/plugins.yaml
41+
---
42+
apiVersion: v1
43+
kind: Service
44+
metadata:
45+
name: grafana
46+
spec:
47+
selector:
48+
app: grafana
49+
ports:
50+
- protocol: TCP
51+
port: 3000
52+
targetPort: 3000
53+
---
54+
apiVersion: v1
55+
kind: ConfigMap
56+
metadata:
57+
name: grafana-provisioning
58+
data:
59+
"datasources": |
60+
apiVersion: 1
61+
datasources:
62+
- uid: local-pyroscope
63+
type: grafana-pyroscope-datasource
64+
name: Pyroscope
65+
url: http://pyroscope:4040
66+
jsonData:
67+
keepCookies: [pyroscope_git_session]
68+
69+
"plugins": |
70+
apiVersion: 1
71+
apps:
72+
- type: grafana-pyroscope-app
73+
jsonData:
74+
backendUrl: http://pyroscope:4040
75+
secureJsonData:

‎examples/grafana-agent-auto-instrumentation/java/kubernetes/java-fast-slow.yaml

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: java-app-jar
5+
binaryData: # holds FastSlow.jar from the sibling "Docker" directory, as a base64 encoded binary
6+
jar: |
7+
UEsDBAoAAAgAAD1iQlkAAAAAAAAAAAAAAAAJAAQATUVUQS1JTkYv/soAAFBLAwQUAAgICAA9YkJZAAAAAAAAAAAAAAAAFAAAAE1FVEEtSU5GL01BTklGRVNULk1G803My0xLLS7RDUstKs7Mz7NSMNQz4OXyTczM03XOSSwutlJwSywuCc7JL+flci5KTSxJTdF1qrRSMAKq0zNU0HBNzsksKE5VcEzJLyjJLM3V5OXi5QIAUEsHCJiPQHxXAAAAVgAAAFBLAwQUAAgICAASYkJZAAAAAAAAAAAAAAAADgAAAEZhc3RTbG93LmNsYXNznVTLUhNBFD0dkgwMAxke8ohPMEgCSkDxGYRSCkqsoItQaJWrzqSBgclMaqaHxwf4I25csXAlsvADLD/HtXp7QggWgpSpSvfcvvece27fW/3t5+FXAM8wpyOGFg1xAwkkGcxNvs3zDnfX86/Lm8KSDMkZ27XlLENLNreqoxVtGnQD7TAYWhd5IEuOt0PeNbvMEM8u5V7q6ERKg2mgC90M1yLOUNpO3vJcK/R94cr8wq6wQun5AUO/K3ZKtrvuiJUNX/BKw8WQzeaK56JLwt+2LVHopmp6DVxCH2mwuOMwDJ2FnSc3Lzui0I4BDGpIG7iMKwyZi2Si+wjCctWmi5nN/oP/jPyLoQx9USCh816FCFNF2xWvwmpZ+CsKx9BRktzaWua1Izte5bbL0Jd9V2z2pyR9urNCbpVBX9i1RE3anhtoyDCMnlMJBR1Ha7jVaE9EueRK4fthTYrKcQypcXi1XOEZJSIzydDbuNgTY1LQMMbQ0zw+gddLXuhbYtGOSmtMzISKpYF77nkykD6vLQu54VUCE9lUEpNq0KYMjEbWPR3TuK/hgYGHeMQw3Mxju9velsgXI4lEwde4Rd3aY2ivnrS+Z4unQPWML7hbcUSQKXreVlgrnL7hs4ArezXxf856yvOxudNeNVglW6rRMZZcV/jzDg8CQT2nk5ELlafh6R8d/1sojXg9GEPUhBjULw6mHghar5J1lXZGe2LsM9gn+iBOWpPRYQva1ONwFDoXQYG+dOIDkun3++l45QBaOqFWvk+eWITtJJxiSEBTTwyuR0xMfd4gHbGIbIp2pafrAB3F8UP0AF/QH8ObpoY6Twdl7YSJXgxHrDHcxIipq3E6opo+KixlpknIW3NArR9//TgWpEcBJoG7GmIQsal6cmSOR6G3cYd2Ve1dOptAnv6PIzjDE8xgFoO/AVBLBwipPrnIuQIAAG0FAABQSwECCgAKAAAIAAA9YkJZAAAAAAAAAAAAAAAACQAEAAAAAAAAAAAAAAAAAAAATUVUQS1JTkYv/soAAFBLAQIUABQACAgIAD1iQlmYj0B8VwAAAFYAAAAUAAAAAAAAAAAAAAAAACsAAABNRVRBLUlORi9NQU5JRkVTVC5NRlBLAQIUABQACAgIABJiQlmpPrnIuQIAAG0FAAAOAAAAAAAAAAAAAAAAAMQAAABGYXN0U2xvdy5jbGFzc1BLBQYAAAAAAwADALkAAAC5AwAAAAA=
8+
---
9+
apiVersion: apps/v1
10+
kind: Deployment
11+
metadata:
12+
name: java-fast-slow
13+
spec:
14+
replicas: 1
15+
selector:
16+
matchLabels:
17+
app: java-fast-slow
18+
template:
19+
metadata:
20+
annotations:
21+
cluster-name: dev-us-east-1
22+
labels:
23+
app: java-fast-slow
24+
spec:
25+
containers:
26+
- name: java-fast-slow
27+
image: openjdk:21-jdk-slim
28+
imagePullPolicy: IfNotPresent
29+
command: [ "java" ]
30+
args: [ "-jar", "/app/FastSlow.jar" ]
31+
volumeMounts:
32+
- name: app-jar
33+
mountPath: /app
34+
volumes:
35+
- name: app-jar
36+
configMap:
37+
name: java-app-jar
38+
items:
39+
- key: jar
40+
path: FastSlow.jar

0 commit comments

Comments
 (0)