Skip to content

Allow attaching GDB to guest VMs launched with KMT - #38503

Merged
dd-mergequeue[bot] merged 16 commits into
mainfrom
usama.saqib/kmt-kernel-debug
Jul 31, 2025
Merged

Allow attaching GDB to guest VMs launched with KMT#38503
dd-mergequeue[bot] merged 16 commits into
mainfrom
usama.saqib/kmt-kernel-debug

Conversation

@usamasaqib

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR extends the kmt.launch-stack task to launch guest VMs with a GDB server. This allows a user to easily debug the running kernel. The PR also adds a helper task kmt.attach-gdb which generates the command to attach to the running VM.

Motivation

Describe how you validated your changes

Possible Drawbacks / Trade-offs

Additional Notes

Corresponding PR in test-infra-definitions DataDog/test-infra-definitions#1579

@usamasaqib
usamasaqib requested a review from a team as a code owner July 7, 2025 15:32
@usamasaqib usamasaqib added changelog/no-changelog No changelog entry needed team/ebpf-platform qa/done QA done before merge and regressions are covered by tests labels Jul 7, 2025
@github-actions github-actions Bot added the medium review PR review might take time label Jul 7, 2025
@cit-pr-commenter

cit-pr-commenter Bot commented Jul 7, 2025

Copy link
Copy Markdown

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 07019b36-c95c-4756-bc68-68856d7b3dbd

Baseline: e6d585c
Comparison: 99d0973
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf experiment goal Δ mean % Δ mean % CI trials links
docker_containers_cpu % cpu utilization -3.00 [-5.91, -0.09] 1 Logs

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
quality_gate_metrics_logs memory utilization +1.31 [+0.92, +1.70] 1 Logs bounds checks dashboard
tcp_syslog_to_blackhole ingress throughput +0.83 [+0.77, +0.90] 1 Logs
file_tree memory utilization +0.18 [+0.14, +0.22] 1 Logs
ddot_logs memory utilization +0.11 [+0.02, +0.20] 1 Logs
quality_gate_idle memory utilization +0.05 [+0.01, +0.10] 1 Logs bounds checks dashboard
file_to_blackhole_1000ms_latency egress throughput +0.03 [-0.59, +0.65] 1 Logs
file_to_blackhole_100ms_latency egress throughput +0.01 [-0.57, +0.59] 1 Logs
docker_containers_memory memory utilization +0.01 [-0.03, +0.05] 1 Logs
uds_dogstatsd_to_api ingress throughput -0.00 [-0.29, +0.28] 1 Logs
tcp_dd_logs_filter_exclude ingress throughput -0.01 [-0.03, +0.01] 1 Logs
file_to_blackhole_0ms_latency egress throughput -0.07 [-0.69, +0.56] 1 Logs
file_to_blackhole_500ms_latency egress throughput -0.10 [-0.74, +0.53] 1 Logs
otlp_ingest_metrics memory utilization -0.21 [-0.35, -0.07] 1 Logs
quality_gate_idle_all_features memory utilization -0.22 [-0.26, -0.17] 1 Logs bounds checks dashboard
ddot_metrics memory utilization -0.23 [-0.34, -0.11] 1 Logs
uds_dogstatsd_20mb_12k_contexts_20_senders memory utilization -0.44 [-0.49, -0.40] 1 Logs
otlp_ingest_logs memory utilization -0.62 [-0.74, -0.50] 1 Logs
quality_gate_logs % cpu utilization -1.26 [-4.02, +1.50] 1 Logs bounds checks dashboard
docker_containers_cpu % cpu utilization -3.00 [-5.91, -0.09] 1 Logs

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed links
docker_containers_cpu simple_check_run 10/10
docker_containers_memory memory_usage 10/10
docker_containers_memory simple_check_run 10/10
file_to_blackhole_0ms_latency lost_bytes 10/10
file_to_blackhole_0ms_latency memory_usage 10/10
file_to_blackhole_1000ms_latency memory_usage 10/10
file_to_blackhole_100ms_latency lost_bytes 10/10
file_to_blackhole_100ms_latency memory_usage 10/10
file_to_blackhole_500ms_latency lost_bytes 10/10
file_to_blackhole_500ms_latency memory_usage 10/10
quality_gate_idle intake_connections 10/10 bounds checks dashboard
quality_gate_idle memory_usage 10/10 bounds checks dashboard
quality_gate_idle_all_features intake_connections 10/10 bounds checks dashboard
quality_gate_idle_all_features memory_usage 10/10 bounds checks dashboard
quality_gate_logs intake_connections 10/10 bounds checks dashboard
quality_gate_logs lost_bytes 10/10 bounds checks dashboard
quality_gate_logs memory_usage 10/10 bounds checks dashboard
quality_gate_metrics_logs cpu_usage 10/10 bounds checks dashboard
quality_gate_metrics_logs intake_connections 10/10 bounds checks dashboard
quality_gate_metrics_logs lost_bytes 10/10 bounds checks dashboard
quality_gate_metrics_logs memory_usage 10/10 bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

Passed. All Quality Gates passed.

  • quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.

@agent-platform-auto-pr

agent-platform-auto-pr Bot commented Jul 7, 2025

Copy link
Copy Markdown
Contributor

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor e6d585c

Successful checks

Info

Quality gate Delta On disk size (MiB) Delta On wire size (MiB)
agent_deb_amd64 $${0}$$ $${699.51}$$ < $${704.84}$$ $${+0}$$ $${177.18}$$ < $${178.58}$$
agent_deb_amd64_fips $${0}$$ $${697.76}$$ < $${703.09}$$ $${+0.02}$$ $${176.58}$$ < $${178.12}$$
agent_heroku_amd64 $${0}$$ $${349.7}$$ < $${355.37}$$ $${-0}$$ $${93.79}$$ < $${95.72}$$
agent_msi $${0}$$ $${974.3}$$ < $${979.61}$$ $${+0.01}$$ $${147.61}$$ < $${149.31}$$
agent_rpm_amd64 $${0}$$ $${699.5}$$ < $${704.83}$$ $${+0.03}$$ $${178.69}$$ < $${180.22}$$
agent_rpm_amd64_fips $${0}$$ $${697.75}$$ < $${703.08}$$ $${-0}$$ $${178.39}$$ < $${179.85}$$
agent_rpm_arm64 $${0}$$ $${688.48}$$ < $${694.74}$$ $${-0.01}$$ $${161.8}$$ < $${163.96}$$
agent_rpm_arm64_fips $${0}$$ $${686.79}$$ < $${693.05}$$ $${+0.03}$$ $${160.85}$$ < $${163.0}$$
agent_suse_amd64 $${0}$$ $${699.5}$$ < $${704.83}$$ $${+0.03}$$ $${178.69}$$ < $${180.22}$$
agent_suse_amd64_fips $${0}$$ $${697.75}$$ < $${703.08}$$ $${-0}$$ $${178.39}$$ < $${179.85}$$
agent_suse_arm64 $${0}$$ $${688.48}$$ < $${694.74}$$ $${-0.01}$$ $${161.8}$$ < $${163.96}$$
agent_suse_arm64_fips $${0}$$ $${686.79}$$ < $${693.05}$$ $${+0.03}$$ $${160.85}$$ < $${163.0}$$
docker_agent_amd64 $${-0}$$ $${781.23}$$ < $${788.65}$$ $${-0}$$ $${269.25}$$ < $${272.01}$$
docker_agent_arm64 $${0}$$ $${793.55}$$ < $${802.0}$$ $${-0}$$ $${256.07}$$ < $${259.7}$$
docker_agent_jmx_amd64 $${-0}$$ $${972.47}$$ < $${979.84}$$ $${-0}$$ $${338.26}$$ < $${340.95}$$
docker_agent_jmx_arm64 $${-0}$$ $${973.39}$$ < $${981.8}$$ $${-0}$$ $${321.06}$$ < $${324.65}$$
docker_cluster_agent_amd64 $${+0}$$ $${214.01}$$ < $${214.5}$$ $${-0}$$ $${72.74}$$ < $${73.51}$$
docker_cluster_agent_arm64 $${+0}$$ $${229.88}$$ < $${230.33}$$ $${+0}$$ $${68.99}$$ < $${69.77}$$
docker_cws_instrumentation_amd64 $${0}$$ $${7.07}$$ < $${7.12}$$ $${-0}$$ $${2.95}$$ < $${3.29}$$
docker_cws_instrumentation_arm64 $${0}$$ $${6.69}$$ < $${6.92}$$ $${0}$$ $${2.7}$$ < $${3.07}$$
docker_dogstatsd_amd64 $${+0}$$ $${38.53}$$ < $${39.57}$$ $${+0}$$ $${14.88}$$ < $${15.76}$$
docker_dogstatsd_arm64 $${+0}$$ $${37.23}$$ < $${38.2}$$ $${+0}$$ $${14.34}$$ < $${14.83}$$
dogstatsd_deb_amd64 $${0}$$ $${29.77}$$ < $${31.4}$$ $${-0}$$ $${7.84}$$ < $${8.95}$$
dogstatsd_deb_arm64 $${0}$$ $${28.37}$$ < $${29.97}$$ $${-0}$$ $${6.81}$$ < $${7.89}$$
dogstatsd_rpm_amd64 $${0}$$ $${29.77}$$ < $${31.4}$$ $${+0}$$ $${7.85}$$ < $${8.96}$$
dogstatsd_suse_amd64 $${0}$$ $${29.77}$$ < $${31.4}$$ $${+0}$$ $${7.85}$$ < $${8.96}$$
iot_agent_deb_amd64 $${0}$$ $${54.12}$$ < $${54.55}$$ $${-0}$$ $${13.64}$$ < $${14.45}$$
iot_agent_deb_arm64 $${0}$$ $${51.44}$$ < $${51.9}$$ $${-0}$$ $${11.8}$$ < $${12.63}$$
iot_agent_deb_armhf $${0}$$ $${50.95}$$ < $${51.42}$$ $${-0}$$ $${11.88}$$ < $${12.74}$$
iot_agent_rpm_amd64 $${0}$$ $${54.12}$$ < $${54.55}$$ $${-0}$$ $${13.66}$$ < $${14.47}$$
iot_agent_rpm_arm64 $${0}$$ $${51.44}$$ < $${51.91}$$ $${-0}$$ $${11.82}$$ < $${12.65}$$
iot_agent_suse_amd64 $${0}$$ $${54.12}$$ < $${54.55}$$ $${-0}$$ $${13.66}$$ < $${14.47}$$

@usamasaqib
usamasaqib requested a review from a team as a code owner July 8, 2025 08:36
@github-actions github-actions Bot added long review PR is complex, plan time to review it and removed medium review PR review might take time labels Jul 8, 2025
@usamasaqib usamasaqib added the ask-review Ask in slack required teams to review this PR label Jul 8, 2025
@github-actions github-actions Bot added medium review PR review might take time and removed long review PR is complex, plan time to review it labels Jul 8, 2025
Comment thread tasks/kernel_matrix_testing/gdb.py Outdated
Comment thread tasks/kernel_matrix_testing/gdb.py Outdated

@chouetz chouetz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question and style suggestion, this is ok for me

Comment thread tasks/kernel_matrix_testing/gdb.py Outdated
gdb_paths.vmlinux.parent.mkdir(exist_ok=True, parents=True)
self.target.download(ctx, "/usr/lib/debug/boot/vmlinux.dbg", f"{gdb_paths.vmlinux}")

ctx.run(f"rm -rf {gdb_paths.kernel_source}")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds "dangerous", should we have something to prevent it from deleting the whole filesystem?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we definitely should! I will rework it.

pip3 install crccheck unicorn capstone ropper keystone-engine tqdm

echo "[+] install seccomp-tools, one_gadget"
if [ "x$(which seccomp-tools)" = "x" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [ "x$(which seccomp-tools)" = "x" ]; then
if [[ -z "$(which seccomp-tools)" ]]; then

I think we should follow these recommendations for testing strings in shell script.


set -euxo pipefail

if [[ $UID = 0 ]] ; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [[ $UID = 0 ]] ; then
if [[ "$UID" == 0 ]] ; then

For clarity, we should use == for equality rather than = even though both work. The former encourages the use of [[ and the latter can be confused with an assignment

sudo gem install seccomp-tools
fi

if [ "x$(which one_gadget)" = "x" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [ "x$(which one_gadget)" = "x" ]; then
if [[ -z "$(which one_gadget)" ]]; then

Comment on lines +27 to +28
if [ "x$(uname -m)" = "xx86_64" ]; then
if [ "x$(which rp-lin)" = "x" ] && [ ! -e /usr/local/bin/rp-lin ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [ "x$(uname -m)" = "xx86_64" ]; then
if [ "x$(which rp-lin)" = "x" ] && [ ! -e /usr/local/bin/rp-lin ]; then
if [[ "$(uname -m)" == "x86_64" ]]; then
if [[ -z "x$(which rp-lin)" ]] && [[ ! -e /usr/local/bin/rp-lin ]]; then

fi

echo "[+] install vmlinux-to-elf"
if [ "x$(which vmlinux-to-elf)" = "x" ] && [ ! -e /usr/local/bin/vmlinux-to-elf ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [ "x$(which vmlinux-to-elf)" = "x" ] && [ ! -e /usr/local/bin/vmlinux-to-elf ]; then
if [[ -z "$(which vmlinux-to-elf)" ]] && [[ ! -e /usr/local/bin/vmlinux-to-elf ]]; then

fi

echo "[+] download gef"
if [ -e ~/.gdbinit-gef.py ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [ -e ~/.gdbinit-gef.py ]; then
if [[ -e ~/.gdbinit-gef.py ]]; then


echo "[+] setup gef"
STARTUP_COMMAND="source ~/.gdbinit-gef.py"
if [ ! -e ~/.gdbinit ] || [ "x$(grep "$STARTUP_COMMAND" ~/.gdbinit)" = "x" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [ ! -e ~/.gdbinit ] || [ "x$(grep "$STARTUP_COMMAND" ~/.gdbinit)" = "x" ]; then
if [[ ! -e ~/.gdbinit ]] || [[ -z "$(grep "$STARTUP_COMMAND" ~/.gdbinit)" ]]; then


# 2. download kernel debug build and kernel sources
codename=$(lsb_release -c | awk '{print $2}')
if [ "${codename}" == "xenial" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [ "${codename}" == "xenial" ]; then
if [[ "${codename}" == "xenial" ]]; then

set -o xtrace

# 1. disable kaslr
[ -f $(grep -q "GRUB_CMDLINE_LINUX=\".*nokaslr" /etc/default/grub) ] && sed -i 's/^GRUB_CMDLINE_LINUX="/&nokaslr /' /etc/default/grub

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
[ -f $(grep -q "GRUB_CMDLINE_LINUX=\".*nokaslr" /etc/default/grub) ] && sed -i 's/^GRUB_CMDLINE_LINUX="/&nokaslr /' /etc/default/grub
[[ -f $(grep -q "GRUB_CMDLINE_LINUX=\".*nokaslr" /etc/default/grub) ]] && sed -i 's/^GRUB_CMDLINE_LINUX="/&nokaslr /' /etc/default/grub

@usamasaqib usamasaqib Jul 9, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out I have to do
[[ ! $(grep -q "GRUB_CMDLINE_LINUX=\".*nokaslr" /etc/default/grub) ]] when using [[

-f does not work

@chouetz chouetz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question and style suggestion, this is ok for me

@@ -0,0 +1,58 @@
#!/bin/bash

set -euxo pipefail

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
set -euxo pipefail
set -euo pipefail

We prefer to not use -x in this codebase, is it mandatory?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not mandatory, but why is it preferred not to use -x or -xtrace?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had some issues related to a token leak some time ago and we prefer to not include these in any script even though the script doesn't contain any secrets as a good practice (to avoid "copy pasting" this into a script that might leak)

pip3 install crccheck unicorn capstone ropper keystone-engine tqdm

echo "[+] install seccomp-tools, one_gadget"
if [ "x$(which seccomp-tools)" = "x" ]; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [ "x$(which seccomp-tools)" = "x" ]; then
if [ "$(which seccomp-tools)" = "" ]; then

This should work fine, right?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(same bellow)

set -o errexit
set -o pipefail
set -o nounset
set -o xtrace

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
set -o xtrace

(same question as above)

@github-actions github-actions Bot removed the medium review PR review might take time label Jul 9, 2025
@github-actions github-actions Bot added the long review PR is complex, plan time to review it label Jul 9, 2025
@usamasaqib

Copy link
Copy Markdown
Contributor Author

/merge

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Jul 9, 2025

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-07-09 09:14:36 UTC ℹ️ Start processing command /merge


2025-07-09 09:14:45 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.


2025-07-09 13:15:05 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

@usamasaqib

Copy link
Copy Markdown
Contributor Author

/merge

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Jul 31, 2025

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-07-31 10:03:07 UTC ℹ️ Start processing command /merge


2025-07-31 10:03:12 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 52m (p90).


2025-07-31 10:43:12 UTC ℹ️ MergeQueue: This merge request was merged

@dd-mergequeue
dd-mergequeue Bot merged commit 663e81f into main Jul 31, 2025
362 of 363 checks passed
@dd-mergequeue
dd-mergequeue Bot deleted the usama.saqib/kmt-kernel-debug branch July 31, 2025 10:43
@github-actions github-actions Bot added this to the 7.70.0 milestone Jul 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ask-review Ask in slack required teams to review this PR changelog/no-changelog No changelog entry needed long review PR is complex, plan time to review it qa/done QA done before merge and regressions are covered by tests team/ebpf-platform

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants