[Backport 7.82.x] fix(dataobs): reuse stored base config across DO_QUERY_ACTIONS updates

[dd-octo-sts]

Backport 83b0223 from #54212.

---

What does this PR do?

Follow-up to #54200 (merged). Fixes a second, distinct duplication bug in
comp/dataobs/queryactions/impl/handler.go, surfaced while verifying #54200: after the first
DO_QUERY_ACTIONS update correctly retires a base config in favor of a DO check, a second
RC update for the same config_id (e.g. editing the monitor's queries) could resurrect the
already-retired base config alongside the new DO check — reintroducing the "parallel checks"
duplication that #54200 fixed for the first-update case.

Root cause: onRCUpdate always re-derived the "base config" via findMatchingConfig, which
searches only currently-active (scheduled) configs. Once the true original is unscheduled after
the first update, the only thing left matching that host is the DO component's own
previously-scheduled check — which also satisfies matchesIdentifier and instanceHasDOEnabled,
so it looks like a legitimate base config from the outside. Adopting it as the new "base" gives it
a different digest than the true original, so reconcileBases's digest-based tracking loses track
of the true original and wrongly restores it.

Fix: added resolveBaseConfig, which reuses the previously-stored baseCfg for an already-active
config_id instead of re-deriving it every update. Falls back to a fresh findMatchingConfig
search if the stored base no longer has a matching instance (e.g. a genuine host change).
Extracted findMatchingInstance so both the fresh-search path and the reuse path share identical
matching logic.

Rebased onto latest main (now including #54215, an orthogonal fix for a same-host/different-port
remainder-matching bug — no overlap with this change; both coexist cleanly).

Motivation

Confirmed live against a real SAP HANA instance, with #54200 already applied: editing a monitor
tied to an already-active DO_QUERY_ACTIONS config (growing its query count) reliably triggered
reconcileBases | Restored original postgres config and a second concurrent
schema-collection/data_observability cycle, exactly reproducing the duplication bug via a
different trigger than #54200 covers.

Describe how you validated your changes

• Added TestOnRCUpdate_SecondUpdateReusesStoredBase, a regression test using a mutable mock
  GetUnresolvedConfigs() that reflects what autodiscovery would actually look like after the
  first update (existing tests' static fixtures couldn't exercise this — they never reflect a
  prior update's changes.Schedule/Unschedule). Verified this test fails with the pre-fix
  behavior (temporarily reverted the fix locally, confirmed the test fails with the same
  "Restored original postgres config" signature seen live, then restored the fix).
• dda inv test --targets=./comp/dataobs/queryactions/impl — all 39 tests pass (41 counting
  table-driven subtests).
• Live-verified end-to-end against a running SAP HANA Express instance: first RC delivery settles
  cleanly (base retired, DO check active), then a real monitor edit triggers a second delivery for
  the same config_id (query count 4 → 5) — before this fix, that reproduced the duplication;
  after, the base config is never rescheduled and exactly one check instance remains active
  throughout, confirmed via schema-collection logs showing only a single running instance.
• Re-verified after rebasing onto latest main (with fix(dataobs): don't drop same-host sibling instances from remainder #54215 merged): build succeeds, all tests
  still pass, agent runs cleanly against the live SAP HANA instance.

Additional Notes

None outstanding — the two known duplication triggers (first update in #54200, second update
here) are both covered now.

[datadog-official]

The new stored-base reuse prevents the reported second-update duplication, but it also ignores a refreshed autodiscovery base while the DO config remains active. A credential or connection-setting change is therefore overwritten with the old values on the next query-action update, which can leave the check unable to authenticate or collecting with stale settings.

_{📊 Validated against 1 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 5333a80 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[datadog-official]

Active DO updates reuse stale base configuration

Credential rotations and other base-instance changes made while query actions remain active are silently discarded on each subsequent RC update, causing authentication failures or stale connection behavior for affected database checks.

Assertion details

• Input: Start with an active postgres DO config derived from a base instance containing user=old-user and password=old-password. Refresh the file-provider/autodiscovery base for the same host to user=new-user and password=new-password, then deliver another DO_QUERY_ACTIONS update for the still-active config_id.
• Expected: The new DO check should be built from the currently active base so refreshed connection settings are retained.
• Actual: The added resolveBaseConfig path calls findMatchingInstance on existing.baseCfg and returns that stored object without consulting the refreshed active config. The scheduled check still contains old-user and old-password; the temporary adversarial test failed on the expected new-user value.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 85.71%
• Overall Coverage: 51.56% (-0.15%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5333a80 | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

Files inventory check summary

File checks results against ancestor c9150d56:

Results for datadog-agent_7.82.0~rc.8.git.7.5333a80.pipeline.128191875-1_amd64.deb:

No change detected

[cit-pr-commenter-54b7da]

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 546f5875-0185-41e5-9588-f978845c6b49

Baseline: c9150d5
Comparison: 5333a80
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	quality_gate_security_idle	memory utilization	+0.66	[+0.60, +0.72]	1	Logs bounds checks dashboard
➖	quality_gate_metrics_logs	memory utilization	+0.30	[+0.05, +0.55]	1	Logs bounds checks dashboard
➖	quality_gate_idle	memory utilization	+0.16	[+0.11, +0.21]	1	Logs bounds checks dashboard
➖	quality_gate_logs	% cpu utilization	+0.00	[-0.98, +0.98]	1	Logs bounds checks dashboard
➖	quality_gate_security_no_fs_load	memory utilization	-0.00	[-0.10, +0.09]	1	Logs bounds checks dashboard
➖	quality_gate_idle_all_features	memory utilization	-0.24	[-0.27, -0.20]	1	Logs bounds checks dashboard
➖	quality_gate_security_mean_fs_load	memory utilization	-0.33	[-0.37, -0.29]	1	Logs bounds checks dashboard

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	observed_value	links
✅	quality_gate_idle	intake_connections	10/10	3 ≤ 4	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	146.46MiB ≤ 154MiB	bounds checks dashboard
✅	quality_gate_idle	total_bytes_received	10/10	733.40KiB ≤ 819.20KiB	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	3 ≤ 4	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	486.48MiB ≤ 495MiB	bounds checks dashboard
✅	quality_gate_idle_all_features	total_bytes_received	10/10	1.13MiB ≤ 1.25MiB	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	4 ≤ 6	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	185.49MiB ≤ 195MiB	bounds checks dashboard
✅	quality_gate_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard
✅	quality_gate_logs	total_bytes_received	10/10	264.13MiB ≤ 292MiB	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	350.90 ≤ 2000	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	3 ≤ 6	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	398.73MiB ≤ 430MiB	bounds checks dashboard
✅	quality_gate_metrics_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard
✅	quality_gate_metrics_logs	total_bytes_received	10/10	0.94GiB ≤ 1.04GiB	bounds checks dashboard
✅	quality_gate_security_idle	cpu_usage	10/10	28.59 ≤ 40	bounds checks dashboard
✅	quality_gate_security_idle	memory_usage	10/10	298.03MiB ≤ 330MiB	bounds checks dashboard
✅	quality_gate_security_mean_fs_load	cpu_usage	10/10	62.74 ≤ 80	bounds checks dashboard
✅	quality_gate_security_mean_fs_load	memory_usage	10/10	276.70MiB ≤ 310MiB	bounds checks dashboard
✅	quality_gate_security_no_fs_load	cpu_usage	10/10	22.29 ≤ 40	bounds checks dashboard
✅	quality_gate_security_no_fs_load	memory_usage	10/10	286.51MiB ≤ 320MiB	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

Replicate Execution Details

We run multiple replicates for each experiment/variant. However, we allow replicates to be automatically retried if there are any failures, up to 8 times, at which point the replicate is marked dead and we are unable to run analysis for the entire experiment. We call each of these attempts at running replicates a replicate execution. This section lists all replicate executions that failed due to the target crashing or being oom killed.

Note: In the below tables we bucket failures by experiment, variant, and failure type. For each of these buckets we list out the replicate indexes that failed with an annotation signifying how many times said replicate failed with the given failure mode. In the below example the baseline variant of the experiment named experiment_with_failures had two replicates that failed by oom kills. Replicate 0, which failed 8 executions, and replicate 1 which failed 6 executions, all with the same failure mode.

Experiment	Variant	Replicates	Failure	Logs	Debug Dashboard
experiment_with_failures	baseline	0 (x8) 1 (x6)	Oom killed		Debug Dashboard

The debug dashboard links will take you to a debugging dashboard specifically designed to investigate replicate execution failures.

❌ Retried Profiling Replicate Execution Failures (ddprof)

Note: Profiling replicas may still be executing. See the debug dashboard for up to date status.

Experiment	Variant	Replicates	Failure	Debug Dashboard
quality_gate_idle_all_features	baseline	10	Oom killed	Debug Dashboard
quality_gate_idle_all_features	comparison	10	Oom killed	Debug Dashboard
quality_gate_logs	baseline	10	Oom killed	Debug Dashboard
quality_gate_logs	comparison	10	Oom killed	Debug Dashboard
quality_gate_metrics_logs	baseline	10	Oom killed	Debug Dashboard
quality_gate_metrics_logs	comparison	10	Oom killed	Debug Dashboard
quality_gate_security_no_fs_load	baseline	10	Crashed (exit code: 134)	Debug Dashboard
quality_gate_security_no_fs_load	comparison	10	Crashed (exit code: 134)	Debug Dashboard

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

• quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
• quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
• quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.