Define dataflows for community-owned integrations (batch 2, individually-maintained Agent checks)

[philjlee]

Jira: TXP-277

What

Adds assets/dataflows.yaml to 43 integrations (45 dataflow entries: 43 metrics, 2 logs).

This is batch 2 of the integrations-extras dataflows backfill. Batch 1 is #3093 (74 Datadog-team-owned integrations). The two batches touch disjoint directories and are both based on master, so they are independent and will not conflict.

Selection criteria

An integration is in scope for this PR if all of the following hold, computed from the repo at 9c197fe:

1. It has a parseable manifest.json (the dataflows validator hard-requires one).
2. It has no existing assets/dataflows.yaml.
3. It has a hard data signal: metadata.csv with at least one data row, and/or a log pipeline under assets/logs/. No data_type was ever guessed.
4. .github/CODEOWNERS resolves <dir>/assets/dataflows.yaml (last-match-wins) to an owner set containing no @DataDog/... team.
5. It is not a RUM tile, not one of the 5 service-check-only directories, and not fluxcd/traefik (README-only, disabled in APW).

Then, within that set of 81, this PR takes the cluster with individually-maintained Agent checks: These paths are owned only by individual GitHub handles or personal email addresses, and the integration ships an Agent check under datadog_checks/.

Content

Every file follows the uniform template already used by the 14 pre-existing files in this repo:

provides:
  - id: <app_id>-<data_type>
    always_on: true
    granular: false
    data_type: <metrics|logs>
    direction: inbound

id is manifest.json's app_id plus -<data_type>. That is the mechanical rule the 17 existing entries follow 17/17 — not the directory name. For all 81 integrations in batch 2 the two happen to coincide after slugification, but app_id is the authority.

metadata.csv rows produce a metrics entry, an assets/logs/ pipeline produces a logs entry, and integrations with both get both (metrics first).

Integrations in this PR

Integration	Dataflows	Evidence	CODEOWNERS
aws_pricing	metrics	100 metric rows	@tsein-bc tsein@brightcove.com
cloudnatix	metrics	7 metric rows	@junm-cloudnatix @kenji-cloudnatix @somik-cloudnatix @rohit-cloudnatix
eventstore	metrics	85 metric rows	@xorima
exim	metrics	2 metric rows	@JeanFred
filebeat	metrics	1 metric rows	@wk8 jean@tripping.com
filemage	metrics	66 metric rows	@devopsec
fluentbit	metrics	12 metric rows	@zhenyami
flume	metrics	36 metric rows	@KealanMaas
gitea	metrics	38 metric rows	@FlorentClarret
grafana	metrics	269 metric rows	@RTJL reubentjl@gmail.com
grpc_check	metrics	2 metric rows	@keisku
hbase_master	metrics+logs	21 metric rows, log pipeline	@everpeace
hbase_regionserver	metrics	159 metric rows	@everpeace
hikaricp	metrics	26 metric rows	@bertaudamien
kernelcare	metrics	4 metric rows	@grubberr schvaliuk@cloudlinux.com
logstash	metrics+logs	58 metric rows, log pipeline	@ervansetiawan ervansetiawan@gmail.com
neutrona	metrics	9 metric rows	@DavidFlamini david@neutrona.com
nextcloud	metrics	22 metric rows	@eplanet
nvml	metrics	15 metric rows	@cep21
octoprint	metrics	10 metric rows	@gwaldo gwaldo@gmail.com
php_apcu	metrics	12 metric rows	@withgod noname@withgod.jp
php_opcache	metrics	25 metric rows	@withgod noname@withgod.jp
pihole	metrics	73 metric rows	@monganai
ping	metrics	2 metric rows	@jstanton617
portworx	metrics	302 metric rows	@pault84 paul@portworx.com
puma	metrics	8 metric rows	@plasticine
radarr	metrics	7 metric rows	@HadrienPatte
resilience4j	metrics	24 metric rows	@willianccs willianccs@gmail.com
resin	metrics	12 metric rows	@brentm5
riak_repl	metrics	50 metric rows	@abtreece
robust_intelligence_ai_firewall	metrics	2 metric rows	@MayankR
sonarr	metrics	14 metric rows	@HadrienPatte
sortdb	metrics	28 metric rows	@namrata4 namrata.deshpande4@gmail.com
speedtest	metrics	13 metric rows	@platinummonkey
storm	metrics	94 metric rows	@platinummonkey
syncthing	metrics	26 metric rows	@sashacmc
trino	metrics	69 metric rows	@ndrluis
unbound	metrics	82 metric rows	@dbyron0 david.byron@avast.com
unifi_console	metrics	41 metric rows	@abruneau
upsc	metrics	12 metric rows	@platinummonkey
wayfinder	metrics	14 metric rows	@jonny-scott80 jonny_scott@outlook.com
zabbix	metrics	80 metric rows	@KosukeKamiya
zenoh_router	metrics	23 metric rows	@sashacmc

Coverage

Assuming batch 1 (#3093) merges, coverage of assets/dataflows.yaml across the 260 integration directories in this repo moves:

Stage	Files	Coverage
Today on master	14	5.4%
After batch 1 (#3093, +74)	88	33.8%
After all of batch 2 (+81, this PR is +43)	169	65.0%

Batch 1 + batch 2 together complete the entire mechanical bucket — every integration in the repo with an unambiguous local data_type signal.

Ownership — please read

None of the directories in this PR have a @DataDog/... team CODEOWNER; they resolve to third-party organizations, vendor support aliases, or individual community maintainers.

Per #2925 ("Define dataflows for saas-integrations") and consistent repo practice, external CODEOWNERS approval is not required to merge here — master carries no branch ruleset requiring it, and Datadog employees routinely land mechanical sweeps across community-owned directories (#2810, #2829, #2878, #2957, #3000). #2925 in particular created 13 of the 14 existing dataflows.yaml files and was approved only by Datadog reviewers.

That said, this PR is intentionally a draft so that no review request has been sent to any external maintainer yet. Marking it ready will fan review requests out to the owners listed in the table above. That should be a deliberate decision, not a side effect.

Validation

These files were validated by running the real upstream validator, not by static inspection.

DataflowsValidationHandler from ddoghq/dd-source@main (domains/integrationscatalog/libs/catalogassetslib/dataflows_validation_handler.go) was executed via go test -overlay, with only the COBS-backed dataflows client and the deployment handler stubbed out. The validation code path itself is byte-for-byte upstream — the handler file was diffed against ddoghq/dd-source@main and is identical, including the current 14-value validDataTypes list (federated_sql_queries included).

Results:

• This PR's tree: 43 new files + the 14 pre-existing = 57 files, 62 dataflow IDs, 0 failures.
• The union of master + batch 1 (Define dataflows for Datadog-owned integrations (batch 1) #3093) + all of batch 2: 169 files, 187 dataflow IDs, 0 failures — which is what proves dataflow ID uniqueness holds globally, across both batches and the pre-existing files. (HandleLibrary enforces "only one app can provide a dataflow".)

The harness was negative-tested first, and correctly rejects all of: missing always_on; a data_type outside the enum; an id violating ^[a-z0-9-]+$; an id shorter than 3 characters; a file with neither provides nor uses; a .yml extension; a missing manifest.json; and the same dataflow ID provided by two apps. A known-good baseline file is accepted. So a pass here means something.

APW also posts a validate-dataflows status check on extras PRs, so there will be pre-merge feedback from the pipeline itself as well.

CI note — run / Validate redness is pre-existing

The run / Validate job is already failing on master and has been for at least the last 10 runs of that workflow. Concrete evidence at the exact SHA this branch is based on:

• https://github.com/DataDog/integrations-extras/actions/runs/30528431295 — master @ 9c197fe2c9ff8864d1fe84274298818aac67853f, job run / Validate failed at "Validate default configuration files" and "Validate configuration models", with 182 is not in sync errors spanning 39 integration directories.

Because the workflow runs with TARGET=changed on PRs, touching a directory pulls that pre-existing drift into this PR's scope even though this PR only adds a new file and changes nothing that ddev validate config/models inspects. 16 of the 43 directories in this PR are already failing those steps on master: cloudnatix, exim, filemage, fluentbit, gitea, grafana, hikaricp, radarr, resilience4j, robust_intelligence_ai_firewall, sonarr, syncthing, trino, unifi_console, wayfinder, zenoh_router. This is unrelated to dataflows and is not something this PR attempts to fix.

Deferred, not in scope

Nothing was dropped from this PR for an ambiguous data_type — every integration in the batch-2 set mapped cleanly to metrics and/or logs. What remains unaddressed repo-wide after batch 1 + batch 2 (91 directories) is the research bucket, which needs schema decisions rather than scripting:

• 17 RUM SDK tiles — there is no rum value in validDataTypes, so no legal file can be written.
• 5 service-check-only directories (cfssl, cybersixgill_actionable_alerts, go_pprof_scraper, isdown, reboot_required) — service checks map to no data_type.
• ~39 outbound notification/webhook tiles — events + direction: outbound is the least-bad fit but has not been ratified by anyone.
• 5 directories with no manifest.json (aerospike_enterprise, scamalytics, warpstream, plus fluxcd/traefik which are README-only and explicitly disabled in APW) — the validator requires a manifest.

Related: #3093 (batch 1), #2925 (the original dataflows PR for saas-integrations).

🤖 Generated with Claude Code

---

Addendum: merge-gating, measured

Two things worth stating precisely, since the "can Datadog land this on community-owned directories?" question is the only real objection to this batch.

1. devflow/mergegate will show "N approvals missing". It is advisory, not a required check. Six of the twenty most recently merged PRs in this repo merged while mergegate was pending or failed:

• Automate post-tag wheel publishing handoff #3077 reached a terminal failure (1 issue: CODEOWNERS missing) and was merged ~15 hours later regardless.
• CloudZero UI update #3022 merged with 1 pending: 11 approvals missing.
• cloudsmith: add billable_only filter to analytics metrics (default: True) #3024, Pre-sort map keys inside list items in _tests.yaml fixtures #3027, Feat/re prometheus endpoint gauges and dbkeys deprecation #3078, Bump ping to 1.0.3 #3081 likewise merged with mergegate still in_progress.

2. There is now a direct precedent for the solely-externally-owned case, not just the co-owned one. #2925 (the original dataflows PR) only ever touched directories that already carried a @DataDog/... team, so it did not settle the harder case that this batch is made of. #3022 does: it modified /cloudzero/, whose CODEOWNERS line is nine external individuals and no Datadog team, and it was approved by Wyrine, sarah-witt, and steveny91 — none of them CloudZero owners — then merged. reviewDecision was APPROVED, which GitHub would not report if CODEOWNERS review were enforced.

What is enforced on master is GitHub's ordinary "at least one approving review" rule, satisfied by any reviewer with write access. The practical requirement for this PR is therefore one Datadog approval, not sign-off from the maintainers in the table above.

Caveat, stated honestly: GET /repos/DataDog/integrations-extras/branches/master/protection returns 404 for non-admin tokens, so the classic branch-protection config could not be read directly. The conclusions above are inferred from ruleset queries (only an org-level tag ruleset exists; rules/branches/master is empty) and from the merge behaviour of 20 recent PRs. Someone with admin access can confirm.

None of this changes the fact that this PR is a draft and no external maintainer has been notified. Marking it ready is what sends the review requests, and that should stay a deliberate step.

[philjlee]

CI note: run / Validate — confirmed pre-existing, not caused by this PR

Now that CI has run on all four batch-2 PRs, here is the observed result. The run / Validate outcome tracks exactly one thing: whether the PR happens to touch a directory whose generated config/model files are already out of sync on master.

PR	Files	Dirs already drifting on master	run / Validate
#3094	4	2 (gatekeeper, kepler)	fail
#3095	24	6	fail
#3096	43	16	fail
#3097	10	0	pass

#3097 is the natural control: same generator, same file shape, zero drifting directories, and run / Validate is green. That isolates the failure to pre-existing drift rather than to anything about dataflows.yaml.

Two further confirmations:

1. The string dataflow does not appear anywhere in the validation output of the failing runs. (It appears 5 times in each log, all in git fetch branch-listing noise.) No validator names a dataflows.yaml file. The failing steps are only "Validate default configuration files" and "Validate configuration models".

2. master emits the identical errors. The last Validate repository run on master — run 30528431295 at 9c197fe, the exact SHA these branches are based on — already reports, verbatim:

   ##[error]File `conf.yaml.example` is not in sync, run "ddev validate config kepler -s"
   ##[error]File `defaults.py` is not in sync, run "ddev validate models gatekeeper -s"
   ##[error]File `instance.py` is not in sync, run "ddev validate models gatekeeper -s"
   

   That master run has 182 is not in sync errors across 39 integration directories, and 10 of the last 10 Validate repository runs on master failed. Because PRs run with TARGET=changed, touching any of those 39 directories pulls the drift into scope.

This PR adds one new file per directory and changes nothing that ddev validate config/models inspects. Fixing the repo-wide codegen drift is out of scope here.