Skip to content

Define dataflows for agent-integrations-owned integrations - #3101

Open
philjlee wants to merge 1 commit into
masterfrom
philip.lee/dataflows-extras-batch1-agent-integrations
Open

Define dataflows for agent-integrations-owned integrations#3101
philjlee wants to merge 1 commit into
masterfrom
philip.lee/dataflows-extras-batch1-agent-integrations

Conversation

@philjlee

@philjlee philjlee commented Aug 6, 2026

Copy link
Copy Markdown

Jira: TXP-277

Adds assets/dataflows.yaml to 9 integrations owned by @DataDog/agent-integrations, declaring 11 dataflows. Mechanical change: no code, no metrics, no manifests touched.

Precedent: #2925 "Define dataflows for saas-integrations", which created 13 of the 14 existing dataflow files in this repo.

Selection criteria

A directory is in this batch only if all of the following hold:

  1. It has no assets/dataflows.yaml today.
  2. It has a parseable manifest.json. The dataflows validator hard-requires one — dataflows_validation_handler.go:56.
  3. The correct data_type is mechanically derivable from a committed artifact, with no judgement call:
    • metadata.csv with at least one data row → metrics
    • a log pipeline under assets/logs/*.yamllogs
    • both → both entries
  4. .github/CODEOWNERS resolves <dir>/assets/dataflows.yaml (last-match-wins) to @DataDog/agent-integrations.

Every directory in this batch has @DataDog/agent-integrations as its Datadog team CODEOWNER (mendix is co-owned with @DataDog/ecosystems-review), so review from @DataDog/agent-integrations covers the whole PR.

Field values

provides:
  - id: <app_id>-<data_type>
    always_on: true
    granular: false
    data_type: <metrics|logs>
    direction: inbound

always_on: true / granular: false / direction: inbound matches 14 of the 17 dataflow entries already in the repo. The only deviation in the repo is vercel, which is always_on: false for a Serverless product-enablement reason that does not apply here.

Dataflow IDs are <app_id>-<data_type>, taking app_id from manifest.json rather than the directory name. All IDs were checked for collisions against every existing ID in the repo, and against each other.

Validation

These files were validated by executing the real validator, DataflowsValidationHandler from dd-source/domains/integrationscatalog/libs/catalogassetslib/dataflows_validation_handler.go, at ddoghq/dd-source@main, against every dataflows.yaml file in the repo working tree with this PR applied. Result: 0 failures, covering per-file unmarshalling, proto constraint validation, and the cross-file HandleLibrary ID-uniqueness check.

The harness was negative-tested first and confirmed to reject: a missing always_on; a data_type outside validDataTypes; an id breaking ^[a-z0-9-]+$; an id under 3 characters; a file with neither provides nor uses; a .yml extension; a missing manifest.json; and the same dataflow ID provided by two apps.

This matters because APW does not post validator comments on integrations-extras PRs (enable_validator_comments is set only for pub-platform-staging and publishing-platform). A malformed dataflows.yaml merges cleanly here and only fails afterwards, in the shared asset pipeline.

No data_type was guessed anywhere. Any integration whose correct value was not unambiguous is deferred to a later batch rather than approximated.

Contents

Metrics only (5)

Integration Dataflow IDs
concourse_ci concourse-ci-metrics
lighthouse lighthouse-metrics
mendix mendix-metrics
redis_sentinel redis-sentinel-metrics
vns3 vns3-metrics

Logs only (2)

Integration Dataflow IDs
lacework lacework-logs
sqreen sqreen-logs

Metrics and logs (2)

Integration Dataflow IDs
auth0 auth0-metrics, auth0-logs
bind9 bind9-metrics, bind9-logs

🤖 Generated with Claude Code

Split of #3093 by CODEOWNERS team.
Jira: TXP-277
@philjlee
philjlee requested review from a team as code owners August 6, 2026 19:18
@philjlee
philjlee requested review from davidfeng-datadog and removed request for a team August 6, 2026 19:18
@philjlee philjlee changed the title Define dataflows for agent-integrations-owned integrations (batch 1b) Define dataflows for agent-integrations-owned integrations Aug 6, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aa82e7c512

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@@ -0,0 +1,6 @@
provides:
- id: lacework-logs

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Declare Lacework's event dataflow

When a customer selects Events Summary, Lacework sends summaries to the Datadog Events platform, as documented in lacework/README.md lines 20-24 and 46-48. Advertising only lacework-logs therefore causes dataflow consumers to omit a supported ingestion path; add a corresponding inbound lacework-events entry.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DataDog fix this

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can only run on private repositories.

@@ -0,0 +1,6 @@
provides:
- id: redis-sentinel-metrics

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Declare Redis Sentinel's failover event dataflow

When a monitored master's IP changes after initialization, _process_master_stats calls self.event(...) for the failover (redis_sentinel.py lines 219-233). Declaring only redis-sentinel-metrics therefore hides a data type that the installed check actually emits; add a corresponding inbound redis-sentinel-events entry.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DataDog fix this

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can only run on private repositories.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant