[ENG-3222] Add GCP Dataflow Job adapter with comprehensive link rules (#4302)

## Summary

- Add a GCP Dataflow Job dynamic adapter that discovers Dataflow jobs
and links them to Pub/Sub topics/subscriptions, BigQuery
tables/datasets, Spanner instances, Bigtable instances, Compute
networks/subnetworks, IAM service accounts, and KMS crypto keys
- This is the first step toward detecting outages like ISSUE-7070 where
a missing Pub/Sub subscription caused a Dataflow job failure — the
Pub/Sub links are the most critical piece
- Adds `roles/dataflow.viewer` to all IAM role configuration surfaces
and enables `dataflow.googleapis.com` in deployment config

## Linear Ticket

- **Ticket**:
[ENG-3222](https://linear.app/overmind/issue/ENG-3222/gcp-dataflow-job-adapter)
— GCP Dataflow Job Adapter
- **Purpose**: Enable infrastructure discovery for GCP Dataflow jobs so
Overmind can map their dependencies and detect blast radius from changes
to connected resources
- **Related**: [ENG-3217](https://linear.app/overmind/issue/ENG-3217) —
the original outage where a missing Pub/Sub subscription broke a
Dataflow job

## Changes

**New files:**
- `sources/gcp/dynamic/adapters/dataflow-job.go` — Dynamic adapter with
12 link rules across Pub/Sub, BigQuery, Spanner, Bigtable, networking,
and IAM
- `sources/gcp/dynamic/adapters/dataflow-job_test.go` — Tests for Get,
Search, ErrorHandling, and StaticTests validating all link rules
- `docs.overmind.tech/docs/sources/gcp/Types/gcp-dataflow-job.md` — Type
documentation page
- `docs.overmind.tech/docs/sources/gcp/data/gcp-dataflow-job.json` —
Type metadata

**Modified files:**
- `sources/gcp/shared/models.go` — `Dataflow` API and `Job` resource
constants
- `sources/gcp/shared/item-types.go` — `DataflowJob` item type
- `sources/gcp/shared/predefined-roles.go` — `roles/dataflow.viewer`
with permissions
- `deploy/modules/ovm-services/gke.tf` — Enable
`dataflow.googleapis.com`
- `docs.overmind.tech/docs/sources/gcp/configuration.md` — Role in docs
table and services list
-
`services/frontend/src/features/settings/sources/details/gcp-scripts.ts`
— Role in setup scripts
- `sources/gcp/setup/scripts/overmind-gcp-roles.sh` — Role in shell
script

**Known limitation:** `spannerDetails.databaseId` and
`bigTableDetails.tableId` return plain names (not resource URIs) in the
Dataflow API, so those links won't auto-resolve for compound-key target
types. Link rules are kept for documentation and `PotentialLinks`
registration. The critical Pub/Sub links work correctly.

## Approved Plan

- **Plan approver**: Elliot
- **Linear ticket**:
[ENG-3222](https://linear.app/overmind/issue/ENG-3222/gcp-dataflow-job-adapter)
(contains the approved plan)

> Deviation analysis and reviewer assignment are handled automatically
by the
> pre-approved PR review automation (see docs/PREAPPROVED_CHANGES.md).

## Pre-PR Review

<details>
<summary>Review findings: 0 Blocking, 1 Warning, 4 Advisories, 0
Failed</summary>

### Security Review (P0)
**Blocking: 0 | Warning: 0**
No security findings. URL construction uses fixed
`dataflow.googleapis.com` host. No new endpoints, no secrets, no auth
changes.

### Architecture / Scope Review (P1)
**Advisory: 3**
1. Cross-cutting scope — PR touches 4 top-level directories (sources/,
deploy/, docs.overmind.tech/, services/frontend/)
2. New adapter is not behind a feature flag — acceptable for additive
discovery capability
3. Existing customers who already ran setup will need to re-grant
`roles/dataflow.viewer` to discover Dataflow jobs

### DevOps / Deployment Review (P1)
**Warning: 1 | Advisory: 1**
1. (Warning) Infrastructure change without documented rollback —
enabling `dataflow.googleapis.com` is low-risk and revertible via PR
revert + terraform apply
2. (Advisory) Consider validating in dogfood before production rollout

</details>

Made with [Cursor](https://cursor.com)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Adds a new GCP discovery adapter plus new IAM role and API enablement
across setup surfaces; risk is mainly around permission rollout and
correctness of new link rules affecting dependency mapping (no write
access or auth flow changes).
>
> **Overview**
> Adds support for discovering **GCP Dataflow Jobs** via a new dynamic
adapter (`gcp-dataflow-job`) with `GET` and location-scoped `SEARCH`,
plus link rules to map job dependencies to Pub/Sub, BigQuery, Spanner,
Bigtable, networking, IAM service accounts, and KMS keys.
>
> Wires this new resource into the system by introducing the `Dataflow`
API / `Job` resource constants and `DataflowJob` item type, adding
`roles/dataflow.viewer` (and required permissions) to predefined roles
and all customer setup script surfaces (frontend templates, shell
scripts, and docs), and enabling `dataflow.googleapis.com` in Terraform
deployment config. Includes comprehensive adapter tests and new type
metadata/docs for the Dataflow Job resource.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
d203d8485f5af18538afd47a02e6ab1ba6f153a7. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

GitOrigin-RevId: f5918383aeae36e45e60c5f3f3abc7ec0b129bc3

Author: dylanratcliffe

docs.overmind.tech/docs/sources/gcp/Types/gcp-dataflow-job.md

@@ -0,0 +1,59 @@
+---
+title: GCP Dataflow Job
+sidebar_label: gcp-dataflow-job
+---
+
+A **Google Cloud Dataflow Job** is a managed Apache Beam pipeline that processes streaming or batch data at scale. Dataflow handles resource provisioning, autoscaling, and fault tolerance, allowing you to run data processing workloads without managing the underlying infrastructure. Jobs can read from and write to Pub/Sub, BigQuery, Spanner, Bigtable, and other GCP services. See the official documentation for full details: https://cloud.google.com/dataflow/docs.
+
+**Terraform Mappings:**
+
+- `google_dataflow_job.job_id`
+- `google_dataflow_flex_template_job.job_id`
+
+## Supported Methods
+
+- `GET`: Get a gcp-dataflow-job by its "locations|jobs"
+- ~~`LIST`~~
+- `SEARCH`: Search for gcp-dataflow-job by location
+
+## Possible Links
+
+### [`gcp-big-query-dataset`](/sources/gcp/Types/gcp-big-query-dataset)
+
+Dataflow jobs that read from or write to BigQuery reference the dataset containing the tables they use. If the dataset is deleted or misconfigured, the job may fail to access data.
+
+### [`gcp-big-query-table`](/sources/gcp/Types/gcp-big-query-table)
+
+Dataflow jobs can read from or write to specific BigQuery tables. If a table is deleted or its schema changes, the job may fail.
+
+### [`gcp-big-table-admin-instance`](/sources/gcp/Types/gcp-big-table-admin-instance)
+
+Dataflow jobs that use Bigtable as a source or sink reference the Bigtable instance. If the instance is deleted or misconfigured, the job may fail.
+
+### [`gcp-cloud-kms-crypto-key`](/sources/gcp/Types/gcp-cloud-kms-crypto-key)
+
+When customer-managed encryption keys (CMEK) are enabled for the Dataflow job environment, the job references the Cloud KMS Crypto Key used for encryption.
+
+### [`gcp-compute-network`](/sources/gcp/Types/gcp-compute-network)
+
+Dataflow worker VMs are attached to a VPC network. If the network is deleted or misconfigured, workers may lose connectivity or fail to start.
+
+### [`gcp-compute-subnetwork`](/sources/gcp/Types/gcp-compute-subnetwork)
+
+Dataflow workers run in a specific subnetwork. If the subnetwork is deleted or misconfigured, workers may lose connectivity or fail to start.
+
+### [`gcp-iam-service-account`](/sources/gcp/Types/gcp-iam-service-account)
+
+Dataflow workers run under a service account that grants them permissions to access other GCP services. If the service account is deleted or its permissions change, the job may fail.
+
+### [`gcp-pub-sub-subscription`](/sources/gcp/Types/gcp-pub-sub-subscription)
+
+Dataflow jobs that consume messages from Pub/Sub reference the subscription. If the subscription is deleted or misconfigured, the job may fail to consume messages.
+
+### [`gcp-pub-sub-topic`](/sources/gcp/Types/gcp-pub-sub-topic)
+
+Dataflow jobs that publish to or consume from Pub/Sub reference the topic. If the topic is deleted or misconfigured, the job may fail to read or write messages.
+
+### [`gcp-spanner-instance`](/sources/gcp/Types/gcp-spanner-instance)
+
+Dataflow jobs that use Spanner reference the Spanner instance. If the instance is deleted or misconfigured, the job may fail.

docs.overmind.tech/docs/sources/gcp/configuration.md

@@ -360,7 +360,7 @@ Overmind requires read-only access to discover and map your GCP infrastructure.
 
 **Read-only viewer roles** for GCP services including:
 
-- Compute Engine, GKE, Cloud Run, Cloud Functions
+- Compute Engine, GKE, Cloud Run, Cloud Functions, Dataflow
 - Cloud SQL, BigQuery, Spanner, Cloud Storage
 - IAM, networking, monitoring, and logging resources
 - And other GCP services
@@ -408,6 +408,7 @@ Here are all the predefined GCP roles that Overmind requires, plus the custom ro
 | `roles/dataform.viewer`                 | Dataform resource discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dataform#dataform.viewer)                                          |
 | `roles/dataplex.catalogViewer`          | Dataplex catalog resource discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dataplex#dataplex.catalogViewer)                           |
 | `roles/dataplex.viewer`                 | Dataplex resource discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dataplex#dataplex.viewer)                                          |
+| `roles/dataflow.viewer`                 | Dataflow job discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dataflow#dataflow.viewer)                                               |
 | `roles/dataproc.viewer`                 | Dataproc cluster discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dataproc#dataproc.viewer)                                           |
 | `roles/dns.reader`                      | Cloud DNS resource discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dns#dns.reader)                                                   |
 | `roles/essentialcontacts.viewer`        | Essential Contacts discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/essentialcontacts#essentialcontacts.viewer)                       |

docs.overmind.tech/docs/sources/gcp/data/gcp-dataflow-job.json

@@ -0,0 +1,31 @@
+{
+  "type": "gcp-dataflow-job",
+  "category": 7,
+  "potentialLinks": [
+    "gcp-big-query-dataset",
+    "gcp-big-query-table",
+    "gcp-big-table-admin-instance",
+    "gcp-cloud-kms-crypto-key",
+    "gcp-compute-network",
+    "gcp-compute-subnetwork",
+    "gcp-iam-service-account",
+    "gcp-pub-sub-subscription",
+    "gcp-pub-sub-topic",
+    "gcp-spanner-instance"
+  ],
+  "descriptiveName": "GCP Dataflow Job",
+  "supportedQueryMethods": {
+    "get": true,
+    "getDescription": "Get a gcp-dataflow-job by its \"locations|jobs\"",
+    "search": true,
+    "searchDescription": "Search for gcp-dataflow-job by location"
+  },
+  "terraformMappings": [
+    {
+      "terraformQueryMap": "google_dataflow_job.job_id"
+    },
+    {
+      "terraformQueryMap": "google_dataflow_flex_template_job.job_id"
+    }
+  ]
+}

sources/gcp/dynamic/adapters/dataflow-job.go

@@ -0,0 +1,80 @@
+package adapters
+
+import (
+	"github.com/overmindtech/cli/go/sdp-go"
+	gcpshared "github.com/overmindtech/cli/sources/gcp/shared"
+)
+
+// Dataflow Job adapter for Google Cloud Dataflow jobs.
+// Reference: https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs#Job
+// GET:  https://dataflow.googleapis.com/v1b3/projects/{project}/locations/{location}/jobs/{jobId}
+// LIST: https://dataflow.googleapis.com/v1b3/projects/{project}/locations/{location}/jobs
+var _ = registerableAdapter{
+	sdpType: gcpshared.DataflowJob,
+	meta: gcpshared.AdapterMeta{
+		SDPAdapterCategory: sdp.AdapterCategory_ADAPTER_CATEGORY_COMPUTE_APPLICATION,
+		LocationLevel:      gcpshared.ProjectLevel,
+		GetEndpointFunc: gcpshared.ProjectLevelEndpointFuncWithTwoQueries(
+			"https://dataflow.googleapis.com/v1b3/projects/%s/locations/%s/jobs/%s",
+		),
+		SearchEndpointFunc: gcpshared.ProjectLevelEndpointFuncWithSingleQuery(
+			"https://dataflow.googleapis.com/v1b3/projects/%s/locations/%s/jobs",
+		),
+		UniqueAttributeKeys: []string{"locations", "jobs"},
+		IAMPermissions:      []string{"dataflow.jobs.get", "dataflow.jobs.list"},
+		PredefinedRole:      "roles/dataflow.viewer",
+	},
+	linkRules: map[string]*gcpshared.Impact{
+		// Pub/Sub links (critical for ENG-3217 outage detection)
+		"jobMetadata.pubsubDetails.topic": {
+			ToSDPItemType: gcpshared.PubSubTopic,
+			Description:   "If the Pub/Sub Topic is deleted or misconfigured: The Dataflow job may fail to read/write messages. If the Dataflow job changes: The topic remains unaffected.",
+		},
+		"jobMetadata.pubsubDetails.subscription": {
+			ToSDPItemType: gcpshared.PubSubSubscription,
+			Description:   "If the Pub/Sub Subscription is deleted or misconfigured: The Dataflow job may fail to consume messages. If the Dataflow job changes: The subscription remains unaffected.",
+		},
+
+		// BigQuery links
+		"jobMetadata.bigqueryDetails.table": {
+			ToSDPItemType: gcpshared.BigQueryTable,
+			Description:   "If the BigQuery Table is deleted or misconfigured: The Dataflow job may fail to read/write data. If the Dataflow job changes: The table remains unaffected.",
+		},
+		"jobMetadata.bigqueryDetails.dataset": {
+			ToSDPItemType: gcpshared.BigQueryDataset,
+			Description:   "If the BigQuery Dataset is deleted or misconfigured: The Dataflow job may fail to access tables. If the Dataflow job changes: The dataset remains unaffected.",
+		},
+
+		// Spanner links
+		"jobMetadata.spannerDetails.instanceId": {
+			ToSDPItemType: gcpshared.SpannerInstance,
+			Description:   "If the Spanner Instance is deleted or misconfigured: The Dataflow job may fail to read/write data. If the Dataflow job changes: The instance remains unaffected.",
+		},
+		// Bigtable links
+		"jobMetadata.bigTableDetails.instanceId": {
+			ToSDPItemType: gcpshared.BigTableAdminInstance,
+			Description:   "If the Bigtable Instance is deleted or misconfigured: The Dataflow job may fail to read/write data. If the Dataflow job changes: The instance remains unaffected.",
+		},
+		// Environment/infra links
+		"environment.serviceAccountEmail": gcpshared.IAMServiceAccountImpactInOnly,
+		"environment.serviceKmsKeyName":   gcpshared.CryptoKeyImpactInOnly,
+		"environment.workerPools.network": gcpshared.ComputeNetworkImpactInOnly,
+		"environment.workerPools.subnetwork": {
+			ToSDPItemType: gcpshared.ComputeSubnetwork,
+			Description:   "If the Compute Subnetwork is deleted or misconfigured: Dataflow workers may lose connectivity or fail to start. If the Dataflow job changes: The subnetwork remains unaffected.",
+		},
+	},
+	terraformMapping: gcpshared.TerraformMapping{
+		Reference: "https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dataflow_job",
+		Mappings: []*sdp.TerraformMapping{
+			{
+				TerraformMethod:   sdp.QueryMethod_GET,
+				TerraformQueryMap: "google_dataflow_job.job_id",
+			},
+			{
+				TerraformMethod:   sdp.QueryMethod_GET,
+				TerraformQueryMap: "google_dataflow_flex_template_job.job_id",
+			},
+		},
+	},
+}.Register()