feat(providerconfig): add stackSecretRef to consume stack connection secrets

[Duologic]

Summary

Allow ProviderConfig resources to reference a connection secret produced by a grafana_cloud_stack resource (managed or observed) via a new stackSecretRef field. This automatically populates URL and ID credential fields, eliminating manual configuration of stack-specific endpoints.

Stack secret coverage of ProviderConfig fields

ProviderConfig field	Covered by stackSecretRef?	Notes
spec.url	Yes	—
spec.oncallUrl	Yes	Remapped from oncall_api_url
spec.fleetManagementUrl	Yes	—
spec.orgId	Yes	—
spec.stackId	Yes	Remapped from id
spec.smUrl	No	Will be added to stack output by grafana/terraform-provider-grafana#2679
spec.cloudApiUrl	No	Defaults to Grafana Cloud; can't be sourced from the stack (chicken/egg: the Cloud API is needed to read the stack)
spec.cloudProviderUrl	No	Will be added by grafana/terraform-provider-grafana#2679
spec.connectionsApiUrl	No	Will be added by grafana/terraform-provider-grafana#2679

5 of 9 ProviderConfig spec fields are covered. The remaining 4 are not stack attributes — they are either environment-specific URLs or not yet exposed by the upstream TF provider.

Key changes:

• Observed (data-source) resources now support ConnectionDetailsFn, enabling them to publish connection details to a secret
• The managed grafana_cloud_stack resource exports all scalar atProvider fields as connection detail keys
• All ProviderConfig types gain an optional stackSecretRef that reads the stack secret and merges it into the credential map with key remapping (oncall_api_url → oncall_url, id → stack_id)

Precedence (lowest → highest)

1. Primary credential secret (credentials.secretRef)
2. Stack secret (stackSecretRef)
3. ProviderConfig spec fields (url, oncallUrl, etc.)

Review Guide

Hand-edited files

File	What changed
config/grafana/cloud.go	Added AdditionalConnectionDetailsFn to grafana_cloud_stack, exporting all scalar TF state attributes
docs/providerconfig-secret-fields.md	Documented stackSecretRef, key remapping, precedence, coverage table
pkg/generateobserved/emit.go	Generator now emits a ConnectionDetailsFn for all observed resources with scalar atProvider fields; includes external name as id
internal/clients/grafana.go	Added mergeStackSecret() helper, key remap map, wired into TerraformSetupBuilder() and ExtractModernConfig()
pkg/tfdatasource/controller.go	Added ConnectionDetailsFn field to Spec, wired into Observe() and Update()
apis/cluster/v1beta1/types.go	Added StackSecretRef *xpv1.SecretReference to ProviderConfigSpec
apis/namespaced/v1beta1/types.go	Added StackSecretRef *xpv1.SecretReference to ProviderConfigSpec
cluster/test/setup-cloud.sh	Rewired cloud e2e setup to use observed Stack + stackSecretRef instead of hardcoded URLs
cluster/test/teardown-cloud.sh	Added cleanup for observed Stack, cloud API ProviderConfig, and related secrets
Makefile	Updated cloud e2e env vars: replaced GRAFANA_URL/GRAFANA_ONCALL_URL with GRAFANA_CLOUD_ACCESS_POLICY_TOKEN/GRAFANA_STACK_SLUG
examples/cloud/v1alpha1/stack-secret-ref.yaml	New e2e test: creates a Folder using a ProviderConfig whose URL comes exclusively from stackSecretRef

Generated files

Everything else in the diff is generated output from make generate:

• internal/controller/namespaced/observed/*/zz_*_spec.go — All observed specs now include ConnectionDetailsFn with external name as id
• apis/*/zz_generated.deepcopy.go — Updated for StackSecretRef
• package/crds/*.yaml — CRDs updated with stackSecretRef
• apis/*/enterprise/v1alpha1/zz_*.go, apis/*/oss/v1alpha1/zz_*.go, config/provider-metadata.yaml — Upstream v4.29.0 schema changes (unrelated to this PR)
• examples-generated/*/oss/v1alpha1/dashboardv2.yaml — Removed by upstream v4.29.0 changes

Usage Examples

Managed Stack → ProviderConfig

# 1. Create a Cloud Stack that publishes its details to a secret
apiVersion: cloud.grafana.m.crossplane.io/v1alpha1
kind: Stack
metadata:
  name: my-stack
  namespace: default
spec:
  forProvider:
    name: my-stack
    slug: my-stack
  writeConnectionSecretToRef:
    name: my-stack-details
---
# 2. Create a ServiceAccountToken for authentication
apiVersion: cloud.grafana.m.crossplane.io/v1alpha1
kind: StackServiceAccountToken
metadata:
  name: my-stack-sa-token
  namespace: default
spec:
  forProvider:
    stackSlugRef:
      name: my-stack
    serviceAccountRef:
      name: my-stack-sa
    name: crossplane
  writeConnectionSecretToRef:
    name: my-stack-token
---
# 3. ProviderConfig consumes both secrets
apiVersion: grafana.m.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: my-stack-config
  namespace: default
spec:
  credentials:
    source: Secret
    secretRef:
      name: my-stack-token
      namespace: default
      key: instanceCredentials
  stackSecretRef:
    name: my-stack-details
    namespace: default

The ProviderConfig automatically gets url, oncall_url, fleet_management_url, org_id, stack_id, and all other stack fields — no manual endpoint configuration needed.

Observed (data-source) Stack → ProviderConfig

For pre-existing stacks not managed by Crossplane:

# 1. Observe an existing stack (data source)
apiVersion: cloud.grafana.o.crossplane.io/v1alpha1
kind: Stack
metadata:
  name: existing-stack
  namespace: default
spec:
  forProvider:
    slug: "my-stack-slug"
  providerConfigRef:
    kind: ClusterProviderConfig
    name: cloud-api
  writeConnectionSecretToRef:
    name: existing-stack-details
---
# 2. ProviderConfig references the observed stack's secret
apiVersion: grafana.m.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: existing-stack-config
  namespace: default
spec:
  credentials:
    source: Secret
    secretRef:
      name: my-auth-secret
      namespace: default
      key: credentials
  stackSecretRef:
    name: existing-stack-details
    namespace: default

Overriding specific fields

Stack secret values can be selectively overridden via ProviderConfig spec fields:

apiVersion: grafana.m.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: custom-config
  namespace: default
spec:
  # Override the URL from the stack secret with a custom one
  url: "https://custom-proxy.internal.example.com"
  credentials:
    source: Secret
    secretRef:
      name: my-stack-token
      namespace: default
      key: instanceCredentials
  stackSecretRef:
    name: my-stack-details
    namespace: default

E2E Test

The stackSecretRef feature is tested end-to-end in the e2e-cloud suite (make e2e-cloud).

How it works

The cloud e2e setup (cluster/test/setup-cloud.sh) now:

1. Creates a cloud API ClusterProviderConfig using GRAFANA_CLOUD_ACCESS_POLICY_TOKEN
2. Observes the existing test stack (cloud.grafana.o.crossplane.io/v1alpha1 Stack) by GRAFANA_STACK_SLUG
3. The observed Stack publishes all atProvider fields (url, oncall_api_url, org_id, etc.) to a connection Secret via writeConnectionSecretToRef
4. Creates an auth-only instance credentials Secret (just {"auth": "..."} — intentionally no URL)
5. Creates both ProviderConfig and ClusterProviderConfig named e2e-cloud-instance with:
   • credentials.secretRef → auth-only secret
   • stackSecretRef → observed stack's connection secret

What it proves

• stack-secret-ref.yaml: Creates a Folder using e2e-cloud-instance. Since the auth secret has no URL, the Folder can only be created if stackSecretRef correctly provides the url from the observed Stack's connection secret.
• oncall-escalation-refs.yaml and oncall-shift-rolling-users.yaml: These existing oncall tests now get oncall_url from the stack secret (remapped from oncall_api_url) instead of a hardcoded GRAFANA_ONCALL_URL env var.

Required env vars

GRAFANA_SA_TOKEN=<service-account-token>
GRAFANA_CLOUD_ACCESS_POLICY_TOKEN=<cloud-access-policy-token-with-stacks:read>
GRAFANA_STACK_SLUG=<stack-slug>
GRAFANA_TEST_USER=<oncall-username>

Running

set -a && source .env && set +a
make e2e-cloud

[suntala]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f4fa22330

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Include datasource id in observed Stack connection details

This connection-details closure never emits an id key, so stackSecretRef cannot populate stack_id when the secret comes from an observed cloud.grafana.o Stack. The read path stores datasource id only as external name (meta.SetExternalName(cr, d.Id())), and mergeStackSecret() relies on an id -> stack_id remap, so any ProviderConfig that depends on observed-stack secrets for stack_id (for example k6/cloud-provider flows) will still fail unless users manually set spec.stackId.

Useful? React with 👍 / 👎.

[Duologic]

Fixed in bd45734: the generator now emits the external name as id in all observed ConnectionDetailsFn closures.

See lines 414-416:

if id := meta.GetExternalName(cr); id != "" {
    cd["id"] = []byte(id)
}