refactor: read the plugin catalog from marketplace-api by ChielTimmermans · Pull Request #408 · fundament-oss/fundament

ChielTimmermans · 2026-09-01T11:04:36Z

Follow-up to #395 (merged). Moves every reader of organization.v1.PluginService onto catalog.v1. organization-api keeps serving the deprecated RPCs, so the two can deploy in either order.

plugin-controller

defclient fetches definitions from marketplace-catalog-api by (organization, plugin) name — the spelling a PluginInstallation carries. ORGANIZATION_API_URL → MARKETPLACE_CATALOG_API_URL.

That rename reaches two deployments: the chart's controller and the shoot-side one cluster-worker provisions (PLUGIN_ORGANIZATION_API_URL → PLUGIN_MARKETPLACE_CATALOG_API_URL). Without the second, a shoot controller would come up missing a URL it now requires — that path landed in #371 while this was in progress. The local sandbox relay is now just plugins sandbox-catalog, bridging the catalog's NodePort.

console

Browsing, detail and the install modal's version picker read catalog.v1 over a separate unauthenticated transport (no cookie, no Fun-Organization header). catalog.v1 returns ids where organization.v1 returned nested names, so publisher and category names are resolved once via ListPublishers/ListCategories rather than per plugin; the detail page maps into a view model to keep its template intact.

One behaviour change: ListPluginVersions returns only published versions, where ListPluginDefinitions returned drafts too despite its name. The picker no longer offers a draft as something to install.

ListPresets has no catalog equivalent and stays on organization.v1.

wiring

externalUrls.marketplace per environment, the console's marketplaceApiUrl config key, and that origin added to CONNECT_SRC — without which the browser's CSP blocks every catalog call.

verification

go build ./..., go test ./cluster-worker/... ./plugin-controller/... ./marketplace-api/..., golangci-lint --new-from-rev origin/master — all clean
console: bunx tsc -p tsconfig.app.json, bun run build, bun run test — tests at baseline (app.spec.ts already fails on master)
bun run lint reports one error in src/connect/observable-client.ts, untouched here and introduced by 853ab48 on master
helm template renders for local and sandbox values

github-actions · 2026-09-01T11:07:57Z

PR Environment

Service	URL
Console	https://console.pr408.fundament.digilab.reviews
Console demo (walkthrough)	https://console-demo.pr408.fundament.digilab.reviews
Authn API	https://authn.pr408.fundament.digilab.reviews
Organization API	https://organization.pr408.fundament.digilab.reviews
Kube API Proxy	https://k8s-api.pr408.fundament.digilab.reviews
Docs	https://docs.pr408.fundament.digilab.reviews
DCIM	https://dcim.pr408.fundament.digilab.reviews
DCIM API	https://dcim-api.pr408.fundament.digilab.reviews
DCIM Auth	https://dcim-authn.pr408.fundament.digilab.reviews
Marketplace Catalog API	https://marketplace-api.pr408.fundament.digilab.reviews

Commit: 0b0fde2ee13bfb3719c07210b0af4a6edadab469-408-merge-2277
Namespace: tn-fundament-pr-408

Environment will be ready once Flux reconciles (~5 min after images are built).

Test credentials (all passwords: password)

alice@acme-corp.com (Acme Corp)
bart@acme-corp.com (Acme Corp)
cindy@acme-corp.com (Acme Corp)
david@globex.com (Globex)
emily@globex.com (Globex)
frank@initech.com (Initech)
grace@initech.com (Initech)

Debugging this environment

Kubeconfig and pinniped setup: see the deployment and infra docs.

1. Connect

export KUBECONFIG=~/.kube/pinniped_digikluster.yaml
export NS=tn-fundament-pr-408
kubectl -n $NS get pods

2. Flux reconciliation

kubectl -n $NS get helmrelease,gitrepository
kubectl -n $NS describe helmrelease fundament
kubectl -n $NS describe gitrepository fundament   # auth / branch / commit issues surface here

# Force a retry
kubectl -n $NS annotate --overwrite helmrelease/fundament \
  reconcile.fluxcd.io/requestedAt="$(date +%s)"

3. Pod / app state

kubectl -n $NS get pods
kubectl -n $NS get events --sort-by=.lastTimestamp
kubectl -n $NS describe pod <pod>   # readiness probes, env refs, and pod events
kubectl -n $NS logs <pod>
kubectl -n $NS logs <pod> --previous   # if the pod crashed

# Narrow to a single service (app.kubernetes.io/name = authn-api, organization-api,
# console-frontend, docs-frontend, cluster-worker, authz-worker, kube-api-proxy, dex, openfga)
kubectl -n $NS get pods -l app.kubernetes.io/name=organization-api

4. OpenFGA / authorization config

# organization-api and kube-api-proxy readiness depend on OpenFGA.
kubectl -n $NS get configmap openfga-config -o yaml
kubectl -n $NS logs -l app.kubernetes.io/name=openfga -c setup
kubectl -n $NS logs -l app.kubernetes.io/name=openfga -c openfga --tail=100

# If consumers are still using an old store/model ID after setup patched openfga-config:
kubectl -n $NS rollout restart deployment/organization-api deployment/kube-api-proxy

5. DB & migrations

kubectl -n $NS get cluster.postgresql.cnpg.io/db
kubectl -n $NS describe cluster.postgresql.cnpg.io/db

# Migrations run as a Job named db-migrations-<helm-revision>, with a
# 5m TTL after success. If the Job is gone, migrations almost
# certainly succeeded.
kubectl -n $NS get jobs -l app.kubernetes.io/name=db-migrations
kubectl -n $NS logs -l app.kubernetes.io/name=db-migrations -c migrations

# OpenFGA blocks on that Job before migrating its schema. If openfga
# is stuck in Init, this is why:
kubectl -n $NS logs -l app.kubernetes.io/name=openfga -c wait-for-reset

6. Routes & TLS

kubectl -n $NS get httproutes
kubectl -n istio-gateway get certificate tn-fundament-pr-408-wildcard-tls

7. Nothing deployed at all?

If the namespace is missing, the PR overlay never reached the cluster. Check the PR overlay in the Flux repo — if it's not there, the CI push to the Flux repo failed; if it is, the cluster-level Flux reconciliation is stuck.

jos-

Looks mostly GTM, but see comments below.

Not tied to a line in the diff

console-frontend/public/assets/config/config.json:3

Is console-frontend/public/assets/config/config.json:3 missing marketplaceApiUrl? Then baseUrl is undefined instead of '' when the chart ConfigMap is not mounted. The transport factory throws instead of falling back to a relative URL.

cluster-worker/README.md:256

cluster-worker/README.md:256 still documents PLUGIN_ORGANIZATION_API_URL / ORGANIZATION_API_URL. After the rename, setting those leaves Config.Enabled() false and shoot provisioning silently off.

jos- · 2026-09-01T12:50:21Z

  dcimApi: https://dcim-api.fundament.projects.digilab.network
  dcim: https://dcim.fundament.projects.digilab.network
  dcimAuthn: https://dcim-authn.fundament.projects.digilab.network
+  marketplace: https://marketplace-api.fundament.projects.digilab.network


externalUrls.marketplace points at marketplace-api.fundament.projects.digilab.network, but apparently this file has no marketplaceCatalogApi block, so the service defaults to disabled and gets no route. The console's plugins page will fail to load on sandbox.

Right — added a marketplaceCatalogApi block (enabled: true, replicas: 2, matching the other APIs in that file). The HTTPRoute already comes from pr-overlay.yaml; helm template -f values-sandbox.yaml now renders the Deployment.

jos- · 2026-09-01T12:50:21Z

  dcimApi: https://dcim-api.fundament.localhost:8443
  dcim: https://dcim.fundament.localhost:8443
  dcimAuthn: https://dcim-authn.fundament.localhost:8443
+  marketplace: https://marketplace-catalog-api.fundament.localhost:8443


According to Claude, the management-cluster plugin-controller lost its in-cluster default and now falls back to https://marketplace-catalog-api.fundament.localhost:8443. That host resolves to the host machine's 127.0.0.1:8443 with no CoreDNS rewrite, and the mkcert CA is not trusted inside the pod, so the controller cannot reach the catalog from local dev.

Needs an explicit pluginController.catalogApiUrl: http://marketplace-catalog-api:8080.

Good catch, and it was a regression: the chart used to default to the in-cluster http://organization-api:8080 and I replaced that with externalUrls.marketplace, which is the external HTTPS host.

Fixed at the source rather than per environment — the chart now defaults to http://marketplace-catalog-api:8080, mirroring the old semantics, so the management-cluster controller stays in-cluster everywhere and only out-of-cluster deployments override. plugins/sandbox/values.yaml keeps its host.k3d.internal:38080 relay, and cluster-worker still passes externalUrls.marketplace for shoots. Verified by rendering all three.

jos- · 2026-09-01T12:50:22Z

          id: backendPlugin.id,
          name: backendPlugin.name,
-          organizationName: backendPlugin.organizationName,
+          organizationName: publisherNames.get(backendPlugin.organizationId) ?? '',


Comment from Claude about this:

publisherNames.get(...) ?? '' silently yields an empty organizationName, which then feeds install identity and installed-state matching. Same pattern at plugin-details.component.ts:157.

The sibling category mapping three lines down falls back to the id instead, which at least keeps the value locatable. Worth matching that, or surfacing the miss rather than swallowing it.

Agreed. Both now fall back to the publisher id, matching the category mapping, with a comment on why: the invariant is that every listed plugin's publisher has a live listing (same RLS gates both), so a miss is an inconsistency — and the id keeps it traceable instead of installing under a nameless --<plugin>.

jos- · 2026-09-01T12:50:22Z

 	}
-	if p.OrganizationAPIURL == "" {
+	if p.CatalogAPIURL == "" {
 		return fmt.Errorf("organization-api URL is empty")


Error text still names the old field.

Suggested change

return fmt.Errorf("organization-api URL is empty")

return fmt.Errorf("catalog-api URL is empty")

ChielTimmermans · 2026-09-01T13:01:21Z

Addressed all six — pushed as ee4116cb.

The two not tied to a diff line:

console-frontend/public/assets/config/config.json — added marketplaceApiUrl: "". You were right about the failure mode: without the key the dev fallback yields undefined, and createConnectTransport throws on it rather than treating it as a relative URL.
cluster-worker/README.md — updated both spots: the shoot Deployment env row (ORGANIZATION_API_URL → MARKETPLACE_CATALOG_API_URL) and the config table (PLUGIN_ORGANIZATION_API_URL → PLUGIN_MARKETPLACE_CATALOG_API_URL), plus the sentence saying Helm wires it from externalUrls.organization, which is now externalUrls.marketplace.

One I did not touch, for a second opinion: docs/funs/FUN-19.adoc still says the controller's ORGANIZATION_API_URL must be organization-api's external URL. That is an ADR recording a decision this PR partly supersedes, so amending it silently in a refactor felt wrong — happy to update it here, or leave it to a FUN that supersedes it properly.

Verification after the changes: go build ./..., go test ./cluster-worker/..., console typecheck and tests all as before, and helm template for local, sandbox and plugins/sandbox/values.yaml each render the URL I expect.

jos-

LGTM 👍

Moves the storefront readers of organization.v1.PluginService onto catalog.v1, which is what #395 deprecated them for. organization-api keeps serving the RPCs; nothing is deleted here, so the two can be deployed in either order. plugin-controller fetches definitions from marketplace-catalog-api by (organization, plugin) name — the spelling a PluginInstallation carries. ORGANIZATION_API_URL becomes MARKETPLACE_CATALOG_API_URL, defaulting to the in-cluster marketplace-catalog-api Service as the old value did. That rename reaches two deployments: the chart's controller, and the shoot-side one cluster-worker provisions, whose PLUGIN_ORGANIZATION_API_URL becomes PLUGIN_MARKETPLACE_CATALOG_API_URL — a shoot controller would otherwise come up without the URL it now requires. Setting the old pluginController.organizationApiUrl key now fails the render rather than silently falling back to the default. The local sandbox relay (`just plugins sandbox-catalog`) bridges the catalog's NodePort instead of organization-api's. The console's plugins page and plugin detail page read catalog.v1 over their own unauthenticated transport: no cookie, no Fun-Organization header. catalog.v1 returns ids where organization.v1 returned nested names, so publisher and category names are resolved once through ListPublishers and ListCategories rather than per plugin, and the detail page maps the response into a view model so its template is unchanged apart from tags, which are plain labels now. The install modal's version picker reads ListPluginVersions, which returns only published versions — organization.v1's ListPluginDefinitions returned drafts too, so a draft is no longer offered as something to install. Still on organization.v1, deliberately: ListPresets, which has no catalog equivalent, and the three cluster-side views (cluster-plugins, cluster-details, shared-plugins-form) that list plugins to name what is installed — the catalog only returns PUBLIC listings, so moving them would hide an organization's own private plugins from its clusters. Adds externalUrls.marketplace per environment, the console's marketplaceApiUrl config key, and that origin to CONNECT_SRC — without which the browser's CSP blocks every catalog call. Regenerates db/fundament.sql from the model: #395 relaxed the three catalog policies in fundament.dbm and in migration 036 but left the schema of record carrying the old published-only predicates, which the next generated migration would have reverted.

ChielTimmermans force-pushed the feat/marketplace-catalog-consumers branch from 876db95 to 014d39a Compare September 1, 2026 11:21

jos- reviewed Sep 1, 2026

View reviewed changes

ChielTimmermans force-pushed the feat/marketplace-catalog-consumers branch from 014d39a to ee4116c Compare September 1, 2026 13:00

jos- previously approved these changes Sep 1, 2026

View reviewed changes

ChielTimmermans dismissed jos-’s stale review via 07fdb82 September 1, 2026 13:35

ChielTimmermans force-pushed the feat/marketplace-catalog-consumers branch from ee4116c to 07fdb82 Compare September 1, 2026 13:35

ChielTimmermans mentioned this pull request Sep 1, 2026

feat(marketplace): serve plugin presets from the catalog #409

Open

Navigation Menu

Uh oh!

refactor: read the plugin catalog from marketplace-api - #408

refactor: read the plugin catalog from marketplace-api#408
ChielTimmermans wants to merge 1 commit into
masterfrom
feat/marketplace-catalog-consumers

ChielTimmermans commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026 •

edited

Loading

Uh oh!

jos- left a comment •

edited

Loading

Uh oh!

jos- Sep 1, 2026

Uh oh!

ChielTimmermans Sep 1, 2026

Uh oh!

jos- Sep 1, 2026

Uh oh!

ChielTimmermans Sep 1, 2026

Uh oh!

jos- Sep 1, 2026

Uh oh!

ChielTimmermans Sep 1, 2026

Uh oh!

jos- Sep 1, 2026

Uh oh!

ChielTimmermans Sep 1, 2026

Uh oh!

ChielTimmermans commented Sep 1, 2026

Uh oh!

jos- left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	return fmt.Errorf("organization-api URL is empty")
	return fmt.Errorf("catalog-api URL is empty")

Uh oh!

Conversation

ChielTimmermans commented Sep 1, 2026

plugin-controller

console

wiring

verification

Uh oh!

github-actions Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Environment

Uh oh!

jos- left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Not tied to a line in the diff

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChielTimmermans commented Sep 1, 2026

Uh oh!

jos- left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Sep 1, 2026 •

edited

Loading

jos- left a comment •

edited

Loading