[Catalog] service not starting with External PostgreSQL #2044
Description
Hi,
I want to deploy catalog to the Jfrog platform using Kubernetes Helm installation. It is a fresh installation and follwing this documentation: https://jfrog.com/help/r/jfrog-installation-setup-documentation/enabling-jfrog-catalog-in-xray-helm-chart
I used the xray chart at version 103.124.26 (also tested with 103.124.13) and using the dependceny for the catalog chart (see values file below).
The external database for catalog is configured the same way like xray database is.
Xray is running, but the catalog service is not able to start. The following error is printed in the container logs ( sensitive infos):
shared.database.password (Environment Variable: JF_SHARED_DATABASE_PASSWORD): *****
shared.database.shouldRunCentralMigrations (Default Value): false
shared.database.type (File): postgresql
shared.database.url (Environment Variable: JF_SHARED_DATABASE_URL): postgres://<database-url-redacted>:5432/artifactory_catalog?sslmode=require
shared.database.username (Environment Variable: JF_SHARED_DATABASE_USERNAME): artifactory_catalog
shared.logging.consoleLog.format (Default Value): jftext
shared.multiTenant.cellid (Default Value): cellid
shared.multiTenant.enabled (Default Value): false
shared.multiTenant.tenantRegistryClient.caCert (Default Value):
shared.multiTenant.tenantRegistryClient.cacheExpirationSecs (Default Value): 21600
shared.multiTenant.tenantRegistryClient.clientCert (Default Value):
shared.multiTenant.tenantRegistryClient.clientCertKey (Default Value): *****
shared.multiTenant.tenantRegistryClient.dialTimeoutSecs (Default Value): 10
shared.multiTenant.tenantRegistryClient.endpoints (Default Value): http://127.0.0.1:2379
shared.multiTenant.tenantRegistryClient.password (Default Value): *****
shared.multiTenant.tenantRegistryClient.requestTimeoutSecs (Default Value): 30
shared.multiTenant.tenantRegistryClient.username (Default Value): arti
shared.newrelic.appName (Default Value):
shared.newrelic.distributedTracerEnabled (Default Value): true
shared.newrelic.enableDebugLog (Default Value): false
shared.newrelic.enabled (Default Value): false
shared.newrelic.licenseKey (Default Value): *****
shared.node.id (Environment Variable: JF_SHARED_NODE_ID): xray-catalog-<redacted>
shared.node.ip (Default Value): <redacted>
shared.security.ExposeApplicationHeaders (Default Value): false
shared.security.bootstrapKeysReadTimeoutSecs (Default Value): 120
shared.security.joinKeyFile (Default Value): etc/security/join.key
shared.security.masterKeyFile (Default Value): etc/security/master.key
shared.system.technicalServerName (Default Value):
system.yaml (Environment Variable: JF_SYSTEM_YAML): /opt/jfrog/catalog/var/etc/system.yaml [startup]
2025-09-25T12:35:24.139Z [jfcat[] [INFO [] [69d4df462be1c495[] [application.go:95 [] [main [] [ ] - Catalog (jfcatalog) service initialization started.
Version: 1.18.1 (revision: 1fa5878, build date: 20250506121456)
PID: 205
Home: /opt/jfrog/catalog
Config: /opt/jfrog/catalog/var/etc/system.yaml
Catalog Mode: singleTenant [startup]
2025-09-25T12:35:24.139Z [jfcat[] [INFO [] [69d4df462be1c495[] [application.go:167 [] [main [] [ [] - Starting singleTenant mode [startup]
2025-09-25T12:35:24.139Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [security_keys.go:117 [] [main [] [ [] - Resolving master key with 2m0s timeout [startup]
2025-09-25T12:35:24.139Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [security_keys.go:118 [] [main [] [ [] - Cluster join: Resolving join key with 2m0s timeout [startup]
2025-09-25T12:35:24.139Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [security_keys.go:127 [] [main [] [ [] - Master key resolved successfully [startup]
2025-09-25T12:35:24.139Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [security_keys.go:130 [] [main [] [ [] - Cluster join: Join key resolved successfully [startup]
2025-09-25T12:35:24.140Z [jfcat[] [INFO [] [69d4df462be1c495[] [http_utils.go:101 [] [main [] [ [] - Pinging access server at http://localhost:8046/access with 120 timeout minutes [startup]
2025-09-25T12:35:24.140Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [http_utils.go:182 [] [main [] [ [] - Cluster join: Pinging service registry on 'http://localhost:8046/access' [startup]
2025-09-25T12:35:24.140Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [http_utils.go:151 [] [main [] [ [] - Pinging service at http://localhost:8046/access/api/v1/system/ping [startup]
2025-09-25T12:35:24.224Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [http_utils.go:190 [] [main [] [ [] - Successfully pinged service registry [startup]
2025-09-25T12:35:24.230Z [jfcat[] [INFO [] [69d4df462be1c495[] [node_info.go:60 [] [main [] [ [] - Catalog Service ID: jfcatalog@<redacted> [startup]
2025-09-25T12:35:24.231Z [jfcat[] [INFO [] [69d4df462be1c495[] [node_info.go:61 [] [main [] [ [] - Catalog Node ID : xray-catalog-<redacted> [startup]
2025-09-25T12:35:24.231Z [jfcat[] [INFO [] [69d4df462be1c495[] [node_info.go:62 [] [main [] [ [] - Catalog Node IP : <redacted> [startup]
2025-09-25T12:35:24.231Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [client.go:84 [] [main [] [ [] - Setting up Db [single_tenant_client]
2025-09-25T12:35:24.275Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [client.go:92 [] [main [] [ [] - Successfully connected to dbx [single_tenant_client]
2025-09-25T12:35:24.275Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [client.go:95 [] [main [] [ [] - Setting dbx.SetConnMaxLifetime to 60 seconds [single_tenant_client]
2025-09-25T12:35:24.275Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [client.go:99 [] [main [] [ [] - Setting dbx.SetConnMaxIdleTime 60 seconds [single_tenant_client]
2025-09-25T12:35:24.275Z [jfcat[] [INFO [] [69d4df462be1c495[] [migrator.go:63 [] [main [] [ [] - Starting execute of blocking DB migrations [single_tenant_client]
2025/09/25 12:35:24 driver.Query: query=SELECT "blocking_migration_versions"."id", "blocking_migration_versions"."created_at", "blocking_migration_versions"."updated_at", "blocking_migration_versions"."migration_number", "blocking_migration_versions"."migration_script", "blocking_migration_versions"."status" FROM "blocking_migration_versions" ORDER BY "blocking_migration_versions"."migration_number" DESC LIMIT 1 args=[]
2025/09/25 12:35:24 driver.Tx(8213c6d8-80ee-4330-a0e8-de0f4e5d0f1d): started
2025-09-25T12:35:24.278Z [jfcat[] [DEBUG[] [69d4df462be1c495[] [migrator.go:129 [] [main [] [ [] - Executing migration: 0_migrations_tables.sql [single_tenant_client]
2025/09/25 12:35:24 Tx(8213c6d8-80ee-4330-a0e8-de0f4e5d0f1d).ExecContext: query=CREATE SCHEMA IF NOT EXISTS catalog;
CREATE TABLE IF NOT EXISTS catalog.blocking_migration_versions (
id SERIAL PRIMARY KEY,
created_at timestamptz NOT NULL DEFAULT NOW(),
updated_at timestamptz NOT NULL DEFAULT NOW(),
migration_number smallint NOT NULL,
migration_script text NOT NULL,
status character varying NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS blocking_migration_versions_migration_number_idx ON catalog.blocking_migration_versions USING btree (migration_number);
CREATE TABLE IF NOT EXISTS catalog.non_blocking_migration_versions (
id SERIAL PRIMARY KEY,
created_at timestamptz NOT NULL DEFAULT NOW(),
updated_at timestamptz NOT NULL DEFAULT NOW(),
migration_number smallint NOT NULL,
migration_script text NOT NULL,
status character varying NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS non_blocking_migration_versions_migration_number_idx ON catalog.non_blocking_migration_versions USING btree (migration_number); args=[]
2025/09/25 12:35:24 Tx(8213c6d8-80ee-4330-a0e8-de0f4e5d0f1d).Query: query=INSERT INTO "blocking_migration_versions" ("created_at", "updated_at", "migration_number", "migration_script", "status") VALUES ($1, $2, $3, $4, $5) RETURNING "id" args=[2025-09-25 12:35:24.125416187 +0000 UTC 2025-09-25 12:35:24.125416376 +0000 UTC 0 0_migrations_tables.sql Finished]
2025/09/25 12:35:24 Tx(8213c6d8-80ee-4330-a0e8-de0f4e5d0f1d): rollbacked
2025-09-25T12:35:24.315Z [catal[] [ERROR[] [ [] [catalog.go:55 [] [main ] - Failed starting application: ERROR: relation "blocking_migration_versions" does not exist (SQLSTATE 42P01)
Failed to mark migration as done: 0_migrations_tables.sql
-- /var/jenkins_home/workspace/build_catalog-build_release_1.18/src/jfrog.com/catalog/internal/migration/db/migrator.go:137 (func1)
failed running with transaction
-- /var/jenkins_home/workspace/build_catalog-build_release_1.18/src/jfrog.com/catalog/internal/app/database/client.go:52 (WithTx)
Failed executing transaction for migration: 0_migrations_tables.sql
-- /var/jenkins_home/workspace/build_catalog-build_release_1.18/src/jfrog.com/catalog/internal/migration/db/migrator.go:144 (executeScript)
Failed to run migration 0_migrations_tables.sql
-- /var/jenkins_home/workspace/build_catalog-build_release_1.18/src/jfrog.com/catalog/internal/migration/db/migrator.go:114 (executeMigrations)
Failed running blocking migration
-- /var/jenkins_home/workspace/build_catalog-build_release_1.18/src/jfrog.com/catalog/internal/migration/db/migrator.go:72 (ExecuteBlockingMigration)
failed to execute blocking migration
-- /var/jenkins_home/workspace/build_catalog-build_release_1.18/src/jfrog.com/catalog/internal/migration/db/migrator.go:44 (Migrate)
failed to migrate database
-- /var/jenkins_home/workspace/build_catalog-build_release_1.18/src/jfrog.com/catalog/internal/app/tenant/single_tenant_client.go:135 (initDatabaseClient)
failed to init database client
-- /var/jenkins_home/workspace/build_catalog-build_release_1.18/src/jfrog.com/catalog/internal/app/tenant/single_tenant_client.go:53 (OnStart)
failed to start tenant client
-- /var/jenkins_home/workspace/build_catalog-build_release_1.18/src/jfrog.com/catalog/internal/app/application.go:199 (startService)
2025-09-25T12:38:22.257Z [shell[] [INFO [] [[] [wrapper.sh:125 [] [main] - Terminate signal received, attempting a graceful shutdown of Service-catalog
2025-09-25T12:38:22.277Z [catal[] [INFO [] [ [] [catalog.go:49 [] [main ] - Received terminated signal, terminating...
2025-09-25T12:38:22.279Z [catal[] [DEBUG[] [ [] [load_config.go:253 [] [main ] - Watcher config changes stopped gracefully
2025-09-25T12:38:22.280Z [jfcat[] [INFO [] [ [] [application.go:162 [] [main [] [ [] - Catalog stopped [application]
2025-09-25T12:38:22.356Z [shell[] [INFO [] [[] [_processHelper.sh:20 [] [main] - Checking if process is alive (attempt #1 / 20)...
2025-09-25T12:38:22.456Z [shell[] [INFO [] [[] [_processHelper.sh:26 [] [main] - Process 205 is not running
find: '/opt/jfrog/catalog/var/log/catalog-error.*.log': No such file or directory
{"log_name": "catalog-error-service.log", "app": {"loglevel": "panic", "class": "wrapper", "datetime": "2025-09-25T12:38:22.517Z","service": "jfcatalog","message": "2025/09/25 12:35:24 driver.Query: query=SELECT "blocking_migration_versions"."id", "blocking_migration_versions"."created_at", "blocking_migration_versions"."updated_at", "blocking_migration_versions"."migration_number", "blocking_migration_versions"."migration_script", "blocking_migration_versions"."status" FROM "blocking_migration_versions" ORDER BY "blocking_migration_versions"."migration_number" DESC LIMIT 1 args=[]\n2025/09/25 12:35:24 driver.Tx(8213c6d8-80ee-4330-a0e8-de0f4e5d0f1d): started\n2025/09/25 12:35:24 Tx(8213c6d8-80ee-4330-a0e8-de0f4e5d0f1d).ExecContext: query=CREATE SCHEMA IF NOT EXISTS catalog;\n\nCREATE TABLE IF NOT EXISTS catalog.blocking_migration_versions (\n id SERIAL PRIMARY KEY,\n created_at timestamptz NOT NULL DEFAULT NOW(),\n updated_at timestamptz NOT NULL DEFAULT NOW(),\n migration_number smallint NOT NULL,\n migration_script text NOT NULL,\n status character varying NOT NULL\n);\n\nCREATE UNIQUE INDEX IF NOT EXISTS blocking_migration_versions_migration_number_idx ON catalog.blocking_migration_versions USING btree (migration_number);\n\nCREATE TABLE IF NOT EXISTS catalog.non_blocking_migration_versions (\n id SERIAL PRIMARY KEY,\n created_at timestamptz NOT NULL DEFAULT NOW(),\n updated_at timestamptz NOT NULL DEFAULT NOW(),\n migration_number smallint NOT NULL,\n migration_script text NOT NULL,\n status character varying NOT NULL\n);\n\nCREATE UNIQUE INDEX IF NOT EXISTS non_blocking_migration_versions_migration_number_idx ON catalog.non_blocking_migration_versions USING btree (migration_number); args=]\n2025/09/25 12:35:24 Tx(8213c6d8-80ee-4330-a0e8-de0f4e5d0f1d).Query: query=INSERT INTO "blocking_migration_versions" ("created_at", "updated_at", "migration_number", "migration_script", "status") VALUES ($1, $2, $3, $4, $5) RETURNING "id" args=[2025-09-25 12:35:24.125416187 +0000 UTC 2025-09-25 12:35:24.125416376 +0000 UTC 0 0_migrations_tables.sql Finished]\n2025/09/25 12:35:24 Tx(8213c6d8-80ee-4330-a0e8-de0f4e5d0f1d): rollbacked\n"}}
find: '/opt/jfrog/catalog/var/log/catalog-error.*.log': No such file or directory
2025-09-25T12:38:22.631Z [shell[] [INFO [] [[] [wrapper.sh:166 [] [main] - Waiting 10 seconds after service shutdown
Installed with the following values.yaml:
replicaCount: 1
waitForDatabase: true
unifiedUpgradeAllowed: true
quota:
enabled: false
common:
persistence:
enabled: false
autoscaling:
enabled: false
global:
masterKeySecretName: "${xray_master_key_secret_name}"
joinKeySecretName: "${artifactory_join_key_secret_name}"
jfrogUrl: "http://artifactory.artifactory.svc.cluster.local:8082"
xray:
podAntiAffinity:
type: "hard"
topologyKey: "kubernetes.io/hostname"
extraSystemYaml:
shared:
database:
maxOpenConns: 40
postgresql:
enabled: false
database:
type: postgresql
driver: org.postgresql.Driver
secrets:
user:
name: "${xray_db_connection_secret_name}"
key: "db-user"
password:
name: "${xray_db_connection_secret_name}"
key: "db-password"
url:
name: "${xray_db_connection_secret_name}"
key: "db-url"
actualUsername:
name: "${xray_db_connection_secret_name}"
key: "db-actualUsername"
analysis:
resources:
requests:
cpu: "100m"
memory: 250Mi
limits:
cpu: "500m"
memory: 1Gi
indexer:
resources:
requests:
cpu: "300m"
memory: 550Mi
limits:
cpu: "500m"
memory: 1Gi
persist:
resources:
requests:
cpu: "100m"
memory: 250Mi
limits:
cpu: "500m"
memory: 1Gi
server:
resources:
requests:
cpu: "300m"
memory: 500Mi
limits:
cpu: "500m"
memory: 1Gi
router:
serviceRegistry:
insecure: true
resources:
requests:
cpu: "60m"
memory: 100Mi
limits:
cpu: "200m"
memory: 400Mi
observability:
resources:
requests:
cpu: "10m"
memory: 25Mi
limits:
cpu: "100m"
memory: 250Mi
panoramic:
resources:
requests:
cpu: "100m"
memory: 250Mi
limits:
cpu: "200m"
memory: 500Mi
sbom:
resources:
requests:
cpu: "100m"
memory: 250Mi
limits:
cpu: "200m"
memory: 500Mi
policyenforcer:
resources:
requests:
memory: "250Mi"
cpu: "60m"
limits:
cpu: "200m"
memory: 800Mi
rabbitmq:
auth:
existingPasswordSecret: "${xray_rabbitmq_password_secret_name}"
persistence:
size: 10Gi
extraConfiguration: |-
vm_memory_high_watermark.absolute = 1G
{{- if not .Values.global.xray.rabbitmq.haQuorum.enabled }}
raft.wal_max_size_bytes = 1048576
{{- end }}
resources:
requests:
cpu: "50m"
memory: 300Mi
limits:
cpu: 1
memory: 1Gi
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- xray
topologyKey: kubernetes.io/hostname
catalog:
enabled: true
createCatalogDb:
enabled: false
extraSystemYaml:
shared:
application:
level: debug
database:
maxOpenConns: 40
database:
type: postgresql
driver: "pgx"
secrets:
user:
name: "${catalog_db_connection_secret_name}"
key: "db-user"
password:
name: "${catalog_db_connection_secret_name}"
key: "db-password"
url:
name: "${catalog_db_connection_secret_name}"
key: "db-url"
resources:
limits:
cpu: 600m
memory: 1Gi
requests:
cpu: 100m
memory: 500Mi
router:
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "500Mi"
cpu: "300m"
initContainers:
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128MiIs there anything that I am missing during configuration?