PMM3 docker image fails to start with "/srv/postgres14/postgresql.conf": No such file or directory

[agiledivops]

Description

Following the direction provided trying to setup PMM3 server using docker-compose

version: '3.2'
services:
  pmm-server:
    image: percona/pmm-server:3
    container_name: pmm-server
    restart: always
    ports:
      - "9000:8443"
    volumes:
      - ./data/pmm/srv:/srv
    networks:
      - mynetwork

• Host OS: macos 14.7.1,
• Hardware: M1
• Rancher Desktop 1.14.1

Expected Results

Accessing https://localhost:9000/graph/ should land on login page or dashboard

Actual Results

Accessing https://localhost:9000/graph/ gives: 500 Internal Server Error

Version

PMM server v3.x

Steps to reproduce

No response

Relevant logs

suvpervisord.log
================
2025-02-04 06:41:18,938 INFO RPC interface 'supervisor' initialized
2025-02-04 06:41:18,940 INFO supervisord started with pid 1
2025-02-04 06:41:19,958 INFO spawned: 'pmm-init' with pid 15
2025-02-04 06:41:19,968 INFO spawned: 'postgresql' with pid 17
2025-02-04 06:41:19,979 INFO spawned: 'clickhouse' with pid 19
2025-02-04 06:41:19,986 INFO spawned: 'grafana' with pid 21
2025-02-04 06:41:19,999 INFO spawned: 'nginx' with pid 23
2025-02-04 06:41:20,046 INFO spawned: 'victoriametrics' with pid 26
2025-02-04 06:41:20,084 INFO spawned: 'vmalert' with pid 30
2025-02-04 06:41:20,125 INFO spawned: 'vmproxy' with pid 34
2025-02-04 06:41:20,149 INFO spawned: 'qan-api2' with pid 36
2025-02-04 06:41:20,201 INFO spawned: 'pmm-managed' with pid 38
2025-02-04 06:41:21,143 INFO success: pmm-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:21,153 INFO success: postgresql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:21,165 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:21,171 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:21,185 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:21,202 INFO success: victoriametrics entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:21,212 INFO success: vmalert entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:21,219 INFO success: vmproxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:21,219 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:21,220 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:21,356 INFO exited: postgresql (exit status 2; not expected)
2025-02-04 06:41:21,482 INFO spawned: 'postgresql' with pid 67
2025-02-04 06:41:22,029 INFO exited: qan-api2 (exit status 1; not expected)
2025-02-04 06:41:22,146 INFO spawned: 'qan-api2' with pid 77
2025-02-04 06:41:22,172 INFO exited: postgresql (exit status 2; not expected)
2025-02-04 06:41:23,308 INFO spawned: 'postgresql' with pid 83
2025-02-04 06:41:23,338 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:23,358 INFO exited: qan-api2 (exit status 1; not expected)
2025-02-04 06:41:23,373 INFO spawned: 'qan-api2' with pid 85
2025-02-04 06:41:23,975 INFO exited: postgresql (exit status 2; not expected)
2025-02-04 06:41:24,438 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:24,603 INFO exited: qan-api2 (exit status 1; not expected)
2025-02-04 06:41:24,965 INFO spawned: 'qan-api2' with pid 99
2025-02-04 06:41:26,168 INFO spawned: 'postgresql' with pid 106
2025-02-04 06:41:26,181 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-04 06:41:26,439 INFO exited: qan-api2 (exit status 1; not expected)
.....


clickhouse-server.log
====================
2025.02.04 06:41:40.627408 [ 95 ] {} <Error> CertificateReloader: Cannot obtain modification time for certificate file /etc/clickhouse-server/server.crt, skipping update. errno: 2, strerror: No such file or directory
2025.02.04 06:41:40.630427 [ 95 ] {} <Error> CertificateReloader: Cannot obtain modification time for key file /etc/clickhouse-server/server.key, skipping update. errno: 2, strerror: No such file or directory
2025.02.04 06:41:40.741125 [ 95 ] {} <Error> CertificateReloader: Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-server/server.key: error:02000002:system library:OPENSSL_internal:No such file or directory (version 23.8.2.7 (official build))
2025.02.04 06:41:40.840652 [ 95 ] {} <Information> Application: Listening for http://[::1]:8123
2025.02.04 06:41:40.840967 [ 95 ] {} <Information> Application: Listening for native protocol (tcp): [::1]:9000
2025.02.04 06:41:40.841162 [ 95 ] {} <Information> Application: Listening for MySQL compatibility protocol: [::1]:9004
2025.02.04 06:41:40.841390 [ 95 ] {} <Information> Application: Listening for PostgreSQL compatibility protocol: [::1]:9005
2025.02.04 06:41:40.842000 [ 95 ] {} <Information> Application: Listening for Prometheus: http://[::1]:9363
2025.02.04 06:41:40.846399 [ 95 ] {} <Information> Application: Listening for http://127.0.0.1:8123
2025.02.04 06:41:40.847875 [ 95 ] {} <Information> Application: Listening for native protocol (tcp): 127.0.0.1:9000
2025.02.04 06:41:40.848223 [ 95 ] {} <Information> Application: Listening for MySQL compatibility protocol: 127.0.0.1:9004
2025.02.04 06:41:40.848488 [ 95 ] {} <Information> Application: Listening for PostgreSQL compatibility protocol: 127.0.0.1:9005
2025.02.04 06:41:40.848713 [ 95 ] {} <Information> Application: Listening for Prometheus: http://127.0.0.1:9363
2025.02.04 06:41:40.849361 [ 95 ] {} <Information> Application: Ready for connections.
2025.02.04 06:41:42.069546 [ 105 ] {} <Error> ServerErrorHandler: Code: 81. DB::Exception: Database pmm does not exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c604bf7 in /usr/bin/clickhouse
1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000711518d in /usr/bin/clickhouse
2. DB::DatabaseCatalog::assertDatabaseExists(String const&) const @ 0x00000000116cb88f in /usr/bin/clickhouse
3. DB::Context::setCurrentDatabase(String const&) @ 0x0000000011610c31 in /usr/bin/clickhouse
4. DB::TCPHandler::runImpl() @ 0x00000000131111c9 in /usr/bin/clickhouse
5. DB::TCPHandler::run() @ 0x000000001311e839 in /usr/bin/clickhouse
6. Poco::Net::TCPServerConnection::start() @ 0x0000000015b104d4 in /usr/bin/clickhouse
7. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b116d1 in /usr/bin/clickhouse
8. Poco::PooledThread::run() @ 0x0000000015c47f07 in /usr/bin/clickhouse
9. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c461dc in /usr/bin/clickhouse
10. ? @ 0x00002aaaab383d32 in ?
11. ? @ 0x00002aaaab4080b4 in ?
 (version 23.8.2.7 (official build))
...

grafana.log
============
logger=settings t=2025-02-04T06:41:26.148823519Z level=info msg="Starting Grafana" version=11.1.8 commit=11.1.8 branch=main compiled=2024-10-24T00:00:00Z
logger=settings t=2025-02-04T06:41:26.23714806Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2025-02-04T06:41:26.237343894Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2025-02-04T06:41:26.237406644Z level=info msg=Target target=[all]
logger=settings t=2025-02-04T06:41:26.23754856Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2025-02-04T06:41:26.237593394Z level=info msg="Path Data" path=/srv/grafana
logger=settings t=2025-02-04T06:41:26.237667602Z level=info msg="Path Logs" path=/srv/logs
logger=settings t=2025-02-04T06:41:26.237715852Z level=info msg="Path Plugins" path=/srv/grafana/plugins
logger=settings t=2025-02-04T06:41:26.237756394Z level=info msg="Path Provisioning" path=/usr/share/grafana/conf/provisioning
logger=settings t=2025-02-04T06:41:26.237806269Z level=info msg="App mode production"
logger=featuremgmt t=2025-02-04T06:41:26.266132519Z level=info msg=FeatureToggles cloudWatchNewLabelParsing=true betterPageScrolling=true publicDashboards=true lokiQueryHints=true annotationPermissionUpdate=true lokiQuerySplitting=true alertingNoDataErrorExecution=true panelTitleSearch=true logRowsPopoverMenu=true promQueryBuilder=true transformationsRedesign=true explore2Dashboard=true prometheusAzureOverrideAudience=true ssoSettingsApi=true kubernetesPlaylists=true topnav=true influxdbBackendMigration=true exploreMetrics=true savedItems=true logsInfiniteScrolling=true managedPluginsInstall=true correlations=true dashgpt=true lokiStructuredMetadata=true panelMonitoring=true recoveryThreshold=true logsExploreTableVisualisation=true nestedFolders=true prometheusDataplane=true awsDatasourcesNewFormStyling=true cloudWatchCrossAccountQuerying=true cloudWatchDynamicLabels=true alertingSimplifiedRouting=true exploreContentOutline=true logsContextDatasourceUi=true prometheusConfigOverhaulAuth=true lokiQueryBuilder=true awsAsyncQueryCaching=true commandPalette=true recordedQueriesMulti=true dataplaneFrontendFallback=true lokiMetricDataplane=true alertingInsights=true prometheusMetricEncyclopedia=true
logger=sqlstore t=2025-02-04T06:41:26.27412256Z level=info msg="Connecting to DB" dbtype=postgres
Error: ✗ dial tcp [::1]:5432: connect: connection refused
....

nginx.log
=========
2025/02/04 08:33:09 [alert] 24#24: unlink() "/run/nginx.pid" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (2: No such file or directory)
2025/02/04 08:33:35 [warn] 25#25: "ssl_stapling" ignored, issuer certificate not found for certificate "/srv/nginx/certificate.crt"
2025/02/04 08:33:35 [emerg] 69#69: io_setup() failed (38: Function not implemented)
2025/02/04 08:33:35 [emerg] 71#71: io_setup() failed (38: Function not implemented)
....

pmm-init.log
============
TASK [initialization : Get current PMM version (aka 'update from')] ************
[WARNING]: Unable to find '/srv/grafana/PERCONA_DASHBOARDS_VERSION' in expected
paths (use -vvvvv to see paths)
ok: [127.0.0.1]

....

pmm-managed.log
===============
time="2025-02-04T06:41:24.087+00:00" level=info msg="Log level: info."
time="2025-02-04T06:41:24.116+00:00" level=info msg="Using default metrics config" component=config
time="2025-02-04T06:41:24.218+00:00" level=info msg="Using default Percona Platform address \"https://check.percona.com\"."
time="2025-02-04T06:41:24.233+00:00" level=info msg="Migrating database..." component=migration
time="2025-02-04T06:41:24.275+00:00" level=warning msg="Failed to migrate database: dial tcp 127.0.0.1:5432: connect: connection refused." component=migration
time="2025-02-04T06:41:25.276+00:00" level=info msg="Migrating database..." component=migration

postgresql14.log
================
postgres: could not access the server configuration file "/srv/postgres14/postgresql.conf": No such file or directory
postgres: could not access the server configuration file "/srv/postgres14/postgresql.conf": No such file or directory
postgres: could not access the server configuration file "/srv/postgres14/postgresql.conf": No such file or directory
....


qan-api2.log
==============
stdlog: qan-api2 v3.0.0.
time="2025-02-04T06:38:53.378+00:00" level=info msg="Log level: info."
time="2025-02-04T06:38:53.390+00:00" level=info msg="DSN: clickhouse://127.0.0.1:9000/pmm" component=main
stdlog: qan-api2 v3.0.0.
.....

victoriametrics.log
===================
pm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:12	build version: victoria-metrics-20250106-062722-pmm-6401-v1.93.4
2025-02-04T06:41:22.978Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:13	command-line flags
2025-02-04T06:41:22.978Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -envflag.enable="true"
2025-02-04T06:41:22.978Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -envflag.prefix="VM_"
2025-02-04T06:41:22.978Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -http.pathPrefix="/prometheus"
2025-02-04T06:41:22.978Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -httpListenAddr="127.0.0.1:9090"
2025-02-04T06:41:22.979Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -promscrape.config="/etc/victoriametrics-promscrape.yml"
2025-02-04T06:41:22.979Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -promscrape.streamParse="true"
2025-02-04T06:41:22.979Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -retentionPeriod="30d"
2025-02-04T06:41:22.979Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -search.disableCache="true"
2025-02-04T06:41:22.979Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -search.latencyOffset="5s"
2025-02-04T06:41:22.979Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -search.logSlowQueryDuration="30s"
2025-02-04T06:41:22.985Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -search.maxQueryDuration="1m30s"
2025-02-04T06:41:22.986Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -search.maxQueryLen="1MB"
2025-02-04T06:41:22.986Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -search.maxQueueDuration="30s"
2025-02-04T06:41:22.986Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -search.maxSamplesPerQuery="1500000000"
2025-02-04T06:41:22.986Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -search.maxUniqueTimeseries="100000000"
2025-02-04T06:41:22.986Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/logger/flag.go:20	  -storageDataPath="/srv/victoriametrics/data"
2025-02-04T06:41:22.986Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/app/victoria-metrics/main.go:78	starting VictoriaMetrics at "127.0.0.1:9090"...
2025-02-04T06:41:22.986Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/app/vmstorage/main.go:109	opening storage at "/srv/victoriametrics/data" with -retentionPeriod=30d
2025-02-04T06:41:23.042Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/memory/memory.go:42	limiting caches to 2464343654 bytes, leaving 1642895770 bytes to the OS according to -memory.allowedPercent=60
2025-02-04T06:41:23.679Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/app/vmstorage/main.go:123	successfully opened storage "/srv/victoriametrics/data" in 0.692 seconds; partsCount: 0; blocksCount: 0; rowsCount: 0; sizeBytes: 0
2025-02-04T06:41:23.702Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/app/vmselect/promql/rollup_result_cache.go:114	loading rollupResult cache from "/srv/victoriametrics/data/cache/rollupResult"...
2025-02-04T06:41:23.752Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/app/vmselect/promql/rollup_result_cache.go:142	loaded rollupResult cache from "/srv/victoriametrics/data/cache/rollupResult" in 0.047 seconds; entriesCount: 0, sizeBytes: 0
2025-02-04T06:41:23.763Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/app/victoria-metrics/main.go:91	started VictoriaMetrics in 0.776 seconds
2025-02-04T06:41:23.764Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/httpserver/httpserver.go:96	starting http server at http://127.0.0.1:9090/
2025-02-04T06:41:23.764Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/httpserver/httpserver.go:97	pprof handlers are exposed at http://127.0.0.1:9090/debug/pprof/
2025-02-04T06:41:23.785Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scraper.go:113	reading Prometheus configs from "/etc/victoriametrics-promscrape.yml"
2025-02-04T06:41:23.988Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/config.go:132	starting service discovery routines...
2025-02-04T06:41:23.994Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/config.go:138	started service discovery routines in 0.006 seconds
2025-02-04T06:41:24.033Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scraper.go:430	static_configs: added targets: 6, removed targets: 0; total targets: 6
2025-02-04T06:41:24.994Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/storage/partition.go:223	creating a partition "2025_02" with smallPartsPath="/srv/victoriametrics/data/data/small/2025_02", bigPartsPath="/srv/victoriametrics/data/data/big/2025_02"
2025-02-04T06:41:25.290Z	info	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/storage/partition.go:232	partition "2025_02" has been created
2025-02-04T06:41:26.266Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:9933/debug/metrics" ({instance="pmm-server",job="qan-api2"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:9933/debug/metrics": Get "http://127.0.0.1:9933/debug/metrics": dial tcp4 127.0.0.1:9933: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2025-02-04T06:41:28.206Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:7773/debug/metrics" ({instance="pmm-server",job="pmm-managed"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:7773/debug/metrics": Get "http://127.0.0.1:7773/debug/metrics": dial tcp4 127.0.0.1:7773: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2025-02-04T06:41:29.421Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:3000/metrics" ({instance="pmm-server",job="grafana"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:3000/metrics": Get "http://127.0.0.1:3000/metrics": dial tcp4 127.0.0.1:3000: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2025-02-04T06:41:32.321Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:9363/metrics" ({instance="pmm-server",job="clickhouse"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:9363/metrics": Get "http://127.0.0.1:9363/metrics": dial tcp4 127.0.0.1:9363: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2025-02-04T06:41:36.216Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:9933/debug/metrics" ({instance="pmm-server",job="qan-api2"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:9933/debug/metrics": Get "http://127.0.0.1:9933/debug/metrics": dial tcp4 127.0.0.1:9933: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2025-02-04T06:41:38.166Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:7773/debug/metrics" ({instance="pmm-server",job="pmm-managed"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:7773/debug/metrics": Get "http://127.0.0.1:7773/debug/metrics": dial tcp4 127.0.0.1:7773: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2025-02-04T06:41:39.421Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:3000/metrics" ({instance="pmm-server",job="grafana"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:3000/metrics": Get "http://127.0.0.1:3000/metrics": dial tcp4 127.0.0.1:3000: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2025-02-04T06:41:42.361Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:9363/metrics" ({instance="pmm-server",job="clickhouse"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:9363/metrics": Get "http://127.0.0.1:9363/metrics": read tcp4 127.0.0.1:33294->127.0.0.1:9363: read: connection reset by peer
2025-02-04T06:41:46.216Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:9933/debug/metrics" ({instance="pmm-server",job="qan-api2"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:9933/debug/metrics": Get "http://127.0.0.1:9933/debug/metrics": dial tcp4 127.0.0.1:9933: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2025-02-04T06:41:48.164Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:7773/debug/metrics" ({instance="pmm-server",job="pmm-managed"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:7773/debug/metrics": Get "http://127.0.0.1:7773/debug/metrics": dial tcp4 127.0.0.1:7773: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2025-02-04T06:41:49.422Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/promscrape/scrapework.go:387	cannot scrape target "http://127.0.0.1:3000/metrics" ({instance="pmm-server",job="grafana"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://127.0.0.1:3000/metrics": Get "http://127.0.0.1:3000/metrics": dial tcp4 127.0.0.1:3000: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2025

Code of Conduct

• I agree to follow Percona Community Code of Conduct

[BupycHuk]

Did you have any data in /srv directory before starting it? Did you migrate from PMM 2 to PMM 3?

[agiledivops]

@BupycHuk thanks , No I am not migrating from PMM2 and I have /srv directory empty and tried with empty directory several times. I also noticed that its expecting volume path to be absolute path and if I manually try to initdb

/usr/pgsql-14/bin/initdb -D /srv/postgres14 --auth=trust --username=postgres

I see following error from within the container:

The files belonging to this database system will be owned by user "pmm".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /srv/postgres14 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 20
selecting default shared_buffers ... 400kB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... 2025-02-04 20:42:21.532 UTC [9946] FATAL: data directory "/srv/postgres14" has wrong ownership
2025-02-04 20:42:21.532 UTC [9946] HINT: The server must be started by the user that owns the data directory.
child process exited with exit code 1
initdb: removing contents of data directory "/srv/postgres14"

[GenLN]

Same issue here, permissions should be fixed by running chmod -R 750 /srv/postgres14 or chmod -R 700 /srv/postgres14

We deployed clean pmm server 3.0 via helm chart and we always need to repair those permissions manually by running chmod commands and using supervisorctl start postgresql

Can you update image and adjust those permissions?
Each pod recreate, pmm server won't start beacuse of this permission issue

[agiledivops]

Unfortunately it did not work for me so I deployed grafana with desired plugins that was very quick

[326068345]

I run pmm in k8s and I encounter the same problem. I can't start pmm again after restarting the container.

Cause:
[pmm@pmm-0]$ cat /srv/logs/postgresql14.log
2025-04-02 11:19:16.703 UTC [598] FATAL: data directory "/srv/postgres14" has invalid permissions
2025-04-02 11:19:16.703 UTC [598] DETAIL: Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).
2025-04-02 11:19:27.164 UTC [611] FATAL: data directory "/srv/postgres14" has invalid permissions
2025-04-02 11:19:27.164 UTC [611] DETAIL: Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).

fix:
[pmm@pmm-0]$ chmod -R 700 /srv/postgres14

Then restart the container.