Cannot start mgmt hub due to failing css-api container #174
Description
When running the mgmt hub via deploy-mgmt-hub.sh, it fails with error code 5 due to a failing css-api container.
Looks like it is erroring because css-api fails to connect to mongo due to invalid connection string format.
Command:
export EXCHANGE_ROOT_PW=Horizon-Rul3s EXCHANGE_IMAGE_TAG=testing EXCHANGE_DATABASE=exchange; \
export EXCHANGE_HUB_ADMIN_PW=hubadminpw EXCHANGE_SYSTEM_ADMIN_PW=ibmadminpw; \
export AGBOT_ID=agbot AGBOT_TOKEN=Abcdefghijklmno1 AGBOT_IMAGE_TAG=e2edev; \
export CSS_IMAGE_TAG=e2edev CSS_INTERNAL_PORT=8080; \
export MONGO_IMAGE_TAG=4.0.6; \
export POSTGRES_IMAGE_TAG=13 POSTGRES_USER=admin; \
export VAULT_IMAGE_TAG=testing VAULT_PORT=8200 VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200 VAULT_ROOT_TOKEN=vault_dev_root_token_id; \
export ANAX_SOURCE=~/repos/forks/anax ANAX_LOG_LEVEL=5; \
export HZN_TRANSPORT=http;
mgmt-hub/deploy-mgmt-hub.sh
Script Output:
------- Checking input and initializing...
Management hub services will listen on http://127.0.0.1
Updating apt-get package index...
Installing prerequisites, this could take a minute...
Certificate /etc/horizon/keys/horizonMgmtHubFDO.crt already exists, so not receating it
------- Downloading template files...
Substituting environment variables into template files...
------- Downloading/starting Horizon management hub services...
Downloading management hub docker images...
Trying to pull openhorizon/amd64_agbot:e2edev ...
Pulling openhorizon/amd64_exchange-api:testing ...
Trying to pull openhorizon/amd64_cloud-sync-service:e2edev ...
Trying to pull postgres:13 ...
Trying to pull mongo:4.0.6 ...
Pulling openhorizon/fdo-owner-services:testing ...
Pulling openhorizon/amd64_vault:testing ...
Starting management hub containers...
[+] Running 15/15
✔ Network hzn_horizonnet Created 0.1s
✔ Volume "hzn_vaultvol" Created 0.0s
✔ Volume "hzn_agbotmsgkeyvol" Created 0.0s
✔ Volume "hzn_postgres" Created 0.0s
✔ Volume "hzn_mongovol" Created 0.0s
✔ Volume "hzn_fdo-ocs-db" Created 0.0s
✔ Volume "hzn_postgresvolfdo" Created 0.0s
✔ Container postgres-fdo-owner-service Started 0.4s
✔ Container postgres Started 0.3s
✔ Container mongo Started 0.4s
✔ Container exchange-api Started 0.1s
✔ Container css-api Started 0.1s
✔ Container vault Started 0.1s
✔ Container fdo-owner-services Started 0.1s
✔ Container agbot Started 0.1s
Waiting for the exchange....
------- Creating the user org, and the admin user in both orgs...
Creating exchange hub admin user, and the admin user and agbot in the system org...
------- Creating a Vault instance and performing all setup and configuration operations ...
Checking Vault service status, initialization, and seal...
A Vault instance has not been initialized. Initializing...
Vault instance is sealed. Unsealing...
Creating KV ver.2 secrets engine openhorizon...
Registering auth plugin openhorizon-exchange to Vault instance...
Generating SHA256 hash of openhorizon-exchange plugin...
Enabling auth method openhorizon-exchange for secrets engine openhorizon...
Configuring auth method openhorizon-exchange for use with the Exchange...
Success! Data written to: auth/openhorizon/config
------- Creating an agbot in the exchange...
[+] Restarting 1/1
✔ Container agbot Started 10.8s
Creating exchange user org and admin user...
------- Downloading/installing/configuring Horizon agent and CLI...
Downloading the Horizon agent and CLI packages...
Installing the Horizon agent and CLI packages...
Configuring the Horizon agent and CLI...
Publishing /tmp/horizon-all-in-1/agent-install.cfg in CSS as public object agent-install.cfg in the IBM org...
Digital sign with SHA256 will be performed for data integrity. It will delay the MMS object publish.
Start hashing the file...
Data hash is generated. Start digital signing with the data hash...
Digital sign finished.
Error: Encountered HTTP error: Put "http://127.0.0.1:9443/api/v1/objects/IBM/agent_files/agent-install.cfg": dial tcp 127.0.0.1:9443: connect: connection refused calling Model Management Service REST API PUT http://127.0.0.1:9443/api/v1/objects/IBM/agent_files/agent-install.cfg. HTTP status: .
Error: exit code 5 from: publishing /tmp/horizon-all-in-1/agent-install.cfg in CSS as a public object
Docker containers:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3e109afdec68 openhorizon/amd64_agbot:e2edev "/bin/sh -c /usr/hor…" 2 minutes ago Up 2 minutes (healthy) 127.0.0.1:3110->8080/tcp, 127.0.0.1:3111->8083/tcp agbot
e30806a9e099 openhorizon/fdo-owner-services:testing "/bin/sh -c $WORKDIR…" 2 minutes ago Up 2 minutes (healthy) 0.0.0.0:8042->8042/tcp, 0.0.0.0:9008->9008/tcp fdo-owner-services
91b08433718e openhorizon/amd64_vault:testing "entrypoint.sh server" 2 minutes ago Up 2 minutes (healthy) 127.0.0.1:8200->8200/tcp vault
57f192b2cacd openhorizon/amd64_cloud-sync-service:e2edev "/usr/edge-sync-serv…" 2 minutes ago Restarting (2) 48 seconds ago css-api
1fb73692c9d2 openhorizon/amd64_exchange-api:testing "/bin/sh -c '/usr/bi…" 2 minutes ago Up 2 minutes (healthy) 8083/tcp, 127.0.0.1:3090->8080/tcp exchange-api
10bbb39c6e89 postgres:13 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes (healthy) 0.0.0.0:5433->5432/tcp postgres-fdo-owner-service
d87d034c9bd4 mongo:4.0.6 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes (healthy) 27017/tcp mongo
09a9538b52c1 postgres:13 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes (healthy) 0.0.0.0:5432->5432/tcp postgres
css-api container logs:
* CSS: 2024/06/18 21:10:22 Loaded configuration:
NodeType CSS
DestinationType Cloud
DestinationID Cloud
OrgID
ListeningType unsecure
ListeningAddress
SecureListeningPort 8080
UnsecureListeningPort 8080
ServerCertificate <...>
ServerKey <...>
CSSOnWIoTP false
UsingEdgeConnector false
LeadershipTimeout 45
AuthenticationHandler dummy
ObjectQueueBufferSize 1000
VerifyQueueBufferSize 500
CommunicationProtocol http
MQTTClientID
MQTTUserName
MQTTPassword
MQTTUseSSL true
MQTTCACertificate <...>
MQTTSSLCert
MQTTSSLKey
MQTTAllowInvalidCertificates false
MQTTBrokerConnectTimeout 300
MQTTParallelMode
PersistenceRootPath /var/edge-sync-service/persist/
BrokerAddress
BrokerPort 0
HTTPPollingInterval 10
HTTPCSSHost
HTTPCSSPort 0
HTTPCSSUseSSL false
HTTPCSSCACertificate
HTTPESSClientTimeout 120
HTTPESSObjClientTimeout 600
HTTPCSSObjDownloadConcurrencyMultiplier 1
HTTPServerReadHeaderTimeout 60
LogLevel INFO
LogRootPath /var/edge-sync-service/log
LogTraceDestination stdout
LogFileName sync-service
TraceLevel INFO
TraceRootPath /var/edge-sync-service/trace
TraceFileName sync-service
LogTraceFileSizeKB 20000
MaxCompressedlLogTraceFilesNumber 50
LogTraceMaintenanceInterval 60
ResendInterval 10
ESSCallSPIRetryInterval 2
ESSSPIMaxRetry 1000
ESSPingInterval 1
RemoveESSRegistrationTime 30
EnableDataChunk true
MaxDataChunkSize 5242880
MaxInflightChunks 1
MongoAddressCsv mongo:27017
MongoAuthDbName admin
MongoDbName d_edge
MongoUsername
MongoPassword
MongoUseSSL false
MongoCACertificate
MongoAllowInvalidCertificates false
MongoSessionCacheSize 1
MongoSleepTimeBetweenRetry 4000
DatabaseConnectTimeout 300
StorageMaintenanceInterval 30
ObjectActivationInterval 30
StorageProvider mongo
ESSConsumedObjectsKept 1000
MessagingGroupCacheExpiration 60
ShutdownQuiesceTime 60
ObjectsDataPath
CSS: 2024/06/18 21:10:22 INFO: Horizon Authenticator starting with exchange authenticated identity
* CSS: 2024/06/18 21:10:22 INFO: CConnecting to mongo...
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 ERROR: Failed to dial mgo. Error: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 ERROR: Retrying to connect to mongo
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x1428f0a]
goroutine 1 [running]:
go.mongodb.org/mongo-driver/mongo.(*Client).Ping(0x26faec0?, {0x1b453c0?, 0xc00021e070?}, 0x62?)
/tmp/anax-gopath/pkg/mod/go.mongodb.org/[email protected]/mongo/client.go:361 +0x4a
github.com/open-horizon/edge-sync-service/core/storage.(*MongoStorage).Init(0xc0000aa000)
/tmp/anax-gopath/pkg/mod/github.com/open-horizon/[email protected]/core/storage/mongoStorage.go:204 +0x725
github.com/open-horizon/edge-sync-service/core/base.Start({0x0, 0x0}, 0xe5?)
/tmp/anax-gopath/pkg/mod/github.com/open-horizon/[email protected]/core/base/base.go:115 +0x673
github.com/open-horizon/edge-sync-service/core/base.StandaloneSyncService({0x1b38890?, 0xc000447f98})
/tmp/anax-gopath/pkg/mod/github.com/open-horizon/[email protected]/core/base/standalone.go:118 +0x5bc
main.main()
/tmp/anax-gopath/src/github.com/open-horizon/anax/css/cmd/cloud-sync-service/main.go:28 +0x2e