Skip to content

Latest commit

 

History

History
237 lines (167 loc) · 6.83 KB

guides-external-database.md

File metadata and controls

237 lines (167 loc) · 6.83 KB

External database setup

This guide explains how to set up external databases for CLP instead of using the Docker Compose managed databases. If the host(s) on which you're running CLP are ephemeral, you should use external databases for metadata storage, and object storage for CLP's archives and streams; this will ensure data is persisted even if a host is replaced.

:::{warning} The CLP Docker Compose project includes MariaDB/MongoDB databases by default. This guide is only for users who want to customize their deployment by using their own database servers or cloud-managed databases (e.g., AWS RDS, Azure Database). :::

CLP requires two databases:

  • MariaDB/MySQL - for storing metadata about archives, files, and jobs.
  • MongoDB - for caching query results.

MariaDB/MySQL setup

CLP is compatible with any MariaDB or MySQL database. The instructions below use Ubuntu as an example, but you can use any compatible database installation or cloud-managed service.

Installing MariaDB on Ubuntu

  1. Install MariaDB server:

    sudo apt update
sudo apt install mariadb-server

  2. Connect to MariaDB as root:

    sudo mysql

  3. Create the CLP database:

    CREATE DATABASE `clp-db`;

  4. Create a user for CLP (replace <password> with a secure password):

    CREATE USER 'clp-user'@'%' IDENTIFIED BY '<password>';

    :::{note} The '%' allows connections from any host. For better security, replace '%' with the specific hostname or IP address from which CLP will connect (e.g., 'clp-user'@'192.168.1.10'). :::

  5. Grant privileges to the user:

    GRANT ALL PRIVILEGES ON `clp-db`.* TO 'clp-user'@'%';
FLUSH PRIVILEGES;

  6. Exit the MariaDB shell:

    EXIT;

Configuring MariaDB for remote connections

If CLP components will connect from a different host, you need to configure MariaDB to accept remote connections:

  1. Edit the MariaDB configuration file:

    sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

  2. Find the bind-address line and change it to allow connections from all interfaces:

    bind-address = 0.0.0.0

  3. Restart MariaDB:

    sudo systemctl restart mariadb

Verifying the MariaDB connection

You can verify the MariaDB connection by running:

mysql -h <mariadb-hostname-or-ip> -u clp-user -p clp-db

Using AWS RDS for MariaDB/MySQL

When using AWS RDS:

  1. Create a MariaDB or MySQL RDS instance in the AWS Console.

  2. Note the endpoint hostname and port (the default is 3306).

  3. Create the database and user using a MySQL client:

    mysql -h <rds-endpoint> -u admin -p

    Then follow steps 2-5 from Installing MariaDB on Ubuntu.

  4. Ensure the RDS security group allows inbound connections on port 3306 from your CLP hosts.

MongoDB setup

CLP is compatible with any MongoDB database. For installation instructions, see the MongoDB installation documentation.

:::{warning} Running an external MongoDB on the same host as CLP (i.e., using localhost or 127.0.0.1 as the results_cache host) is not supported. CLP's results-cache-indices-creator initializes a MongoDB replica set using the configured hostname, which MongoDB must be able to resolve to itself; localhost from inside a Docker container does not resolve to the host machine.

Instead, either:

  • Keep results_cache in the bundled list (recommended for single-host deployments).
  • Use a truly remote MongoDB instance and specify its hostname or IP.
  • If you must use a same-host MongoDB, configure results_cache.host in clp-config.yaml to the host's non-loopback IP address (e.g., 192.168.1.10) and ensure MongoDB is bound to that address. :::

Creating the CLP database in MongoDB

MongoDB automatically creates databases and collections when first accessed, so no manual database creation is needed. CLP will create the necessary database and collections (clp-query-results by default) when it first connects.

Configuring MongoDB for remote connections

If CLP components will connect from a different host:

  1. Edit the MongoDB configuration file:

    sudo nano /etc/mongod.conf

  2. Find the net.bindIp setting and change it to allow connections from all interfaces:

    net:
  port: 27017
  bindIp: 0.0.0.0

  3. Restart MongoDB:

    sudo systemctl restart mongod

:::{warning} For production deployments, it's highly recommended to enable authentication and SSL/TLS for MongoDB. See the MongoDB security documentation for details. :::

Verifying the MongoDB connection

You can verify the MongoDB connection by running:

mongosh "mongodb://<mongodb-hostname-or-ip>:27017/clp-query-results"

Using AWS DocumentDB or MongoDB Atlas

When using AWS DocumentDB or MongoDB Atlas:

  1. Create a cluster in the AWS Console or MongoDB Atlas.
  2. Note the connection string/endpoint provided.
  3. Ensure the security group or IP access list allows connections from your CLP hosts.
  4. Use the provided connection string when configuring CLP (see below).

Configuring CLP to use external databases

After setting up your external databases, configure CLP to use them:

  1. Edit etc/clp-config.yaml to specify which services are bundled (managed by the clp-package Docker Compose project):

    # Remove "database" and "results_cache" from this list to use external instances
bundled:
  # - "database"
  - "queue"
  - "redis"
  # - "results_cache"

  2. Configure the connection details for your external databases in etc/clp-config.yaml:

    database:
  host: "<mariadb-hostname-or-ip>"
  port: <mariadb-port>

results_cache:
  host: "<mongodb-hostname-or-ip>"
  port: <mongodb-port>

  3. Set the credentials in etc/credentials.yaml:

    database:
  username: "clp-user"
  password: "<your-mariadb-password>"

:::{note} When using external databases in a multi-host deployment, you do not need to start the database and results-cache Docker Compose services. Skip those services when following the multi-host deployment guide. However, you still need to run the database initialization jobs (db-table-creator and results-cache-indices-creator). :::