docker-ckan

This repository provides a complete setup for deploying CKAN using Docker Compose. It is based on the official Docker images for CKAN and the official Docker Compose setup for CKAN. CKAN is an open-source data management system that makes data accessible by providing tools to streamline publishing, sharing, finding, and using data.

The deployment is composed of several Docker services, including CKAN itself, PostgreSQL for database management, Solr for search functionality, Redis for caching, and Nginx for handling SSL traffic. Additionally, the deployment includes the DataPusher service for handling resource uploads.

We use Make as main entry point and command line interface to build, run and destroy docker-ckan instances.

Quick Start

git clone https://github.com/TIBHannover/docker-ckan
cd docker-ckan
cp .env.example .env

Create a docker-compose.override.yml with the following content:

services:
  ckan:
    ports:
      - 5000:5000

Run:

make build up

Open your browser at https://localhost:8443. Login as ckan_admin with password test1234 (see .env).

Key Features

Dockerized CKAN Setup

CKAN and its dependencies (PostgreSQL, Solr, Redis, Nginx) are fully containerized using Docker Compose, making it easy to spin up and manage the entire stack.

Custom Initialization and Configuration

• The setup includes custom entrypoint scripts that handle initialization tasks, such as:

  • Setting up plugins, including the CKAN datapusher plugin.

  • Automatically generating and configuring security tokens (e.g., JWT, session secrets).

  • Configuring system settings through environment variables and initialization scripts.

• The initialization process uses Python-based scripts located in the ckan/setup/ folder to manage essential CKAN setup tasks.

Modular Service Architecture

• The repository is divided into distinct contexts for each service:

  • nginx/: Configures Nginx to act as a reverse proxy for CKAN, handling SSL termination.

  • ckan/: Builds CKAN from the ckan-base image, installs extensions, and applies patches. It also includes custom scripts in docker-entrypoint.d/ to handle plugin setup (like DataPusher).

  • postgresql/: Sets up a PostgreSQL instance with a main CKAN database and a DataStore database.

  • solr/: Provides a Solr instance to power CKAN’s search functionality.

  • redis/: Configures Redis for caching CKAN’s data.

Makefile for Easy Management

A Makefile is provided to streamline the management of Docker Compose commands. Some key targets include:

• make build: Build the containers. Required after changing Dockerfiles or files added to the containers.

• make up: Starts the CKAN stack in the background.

• make down: Stops and removes all containers, networks, and volumes (keeping data).

• make show-logs: Shows logs from all services in real-time.

• make destroy: Completely tears down the environment, removing all associated volumes (including persistent data).

• make bash: Bash into the running CKAN container.

Patches and Customizations

The ckan/ context includes support for applying patches to CKAN core and installed extensions. These patches are applied automatically during the build process, allowing for easy customization and updates.

Environment-Driven Configuration

The configuration of services is driven by environment variables defined in the .env file. This allows for flexible configuration of:

• Service ports, database credentials, plugin activation, and more.

• CKAN’s core functionality, such as enabling plugins or configuring API tokens for services like DataPusher.

See .env.example for a list of available environment variables.

CKAN environment configuration settings are controlled through YAML files in the configuration repository and are applied as Ansible variables. These variables can be defined at different levels (layers), allowing flexible overrides depending on scope.

Configuration Layers

Configuration values can be set at multiple levels:

• Project level (recommended for most changes)

• Instance level (alternative, depending on deployment needs)

Modifying Environment Configuration

To change an existing CKAN configuration:

1. Locate the appropriate project-level YAML file.

2. Add or modify the required configuration key.

3. Ensure the change is committed and redeployed.

4. Adding Missing Configuration

If a configuration option is not already defined:

Add it within the Docker image configuration structure:

docker_image/ box/ config/

Define the setting in the configuration repository as an Ansible variable.

Assign the value at the desired layer (e.g., project level).

Example

To enable user creation via the web interface:

ckan.auth.create_user_via_web: true

This setting was added/modified in the project-level YAML file and applied during deployment.

ckan-config/ inventory/ group_vars/ project<name>.yml

Notes

• For major changes, Checking the docker container should always be the first-step

• Configuration values are managed as Ansible variables, enabling layered overrides.

• Project-level settings take precedence in most use cases.

• Instance-level overrides are also supported if needed.

Folder Structure

• nginx/: Contains the Nginx Dockerfile and configuration files for serving CKAN over HTTPS.

• ckan/: Includes the CKAN Dockerfile, custom entrypoint scripts (in docker-entrypoint.d), patches, and setup scripts.

• postgresql/: Contains the Docker setup for PostgreSQL, including database initialization.

• setup/: Houses the initialization scripts that handle database setup, plugin configuration, Solr checks, and sysadmin creation.

Folder Structure

• nginx/: Contains the Nginx Dockerfile and configuration files for serving CKAN over HTTPS.

• ckan/: Includes the CKAN Dockerfile, custom entrypoint scripts (in docker-entrypoint.d), patches, and setup scripts.

• postgresql/: Contains the Docker setup for PostgreSQL, including database initialization.

• setup/: Houses the initialization scripts that handle database setup, plugin configuration, Solr checks, and sysadmin creation.

Developing

In the following section, we show how changes can be made to the CKAN extension "ckanext-organisation-group" and how these are immediately visible in the running containers. The customisation of CSS classes in a template serves as an example. Before we can make the actual changes, we need to make some preparations:

1) Clone the extension code next to CKAN Docker (docker-ckan)

To be able to make changes independently of CKAN, it makes sense to check out the extension alongside CKAN Docker. This makes it easy to commit changes to the extension without having to change CKAN Docker.

A typical directory might look like this:

drwxrwxrwx  6 alex  502 4096 Feb 17 18:17 ckanext-organization-group
drwxr-xr-x  5 alex alex 4096 Jan 20 08:45 ckanext-system-stats
drwxr-xr-x  5 alex alex 4096 Jan 23 10:52 ckanext-tif-imageview
drwxr-xr-x  5 alex alex 4096 Jan 20 16:46 ckanext-user-manual
drwxr-xr-x  7 alex alex 4096 Feb 17 16:05 docker-ckan

2) Add a volume to docker-compose.override.yaml

Now we want to replace the code of the extension included in the container with the checked-out code. To do this, we use a so-called "volume" which is specified in docker-compose.override.yaml. The part before the colon (~/_dev/github.com/TIBHannover/ckanext-organization-group) in the example must be replaced by the abosulte path in the local environment.

services:
  ckan:
    ports:
      - 5000:5000
    volumes:
      - ~/_dev/github.com/TIBHannover/ckanext-organization-group:/srv/app/src/ckanext-organization-group

3) Add a temporary startup script for the CKAN container

Add a startup script ckan/docker-entrypoint.d/99_dev.sh with the following content. This script ensures that the correct file permissions and ownership are set, and that the installation is complete:

#!/bin/bash

chgrp -R ckan-sys /srv/app/src
find /srv/app/src -type d -exec chmod 777 {} \;
pip install -e /srv/app/src/ckanext-organization-group
pip install -r /srv/app/src/ckanext-organization-group/requirements.txt

4) Build and run the conainers

Run make build bash to build and run the containers including the volume and the startup script. All changes to the extension code are now "mirrored" directly in the container and are generally visible immediately.