Auto migration after app update

[jslocomotor]

Is your feature request related to a problem? Please describe.
We use an automated deployment for our ERPNext instances. When modifying custom images, i.e. updating apps, it is best practice to run bench migrate in order to update the database schema or DocTypes.
Most instructions show executing the command from docker host or backend container, which interferes with automated deployment.

Describe the solution you'd like
Find a good way to automate migration. Our current approach:
Adding a "migrator" container to the stack:

migrator:
  <<: *backend_defaults
  entrypoint:
    - bash
    - -c

  command:
    - >
      if [ -d "sites/$$SITE_NAME" ]; then
        bench --site $$SITE_NAME migrate;
      else
        echo "Site $$SITE_NAME does not exist yet, skipping migrate.";
      fi

  depends_on:
    configurator:
      condition: service_completed_successfully

    db:
      condition: service_healthy

    redis-cache:
      condition: service_started

    redis-queue:
      condition: service_started

  restart: on-failure

This will run once after startup or update, check if the defined site exists and then run migration.

Feel free to point out other/ better solutions or problems with the above approach.

[DanielRadlAMR]

The migrator should only depend on the configurator.

One concern: with this setup, migrator will run on every docker compose up. That’s usually what you want during upgrades, but some users may prefer to trigger migrations manually. We should make this behavior opt‑out/opt‑in – for example by gating it behind an AUTO_MIGRATE_ON_START env variable or moving the service into a separate compose.auto-migrate.yaml that people include only when they want automatic migrations.

[jslocomotor]

You mean because the configurator itself depends on those three containers? If it has completed successfully, the migrator should be safe to run as well?

Yes, it will run every time the stack starts, but in production that shouldn’t happen too often, right? And even then, what do you really lose compared to the safety you gain?

I do like the idea of a separate compose file and an env-based switch.

[Rocket-Quack]

I think an optional compose override would be the best solution, so it would be up to users to decide when the migrations run and how

[DanielRadlAMR]

So a separate overrider it is.

And yeah, I did mean that the configurator already depends on all these services, so there’s not really a point in depending on them again, in my opinion. Furthermore, Redis and the database are only added by overriders, we can’t make any services in the main compose file depend on "optional" services.

[ews-pgasser]

@jslocomotor could you please share your full compose.custom.yaml?

[jslocomotor]

Of course, it's close to the default with added redis, noproxy and mariadb overrides. But combined into one file for easier automated deployment:

x-customizable-image: &customizable_image
  image: ${CUSTOM_IMAGE}:${CUSTOM_TAG}
  pull_policy: ${PULL_POLICY:-always}
  restart: ${RESTART_POLICY:-unless-stopped}

x-depends-on-configurator: &depends_on_configurator
  depends_on:
    configurator:
      condition: service_completed_successfully

x-backend-defaults: &backend_defaults
  <<: [*depends_on_configurator, *customizable_image]
  volumes:
    - sites:/home/frappe/frappe-bench/sites
    - logs:/home/frappe/frappe-bench/logs

services:
  configurator:
    <<: *customizable_image
    entrypoint:
      - bash
      - -c
    # add redis_socketio for backward compatibility
    command:
      - >
        ls -1 apps > sites/apps.txt;
        bench set-config -g db_host $$DB_HOST;
        bench set-config -gp db_port $$DB_PORT;
        bench set-config -g redis_cache "redis://$$REDIS_CACHE";
        bench set-config -g redis_queue "redis://$$REDIS_QUEUE";
        bench set-config -g redis_socketio "redis://$$REDIS_QUEUE";
        bench set-config -gp socketio_port $$SOCKETIO_PORT;
        bench set-config -g chromium_path /usr/bin/chromium-headless-shell;
    environment:
      DB_HOST: db
      DB_PORT: "3306"
      REDIS_CACHE: redis-cache:6379
      REDIS_QUEUE: redis-queue:6379
      SOCKETIO_PORT: 9000
    depends_on:
      db:
        condition: service_healthy
      redis-cache:
        condition: service_started
      redis-queue:
        condition: service_started
    restart: on-failure
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs

  migrator:
    <<: *backend_defaults
    entrypoint:
      - bash
      - -c
    command:
      - >
        if [ -d "sites/$$SITE_NAME" ]; then
          bench --site $$SITE_NAME migrate;
        else
          echo "Site $$SITE_NAME does not exist yet, skipping migrate.";
        fi
    environment:
      SITE_NAME: ${SITE_NAME}        
    depends_on:
      configurator:
        condition: service_completed_successfully      
      db:
        condition: service_healthy
      redis-cache:
        condition: service_started
      redis-queue:
        condition: service_started
    restart: on-failure

  db:
    image: mariadb:11.8
    healthcheck:
      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
      start_period: 5s
      interval: 5s
      timeout: 5s
      retries: 5
    restart: unless-stopped
    command:
      - --character-set-server=utf8mb4
      - --collation-server=utf8mb4_unicode_ci
      - --skip-character-set-client-handshake
    environment:
      MYSQL_ROOT_PASSWORD: ${DB_PASSWORD}
      MARIADB_AUTO_UPGRADE: 1
    volumes:
      - db-data:/var/lib/mysql

  redis-cache:
    image: redis:6.2-alpine
    restart: unless-stopped

  redis-queue:
    image: redis:6.2-alpine
    restart: unless-stopped
    volumes:
      - redis-queue-data:/data
  
  backend:
    <<: *backend_defaults

  frontend:
    <<: *customizable_image
    command:
      - nginx-entrypoint.sh
    environment:
      BACKEND: backend:8000
      SOCKETIO: websocket:9000
      FRAPPE_SITE_NAME_HEADER: ${FRAPPE_SITE_NAME_HEADER:-$$host}
      UPSTREAM_REAL_IP_ADDRESS: ${UPSTREAM_REAL_IP_ADDRESS:-127.0.0.1}
      UPSTREAM_REAL_IP_HEADER: ${UPSTREAM_REAL_IP_HEADER:-X-Forwarded-For}
      UPSTREAM_REAL_IP_RECURSIVE: ${UPSTREAM_REAL_IP_RECURSIVE:-off}
      PROXY_READ_TIMEOUT: ${PROXY_READ_TIMEOUT:-120}
      CLIENT_MAX_BODY_SIZE: ${CLIENT_MAX_BODY_SIZE:-50m}
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs
    depends_on:
      - backend
      - websocket
    ports:
      - "8080:8080"

  websocket:
    <<: [*depends_on_configurator, *customizable_image]
    command:
      - node
      - /home/frappe/frappe-bench/apps/frappe/socketio.js
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs

  queue-short:
    <<: *backend_defaults
    command: bench worker --queue short,default

  queue-long:
    <<: *backend_defaults
    command: bench worker --queue long,default,short

  scheduler:
    <<: *backend_defaults
    command: bench schedule

# ERPNext requires local assets access (Frappe does not)
volumes:
  db-data:
  redis-queue-data:
  sites:
  logs:

[ews-pgasser]

Thanks for sharing! I assume you currently create the site manually as in docs, right?

Some time ago on 11th February 2025 Hussain released a compose template for Dokploy https://frappe.io/blog/tutorial/self-hosting-frappe-erpnext-apps-with-dokploy, where he also defined a create-site service:

  create-site:
    <<: *custom_image
    deploy:
      mode: replicated
      replicas: ${CREATE_SITE:-0}
      restart_policy:
        condition: none
    entrypoint: ["bash", "-c"]
    command:
      - >
        wait-for-it -t 120 $$DB_HOST:$$DB_PORT;
        wait-for-it -t 120 redis-cache:6379;
        wait-for-it -t 120 redis-queue:6379;
        export start=`date +%s`;
        until [[ -n `grep -hs ^ sites/common_site_config.json | jq -r ".db_host // empty"` ]] && \
          [[ -n `grep -hs ^ sites/common_site_config.json | jq -r ".redis_cache // empty"` ]] && \
          [[ -n `grep -hs ^ sites/common_site_config.json | jq -r ".redis_queue // empty"` ]];
        do
          echo "Waiting for sites/common_site_config.json to be created";
          sleep 5;
          if (( `date +%s`-start > 120 )); then
            echo "could not find sites/common_site_config.json with required keys";
            exit 1
          fi
        done;
        echo "sites/common_site_config.json found";
        [[ -d "sites/${SITE_NAME}" ]] && echo "${SITE_NAME} already exists" && exit 0;
        bench new-site --mariadb-user-host-login-scope='%' --admin-password=$${ADMIN_PASSWORD} --db-root-username=root --db-root-password=$${DB_ROOT_PASSWORD} $${INSTALL_APP_ARGS} $${SITE_NAME};
    volumes:
      - sites:/home/frappe/frappe-bench/sites
    environment:
      SITE_NAME: ${SITE_NAME}
      ADMIN_PASSWORD: ${ADMIN_PASSWORD}
      DB_HOST: ${DB_HOST:-db}
      DB_PORT: "${DB_PORT:-3306}"
      DB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
      INSTALL_APP_ARGS: ${INSTALL_APP_ARGS}
    networks:
      - bench-network

It may need some adjustment, but if i we add the migrator OPT-IN/OPT-OUT, we could also add a create-site OPT-IN/OPT-OUT.

[DanielRadlAMR]

Sounds good to me too, just please handle it in a separate PR 🙂

[jslocomotor]

Thanks for sharing! I assume you currently create the site manually as in docs, right?

Some time ago on 11th February 2025 Hussain released a compose template for Dokploy https://frappe.io/blog/tutorial/self-hosting-frappe-erpnext-apps-with-dokploy, where he also defined a create-site service:

create-site:
<<: *custom_image
deploy:
mode: replicated
replicas: ${CREATE_SITE:-0}
restart_policy:
condition: none
entrypoint: ["bash", "-c"]
command:
- >
wait-for-it -t 120 $$DB_HOST:$$DB_PORT;
wait-for-it -t 120 redis-cache:6379;
wait-for-it -t 120 redis-queue:6379;
export start=date +%s;
until [[ -n grep -hs ^ sites/common_site_config.json | jq -r ".db_host // empty" ]] &&
[[ -n grep -hs ^ sites/common_site_config.json | jq -r ".redis_cache // empty" ]] &&
[[ -n grep -hs ^ sites/common_site_config.json | jq -r ".redis_queue // empty" ]];
do
echo "Waiting for sites/common_site_config.json to be created";
sleep 5;
if (( date +%s-start > 120 )); then
echo "could not find sites/common_site_config.json with required keys";
exit 1
fi
done;
echo "sites/common_site_config.json found";
[[ -d "sites/${SITE_NAME}" ]] && echo "${SITE_NAME} already exists" && exit 0;
bench new-site --mariadb-user-host-login-scope='%' --admin-password=$${ADMIN_PASSWORD} --db-root-username=root --db-root-password=$${DB_ROOT_PASSWORD} $${INSTALL_APP_ARGS} $${SITE_NAME};
volumes:
- sites:/home/frappe/frappe-bench/sites
environment:
SITE_NAME: ${SITE_NAME}
ADMIN_PASSWORD: ${ADMIN_PASSWORD}
DB_HOST: ${DB_HOST:-db}
DB_PORT: "${DB_PORT:-3306}"
DB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
INSTALL_APP_ARGS: ${INSTALL_APP_ARGS}
networks:
- bench-network

It may need some adjustment, but if i we add the migrator OPT-IN/OPT-OUT, we could also add a create-site OPT-IN/OPT-OUT.

I mean, sure, we could offer that. Take a look at the pwd.yaml — it's there as well.

But in that case, users would still need to install their apps manually or add them to the container commands.

Migrations are required for every image or app update, while site creation usually only happens a few times during a company's lifetime.