Asset mismatch between backend and frontend after running `bench migrate`

[silicht]

Description of the issue

I run a CI powered, docker based setup with a custom image.

In my CI environment, I build the custom container image. After the image is built, I run some docker compose commands on my server:

docker compose down
docker compose pull
docker compose up -d
docker compose exec backend bench migrate
docker compose exec redis-cache redis-cli FLUSHALL

This worked well, until #1879. Now, the bench migrate command creates a new assets.json, which is stored with the frontend container as it is stored in /home/frappe/frappe-bench/sites/assets/ folder, and with that in the sites volume.

The css/js files itself are stored in /home/frappe/frappe-bench/apps/, which is not shared between the containers. There are symlinks from the assets forlder to the apps folder. So after running the migration in the backend container, the css/js files in the backend container are updated, but not that files in the frontend container. But as the assets.json file is shared, the frontend cannot find the assets and the user get's 404 errors.

In the past, it looks like it worked by accident. I run bench migrate in the backend container, and bench updated css/js in the backend container, and the assets.json in a unnamed volume, which was not shared with the frontend container. So the frontend container still served the "old" version of these files.

I understand that the css/js files should not be build in productive environment. But I need to run the migrations.

How to run migrations without building the assets?

Context information (for bug reports)

Steps to reproduce the issue

1. Build a custom app image
2. Update container using that image
3. Run bench migrate in the backend container

Observed result

assets.json and accessable files are out of sync.

Expected result

bench migrate should not manipulate assets.json file.

Stacktrace / full error message if available

[ews-pgasser]

I have tried to recreate it and wrote the following shell-script to do so:

#!/bin/bash
PROJECT=$(docker ps --format '{{.Names}}' | grep backend | head -1 | sed 's/-backend-[0-9]*//')
SITE=$(ls /var/lib/docker/volumes/${PROJECT}_sites/_data/ | grep -v apps | grep -v common | grep "\.")
BACKEND="${PROJECT}-backend-1"
FRONTEND="${PROJECT}-frontend-1"
REDIS_CACHE="${PROJECT}-redis-cache-1"
ASSETS_JSON="/var/lib/docker/volumes/${PROJECT}_sites/_data/assets/assets.json"
ASSETS_DIR="/var/lib/docker/volumes/${PROJECT}_sites/_data/assets"
IMAGE_SHA=$(docker inspect $FRONTEND --format '{{.Image}}')

BUNDLE_PATTERN="desk.bundle"

# Bundle filenames (full names with hash)
ASSETS_BUNDLE=$(cat $ASSETS_JSON | grep "${BUNDLE_PATTERN}.css" | grep -o "${BUNDLE_PATTERN}\.[A-Z0-9]*\.css" | head -1)
HTML_BUNDLE=$(curl -sk https://$SITE/login | grep -o "${BUNDLE_PATTERN}\.[A-Z0-9]*\.css" | head -1)
FILE_BUNDLE=$(docker exec $FRONTEND ls /home/frappe/frappe-bench/sites/assets/frappe/dist/css/ | grep "${BUNDLE_PATTERN}" | head -1)

# Hashes (extracted from bundle filenames)
ASSETS_HASH=$(echo "$ASSETS_BUNDLE" | grep -o '[A-Z0-9]\{8\}')
HTML_HASH=$(echo "$HTML_BUNDLE" | grep -o '[A-Z0-9]\{8\}')
FILE_HASH=$(echo "$FILE_BUNDLE" | grep -o '[A-Z0-9]\{8\}')

# Locations
ASSETS_LOC="$ASSETS_JSON (host volume: ${PROJECT}_sites)"
HTML_LOC="https://$SITE/login (served via $FRONTEND, cached in $REDIS_CACHE)"
FILE_LOC="$FRONTEND:/home/frappe/frappe-bench/sites/assets/frappe/dist/css/"

echo "================================================================================="
echo "🔍 Asset Hash Check"
echo "================================================================================="
echo "🎯 Bundle pattern       : ${BUNDLE_PATTERN}"
echo "---------------------------------------------------------------------------------"
echo "📄 assets.json (volume)  : $ASSETS_HASH  ← what Frappe thinks the file is"
echo "   Bundle file          : $ASSETS_BUNDLE"
echo "   Location             : $ASSETS_LOC"
echo "---------------------------------------------------------------------------------"
echo "🌐 HTML/Redis (browser)  : $HTML_HASH  ← what browser is told to load"
echo "   Bundle file          : $HTML_BUNDLE"
echo "   Location             : $HTML_LOC"
echo "---------------------------------------------------------------------------------"
echo "📦 Actual file (frontend): $FILE_HASH  ← what actually exists on disk"
echo "   Bundle file          : $FILE_BUNDLE"
echo "   Location             : $FILE_LOC"
echo "================================================================================="
if [ "$ASSETS_HASH" == "$HTML_HASH" ] && [ "$HTML_HASH" == "$FILE_HASH" ]; then
  echo "✅ ALL MATCH - No asset mismatch"
else
  echo "❌ MISMATCH DETECTED - Auto-fixing..."
  if [ "$ASSETS_HASH" != "$FILE_HASH" ]; then
    echo ""
    echo "→ Copying fresh assets.json from image..."
    docker create --name temp_assets $IMAGE_SHA
    docker cp temp_assets:/home/frappe/frappe-bench/sites/assets/assets.json $ASSETS_DIR/
    docker rm temp_assets
    echo "  ✅ assets.json updated"
  fi
  echo ""
  echo "→ Flushing Redis cache..."
  docker exec $REDIS_CACHE redis-cli FLUSHALL
  echo "  ✅ Redis flushed"
  echo ""
  echo "→ Clearing bench cache..."
  docker exec $BACKEND bench --site $SITE clear-cache
  echo "  ✅ Bench cache cleared"
  echo ""
  echo "================================================================================="
  echo "🔁 Re-checking after fix..."
  echo "================================================================================="
  ASSETS_BUNDLE=$(cat $ASSETS_JSON | grep "${BUNDLE_PATTERN}.css" | grep -o "${BUNDLE_PATTERN}\.[A-Z0-9]*\.css" | head -1)
  HTML_BUNDLE=$(curl -sk https://$SITE/login | grep -o "${BUNDLE_PATTERN}\.[A-Z0-9]*\.css" | head -1)
  FILE_BUNDLE=$(docker exec $FRONTEND ls /home/frappe/frappe-bench/sites/assets/frappe/dist/css/ | grep "${BUNDLE_PATTERN}" | head -1)
  ASSETS_HASH=$(echo "$ASSETS_BUNDLE" | grep -o '[A-Z0-9]\{8\}')
  HTML_HASH=$(echo "$HTML_BUNDLE" | grep -o '[A-Z0-9]\{8\}')
  FILE_HASH=$(echo "$FILE_BUNDLE" | grep -o '[A-Z0-9]\{8\}')
  echo "📄 assets.json : $ASSETS_HASH  ($ASSETS_BUNDLE)"
  echo "   Location    : $ASSETS_LOC"
  echo "🌐 HTML/Redis  : $HTML_HASH  ($HTML_BUNDLE)"
  echo "   Location    : $HTML_LOC"
  echo "📦 File        : $FILE_HASH  ($FILE_BUNDLE)"
  echo "   Location    : $FILE_LOC"
  if [ "$ASSETS_HASH" == "$HTML_HASH" ] && [ "$HTML_HASH" == "$FILE_HASH" ]; then
    echo "✅ ALL MATCH - Fixed successfully"
  else
    echo "❌ STILL MISMATCHED - Manual intervention required"
  fi
fi

Could you please run and show me the output?
For me the frontend (erpnext-frontend-1:/home/frappe/frappe-bench/sites/assets/frappe/dist/css/) has the updatedcss from the new docker image, but the assets.json volume in the host dir (/var/lib/docker/volumes/erpnext-sites/_data/assets/assets.json) still holds the old ones (for me it only happens to be the css).

The above script should fix the issue for now. I am currently working on a cleaner long-term fix that can get integrated into an CI/CD-Pipeline without manual "afterwork".
If anyone already worked on it lmk.

[Uninvited3834]

After running your script everything works again. Here is the Output:

=================================================================================
🔍 Asset Hash Check

🎯 Bundle pattern : desk.bundle

📄 assets.json (volume) : Q5W57O4L ← what Frappe thinks the file is
Bundle file : desk.bundle.Q5W57O4L.css
Location : /var/lib/docker/volumes/frappe_docker_sites/_data/assets/assets.json (host volume: frappe_docker_sites)

🌐 HTML/Redis (browser) : Q5W57O4L ← what browser is told to load
Bundle file : desk.bundle.Q5W57O4L.css
Location : https://erp..eu/login (served via frappe_docker-frontend-1, cached in frappe_docker-redis-cache-1)

📦 Actual file (frontend): LSKBXZ7P ← what actually exists on disk
Bundle file : desk.bundle.LSKBXZ7P.css
Location : frappe_docker-frontend-1:/home/frappe/frappe-bench/sites/assets/frappe/dist/css/

❌ MISMATCH DETECTED - Auto-fixing...

→ Copying fresh assets.json from image...
e1cd069692b4226bf516d051ba6d37040c19ed0ced909bf9d49c4caccd99
Successfully copied 4.17kB (transferred 6.14kB) to /var/lib/docker/volumes/frappe_docker_sites/_data/assets/
temp_assets
✅ assets.json updated

→ Flushing Redis cache...
OK
✅ Redis flushed

→ Clearing bench cache...
✅ Bench cache cleared

=================================================================================
🔁 Re-checking after fix...

📄 assets.json : LSKBXZ7P (desk.bundle.LSKBXZ7P.css)
Location : /var/lib/docker/volumes/frappe_docker_sites/_data/assets/assets.json (host volume: frappe_docker_sites)
🌐 HTML/Redis : LSKBXZ7P (desk.bundle.LSKBXZ7P.css)
Location : https://erp..eu/login (served via frappe_docker-frontend-1, cached in frappe_docker-redis-cache-1)
📦 File : LSKBXZ7P (desk.bundle.LSKBXZ7P.css)
Location : frappe_docker-frontend-1:/home/frappe/frappe-bench/sites/assets/frappe/dist/css/
✅ ALL MATCH - Fixed successfully

[ews-pgasser]

Background

Previously sites/assets was declared as a separate VOLUME in the Dockerfile, which meant it got its own named volume. On every deploy you could simply run:

docker volume rm frappe_docker_sites_assets
docker compose up -d

Docker would recreate the volume and automatically populate it fresh from the image. Clean, no extra services needed.

When sites/assets was removed from the Dockerfile VOLUME declaration, it fell under the parent sites volume. This makes it impossible to nuke and repopulate just the assets anymore, as deleting the sites volume would also wipe all site data under sites/<your-site>/ including private and public files.

---

Current workaround

Since Docker never refreshes an existing volume from the image on redeployment, assets go stale after every image update. To solve this without touching site data, assets-sync mounts the sites volume at /tmp_sites instead of the normal /home/frappe/frappe-bench/sites. This gives the container two distinct paths simultaneously:

• /home/frappe/frappe-bench/sites/assets/ -> image layer, fresh assets, not shadowed by any mount
• /tmp_sites/assets/ -> the live sites volume that all other containers read from

assets-sync:
  <<: *x-customizable-image
  restart: "no"
  entrypoint: ["bash", "-c"]
  command:
    - >
      mkdir -p /tmp_sites/assets/;
      cp -rf /home/frappe/frappe-bench/sites/assets/. /tmp_sites/assets/;
      echo "======SYNC COMPLETED======";
  volumes:
    - sites:/tmp_sites

---

Proper fix

Not sure what to think about both solutions. What do you think @DanielRadlAMR @Rocket-Quack?

[DanielRadlAMR]

Here is what i would do:

Basically what @ews-pgasser suggested, slightly adapted:

Append the compose by a new service:

update-assets:
    <<: *customizable-image
    command: >
      bash -c "rm -rf /mnt/sites/assets/ && cp -r /home/frappe/frappe-bench/sites/assets /mnt/sites/"
    volumes:
      -sites:/mnt/sites   # mounted at a different path!
    restart: on-failure

and add a depends_on:

x-depends-on-configurator: &depends_on_configurator
  depends_on:
    configurator:
       condition: service_completed_successfully
     update-assets:
        condition: service_completed_successfully

This keeps things quite clean and straightforward in my opinion. It keeps the sites folder aligned with the standard Frappe Framework setup, which feels more predictable operationally.
Introducing an additional volume for assets would add a bit of cognitive and operational overhead.

@Rocket-Quack what do you think about it?

There are two things I’m still unsure about:

• we introduce an additional lifecycle step on top of Docker volumes (syncing image → volume at runtime instead of keeping it purely declarative)
• the number of auxiliary services is increasing (configurator, asset update, migrator, site creation), which adds orchestration overhead

You definitely know Docker better than I do, so I’m curious about your take on this. It feels a bit off to me architecturally, but maybe this is actually a common pattern in practice?

[jslocomotor]

As @silicht mentioned bench migrate is altering the assets.json file. Shouldn't we dig deeper into that instead of working on a temporary fix?

[Rocket-Quack]

I would be very reluctant to implement a new cycle on top of the existing one, as we’ve already noted, since it would serve only as a temporary fix.
Because, as you said @DanielRadlAMR that would just tie us into even more complexity down the road.

When I look at it this way and approach it with KISS in mind I think we just need to ensure that the assets don’t remain persistent, whereconfigs, etc., do need to be persistent. All while keeping in mind that everything is located under a single directory.
The big problem is also that this won’t be a pure volume fix, since we can’t exclude paths and we don’t want to bring back the original bug from #1879.

My idea (Spoiler: doesnt work ☹️) or suggestion is to separate everything using symlinks. Using an EntryPoint script, we could then make the whole process pathsafe for the build process. Everything except assets would then be stored as fixed volumes for example in a directory of our choosing that we can remount. This way assets come from the image while the rest remains persistent.
As I’m writing this, it occurs to me that this wouldnt work either since the sites are created in the backend container at runtime that would break the entry script with the symlinks. I cant think of a way to cover dynamic site creation with my idea right now so lets put that the back in the bucket.

In my view the most elegant solution and simplest (for us) approach would be for the Frappe architecture to change so that this isnt all located under “sites”.
Instead the whole thing should be separated from the frappe architecture, or the directory structure should be adjusted accordingly.

How do we estimate the effort involved? The biggest problem is creating an issue and PR for this, since it represents a very profound change.

But I’m also open to other possibilities if we can’t find anything else. Otherwise from a Docker perspective we hit our limits without integrating our own storage driver (and thats is a scale of antohger dimension)

[DanielRadlAMR]

@jslocomotor I understand what you mean, but bench migrate isn’t the issue here. Furthermore, I didn’t consider my suggestion to be a temporary fix, but rather an architectural choice for our setup.

The current situation is that we mount the assets as a volume; therefore, it is persistent data and does not get updated from the image’s assets.

But why did updating assets.json fix this then?

This happens because the actual assets (i.e. the .css files) are not “real” files in the assets folder. The app-specific assets are just symlinks to the app’s public folder. Since the app folder is not mounted as a persistent volume, those files get updated. However, other files like assets.json are actual files in the assets folder, so they persist and are not overwritten.

[DanielRadlAMR]

@Rocket-Quack I don’t think the root issue is that assets live in the sites folder. For me, the problem is that assets currently need to satisfy two requirements that oppose each other:

• be shared between containers (backend and frontend)
• behave like non-persistent runtime data, retrieved directly from the image

To share a folder between containers you need to mount a volume - and a mounted volume is persistent. I haven’t found a “clean” way around that; that’s just how Docker is designed to work, right?

---

To be completly honest, i still do not get when you run into this issue: #1850
In my understanding, assets should be build-time artifacts and not change during runtime.

In that case, I would agree with you that it would be beneficial to move assets somewhere outside the sites folder.

---

But given the current requirements for assets, my suggested approach is the simplest I could come up with:

1. Two containers need to share the folder:
   → make it a mounted volume

2. The folder should still be updated when the container is recreated with new assets:
   → overwrite the assets folder once on container recreation

[orey-aajaamu]

Jumping in because I've been hitting the same 404s and I think the thread is converging on a fix (#1888) without resolving a more fundamental question @DanielRadlAMR raised.

On whether bench migrate is the cause: I tested the sequence in isolation. My deploy is docker compose down && pull && up -d && redis FLUSHALL, with bench migrate run separately and manually only on schema changes. I hit the exact 404s after pulling a post-#1879 image, without running migrate at all. The mismatch is already present right after up -d.

The cause is the volume lifecycle change from #1879. Pre-#1879, each container had its own anonymous volume for sites/assets, down freed them and up populated fresh ones from the new image. Post-#1879, sites/assets lives inside the persistent named sites volume, which survives down. Docker's copy-on-create doesn't run on a non-empty named volume, so the old image's assets.json remains. bench migrate gets blamed because it's the last non-flush command in silicht's sequence, but it's not the one writing the stale state, the stale state is already there from up -d.

So @jslocomotor instinct to dig deeper was good, but migrate isn't actually the bug to chase. @DanielRadlAMR is right that migrate isn't the cause, and I don't think it's worth pursuing upstream.

@DanielRadlAMR's "I still don't get when you run into this issue": I think this is the key question, and I suspect #1850 was misdiagnosed. The original repro says "Install a custom app (e.g. Mint) or run bench build" these aren't equivalent:

• bench build in a running container writes to sites/assets at runtime. The FAQ explicitly warns against this.
• bench --site install-app on a pre-baked app only writes DocTypes to the DB. It doesn't touch sites/assets.

I run a layered image with multiple custom apps baked in at build time, single-host compose, and I was never able to reproduce #1850's symptoms on the documented workflow. The nested VOLUME was already enforcing what @DanielRadlAMR said it should "assets should be build-time artifacts and not change during runtime" by isolating each container's sites/assets as a per-container anonymous volume. Runtime writes (from someone running bench build against the FAQ's warning) stayed contained to one container and didn't corrupt shared state. down/up cycles refreshed assets for free because new containers got new anonymous volumes populated from the new image.

On the "two opposing requirements" framing: I'd push back gently. Pre-#1879, backend and frontend didn't actually share sites/assets - each had its own anonymous volume with identical content from the same image. They didn't need to share because nothing was supposed to mutate at runtime. Treating this as a "shared folder" problem forces a persistent-volume solution. Treating it as "each container needs the same build-time artifacts from its own image" has a much simpler answer: per-container anonymous volume, refreshed on recreate. Which is exactly what #1879 removed.

On #1888 : @Rocket-Quack hesitation reads right to me. The update-assets service works, but it's adding an auxiliary lifecycle step to every deploy to solve a problem that didn't exist before #1879 - and every workaround in this thread is pushing complexity from the image/volume layer, where Docker handled it for free, onto every user's orchestration layer.

My suggestion: before merging #1888, consider whether #1879 should be reverted and the FAQ strengthened instead. The patterns behind reports like #1850 (bench build in running containers, runtime asset modifications, skipping Redis flush) are documented workflow violations.

[DanielRadlAMR]

thanks for the input @orey-aajaamu

Maybe @remichristiaan could join the discussion and elaborate on how he encountered the issue? Did you run bench build or did you install a new app to the bench in a running container? Because both of those things shouldn’t be done in running containers. Yet even if you do, this still shouldn’t nuke your setup as it did previously.

I clearly think assets should be build-time artefacts, yet I still would not revert #1879. The setup worked before that, but it was just a lucky break, not a real solution to the problem. And if you don’t clean up volumes before redeploying, you could still run into issues.

---

While thinking about it today, I had another idea that comes from the funny behaviour described in this issue, which leads to a mismatch between assets.json and the actual app assets. This didn’t make much sense to me at first, but then I stumbled over the fact that the app assets are symlinks.

While, as @Rocket-Quack already mentioned, moving the assets out of the sites folder would be a huge improvement, that’s nothing we can control in this repo. I also think it will take quite some time before the Frappe team considers something like that seriously. Since our images are currently broken, we should fix this sooner rather than later.

So here’s my idea: we move the assets out of the sites folder during our image build process, and then add a symlink back into the sites folder so it stays compatible with the current Frappe architecture.

This way, the sites folder remains persistent storage, but the assets are no longer stored inside it. Instead, sites contains only a link pointing to the external assets folder created at build time, which is not persistent. This is essentially what we want.

Drawbacks:

• still cannot run bench build or similar commands in running containers
• existing running instances need a migration step (create the symlink in the mounted volume)

[remichristiaan]

@DanielRadlAMR, indeed did the issue show up when running bench build when adding mint to the setup ... later I learned that that should not be done in a container setup (RTFM). Nevertheless, I agree with you that volumes need to be clean, lucky breaks are not way to go. I've not looked into this further due to time constraints, but will if time allows.

[dandax123]

I created this PR #1894 and this should fix all the issues