Skip to content

Add daemon.json with overlay2 to fix container startup issue#351

Merged
cheeyanglee merged 1 commit intoopen-edge-platform:3.0-devfrom
polmoorx:fix-docker-startup-overlay2
Nov 17, 2025
Merged

Add daemon.json with overlay2 to fix container startup issue#351
cheeyanglee merged 1 commit intoopen-edge-platform:3.0-devfrom
polmoorx:fix-docker-startup-overlay2

Conversation

@polmoorx
Copy link
Copy Markdown
Contributor

@polmoorx polmoorx commented Jul 28, 2025

Merge Checklist

All boxes should be checked before merging the PR

  • The changes in the PR have been built and tested
  • [] cgmanifest file has been updated if required
  • Ready to merge

Description

On EMT OS, when we tried to start Docker, it failed because the default overlay2 storage driver requires a writable, overlay-compatible filesystem, but /var/lib/docker resides under /var, which on EMT OS can be mounted as read-only or which isn’t compatible with overlay operations. This prevented Docker from launching. To fix this, we configured Docker to store its data in /opt/docker-data, a persistent and writable location that supports overlay2, by adding a daemon.json file and updating the RPM spec to create the required directories, install the configuration, and safely migrate any existing data from /var/lib/docker using rsync. With this change, Docker now starts and runs reliably on both RT and non-RT EMT OS, using a stable and compatible storage location.

Any Newly Introduced Dependencies

No

How Has This Been Tested?

Tested manually.

@polmoorx polmoorx requested a review from a team as a code owner July 28, 2025 09:21
@polmoorx polmoorx requested a review from a team as a code owner July 29, 2025 04:19
@polmoorx polmoorx force-pushed the fix-docker-startup-overlay2 branch 2 times, most recently from 6b58436 to 5110a47 Compare July 30, 2025 06:07
@ssinghax ssinghax requested a review from anujm1 July 30, 2025 06:34
@polmoorx polmoorx force-pushed the fix-docker-startup-overlay2 branch 4 times, most recently from ee1ab22 to 34d3f78 Compare August 5, 2025 14:08
@polmoorx polmoorx force-pushed the fix-docker-startup-overlay2 branch 3 times, most recently from 4a3f4e2 to 686f4fa Compare August 13, 2025 05:01
@polmoorx polmoorx force-pushed the fix-docker-startup-overlay2 branch 5 times, most recently from a2838e3 to f720386 Compare September 1, 2025 04:41
@polmoorx polmoorx force-pushed the fix-docker-startup-overlay2 branch from f720386 to e876a84 Compare September 9, 2025 04:05
@polmoorx polmoorx force-pushed the fix-docker-startup-overlay2 branch 2 times, most recently from c4934ed to 46d9f21 Compare September 18, 2025 16:30
@yepmunchun
Copy link
Copy Markdown
Contributor

Before switching data-root to new location, have you tried to add /var/lib/docker in layout.env's PERSISTENT_BIND_PATHS ?

@polmoorx polmoorx force-pushed the fix-docker-startup-overlay2 branch 2 times, most recently from 9022047 to 5319537 Compare October 23, 2025 08:06
@polmoorx
Copy link
Copy Markdown
Contributor Author

Before switching data-root to new location, have you tried to add /var/lib/docker in layout.env's PERSISTENT_BIND_PATHS ?

We did try adding /var/lib/docker to PERSISTENT_BIND_PATHS before changing the data-root to /opt/docker-data. But because on EMT OS /var is often temporary (tmpfs) early in boot, Docker’s data under /var/lib/docker wasn’t actually persistent, so the overlayfs errors continued.

Changing Docker’s data-root to /opt/docker-data and ensuring it is persistent and created at runtime fixed the problem.

@Jkintalix Jkintalix requested a review from yepmunchun October 24, 2025 04:12
@polmoorx polmoorx force-pushed the fix-docker-startup-overlay2 branch 2 times, most recently from 00d09b3 to 7b082d9 Compare October 30, 2025 04:25
Copy link
Copy Markdown
Contributor

@andy-vm andy-vm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cheeyanglee cheeyanglee merged commit 8601739 into open-edge-platform:3.0-dev Nov 17, 2025
14 of 18 checks passed
@aaroncyew aaroncyew added x-oauth2: v0.10.0: CVE-2025-22868 aws-sdk-go: v1.17.6: CVE-2022-2582 Component aws-sdk-go labelled for run 7915105 containerd: v1.7.12: CVE-2024-40635 Component containerd labelled for run 7915105 and removed x-oauth2: v0.10.0: CVE-2025-22868 labels Nov 18, 2025
@aaroncyew
Copy link
Copy Markdown
Member

Automated Messages: Label 'moby-engine' has been added to this Pull Request.

1 similar comment
@aaroncyew
Copy link
Copy Markdown
Member

Automated Messages: Label 'moby-engine' has been added to this Pull Request.

@aaroncyew aaroncyew added contrib-instrumentation-g: v0.45.0: CVE-2023-47108 Component contrib-instrumentation-g labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2018-16886 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15106 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15112 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15113 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15114 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15115 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15136 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2023-32082 Component etcd labelled for run 7915212 golang-jwt: v4.4.2: CVE-2024-51744 Component golang-jwt labelled for run 7915212 golang-jwt: v4.4.2: CVE-2025-30204 Component golang-jwt labelled for run 7915212 protobuf: v1.5.3: CVE-2019-15544 Component protobuf labelled for run 7915212 protobuf: v1.5.3: CVE-2025-53605 Component protobuf labelled for run 7915212 protobuf-go: v1.31.0: CVE-2024-24786 Component protobuf-go labelled for run 7915212 runc: v1.1.12: CVE-2024-45310 Component runc labelled for run 7915212 labels Nov 18, 2025
@aaroncyew
Copy link
Copy Markdown
Member

Automated Messages: Label 'moby-engine' has been added to this Pull Request.

1 similar comment
@aaroncyew
Copy link
Copy Markdown
Member

Automated Messages: Label 'moby-engine' has been added to this Pull Request.

@aaroncyew aaroncyew added etcd: v3.3.10+incompa: CVE-2018-16886 Component etcd labelled for run 7915497 etcd: v3.3.10+incompa: CVE-2020-15106 Component etcd labelled for run 7915497 etcd: v3.3.10+incompa: CVE-2020-15112 Component etcd labelled for run 7915497 labels Nov 18, 2025
@aaroncyew
Copy link
Copy Markdown
Member

Automated Messages: Label 'moby-engine' has been added to this Pull Request.

1 similar comment
@aaroncyew
Copy link
Copy Markdown
Member

Automated Messages: Label 'moby-engine' has been added to this Pull Request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aws-sdk-go: v1.17.6: CVE-2022-2582 Component aws-sdk-go labelled for run 7915105 containerd: v1.7.12: CVE-2024-25621 Component containerd labelled for run 7918506 containerd: v1.7.12: CVE-2024-40635 Component containerd labelled for run 7915105 containerd: v1.7.12: CVE-2025-64329 Component containerd labelled for run 7918506 contrib-instrumentation-g: v0.45.0: CVE-2023-47108 Component contrib-instrumentation-g labelled for run 7915212 etcd: v3.3.10+incompa: CVE-2018-16886 Component etcd labelled for run 7915497 etcd: v3.3.10+incompa: CVE-2020-15106 Component etcd labelled for run 7915497 etcd: v3.3.10+incompa: CVE-2020-15112 Component etcd labelled for run 7915497 etcd: v3.3.10+incompa: CVE-2020-15113 Component etcd labelled for run 7915497 etcd: v3.3.10+incompa: CVE-2020-15114 Component etcd labelled for run 7915497 etcd: v3.3.10+incompa: CVE-2020-15115 Component etcd labelled for run 7915497 etcd: v3.3.10+incompa: CVE-2020-15136 Component etcd labelled for run 7915497 etcd: v3.3.10+incompa: CVE-2023-32082 Component etcd labelled for run 7915497 etcd: v3.3.10+incompatible: CVE-2018-16886 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15106 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15112 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15113 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15114 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15115 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2020-15136 Component etcd labelled for run 7915212 etcd: v3.3.10+incompatible: CVE-2023-32082 Component etcd labelled for run 7915212 golang-jwt: v4.4.2: CVE-2024-51744 Component golang-jwt labelled for run 7915212 golang-jwt: v4.4.2: CVE-2025-30204 Component golang-jwt labelled for run 7915212 moby-engine Package: moby-engine protobuf: v1.5.3: CVE-2019-15544 Component protobuf labelled for run 7915212 protobuf: v1.5.3: CVE-2025-53605 Component protobuf labelled for run 7915212 protobuf-go: v1.31.0: CVE-2024-24786 Component protobuf-go labelled for run 7915212 runc: v1.1.12: CVE-2024-45310 Component runc labelled for run 7915212 shlex: v0.0.0-20191202: CVE-2024-58266 Component shlex labelled for run 7915497 x-crypto: v0.17.0: CVE-2024-45337 Component x-crypto labelled for run 7915497 x-crypto: v0.17.0: CVE-2025-22869 Component x-crypto labelled for run 7915497 x-crypto: v0.17.0: CVE-2025-47913 Component x-crypto labelled for run 7915497 x-net: v0.17.0: CVE-2023-45288 Component x-net labelled for run 7915497 x-net: v0.17.0: CVE-2024-45338 Component x-net labelled for run 7915497 x-net: v0.17.0: CVE-2025-22870 Component x-net labelled for run 7915497 x-net: v0.17.0: CVE-2025-22872 Component x-net labelled for run 7915497 x-oauth2: v0.10.0: CVE-2025-22868 Component x-oauth2 labelled for run 7915497

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants