perf(platform): build the backend image once instead of eight times - #14457
Draft
Bentlybro wants to merge 1 commit into
Draft
perf(platform): build the backend image once instead of eight times#14457Bentlybro wants to merge 1 commit into
Bentlybro wants to merge 1 commit into
Conversation
Eight services declare byte-identical builds -- same context, same Dockerfile, target: server, no differing build args -- and differ only in the command each container runs. Compose has no way to know they are the same image, so it builds and then exports that image once per service. The export is where the time goes. From a docker compose build of the full stack: #80 DONE 2023.8s [rest_server] exporting to image #74 DONE 2019.9s [notification_server] exporting to image #78 DONE 2019.5s [scheduler_server] exporting to image #75 DONE 2019.4s [copilot_executor] exporting to image Roughly 33 minutes each, writing the same ~3.7 GB of layers and contending for the same disk. With one image name those eight exports become one, and the slowest step in the equivalent build fell to 762s. Behaviour is unchanged: each service still runs its own command from the same image, which is what it already did in practice. Measured on two identically specced VMs (12 cores, 32 GB), cold build and start of the whole stack: 56m18s before, 32m33s after. Not a single-machine A/B, and the dev commit moved between the runs, so treat the wall-clock as indicative and the export collapse as the mechanism.
github-project-automation
Bot
moved this to 🆕 Needs initial review
in AutoGPT development kanban
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #14457 +/- ##
=======================================
Coverage 81.51% 81.51%
=======================================
Files 3561 3561
Lines 266076 266075 -1
Branches 24637 24636 -1
=======================================
+ Hits 216882 216886 +4
+ Misses 43835 43743 -92
- Partials 5359 5446 +87
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why / What / How
Why:
docker compose buildon the platform spends most of its time exporting the same image over and over. Eight services (rest_server,executor,copilot_executor,websocket_server,database_manager,scheduler_server,notification_server,platform_linking_manager) declare byte-identical builds — same context, same Dockerfile,target: server, no differing build args — and differ only in thecommandeach container runs. Compose has no way to know they are the same image, so it builds it, then exports it once per service.From a full-stack build, the slowest steps are not compiling or downloading:
Roughly 33 minutes each, writing the same ~3.7 GB of layers and contending for the same disk.
What: gives those eight services one
image:name, so compose builds and exports the image a single time.How: one line per service. Nothing else changes — each container still runs its own
commandfrom that image, which is exactly what it already did; they were identical images with eight different auto-generated names.Changes 🏗️
autogpt_platform/docker-compose.platform.yml: the eighttarget: serverservices now shareimage: autogpt_platform-backend:latest.Agents and large language models used
Claude Code with Claude Fable 5.1
Checklist 📋
For code changes:
deploy-pr dev(wipe all images and build cache,docker compose buildthenup -d) on a 12-core/32 GB VM, before and after: 56m18s → 32m33s, slowest single step 2023s → 762s/docsrespondingdocker compose configresolves; everytarget: serverservice reports the same image, and each keeps its own distinctcommandplatform_linking_manager(the one behindprofiles: ["bot"]) still resolves correctlyA caveat on the timing, since it matters: the two runs were on two identically specced VMs rather than a single-machine A/B, and the
devcommit moved between them. Treat the wall-clock as indicative; the export collapse from eight to one is the mechanism, and that part is directly visible in the build output above.For configuration changes:
.env.defaultis updated or already compatible with my changesdocker-compose.ymlis updated or already compatible with my changes