feat(codebuild): real buildspec execution in Docker containers#2168
Open
vieiralucas wants to merge 4 commits into
Open
feat(codebuild): real buildspec execution in Docker containers#2168vieiralucas wants to merge 4 commits into
vieiralucas wants to merge 4 commits into
Conversation
StartBuild now runs the build for real in a background task: resolves the environment image, parses the buildspec phases, executes each phase's commands in a Docker/Podman container, streams output to CloudWatch Logs, uploads S3 artifacts, and settles buildStatus on the real container exit codes (with real per-phase phases[] results). Honors CodeBuild phase-failure semantics (post_build always runs). StopBuild kills the container. Build batches and retries mirror the single-build path. Env-gated: the real backend runs only when a container CLI is available and FAKECLOUD_CODEBUILD_DISABLE_BACKEND is not set. When disabled (conformance points FAKECLOUD_CONTAINER_CLI at a non-existent binary; the tfacc harness sets the disable flag) builds fall back to the deterministic settle-to-SUCCEEDED-on-read path, keeping response shapes and conformance (59/59 ops, 2051/2051 variants) unchanged. Restart reconcile: a build persisted IN_PROGRESS whose container is gone is flipped to FAILED on load instead of left a zombie. E2E (Docker-gated) proves a real command runs to SUCCEEDED with the real phase breakdown and that a failing build phase settles FAILED, plus restart reconcile.
1. Cross-phase shell state: run ALL phases in ONE continuous shell (phase functions in the current shell + marker-delimited exit codes) so cd/export persist across phases like AWS, while preserving real per-phase status/timing. 2. Artifact globs: copy the artifact base dir out and match patterns with the glob crate (**/*, target/*.jar, base-directory, discard-paths); error when declared files match nothing (previously silently uploaded zero). 3. Keepalive sized to the configured timeout (+buffer) instead of a fixed 1h. 4. Timeout kills the exec (kill_on_drop) and force-removes the container, so nothing runs past the deadline (one exec means no POST_BUILD race). 5. Honor timeoutInMinutes: settle TIMED_OUT at the configured limit (5..=480). 6. BuildBatch emits BuildBatch-shaped phases (DOWNLOAD_BATCHSPEC/IN_PROGRESS/ COMBINE_ARTIFACTS) and keeps logConfig (no spurious logs field). 7. Resolve PARAMETER_STORE / SECRETS_MANAGER env vars cross-service from fakecloud-ssm / fakecloud-secretsmanager and inject into the container. 8. BuildGuard (Drop) settles the build FAILED and force-removes the container on any abnormal task exit (panic), so no leak / stuck IN_PROGRESS. 9. Read-side safety net: a real-backed build IN_PROGRESS past timeout+grace is settled TIMED_OUT on BatchGetBuilds. E2E adds cross-phase (export in pre_build used in build -> SUCCEEDED) and glob artifact upload (target/*.txt lands in S3, .bin does not) tests. Disabled-backend path unchanged: conformance 59/59, tfacc codebuild_acceptance PASS.
The round-2 single-continuous-shell rewrite ran each phase's commands in the current shell. A user command that calls `exit N` (e.g. a buildspec whose BUILD phase runs `exit 1`) terminated the whole script BEFORE the BUILD phase-end marker was printed, so the parser never recorded BUILD, build_failed stayed false, and the build wrongly settled SUCCEEDED with the BUILD phase missing. Fix: - Run each phase's commands in a SUBSHELL so a failing command — including `exit N` — fails only that phase; the phase-end marker with the real exit code is ALWAYS emitted. Cross-phase state still persists: cwd + exported vars are threaded phase-to-phase via a state file (sourced at phase start, written back on success), so cd/export carry across phases like AWS. - Parser records a phase with a start but NO end marker as FAILED (shell died mid-phase), never dropping it -> an unknown outcome can never settle SUCCEEDED. - buildStatus FAILED if any phase reports non-zero OR the build script itself exits non-zero (belt-and-suspenders). Tests: added parser unit tests (non-zero phase -> FAILED phase + FAILED build; missing end-marker -> FAILED). E2E failing-command test now also asserts the BUILD phase is present. Cross-phase-state + success + glob-artifact tests unchanged. Green: build, fmt, clippy --workspace --all-targets -D warnings, codebuild unit tests (52), conformance 59/59, tfacc codebuild_acceptance PASS.
…ckend-disable envs)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the status-flip stub (StartBuild fabricated SUCCEEDED with no container engine) with REAL build execution — CodeBuild's product actually runs now.
docker exec sh -c), socd/exportpersist across phases like AWS. Real per-phase status/timing breakdown (INSTALL/PRE_BUILD/BUILD/POST_BUILD...), real exit codes settle SUCCEEDED/FAILED; POST_BUILD always runs.logsLogsLocation is real. Artifacts:artifacts.filesglobs matched via the glob crate (base-directory/discard-paths honored), uploaded to S3; a pattern matching nothing fails the build (no silent empty).timeoutInMinutes(5..480), settles TIMED_OUT; execkill_on_drop+ container force-remove so nothing runs past the deadline.logConfigshape.FAKECLOUD_CODEBUILD_DISABLE_BACKEND, set in tfacc harness; conformance probe has no container CLI) — disabled path keeps the deterministic settle, byte-identical shapes.Incorporates fixes for 9 code-review findings (cross-phase shell state, artifact globs, timeout cap/kill, BuildBatch shape, secret resolution, panic guard, read-side fallback).
Test plan
cargo build -p fakecloud,clippy --workspace --all-targets -D warnings,fmt: clean;cargo test -p fakecloud-codebuild50 passHonest limitation: BuildBatch doesn't fan out a child-build matrix (single resolved buildspec); no real source repo (DOWNLOAD_SOURCE immediate). Documented in codebuild.md.
Summary by cubic
Run CodeBuild buildspecs for real in Docker/Podman. StartBuild returns IN_PROGRESS immediately while a background task runs the build with real logs, artifacts, timeouts, and statuses.
New Features
fakecloud-logs; uploadS3artifacts withartifacts.filesglobs (respectbase-directory/discard-paths); fail when patterns match nothing.timeoutInMinutes(5–480) and settleTIMED_OUT;StopBuildkills the container and cleans up.PARAMETER_STORE/SECRETS_MANAGERenv vars viafakecloud-ssm/fakecloud-secretsmanager; correctBuildBatchphases andlogConfig.IN_PROGRESSbuilds flip toFAILED; stalled builds past timeout settle on read; setFAKECLOUD_CODEBUILD_DISABLE_BACKEND=1to disable the backend and keep deterministic settle.Bug Fixes
SUCCEEDED: each phase now runs in a subshell, the parser marks a started-but-missing end marker asFAILED, and the build settlesFAILEDif any phase or the script exits non-zero.Written for commit b023c5e. Summary will update on new commits.