Skip to content

Conversation

@Pnkcaht
Copy link
Contributor

@Pnkcaht Pnkcaht commented Jan 26, 2026

Fix: kill docker container on pipeline cancellation

What was happening

When a pipeline step was canceled (e.g. manual cancel or agent shutdown), the Docker backend would stop waiting for the container but would not actively terminate it.
This caused containers running long-lived commands (like sleep, tail -f, servers, etc.) to continue running in the background.

Related Issue

Closes #6016

What this PR changes

This PR ensures that when the step context is canceled, the running Docker container is immediately killed.

  • Listens to ctx.Done() in WaitStep
  • Sends a ContainerKill using a non-cancelable context
  • Preserves existing behavior for normal container exit
  • Prevents orphaned containers after pipeline cancellation

Why this approach

Docker containers do not automatically stop when the caller context is canceled.
Explicitly killing the container on ctx.Done() guarantees correct cleanup and matches expected CI behavior.

Implementation overview

sequenceDiagram
    participant Engine as Pipeline Engine
    participant Docker as Docker Backend
    participant Ctx as Step Context
    participant C as Container

    Engine->>Docker: StartStep
    Docker->>C: docker run
    Docker->>Docker: WaitStep (ContainerWait)

    alt Pipeline canceled
        Ctx-->>Docker: ctx.Done()
        Docker->>C: ContainerKill (SIGKILL)
        Docker->>Docker: ContainerWait returns
    else Step finishes normally
        C-->>Docker: container exits
    end

    Docker-->>Engine: Step state (exit code)
Loading

How it was tested

  • Manually reproduced the issue with long-running containers
  • Verified that containers remain running without this fix
  • Verified that containers are killed immediately after cancellation with this fix
  • Ran go test ./pipeline/backend/docker

Test pipeline configuration

To reproduce the issue and validate the fix, a minimal Woodpecker pipeline was used with skip_clone: true and a long-running step.

skip_clone: true

steps:
  sleep_30s:
    image: bash
    commands:
      - echo wait 30s
      - sleep 30s
      - echo wait done

Terminal Output

The behavior was validated using a long-running Docker container to simulate a pipeline step that does not exit on its own.

Before the fix

  • Canceling the step would stop waiting, but the container kept running.

After the fix

  • Canceling the step immediately terminates the running container.

Example manual verification:

image

@6543 6543 added bug Something isn't working backend/docker labels Jan 26, 2026
@codecov
Copy link

codecov bot commented Jan 26, 2026

Codecov Report

❌ Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 21.86%. Comparing base (3786077) to head (1b35568).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
pipeline/backend/docker/docker.go 0.00% 15 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6018      +/-   ##
==========================================
- Coverage   21.88%   21.86%   -0.02%     
==========================================
  Files         432      432              
  Lines       39241    39254      +13     
==========================================
- Hits         8586     8581       -5     
- Misses      29847    29865      +18     
  Partials      808      808              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@6543 6543 merged commit b1cbd96 into woodpecker-ci:main Jan 26, 2026
7 of 9 checks passed
@6543 6543 changed the title Fix/docker kill container on cancel Fix docker backend to kill container on cancel Jan 26, 2026
@woodpecker-bot woodpecker-bot mentioned this pull request Jan 26, 2026
1 task
@6543 6543 changed the title Fix docker backend to kill container on cancel Docker backend kills running containers if agent gets killed Agent now Jan 28, 2026
@6543 6543 changed the title Docker backend kills running containers if agent gets killed Agent now Fix: Agent now gracefully handles running containers when killed Jan 28, 2026
6543 added a commit to 6543-forks/woodpecker that referenced this pull request Jan 28, 2026
@6543
Copy link
Member

6543 commented Jan 28, 2026

hotfix removed in #6011 as solved propperly

(as per #6018 (comment))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/docker bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent does not shutdown docker backend propperly

2 participants