Repository sync fetches remote but leaves local branch stale, causing outdated task execution

[Moep90]

Summary

When using a persistent repository checkout on a Semaphore runner, the repository sync mechanism successfully fetches updates from the remote (origin/master advances, FETCH_HEAD is written), but the local branch (master) is not fast-forwarded to match. As a result, tasks execute stale code even though the runner has the latest commits locally.

The runner's internal sync has access to the deploy key (fetch works), but a shell session on the runner does not have the same key, so manual git pull fails. This confirms the bug is specifically in Semaphore's branch-update logic after fetch, not a network/auth problem.

Environment

Component	Value
Semaphore version	v2.18.8^0-459ccee-1780420795
Go version	go1.24.11 (linux/amd64)
Git client	cmd_git
Temp path	/tmp/semaphore
Home dir mode	template_dir
Schedule timezone	UTC
Ansible core	2.20.6
Ansible Python	3.12.13
Jinja	3.1.6
Database dialect	postgres
Runner deployment	Docker-based runner with persistent /tmp/semaphore volume
Repository type	Git repository (GitHub, SSH transport)
Runner checkout path	/tmp/semaphore/project_2/repository_3_template_1/

Steps to Reproduce

1. Push a commit to master that fixes a bug in an Ansible task.
2. Run a Semaphore task template that uses this repository.
3. Observe that the task fails with the old bug (pre-fix behavior).
4. SSH into the runner and check the checkout:

cd /tmp/semaphore/project_2/repository_3_template_1/
git log --oneline --decorate -n 5

Expected Behavior

The local master branch should point to the same commit as origin/master after sync, and the task should execute the latest code.

Actual Behavior

The local master branch remains on an older commit while origin/master has advanced.

Evidence from the runner

$ git log --oneline --decorate -n 5
4aa4863 (HEAD, origin/master, origin/HEAD) fix(validate): update regex
0d9c713 fix(system): ensure parent directory exists before touch
daac53a (master) fix(discover): add path discovery
33e5a1a fix(validate): enhance cert validation
f464412 fix(system): update file module

Notice that:

• origin/master is at 4aa4863 (latest, already fetched)
• The local master branch is at daac53a (2 commits behind)
• The fix commit 0d9c713 exists in the fetched history but master never advanced to include it

Fetch timestamp confirms sync happened:

-rw-r--r-- 1 semaphore root 119 Jun  3 21:37 FETCH_HEAD

The runner fetched updates from the remote at 21:37, but the local branch was not updated.

Manual git pull fails because the shell lacks the deploy key:

$ git checkout master
Switched to branch 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.

$ git pull
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

This is critical: the runner's internal sync can authenticate and fetch, but it does not update the local branch afterward. A manual fix via git pull is impossible in a shell because the shell session lacks the SSH key.

Task failure confirms stale execution:

The task log shows the failure originating from tasks/main.yml:18 — which in the stale checkout points to the old (pre-fix) task. In the latest commit, that same task was rewritten and moved to a different line.

Impact

This makes Semaphore unreliable for CI/CD: a task run after a bugfix push may still execute broken code because the checkout is stale. It is especially dangerous with persistent runner volumes where the repository is reused across task executions.

Workaround

Since git pull is not possible (shell lacks the deploy key), use git reset against the already-fetched remote tracking branch:

cd /tmp/semaphore/project_2/repository_3_template_1/
git checkout master
git reset --hard origin/master

This requires no network access and uses the commits already fetched by Semaphore's internal sync. After this, the Semaphore task passes because it now runs the latest commit.

Alternatively, delete the entire checkout directory and let Semaphore re-clone fresh.

Proposed Fix

Semaphore's repository sync phase should ensure the checked-out branch is fast-forwarded (or hard-reset) to match the fetched remote tracking branch after git fetch completes. Simply running git fetch is insufficient when the local branch is behind.