Skip to content

Gitlab MR pipeline status is not updated #5228

Open
@rhariady

Description

@rhariady

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

In gitlab, when i create 2 MR using 2 difference branch but with same HEAD commit, atlantis will only update MR pipeline status for first MR. This is blocking us when we set gitlab repo to enforce pipeline success before merge.

Reproduction Steps

  • Checkout branch A (head at commit X), and push to gitlab
  • Open a new MR 1 from branch A
  • Run atlantis plan on MR 1
    • Atlantis post a comment with plan result to MR 1
    • Atlantis update MR 1 pipeline status to success
  • Run atlantis unlock from MR 1
  • Checkout another branch B (head also at commit X), and push to gitlab
  • Create another MR 2 from branch B
  • Run atlantis plan on MR 2
    • Atlantis post a comment with plan result to MR 2
    • Atlantis do not update MR 2 pipeline status (it keep showing Checking pipeline status.

image

Logs

Logs
[DBUG] Getting GitLab merge request 84
[DBUG] GET /projects/<repo_url>/merge_requests/84 returned: 200
[INFO] Pre-workflow hooks configured, running...
[DBUG] got workspace lock
[DBUG] clone directory '/home/atlantis/.atlantis/repos/<repo_url>/84/default' already exists, checking if it's at the right commit
[DBUG] repo is at correct commit '888407c1cf7a31adb584c17b0a1bfc6c6efef6df' so will not re-clone
[INFO] Updating GitLab commit status for 'atlantis-nonprod/plan' to 'running'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[DBUG] Processing pre workflow hook 'Generate repo config', Command 'plan', Target commands []
[DBUG] Running pre workflow hook: 'Generate repo config'
[DBUG] Setting shell to default: ''
[DBUG] Setting shellArgs to default: ''
[INFO] Updating GitLab commit status for 'atlantis-nonprod/pre_workflow_hook: Generate repo config' to 'running'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[INFO] Updating GitLab commit status for 'atlantis-nonprod/pre_workflow_hook: Generate repo config' to 'success'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[INFO] Pre-workflow hooks completed successfully
[DBUG] Checking if GitLab merge request 84 is approved
[DBUG] GET /projects/<repo_url>/merge_requests/84/approvals returned: 200
[DBUG] Checking if GitLab merge request 84 is mergeable
[DBUG] GET /projects/<repo_url>/merge_requests/84 returned: 200
[DBUG] GET /projects/42481065 returned: 200
[DBUG] GET /projects/42481065/commits/888407c1cf7a31adb584c17b0a1bfc6c6efef6df/statuses returned: 200
[INFO] Updating GitLab commit status for 'atlantis-nonprod/plan' to 'running'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[WARN] unable to update commit status: POST https://gitlab.com/api/v4/projects/<repo_url>/statuses/888407c1cf7a31adb584c17b0a1bfc6c6efef6df: 400 {message: Cannot transition status via :run from :running (Reason(s): Status cannot transition via "run")}
[DBUG] Building plan command for all affected projects
[DBUG] Getting modified files for GitLab merge request 84
[DBUG] GET projects/<repo_url>/merge_requests/84/changes returned: 200
[DBUG] 1 files were modified in this pull request. Modified files: [nonprod.terraform.tfvars.yaml]
[DBUG] got workspace lock
[DBUG] clone directory '/home/atlantis/.atlantis/repos/<repo_url>/84/default' already exists, checking if it's at the right commit
[DBUG] repo is at correct commit '888407c1cf7a31adb584c17b0a1bfc6c6efef6df' so will not re-clone
[INFO] successfully parsed atlantis.yaml file
[DBUG] moduleInfo for '/home/atlantis/.atlantis/repos/<repo_url>/84/default' (matching '') = map[]
[DBUG] found downstream projects for "nonprod.terraform.tfvars.yaml": []
[DBUG] checking if project at dir "." workspace "nonprod" was modified
[DBUG] file "nonprod.terraform.tfvars.yaml" matched pattern
[INFO] 1 projects are to be planned based on their when_modified config
[DBUG] determining config for project at dir: '.' workspace: 'nonprod'
[DBUG] MergeProjectCfg started
[DBUG] setting import_requirements: [approved,mergeable,undiverged] from repos[1], id: /gitlab\.com\/<repo_group>\/.*/
[DBUG] setting repo_locks: this is a bug from default server config
[DBUG] setting plan_requirements: [undiverged] from repos[1], id: /gitlab\.com\/<repo_group>\/.*/
[DBUG] setting workflow: "default" from default server config
[DBUG] setting allowed_overrides: [workflow] from repos[1], id: /gitlab\.com\/<repo_group>\/.*/
[DBUG] setting allow_custom_workflows: false from default server config
[DBUG] setting delete_source_branch_on_merge: true from repos[1], id: /gitlab\.com\/<repo_group>\/.*/
[DBUG] setting policy_check: false from default server config
[DBUG] setting custom_policy_check: false from default server config
[DBUG] setting apply_requirements: [approved,mergeable,undiverged] from repos[1], id: /gitlab\.com\/<repo_group>\/.*/
[DBUG] overriding server-defined workflow with repo-specified workflow: "bpv1"
[DBUG] MergeProjectCfg completed
[DBUG] final settings: plan_requirements: [undiverged], apply_requirements: [approved,mergeable,undiverged], import_requirements: [approved,mergeable,undiverged], workflow: bpv1, delete_source_branch_on_merge: true, repo_locking: on_plan, policy_check: false, custom_policy_check: false, silence_pr_comments: []
[DBUG] Building project command context for plan
[INFO] cannot determine which version to use from terraform configuration, detected 0 possibilities.
[DBUG] deleting previous plans and locks
[INFO] Updating GitLab commit status for 'atlantis-nonprod/plan: nonprod' to 'running'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[INFO] Acquired lock with id '<repo_url>/./nonprod'
[DBUG] acquired lock for project
[DBUG] clone directory '/home/atlantis/.atlantis/repos/<repo_url>/84/nonprod' already exists, checking if it's at the right commit
[DBUG] repo is at correct commit '888407c1cf7a31adb584c17b0a1bfc6c6efef6df' so will not re-clone
[INFO] Updating GitLab commit status for 'atlantis-nonprod/plan: nonprod' to 'success'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[INFO] plan success. output available at: https://gitlab.com/<repo_url>/-/merge_requests/84
[DBUG] hiding previous plan comments for command: 'Plan', directory: ''
[DBUG] Hiding previous command comments on GitLab merge request 84
[DBUG] /projects/<repo_url>/merge_requests/84/notes
[DBUG] GET /projects/<repo_url>/merge_requests/84/notes returned: 200
[DBUG] Updating merge request note: Repo: '<repo_url>', MR: '84', comment ID: '2290094251'
[DBUG] PUT /projects/<repo_url>/merge_requests/84/notes/2290094251 returned: 200

PS: I have retract some information from the log

Environment details

  • Atlantis version: v0.32.0 (commit: ea838ac) (build date: 2024-12-20T02:51:00.531Z)
  • Deployment method: docker on google cloud run
  • If not running the latest Atlantis version have you tried to reproduce this issue on the latest version:
  • Atlantis flags:

Atlantis server-side config file:

ATLANTIS_WEB_BASIC_AUTH=true
ATLANTIS_AUTOMERGE=true
ATLANTIS_EMOJI_REACTION=trident
ATLANTIS_REPO_CONFIG=/app/atlantis/cfg/atlantis-cfg-repos/atlantis-cfg-repos.yaml
ATLANTIS_HIDE_PREV_PLAN_COMMENTS=true
ATLANTIS_SILENCE_NO_PROJECTS=true
ATLANTIS_FAIL_ON_PRE_WORKFLOW_HOOK_ERROR=true
ATLANTIS_VCS_STATUS_NAME=atlantis-nonprod
ATLANTIS_CONFIG=/app/atlantis/cfg/atlantis-cfg-config/atlantis-cfg-config.yaml
ATLANTIS_WRITE_GIT_CREDS=true
ATLANTIS_CHECKOUT_STRATEGY=merge
ATLANTIS_ENABLE_DIFF_MARKDOWN_FORMAT=true
ATLANTIS_SILENCE_VCS_STATUS_NO_PLANS=true

Additional Context

I try to investigate from the log, and it seems this bug is happen because atlantis found out that there was another pipeline for the commit X already (despite it's from another branch), and will try to use it instead of creating another one for different MR.

for i := 0; i <= retries; i++ {
commit, resp, err = g.Client.Commits.GetCommit(repo.FullName, pull.HeadCommit, nil)
if resp != nil {
logger.Debug("GET /projects/%s/repository/commits/%d: %d", pull.BaseRepo.ID(), pull.HeadCommit, resp.StatusCode)
}
if err != nil {
return err
}
if commit.LastPipeline != nil {
logger.Info("Pipeline found for commit %s, setting pipeline ID to %d", pull.HeadCommit, commit.LastPipeline.ID)
// Set the pipeline ID to the last pipeline that ran for the commit
setCommitStatusOptions.PipelineID = gitlab.Ptr(commit.LastPipeline.ID)
break
}
if i != retries {
logger.Info("No pipeline found for commit %s, retrying in %s", pull.HeadCommit, delay)
time.Sleep(delay)
} else {
// If we've exhausted all retries, set the Ref to the branch name
logger.Info("No pipeline found for commit %s, setting Ref to %s", pull.HeadCommit, pull.HeadBranch)
setCommitStatusOptions.Ref = gitlab.Ptr(pull.HeadBranch)
}
}

I think we can fix this by checking if the last pipeline was indeed from the same ref first. If the last pipeline was from different branch (ref), then try to create one instead.

example Gitlab API GET /projects/%s/repository/commits/%d response

{
  "id": "6104942438c14ec7bd21c6cd5bd995272b3faff6",
  "short_id": "6104942438c",
  "title": "Sanitize for network graph",
  "author_name": "randx",
  "author_email": "[email protected]",
  "committer_name": "Dmitriy",
  "committer_email": "[email protected]",
  "created_at": "2021-09-20T09:06:12.300+03:00",
  "message": "Sanitize for network graph",
  "committed_date": "2021-09-20T09:06:12.300+03:00",
  "authored_date": "2021-09-20T09:06:12.420+03:00",
  "parent_ids": [
    "ae1d9fb46aa2b07ee9836d49862ec4e2c46fbbba"
  ],
  "last_pipeline" : {
    "id": 8,
    "ref": "main",
    "sha": "2dc6aa325a317eda67812f05600bdf0fcdc70ab0",
    "status": "created"
  },
  "stats": {
    "additions": 15,
    "deletions": 10,
    "total": 25
  },
  "status": "running",
  "web_url": "https://gitlab.example.com/janedoe/gitlab-foss/-/commit/6104942438c14ec7bd21c6cd5bd995272b3faff6"
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions