v0.22.2+: `atlantis/post_workflow_hook` errors after `apply` when using `--automerge` flag due to deleted directory

[Yasmine92]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

When using the flag --automerge, the Postworkflow hooks are executed after the PR is merged (the PR directory is deleted, and the branch is deleted). which results in errors in logs about not being able to read the current directory, or fetching from origin.

Reproduction Steps

• enable automerging mode, by adding the flag --automerge
• add a post-workflow-hook
• open a Pull request, and run atlantis apply

Logs

Logs

{"level":"info","ts":"2023-01-24T09:58:33.477Z","caller":"events/events_controller.go:542","msg":"parsed comment as command=\"apply\" verbose=false dir=\"\" workspace=\"\" project=\"\" flags=\"\"","json":{"gh-request-id":"X-Github-Delivery=a9221ee0-9bcd-11ed-9d8e-31769df0a119"}}
{"level":"info","ts":"2023-01-24T09:58:37.115Z","caller":"terraform/terraform_client.go:317","msg":"Cannot determine which version to use from terraform configuration, detected 2 possibilities.","json":{"repo":"example/sandbox-project","pull":"19"}}
{"level":"info","ts":"2023-01-24T09:58:37.116Z","caller":"terraform/terraform_client.go:317","msg":"Cannot determine which version to use from terraform configuration, detected 2 possibilities.","json":{"repo":"example/sandbox-project","pull":"19"}}
{"level":"info","ts":"2023-01-24T09:58:37.460Z","caller":"runtime/apply_step_runner.go:39","msg":"starting apply","json":{"repo":"example/sandbox-project","pull":"19"}}
{"level":"info","ts":"2023-01-24T09:58:41.988Z","caller":"models/shell_command_runner.go:156","msg":"successfully ran \"/usr/local/bin/terraform apply -input=false \\\"/atlantis-data/repos/example/sandbox-project/19/dev/terraform/dev-dev.tfplan\\\"\" in \"/atlantis-data/repos/example/sandbox-project/19/dev/terraform\"","json":{"repo":"example/sandbox-project","pull":"19"}}
{"level":"info","ts":"2023-01-24T09:58:41.989Z","caller":"runtime/apply_step_runner.go:58","msg":"apply successful, deleting planfile","json":{"repo":"example/sandbox-project","pull":"19"}}
{"level":"info","ts":"2023-01-24T09:58:42.363Z","caller":"events/instrumented_project_command_runner.go:82","msg":"apply success. output available at: https://github.com/example/sandbox-project/pull/19","json":{"repo":"example/sandbox-project","pull":"19"}}
{"level":"info","ts":"2023-01-24T09:58:45.244Z","caller":"events/automerger.go:32","msg":"automerging pull request","json":{"repo":"example/sandbox-project","pull":"19"}}
{"level":"info","ts":"2023-01-24T09:58:47.986Z","caller":"events/instrumented_pull_closed_executor.go:45","msg":"Initiating cleanup of pull data.","json":{"repository":"example/sandbox-project","pull-num":"19"}}
{"level":"warn","ts":"2023-01-24T09:58:48.154Z","caller":"events/working_dir.go:168","msg":"getting remote update failed: Fetching origin\nerror: cannot open '.git/FETCH_HEAD': No such file or directory\nerror: could not fetch origin\nFetching head\nfatal: Unable to read current working directory: No such file or directory\nerror: could not fetch head\nfatal: Unable to read current working directory: No such file or directory\n","json":{"repo":"example/sandbox-project","pull":"19"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*FileWorkspace).warnDiverged\n\tgithub.com/runatlantis/atlantis/server/events/working_dir.go:168\ngithub.com/runatlantis/atlantis/server/events.(*FileWorkspace).Clone\n\tgithub.com/runatlantis/atlantis/server/events/working_dir.go:117\ngithub.com/runatlantis/atlantis/server/events.(*DefaultPostWorkflowHooksCommandRunner).RunPostHooks\n\tgithub.com/runatlantis/atlantis/server/events/post_workflow_hooks_command_runner.go:69\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:298"}
{"level":"info","ts":"2023-01-24T09:58:49.011Z","caller":"events/events_controller.go:470","msg":"deleted locks and workspace for repo example/sandbox-project, pull 19","json":{"gh-request-id":"X-Github-Delivery=b183cfc0-9bcd-11ed-9de7-c11f64f1d767"}}
{"level":"error","ts":"2023-01-24T09:58:49.101Z","caller":"events/command_runner.go:301","msg":"Error running post-workflow hooks chdir /atlantis-data/repos/example/sandbox-project/19/default: no such file or directory: running \"ls -l /etc/atlantis/repos.yaml\" in \"/atlantis-data/repos/example/sandbox-project/19/default\": \n.","json":{"repo":"example/sandbox-project","pull":"19"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:301"}

Environment details

• Atlantis version: 0.22.3
• Atlantis flags: --enable-diff-markdown-format --automerge --checkout-strategy=merge --hide-prev-plan-comments --enable-policy-checks --gh-allow-mergeable-bypass-apply --quiet-policy-checks

post-worklflow hook configuration from the server config:

post_workflow_hooks:
   - run: ls -l /etc/atlantis/repos.yaml

Repo atlantis.yaml file:

---
# atlantis.yaml
version: 3
parallel_plan: true
projects:
- name: dev
  dir: terraform
  autoplan:
    when_modified: ["*.tf*"]
  workflow: dev
  workspace: dev
workflows:
  dev:
    plan:
      steps:
      - init:
      - plan:
          extra_args: ["-var-file", "environments/dev.tfvars"]

Additional Context

[nitrocode]

Could you include your yaml configuration such as your post workflow hook?

Looks like this is the main error

Error running post-workflow hooks chdir /atlantis-data/repos///7/default: no such file or directory: running "rm -rf /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM"

[jamengual]

it looks like you have an additional / in the path on that command

…

On Mon, Jan 23, 2023, 6:27 a.m. nitrocode ***@***.***> wrote: Could you include your yaml configuration such as your post workflow hook? Looks like this is the main error Error running post-workflow hooks chdir /atlantis-data/repos///7/default: no such file or directory: running "rm -rf /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM" — Reply to this email directly, view it on GitHub <#3031 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3ERG3EO4DB4IPMWUUHQTWT2IMBANCNFSM6AAAAAAUD2FUUA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Yasmine92]

/atlantis-data/repos///7/default:

no, I just tried to hide the repo and organisation name and it resulted in that 🤦. I'll update the issue with a different log, using a simple post workflow hook

[Yasmine92]

@jamengual and @nitrocode I updated the issue with another simpler example (different and simpler post workflow), and the log from the apply time.

[nitrocode]

@Yasmine92 can you ssh to the pod and try to access the directory it's complaining about? Can you cd to it?

cd /atlantis-data/repos/example/sandbox-project/19/default

[Yasmine92]

@Yasmine92 can you ssh to the pod and try to access the directory it's complaining about? Can you cd to it?

cd /atlantis-data/repos/example/sandbox-project/19/default

Well from my understanding, the directory for the PR is cleaned up after the automerge at this step:

{"level":"info","ts":"2023-01-24T09:58:45.244Z","caller":"events/automerger.go:32","msg":"automerging pull request","json":{"repo":"example/sandbox-project","pull":"19"}}
{"level":"info","ts":"2023-01-24T09:58:47.986Z","caller":"events/instrumented_pull_closed_executor.go:45","msg":"Initiating cleanup of pull data.","json":{"repository":"example/sandbox-project","pull-num":"19"}}

so it's normal that the directory is already gone after merge:

$ ls /atlantis-data/repos/example/sandbox-project/19           
ls: /atlantis-data/repos/example/sandbox-project/19: No such file or directory

[Yasmine92]

but the thing is that events/working_dir.go is called before running the post-workflow-hook, so it always try to cd to the directory for the PR (that is already merged) before executing the post-workflow action that is not necessarily related to that directory, like the example I showed ls -l /etc/atlantis/repos.yaml
So I think it would be better to change the logic of the code to execute the post-workflow-hooks before automerging.

[nitrocode]

@Fabianoshz for your thoughts here.

@Yasmine92 please feel free to test a change locally and propose it. What's odd is that I'm using the latest version, most of the above flags, and not experiencing this. I wonder what condition would make the cleanup happen prior to the post workflow run?

[Yasmine92]

@Fabianoshz for your thoughts here.

@Yasmine92 please feel free to test a change locally and propose it. What's odd is that I'm using the latest version, most of the above flags, and not experiencing this. I wonder what condition would make the cleanup happen prior to the post workflow run?

@nitrocode are you using both --automerge and a post-workflow-hook? how does it look in the logs when you do an apply?
Sure, I'll give it a try :)

[Fabianoshz]

@Yasmine92 can you check if you can access the PR directory?

• This should work: ls /atlantis-data/repos/example/sandbox-project/19/ - This is the PR directory.
• This should not: ls atlantis-data/repos/example/sandbox-project/19/default

Taking from memory I believe the project directory is deleted right after the apply, while the PR directory lives until we receive a merged event. Again, taking from memory I might be wrong.

[Yasmine92]

@Yasmine92 can you check if you can access the PR directory?

• This should work: ls /atlantis-data/repos/example/sandbox-project/19/ - This is the PR directory.
• This should not: ls atlantis-data/repos/example/sandbox-project/19/default

Taking from memory I believe the project directory is deleted right after the apply, while the PR directory lives until we receive a merged event. Again, taking from memory I might be wrong.

Thanks @Fabianoshz , the PR directory /atlantis-data/repos/example/sandbox-project/19/ is deleted after apply, because the merge event comes automatically after apply, because of the flag "--automerge". and that's the exact bug I'm pointing out, using --automerge combined with a post-workflow makes the dir of the Pr deleted before the post-workflow is executed.