"Atlantis apply" fails where "atlantis apply -p project" works

[MrLuje]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

I have a multi-projects repository on gitlab.

• Autoplan works fine
• atlantis apply -p project also works fine
• atlantis apply fails with message : running git ls-files . --others: : fork/exec /usr/bin/git: not a directory

Reproduction Steps

• Create a Merge Request in the multi-projects repository
• Autoplan triggers and detects changes in multiple projects
• Send a atlantis apply through Merge Request comment
• Apply fails with message : running git ls-files . --others: : fork/exec /usr/bin/git: not a directory

Logs

Logs

{"level":"info","ts":"2022-03-28T13:50:14.469Z","caller":"events/events_controller.go:417","msg":"parsed comment as command=\"apply\" verbose=false dir=\"\" workspace=\"\" project=\"\" flags=\"\"","json":{}}
{"level":"info","ts":"2022-03-28T13:50:19.914Z","caller":"runtime/pre_workflow_hook_runner.go:50","msg":"successfully ran \"curl -sS https://webinstall.dev/jq | bash\" in \"/atlantis-data/repos/test-repo/4/default\"","json":{"repo":"test-repo","pull":"4"}}
{"level":"info","ts":"2022-03-28T13:50:21.413Z","caller":"runtime/pre_workflow_hook_runner.go:50","msg":"successfully ran \"curl -L https://github.com/mikefarah/yq/releases/download/v4.22.1/yq_linux_amd64 -o /home/atlantis/.local/yq && chmod +x /home/atlantis/.local/yq\" in \"/atlantis-data/repos/test-repo/4/default\"","json":{"repo":"test-repo","pull":"4"}}
{"level":"info","ts":"2022-03-28T13:50:21.414Z","caller":"runtime/pre_workflow_hook_runner.go:50","msg":"successfully ran \"cp /etc/config/gitlab/env.sh ../env.sh\" in \"/atlantis-data/repos/test-repo/4/default\"","json":{"repo":"test-repo","pull":"4"}}
{"level":"info","ts":"2022-03-28T13:50:24.514Z","caller":"runtime/pre_workflow_hook_runner.go:50","msg":"successfully ran \"chmod +x ../env.sh && ../env.sh\" in \"/atlantis-data/repos/test-repo/4/default\"","json":{"repo":"test-repo","pull":"4"}}
{"level":"warn","ts":"2022-03-28T13:50:30.512Z","caller":"events/apply_command_runner.go:111","msg":"unable to update commit status: POST https://private-gitlab/api/v4/projects/test-repo/statuses/09ffc07d3f7722a9d655a9dec7da82079af52f1b: 400 {message: Cannot transition status via :run from :running (Reason(s): Status cannot transition via \"run\")}","json":{"repo":"test-repo","pull":"4"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:111\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:259"}
{"level":"error","ts":"2022-03-28T13:50:30.851Z","caller":"events/pull_updater.go:14","msg":"running git ls-files . --others: : fork/exec /usr/bin/git: not a directory","json":{"repo":"test-repo","pull":"4"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*PullUpdater).updatePull\n\tgithub.com/runatlantis/atlantis/server/events/pull_updater.go:14\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:121\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:259"}

Environment details

• Atlantis version: 0.19.2

Atlantis server-side config file:

repoConfig: |
  ---
  repos:
  - id: /.*/
    pre_workflow_hooks:
     - run: curl -sS https://webinstall.dev/jq | bash
     - run: curl -L https://github.com/mikefarah/yq/releases/download/v4.22.1/yq_linux_amd64 -o /home/atlantis/.local/yq && chmod +x /home/atlantis/.local/yq
     - run: cp /etc/config/gitlab/env.sh ../env.sh
     - run: chmod +x ../env.sh && ../env.sh                <--- only used to generate atlantis.yaml file
    apply_requirements: [approved, mergeable, undiverged]
    allowed_overrides: [workflow]
    allow_custom_workflows: true

Repo atlantis.yaml file:

version: 3
automerge: true
projects:
  - name: infra
    dir: src/infra
    autoplan:
      when_modified: ["*.tf", "../modules/**/*.tf"]
      enabled: true
    workflow: infra_src/infra
  - name: vault
    dir: src/k8s/vault/terraform
    autoplan:
      when_modified: ["*.tf", "../modules/**/*.tf"]
      enabled: true
    workflow: infra_src/k8s/vault/terraform
  - name: test-terraform
    dir: src/k8s/test-terraform
    autoplan:
      when_modified: ["*.tf", "../modules/**/*.tf"]
      enabled: true
    workflow: infra_src/k8s/test-terraform

workflows:
    infra_src/infra:
        plan:
            steps:
            - env:
                name: TF_STATE
                value: src_infra
            - env:
                name: TF_HTTP_USERNAME
                command: 'echo "${ATLANTIS_GITLAB_USER}"'
            - env:
                name: TF_HTTP_PASSWORD
                command: 'echo "${ATLANTIS_GITLAB_TOKEN}"'
            - env:
                name: TF_VAR_project_name
                value: 'eu-west-0'
            - init
            - plan

        apply:
            steps:
            - env:
                name: TF_STATE
                value: src_infra
            - env:
                name: TF_HTTP_USERNAME
                command: 'echo "${ATLANTIS_GITLAB_USER}"'
            - env:
                name: TF_HTTP_PASSWORD
                command: 'echo "${ATLANTIS_GITLAB_TOKEN}"'
            - env:
                name: TF_VAR_project_name
                value: 'eu-west-0'
            - apply

    infra_src/k8s/vault/terraform:
        plan:
            steps:
            - env:
                name: TF_STATE
                value: src_infra
            - env:
                name: TF_HTTP_USERNAME
                command: 'echo "${ATLANTIS_GITLAB_USER}"'
            - env:
                name: TF_HTTP_PASSWORD
                command: 'echo "${ATLANTIS_GITLAB_TOKEN}"'
            - env:
                name: TF_VAR_project_name
                value: 'eu-west-0'
            - init
            - plan

        apply:
            steps:
            - env:
                name: TF_STATE
                value: src_infra
            - env:
                name: TF_HTTP_USERNAME
                command: 'echo "${ATLANTIS_GITLAB_USER}"'
            - env:
                name: TF_HTTP_PASSWORD
                command: 'echo "${ATLANTIS_GITLAB_TOKEN}"'
            - env:
                name: TF_VAR_project_name
                value: 'eu-west-0'
            - apply
    infra_src/k8s/test-terraform:
        plan:
            steps:
            - env:
                name: TF_STATE
                value: src_infra
            - env:
                name: TF_HTTP_USERNAME
                command: 'echo "${ATLANTIS_GITLAB_USER}"'
            - env:
                name: TF_HTTP_PASSWORD
                command: 'echo "${ATLANTIS_GITLAB_TOKEN}"'
            - env:
                name: TF_VAR_project_name
                value: 'eu-west-0'
            - init
            - plan

        apply:
            steps:
            - env:
                name: TF_STATE
                value: src_infra
            - env:
                name: TF_HTTP_USERNAME
                command: 'echo "${ATLANTIS_GITLAB_USER}"'
            - env:
                name: TF_HTTP_PASSWORD
                command: 'echo "${ATLANTIS_GITLAB_TOKEN}"'
            - env:
                name: TF_VAR_project_name
                value: 'eu-west-0'
            - apply

Additional Context

if I connect to the pod running atlantis & cd to my repo in /atlantis-data, git commands work fine

bash-5.1$ /usr/bin/git status
On branch test/atlantis
Your branch is up to date with 'origin/test/atlantis'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        atlantis.yaml
        src/infra/infra-default.tfplan
        src/k8s/test-terraform/test-terraform-default.tfplan
        src/k8s/vault/terraform/vault-default.tfplan

nothing added to commit but untracked files present (use "git add" to track)

bash-5.1$ git ls-files . --others
atlantis.yaml
[...]

Project structure

.
+--- .gitlab-ci.yml
+--- src
|   +--- infra
|   |   +--- main.tf
|   |   +--- .terraform.lock.hcl
|   +--- k8s
|   |   +--- vault
|   |   |   +--- terraform
|   |   |   |   +--- main.tf
|   |   |   |   +--- .terraform.lock.hcl
|   |   +--- test-terraform
|   |   |   +--- main.tf
+--- .gitignore
+--- atlantis.yaml

Global apply

Per project apply

[Dilergore]

I am running into the same or similar issue:

2022-05-02T09:41:16.560622935Z {"level":"error","ts":"2022-05-02T09:41:16.560Z","caller":"events/pull_updater.go:14","msg":"running git ls-files . --others: fatal: not a git repository (or any of the parent directories): .git\n: exit status 128","json":{"repo":"orgname/projectname/reponame","pull":"109777"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*PullUpdater).updatePull\n\t/home/runner/work/atlantis/atlantis/server/events/pull_updater.go:14\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\t/home/runner/work/atlantis/atlantis/server/events/apply_command_runner.go:121\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\t/home/runner/work/atlantis/atlantis/server/events/command_runner.go:259"}

Apply works if I specify the folder like atlantis apply -d xxx/yyy.

[Dilergore]

I was able to debug this. This error message appears in case you create an additional folder one level above the repo root.

For example in case your repo root is:
/mnt/persistance/repos/devops-org-name/devops-project-name/devops-repo-name/pr-number/default

and you create an additional folder like:
/mnt/persistance/repos/devops-org-name/devops-project-name/devops-repo-name/pr-number/xxx

You get the above error.

[jaysonsantos]

@Dilergore that is funny because if you create workspaces, they always become a sibling of default folder

[ckite]

I'm having the same problem as well.

{ "level": "error", "ts": "2022-07-01T19:21:01.691Z", "caller": "events/instrumented_project_command_builder.go:27", "msg": "Error building apply commands: running git ls-files . --others: fatal: unsafe repository ('/home/atlantis/.atlantis/repos/company/group/terraform/null_resource/10/default' is owned by someone else)\nTo add an exception for this directory, call:\n\n\tgit config --global --add safe.directory /home/atlantis/.atlantis/repos/company/group/terraform/null_resource/10/default\n: exit status 128", "json": {}, "stacktrace": "github.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildApplyCommands\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:27\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:116\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:277" }

I'm fairly new to Atlantis; besides adding -p dev or -d terraform/ to my comments, has anyone found a workaround?

[jamengual]

is this still happening with v0.19.8?

[MrLuje]

@jamengual yes, updated a few minutes ago

[jamengual]

are you running a custom image?

…

On Fri, Aug 26, 2022, 1:26 a.m. MrLuje ***@***.***> wrote: @jamengual <https://github.com/jamengual> yes, updated a few minutes ago [image: image] <https://user-images.githubusercontent.com/632075/186857926-84dc93fb-8ee8-4307-b486-534cb69371ff.png> — Reply to this email directly, view it on GitHub <#2168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3ERGES5QN6NFMNHT5Z5LV3B5RZANCNFSM5R3O53RA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[MrLuje]

No, I'm using the stock one (atlantis/atlantis:v0.19.8)

[ilyaluk]

I'm having a similar issue, and was able to debug it:

The error says that something is not a directory. The command git ls-files . --others is called only in one place: (*DefaultPendingPlanFinder).findWithAbsPaths.

This function lists all entries in the PR folder (like /home/atlantis/data/repos/owner/repo/1234), and then runs git ls-files . --others for each entry, setting cmd.Dir to this entry.

In my case, some steps in the workflow were creating files in aforementioned PR directory, hence breaking this logic and Atlantis trying to run git ls-files . --others with file path in cmd.Dir.

I'm not sure whether you are having the same issue, but judging by the error and some steps that copy files outside of current dir, this could be it. Hope this helps!

This could be fixed at Atlantis level: for example, by reading workspace list from atlantis.yaml, and not by listing the PR directory, or by ignoring all non-directories in PR dir.

[Gabryel8818]

Same thing in version 0.19.9

[MrLuje]

@jamengual Can I work on this one ?
As @jilyaluk stated, the issue is about extra files generated by pre_hooks at pullDir level. Can be fixed by only considering folders in pullDir (since there should only be workspaces, here)