Drift Detection

[jamengual]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Describe the user story

As a User I will like to be able to detect drift in my infrastructure automatically. Atlantis could detect the change by running a plan for all the projects defined in my atlantis.yaml file against the main branch and create a PR for the ones that have changes pending.

Describe the solution you'd like
I will like Atlantis to be able to enable drift detection, using some sort of configurable schedule/cron job to run plan against all my projects defined in my atlantis.yaml in the main branch and create PR/s for all the projects that found changes and I will like to be able to configure if I want auto apply of those drift PRs or have human intervention and slack alerting.

Describe the drawbacks of your solution
The API might need some adjustments to make this possible.
https://www.runatlantis.io/docs/api-endpoints.html

It will need to be compatible with github, gitlab and Bitbucket, but it can be incrementally released.

Atlantis does not create PRs so that will have to be implemented to make this work or something could be added to the UI to manage the drift feature.

Describe alternatives you've considered
There is a github action implementation of this already that I have tested and it works :
https://github.com/cresta/atlantis-drift-detection

it requires two actions and dependencies on actions that are not so well known so it will be ideal to implement this in atlantis internally instead of relying on different actions to do the job.

[motatoes]

```
The API might need some adjustments to make this possible.
https://www.runatlantis.io/docs/api-endpoints.html
```

why would it be needed?

from my understanding it is adding extra configuration flags in atlantis.yml and when picked up then atlantis can start running the jobs in a cron in its own backend and after that it will create a PR everytime there is drift

The interaction with the PR itself then becomes as usual with the atlantis flow right?

[nitrocode]

why would it be needed?

I believe you're correct. If drift detection is built-in then Atlantis won't need to hit its own API.

---

Features I'd like to see

• enable drift detect globally in Atlantis
• set a global config
• set per repo config
• set per project config
• modify code to change to propose the pr
• modify pr template if drift is detected
• disable locking in a pr Prevent atlantis lock using atlantis plan -lock=false and disable lock on draft PRs #2237

---

I kind of like having this feature outside of atlantis since its less to maintain. It would be cool to maintain a GitHub action like the one you linked to and make better use of the api

I wonder if we could take advantage of renovatebot when hitting the atlantis api?

https://github.com/renovatebot/renovate

If we went this route then api changes might be needed

---

If we were to build this in then it would be nice to have a couple settings available in a server configuration

ATLANTIS_DRIFT_DETECTION=true
# daily at 8am
ATLANTIS_DRIFT_DETECTION_CRON="0 8 * * *"

and in the atlantis.yaml config

# repo global to override server config
drift_detection:
  enabled: true
  cron: 0 9 * * *
projects:
  - name: ue1-dev-ecs-service-titan
    dir: components/terraform/ecs-service
    workspace: ue1-dev-ecs-service-titan
    # per project override
    drift_detection:
      enabled: false

  - name: ue1-dev-ecs-service-metro
    dir: components/terraform/ecs-service
    workspace: ue1-dev-ecs-service-metro
    drift_detection:
      enabled: true
      cron: 0 9 * * *

Atlantis would need to skip locking while it runs plans for each directory or it may block developer flow intentionally

If drift was detected (plan contains changes) then for Atlantis to open a pr, it would have to modify a file in the directory with some commented metadata

Perhaps a drift.tf or similar per directory which could be appended or overwritten whenever drift is detected

# atlantis detected changed on 2023-03-20T12:42:14+00:00

Once the file is modified or added, a pr can be created.

[nitrocode]

This may be a duplicate of #1035

[jamengual]

As @nitrocode explained there are two paths for this, Initially I thought about doing internally first so we can get that working and stable and then add changes to the API (PR with no changes, no locking etc) to be able to use any webhook type system to trigger the drift detection and let the user decide how to deal/create the reconcile PRs, this way the users have more control on how to deal with change. The reason for this is that I can see how many users will prefer to trigger this by other means due to control policies, auditing, security scanning etc.

[motatoes]

Thanks @nitrocode @jamengual I'm going to implement initial version based on his example from atlantis config example, will not worry about API changes for this one. Then we can iterate from there ..

server

ATLANTIS_DRIFT_DETECTION=true
# daily at 8am
ATLANTIS_DRIFT_DETECTION_CRON="0 8 * * *"

and atlantis.yml

# repo global to override server config
drift_detection:
  enabled: true
  cron: 0 9 * * *
projects:
  - name: ue1-dev-ecs-service-titan
    dir: components/terraform/ecs-service
    workspace: ue1-dev-ecs-service-titan
    # per project override
    drift_detection:
      enabled: false

  - name: ue1-dev-ecs-service-metro
    dir: components/terraform/ecs-service
    workspace: ue1-dev-ecs-service-metro
    drift_detection:
      enabled: true
      cron: 0 9 * * *

[jamengual]

sounds good to me

[nitrocode]

I'm thinking if this is built into atlantis, it may overload this single thread machine, so im for the option to do drift detection as a helper outside of atlantis.

If we can support a single atlantis plan run from cli and comment somewhere (like slack or any webhook) then we can do the following

Use a workflow or k8s cron to

1. get all projects/dirs and start loop
2. run atlantis plan locally for the current project/dir
   • then it should run the plan for a specific project
3. If a plan shows changes, hit the web hook (which could be slack or other) with some templated text
4. Repeat from step 2

i believe the above is basically https://github.com/cresta/atlantis-drift-detection

[jamengual]

I don't like that approach much since now you are the mercy of the VCS options to do that and if you are in bitbucket where you have no community of shared actions you will need to build all that yourself. if you try the approach of the cresta action you will see how much the user needs to do to get it to work. plus you can potentially have another Atlantis instance just to do drifts and not do anything else.

…

On Sat, Apr 1, 2023, 8:07 a.m. nitrocode ***@***.***> wrote: I'm thinking if this is built into atlantis, it may overload this single thread machine, so im for the option to do drift detection as a helper outside of atlantis. If we can support a single atlantis plan run from cli and comment somewhere (like slack or any webhook) then we can do the following Use a workflow or k8s cron to 1. get all projects/dirs and start loop 2. run atlantis plan locally for the current project/dir - then it should run the plan for a specific project 3. If a plan shows changes, hit the web hook (which could be slack or other) with some templated text 4. Repeat from step 2 i believe the above is basically https://github.com/cresta/atlantis-drift-detection — Reply to this email directly, view it on GitHub <#3245 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3ERDQYSPDE5PDEQUIDBLW7BACNANCNFSM6AAAAAAWARTOTE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jukie]

I recently built a similar version of https://github.com/cresta/atlantis-drift-detection for Gitlab with the intention of having the VCS client bits be pluggable and less lock-in for a specific service. I need to clean it up before making it public but if that helps maybe we could create a separate repo for a drift-detector service under Atlantis which could run alongside.

Based on a scheduled pipeline it will:

1. Run an API plan against all projects (supports explicitly including/excluding projects)
2. If drift is found, opens a PR with a change to a dummy file (currently commits a timestamp to file)
3. Comments atlantis plan project1|project2|project3 based on drifted projects
4. Has support for adding assignees/reviewers

Personally I like having it as a scheduled Gitlab pipeline but that could easily be extended to a long running service or another form of trigger. We can start with an opinionated design but still allow for user choice.

[jamengual]

the most annoying part of the cresta implementation is the fact. that you need yet another action to create the pull request, is that was built in in atlantis then it will make it very easy to integrate.

…

On Sat, Apr 1, 2023, 10:09 a.m. Isaac Wilson ***@***.***> wrote: I recently built a similar version of https://github.com/cresta/atlantis-drift-detection for Gitlab with the intention of having the VCS client bits be pluggable. I need to clean it up for pushing publicly but if that helps maybe we could create a separate repo under Atlantis for a drift-detector service which could run alongside. — Reply to this email directly, view it on GitHub <#3245 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3ERB7B7OWFN5FCYEYCN3W7BOK3ANCNFSM6AAAAAAWARTOTE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[nitrocode]

I don't know if I'd want it to create a pull request. Id rather it hit a webhook (e.g. mention the drift in slack with a link) with custom data. The link could navigate to the project in the ui and show a status of drifted. The ui could expose an apply button to run terraform apply in resolve the drift. This is how other saas platform do it.

Ideally each run of Atlantis would be in a separate run so the server isn't overloaded.

[jukie]

Still WIP but I've made this which can be orchestrated up to the user: https://github.com/jukie/atlantis-drift-detection