[Feature request] Update a stack through webhook/deployment tokens

[Beanow]

Is your feature request related to a problem? Please describe.
Currently I am managing stacks through a CI tool to build new images, and calling PUT /stacks/{id} to take the stack's compose file from the git repository that triggered this build and deploy it.

This requires some steps to set up, particularly creating a new user, granting them access, and knowing the stack ID. The flow would be to obtain a JWT for this user then PUT the stack.

Describe the solution you'd like
Similar to how webhooks currently work to restart services, being able to generate a token for this specific stack and operation will make administration a lot simpler by removing the need for deploy-users. A bit safer as capabilities are reduced compared to a user. And simpler to implement as a single HTTP request.

For the API there's lots of options, but I would suggest using the existing POST /webhooks/{token} for a new type StackWebhook.

It would have an optional body that matches PUT /stacks/{id}'s body.
When the body is empty, this is the equivalent of updating every existing service in this stack.
When a body is provided, it's the equivalent of a PUT /stacks/{id} request.

This allows for both clients that don't know Portainer's API to trigger updates, and for clients that do to be able to change the stack's contents.

I'm happy to have a crack at this, but some opinions on the API would be appreciated.

[Beanow]

Some related issues,

Implementing #1753 and a webhook without payload would allow you to update the stack after a build is completed. However, the git reference may not match the trigger for the CI build. And the stack file can't be generated as a build artifact (for example if you've got a need to generate it from templates), it needs to be committed.

This may cover the use cases for #2752

It may also provide an alternative approach to #2238, where builds can be run externally, for example on Drone CI and deployed through this call.

[deviantony]

Hi @Beanow

Definitely a must have !

However, the git reference may not match the trigger for the CI build. And the stack file can't be generated as a build artifact (for example if you've got a need to generate it from templates), it needs to be committed.

I believe that if we can support #1753 (and thus, #1752 first), we'd be able to trigger a git pull and docker stack deploy via a webhook. I wonder why you think it might not be enough?

I also don't think that it covers #2752 as this one covers service webhooks, which are used to update a single service (and this enhancement's goal is to be able to provide a different image name to use for a service).

I'm keen to discuss this on Slack, feel free to PM me and then we can share our discussion/ideas in here.

[Beanow]

Sorry to have gone off grid for a while.
In the past I've glued together both approaches.

• Triggering to fetch a stack.yml file from the master branch on build completion.
• Sending the stack to an API as a deploy step.

Both worked well for a straightforward use-case. But I think there are some differences that might make the later preferable for more involved setups.

One big difference is handling race conditions. Whether 20s or 20m, your CI build will take some time to complete. Imagine two merges going to master in quick succession. And the second one is a breaking overhaul of the container's configuration, reflected in the environment vars of the service in the stack. Completing the first build pipeline, may cause the second stack configuration to be deployed before the corresponding code changes are built, breaking the deployment. Had the stack been submitted from the build pipeline instead, that would have allowed matching the correct stack with the code we've just built.
This comes with a downside though, because now your builds must complete in the correct order if they deploy to the same stack, so either portainer will need to offer some kind of conditional update (if this stack is newer, deploy it) or your CI platform needs to enforce sending stacks to the webhook happens in git commit order when doing parallel builds.

Another is having a more declarative pipeline. When the stack is pushed, you specify exactly what you want your end result to be. While the git pull approach means your pipeline comes to depend on assumptions about the side-effects of triggering the webhook.

Finally, complex deployment schemes don't require this complexity to be implemented in portainer. For example, if you'd like to maintain a stack serving a preview service for each open pull request. Could be achieved with a script in your pipeline generating the necessary stack for PR triggers but would be a standardization nightmare to do based on pulling multiple files from git and merging them somehow.

[derekpovah]

We're currently calling a handful of service webhooks from CI after each successful build, but as we scale up, this is going to become a nightmare to manage. A single webhook URL for the stack that updates every service (or a subset of services based on the body of the request) would be super useful.

[huib-portainer]

We have a preview version available that allows you to pull compose files from Git and either poll for changes or use a webhook to pull the latest compose file from Git and redeploy the stack.

Please give it a try by using the image portainerci/portainer:pr5340 and let us know how it's working.
Note that this is a development build and should not be used in a production environment.

[lukasmrtvy]

@huib-portainer are notifications ( slack,webhook,email,etc ) on the roadmap?

For example:

• send notification if update for stack is available
• send notification if stack is auto updated

Would be nice to support also Prometheus metrics for these states, but afaik Prometheus integration is not planned..

Thanks

[huib-portainer]

For the short term sending notifications is not part of this feature.
But monitoring and alerting is something we're still considering for the long term.

What is your use case for needing notifications, if you don't mind me asking?
How does your current workflow hang together?

[lukasmrtvy]

I am using industry standard for k8s

• https://github.com/argoproj/argo-cd ( sync from git ; provide metrics ) ( sync from git ; provide metrics )
• https://github.com/argoproj-labs/argocd-notifications ( provide notifications )

Notifications

• New version of an application {{.app.metadata.name}} is up and running.
• Failed to sync application {{.app.metadata.name}}.
• Start syncing application {{.app.metadata.name}}.
• Application {{.app.metadata.name}} has been successfully synced.
• Application {{.app.metadata.name}} sync status is 'Unknown'
• Application {{.app.metadata.name}} has degraded.
• Portainer can probably add also notification if new version is available

,but there is nothing for standalone docker. And these notification states should correlate also to metrics ...

• portainer_stack_available_update{'app.metadata.name'='foo'} 0/1
• portainer_stack_running{'app.metadata.name'='foo'} 0/1
• portainer_stack_updated{'app.metadata.name'='foo'} 0/1
• etc

These are related to continious deployment ( portainer ), metrics from node-exporter, cadvisor or dockerd should not replace them, only to complete the observability itself.

[lukasmrtvy]

@deviantony ^ what do you think ?