[Feature] Add an API endpoint to update containers in a stack (including web editor managed stacks) to the latest image

[callumgare]

It would be great if you could ping an endpoint in the portainer API and, using the supplied stack ID, it would pull the latest images for the containers in that stack and then update that stack. Even better (at least for me) would be adding a webhook that would function in the same way as the webhooks for swarm services (https://documentation.portainer.io/v2.0/webhooks/create/).

I have portainer setup with standalone docker which amoung other things, runs a number of applications I have writen. I have GitLab setup to run a CI process to build a new image for those apps whenever new code has been pushed to the repo. I would love it if it could also trigger an image pull for a given stack in portainer after it has finished building a new image.

[deviantony]

Hi @callumgare,

Stack webhooks are in our plans and should be coming soon :-)

We've recently introduced the ability to pull and redeploy a stack definition from Git via #1753

In there, I've provided screenshots for that upcoming stack webhooks, you can see it and get more details in that comment : #1753 (comment)

Closing this issue in favor of #2835, feel free to discuss in #2835 !

[callumgare]

Rad! Super excited to hear it! Thanks :)

[callumgare]

@deviantony Looking though those issues it looks like the scope might be slightly different. Correctly me if I'm wrong but it looks like those rely on your stack being pulled in from a git repo? In my case I'm using the standard stack web editor in portainer to manage my composer files.

[deviantony]

You're right, I'll re-open to consider adding this to regular stacks.

However this won't be our primary focus, we're heavily gearing towards automation/webhooks for anything that is deployed through Git at the moment so I would recommend making use of Git at that point if you want have access to this feature.

[callumgare]

Cheers, I've update the title to reflect that aspect specifically. Fair enough that this might not be your primary focus :)

I'll see if git managed stacks will work in my workflow when #2835 has rolled out. Cheers!

[huib-portainer]

@callumgare you mention you're using GitLab as part of your CI process. I'm interested to find out why it would be easier for you to keep using the web editor to define stacks, instead of having the compose files defined in GitLab.

And I'm curious as well if the following flow would work for you:
As part of CI/CD you would check the compose files into Git.
Have the CI pipeline create an new image with a certain tag.
Have the CD pipeline commit the new tag into a docker-compose.override.yml and invoke the webhook.
Portainer then pulls the latest compose files from git and redeploys the stack with the new images.

[callumgare]

Hi @huib-portainer thanks for your detailed response. I am actually really interesting to understand why other people find it easier to have their compose file managed in a repo!

I figure most web apps are going to have some instance specific config that surely doesn’t make sense to commit into your main codebase repo. For example
I use traefik for my reverse proxy (a fairly common setup with docker web service hosting I understand) and that uses labels in the compose file. Some of my apps also use Google analytics and take in a key via an env var. Those are things that are gonna be different depending on where I deploy the app to.

I could have another repo that is separate from my code repo which contains all my compose configuration but at that point I don’t really gain anything (I don’t need a version history or other people to review my config changes every time) and it just seems like more hassle than using the web editor to manage that config.

Interested in your proposed workaround but not sure how it would help me with instance specific config. Where would that live?

[huib-portainer]

Thanks for taking the time to describe your use case.

In my mind there's a trade-off between convenience and reliability, because having reviewers might reduce the risk of making any trivial mistakes, which could affect your production environment. But I hear what you're saying when you just want to make a quick change.

Let me describe the flow I was thinking about.
Just like you I'm happy to have both the code and the config in the same repo:

The docker-compose.yml then defines the application and reverse proxy. e.g.

docker-compose.yml:
version: '3.1'
services:
  webapp:
    restart: always
    environment:
      VIRTUAL_PORT: 80
    networks:
      - proxy
    ports:
      - 5050:80
  nginx-proxy:
    image: jwilder/nginx-proxy
    restart: always
    ports:
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/tmp/docker.sock:ro"
      - "./vhost.d:/etc/nginx/vhost.d:ro"
      - ./certs:/etc/nginx/certs
    networks:
      - proxy
  proxy-companion:        
    restart: always
    image: jrcs/letsencrypt-nginx-proxy-companion
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - proxy
networks:
  proxy:

And in each dev/qa/etc folder I would define the image and env vars specific to where it's deployed.
docker-compose.override.yml for instance contains

version: '3.1'
services:
  webapp:
    image: nginx:latest
    environment:
      description: 'DEV environment'
      client_id: ODddLfSNKzJl2cOzF
      VIRTUAL_HOST: 'fc4351c4f8bb.ngrok.io'

In Portainer I would deploy that as a stack

Finally, if I want to update the version that's deployed to QA, I would make changes to the docker-compose.override.yml in the QA folder (either direct or via a pull request), which invokes the webhook and causes Portainer to pull the latest yml & image and redeploy the stack.

[hhromic]

@huib-portainer what you described is exactly the workflow we are looking for in my team.
I'm looking very forward for the version of Portainer that brings this. Congratulations for the great job Portainer is doing.

[callumgare]

@huib-portainer Right, but that still includes committing config into your code repo no? Which is what I want to avoid. I’m not mad about the idea of including secrets in my code repo. To me it seems kind of icky mixing the two. I want the option of being able to share that code base without exposing any secrets. If I was ever to open source any of those repos I’d have to come up with a different workflow then anyway plus go to the extra effort of scrubbing secrets from history git history (which isn’t ideal) or going and invalidating all those keys and regenerating them.

I appreciate your answer though! And maybe I’ll give that a shot but just store it in a different repo purely for config.

[hhromic]

@callumgare , the way we work with secrets is by considering them all as external to our stacks:

version: '3.8'

secrets:
  db-password:
    name: prod-db-password-20210123
    external: true

In this way, every stack assumes that the given secrets are already setup in the Swarm by an operator (or developer), and we can fully separate the storage/handling of secrets from the storage/handling of stacks. In addition, given that we name our secrets using a date numbering, we also know precisely which secret it was meant to work with. This aids a lot with secret rotations.

No secret is ever committed to any Git repository apart from its name. All you need is to make sure that the user of your stack understands that a secret must be created into the swarm with a given name before deploying the stack.

[callumgare]

@hhromic Ah, makes sense! Unfortunately I’m using docker standalone (which doesn’t support secrets) not docker swarm. If I was using swarm I’d be able to use the service webhook as an endpoint to update a stack, thus negating the need for this ticket in the first place. Like the date numbering system though, very smart.

…

On Sat, 10 Jul 2021 at 20:29, Hugo Hromic ***@***.***> wrote: @callumgare <https://github.com/callumgare> , the way we work with secrets is by considering them all as external to our stacks: version: '3.8' secrets: db-password: name: prod-db-password-20210123 external: true In this way, every stack assumes that the given secrets are already setup in the Swarm by an operator (or developer), and we can fully separate the storage/handling of secrets from the storage/handling of stacks. In addition, given that we name our secrets using a date numbering, we also know precisely which secret it was meant to work with. This aids a lot with secret rotations. No secret is ever committed to any Git repository apart from its name. All you need is to make sure that the user of your stack understands that a secret must be created into the swarm with a given name before deploying the stack. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5199 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACURZEJHWIVDYNAD7CYFJLTXAOHFANCNFSM46YRJGVA> .

[huib-portainer]

@callumgare you mention that you currently are using the web editor to create a stack. How do you handle your secrets then in that way?

When adding a Stack in Portainer from a Git repo, as per the screen design above, you can still specify the substitution variables in Portainer. So that's independent from the content in the Git repo.

If your docker-compose.yml in Git for example contains:

version: '3'
services:
  example:
    image: nginx:alpine
    environment:
      PORT: "${PORT}"
    ports:
      - "${PORT}:80"

Then ${PORT} will be substituted with the environment variables provided when creating the stack in Portainer:

[deviantony]

@callumgare it is also worth noting that docker-compose supports secrets (only through file though) when running Docker standalone.

I've never used them much on standalone but maybe you can find some way to leverage that.

https://serverfault.com/a/936262

[huib-portainer]

We have a preview version available that allows you to pull compose files from Git and either poll for changes or use a webhook to pull the latest compose file from Git and redeploy the stack.

Please give it a try by using the image portainerci/portainer:pr5340 and let us know how it's working.
Note that this is a development build and should not be used in a production environment.

[derekpovah]

@callumgare @huib-portainer

We tag our Docker images for each release stage and not with commit hashes, so our compose files don't change with every build. We also often use the same docker image with different commands multiple times in the same stack. Having one webhook at the stack level that updates all of the services via something like docker stack deploy with --resolve-image=changed would be super useful to us.

If I was using swarm I’d be able to use the service webhook as an endpoint to update a stack, thus negating the need for this ticket in the first place.

I don't think using swarm completely negates the need for this ticket. Consider a stack like my example above that has a bunch of different services all with their own unique webhook. Configuring CI to hit each of these endpoints after a build can be cumbersome. Adding services to the stack down the line is also non-trivial.

[huib-portainer]

Thanks for the feedback.

Here's the automatic update logic that we're currently using for stacks deployed from Git, in order to avoid redeploying too often:
The Pull & redeploy button: redeploys even if the branch in Git has not changed
Webhook: redeploys only when the branch in Git has changed
Polling: redeploys only when the branch in Git has changed

[derekpovah]

Thanks for the clarification.

I really think the update logic should be based on Docker image digests and not on a Git repository. There are too many popular deployment strategies where the compose files are decoupled from the actual code base; defining the stack in the Portainer GUI being the main one here. Being able to pull your stack's configuration from Git is definitely cool, but Portainer shouldn't use that repository to determine if something has changed or not because Docker already has a mechanism in place to do that.