docker cap_add seems to work only with replica to 1 and not in scheduled mode

[NicolasLeCorre]

Hi, and thanks for your work !

We use cronjob to start jobs in our clusters and for some reasons we need for one of our container to set the mac_address (licensing stuff). We found that adding cap_add: - NET_ADMIN allow us to override that value and we are ok with that.

We then tried to deploy such a container in our swarm and found that :

• if the container is started with deploy: replica: 1, it is running well
• if the container is started by the scheduler : swarm.cronjob.schedule=* * * * * for example, we see the error : ip: ioctl 0x8914 failed: Operation not permitted

In the docker documentation, we can see that :
`Note when using docker stack deploy

The cap_add and cap_drop options are ignored when deploying a stack in swarm mode`

But it seems that with our used version of docker (20.10.10) and cronjob (1.10.0, latest) the cap_add is NOT ignored, or at least for the replica :1 option.

Here is an example of our iml file :

version: '3.8'

services:
  stream:
    image: alpine
    entrypoint: [ "/bin/sh","-c" ]
    command: >
        "ip link set dev eth0 down
        && ip link set dev eth0 address fa:16:3e:87:02:d7
        && ip link set dev eth0 up"
    deploy:
      replicas: 0
      labels:
        - "swarm.cronjob.enable=true"
        - "swarm.cronjob.schedule=* * * * *"
        - "swarm.cronjob.skip-running=true"
    cap_add:
      - NET_ADMIN

[crazy-max]

Apparently capabilities are not handled for swarm services. In #203 I have added a new label swarm.cronjob.capabilities where we can set kernel capabilities when the service is updated but does not seem to work:

# cap.yml
version: "3.8"

services:
  test:
    image: alpine:edge
    command: >
      /bin/sh -c "apk add libcap-utils && capsh --print | grep Current: | cut -d' ' -f2"
    deploy:
      replicas: 0
      labels:
        - "swarm.cronjob.enable=true"
        - "swarm.cronjob.schedule=*/5 * * * * *"
        - "swarm.cronjob.skip-running=true"
        - "swarm.cronjob.capabilities=NET_ADMIN"
    cap_add:
      - NET_ADMIN

$ docker service logs cap_test
...
cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep

Also looking at the docs it seems it's not supported when deploying a stack in swarm mode: https://docs.docker.com/compose/compose-file/compose-file-v3/#cap_add-cap_drop

I'm not sure why it works with replicas: 1 though.

[NicolasLeCorre]

Thanks for your reply :)
We managed to use a different solution (not requiring NET_ADMIN) but it may be interesting to be able to use it anyways.

You said that even with the dedicated label it still does not work; it is because of swarm ?
Do you think it is a "bug" that it works with replicas: 1 ?