Error on container creation when /var/vcap/instance/dns does not exist

[rkoster]

D, [2025-03-04T09:08:49.583593 #28921] [canary_update(worker/13090775-5621-4ec9-9987-cc81957d9cbb (0))] DEBUG -- DirectorJobRunner: [external-cpi] [cpi-319346] response: {"result":null,"error":{"type":"Bosh::Clouds::CloudError","message":"
Attaching disk '{{vol-6cc0a940-8038-43c5-6b91-190d5bcfdc7d}}' to VM '{{c-b6f66eb3-00dc-4e80-6492-74e5ba9c005d}}': Restoring records.json: Copying to container: Error response from daemon: Could not find the file /var/vcap/instance/dns in c
ontainer c-b6f66eb3-00dc-4e80-6492-74e5ba9c005d","ok_to_retry":false},"log":""}, err: [File System] 2025/03/04 09:08:44 DEBUG - Reading file /var/vcap/jobs/docker_cpi/config/cpi.json

Ideally this file should only be copied when it exists. Or an empty file should be created on initial container creation.

[jpalermo]

Are you using 0.0.20? 0.0.19 had a problem where it didn't mkdir -p the destination before copying the dns records back, but that was fixed in 0.0.20

In either version, we guard against an empty/non-existent file in the container and just skip uploading it.

[rkoster]

I was using a local build based on: 9f9e380 so that's fairly recent.

[jpalermo]

We had a thought.

Since the container is recreated between the read and the write, maybe /var/vcap/instance/dns doesn't yet exist on the new container.

We couldn't follow that thought to anything productive though because the bosh-agent doesn't create that folder on startup, it only creates it during a sync-dns action on the agent, which I would expect NOT to generally be happening during attach_disk. So if this were the problem, I'd think we'd be seeing it all the time.

That makes me think that the docker API that's being called to put the file there in the container must be doing a mkdir.

None of this really explains how you got the error unless whatever version of docker you're using behaves differently.

[rkoster]

Unless, hear me out, you are really smart and have put a brilliant temporary fix in place which everybody had forgotten about:

  - path: /run/tas-dns-fix.sh
    permissions: "0700"
    owner: root
    content: |
      #!/bin/bash
      set -eux
      set -o pipefail
      director_container="$(docker ps -q | xargs -n 1 docker inspect --format '{{ .NetworkSettings.Networks.opsman.IPAddress }}-{{ .Id }}' | grep 10.10.0.6 | cut -d- -f2)"
      if [[ -z "$director_container" ]]; then
        echo "Could not find director; aborting"
        exit 1
      fi
      docker exec "$director_container" /var/vcap/jobs/director/bin/trigger-one-time-sync-dns
  - path: /etc/systemd/system/tas-dns-fix.service
    permissions: "0644"
    owner: root
    content: |
      [Unit]
      Description=A timer which periodically applies a workaround bosh a bosh bug
      After=opsman.service docker.socket docker.service
      Requires=opsman.service docker.socket docker.service
      [Service]
      Type=oneshot
      ExecStart=/run/tas-dns-fix.sh
  - path: /etc/systemd/system/tas-dns-fix.timer
    permissions: "0644"
    owner: root
    content: |
      [Unit]
      Description=A timer which periodically applies a workaround bosh a bosh bug
      After=opsman.service docker.socket docker.service
      Requires=opsman.service docker.socket docker.service
      [Timer]
      Unit=tas-dns-fix.service
      OnCalendar=*:*:0/15
      [Install]
      WantedBy=timers.target

[jpalermo]

So your fix was mucking with the /var/vcap/instance/dns file while it was being replaced by the CPI?