Unable to access main dokploy server

[rothnic]

Steps:

• I first tried setting up coolify and for some reason selected only 1 worker. I was able to reach the main instance then
• I destroyed the stack
• Tried again with coolify, but with 3 workers this time, and was never able to reach the main instance
• I destroyed the stack
• I found the dokploy repository, so tried that with 3 workers
• Again, not able to reach the main instance
• Tried pinging the main instance public IP and it doesn't respond

Could there be some kind of network configuration issue triggered by the number of worker nodes?

[rothnic]

Additional troubleshooting:

• able to ssh into server
• firewall isn't active
• no application listening to port 3000
• docker command not found

When through some more steps with chatgpt and here is the summary of that:

Issue: Cloud-init script fails to install Docker/Dokploy on OCI free-tier VM

Symptoms

• Fresh stack deploys, but Dokploy is not reachable on port 3000.

• On the instance:

  sudo ss -tulpn

  shows no process listening on 3000, only sshd, rpcbind, etc.

• docker command is missing:

  $ docker ps
  bash: docker: command not found

• Cloud-init logs show the installer failing:

  grep -iE 'curl|docker|swarm|failed' /var/log/cloud-init-output.log
  curl: (7) Couldn't connect to server
  sh: 130: docker: not found
  Error: Failed to initialize Docker Swarm

• User-data script in /var/lib/cloud/instance/user-data.txt contains:

  # Install Dokploy
  curl -sSL https://dokploy.com/install.sh | sh

  This ran before Docker was present, so docker swarm init failed.

Root Causes

1. Race with network availability – at first boot, curl hit (7) Couldn't connect to server. Later manual curl tests worked fine, so it’s a timing issue.
2. No explicit Docker installation before Dokploy – the script assumes Dokploy’s installer will pull in Docker, but when curl fails, nothing is installed.
3. iptables changes – the script re-adds a blanket REJECT rule to the FORWARD chain, which can interfere with Docker networking.

Debugging steps we ran

• Checked what ports were listening:

  sudo ss -tulpn

  → no 3000/tcp listener.

• Inspected cloud-init logs:

  sudo cloud-init status --long
  sudo less /var/log/cloud-init-output.log
  sudo journalctl -u cloud-init -b | grep -i docker

  → confirmed curl failure and “docker not found”.

• Inspected injected user-data:

  sudo sed -n '1,200p' /var/lib/cloud/instance/user-data.txt

  → showed Dokploy install step runs immediately in user-data.

• Inspected iptables rules:

  sudo iptables -S

  → found custom REJECT --reject-with icmp-host-prohibited rules on FORWARD.

Proposed Fix

1. Install Docker explicitly before Dokploy

Use the official Docker convenience script with retries:

curl --fail --retry 10 --retry-all-errors -fsSL https://get.docker.com | sh
systemctl enable --now docker

2. Run Dokploy install only after network-online

Wrap the Dokploy installer in a systemd unit that requires network-online.target. Example:

#cloud-config
write_files:
  - path: /usr/local/sbin/dokploy-install.sh
    permissions: '0755'
    content: |
      #!/usr/bin/env bash
      set -euo pipefail
      log=/var/log/dokploy-install.log
      exec > >(tee -a "$log") 2>&1

      echo "== $(date -Is) installing docker + dokploy =="

      curl --fail --retry 10 --retry-all-errors -fsSL https://get.docker.com | sh
      systemctl enable --now docker

      curl --fail --retry 10 --retry-all-errors -fsSL https://dokploy.com/install.sh | sh

  - path: /etc/systemd/system/dokploy-install.service
    permissions: '0644'
    content: |
      [Unit]
      Description=Install Dokploy after network is online
      Wants=network-online.target
      After=network-online.target

      [Service]
      Type=oneshot
      ExecStart=/usr/local/sbin/dokploy-install.sh

      [Install]
      WantedBy=multi-user.target

runcmd:
  - systemctl daemon-reload
  - systemctl enable dokploy-install.service
  - systemctl start dokploy-install.service

3. Avoid breaking Docker’s FORWARD chain

Instead of appending a blanket REJECT:

iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited

→ Remove this step. Let Docker manage FORWARD, or if needed, add policy in the DOCKER-USER chain.

Unknowns / open questions

• Does Dokploy’s installer always bootstrap Docker itself? (If yes, then the curl race is the only blocker. If no, Docker must be installed first.)
• Should the stack expose Dokploy via port 3000 directly, or standardize on 80/443 with a reverse proxy? (Current terraform only exposes 3000.)

Next Steps

• Update the Terraform repo’s cloud-init template to:

  • Install Docker before Dokploy.
  • Wrap Dokploy install in a systemd unit bound to network-online.target.
  • Remove the iptables FORWARD reject rules.

• Test on a fresh VM: expect docker ps to show Dokploy containers and curl http://<public-ip>:3000 to succeed.