Issues with fresh OSS install

[gonzalesMK]

I executed the following command from a fresh repository clone

sudo docker compose --profile puppet up

but I was unable to run the testing command:

docker compose --profile test run testing puppet agent -t

because I got those errors in the command line:

sudo docker compose --profile test run testing puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: /puppet/v3/node/crafty-testing. (method :get). Please see the server logs for details.
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from puppet:8140 (172.20.0.4)
Notice: Catalog compiled by puppet.
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: /puppet/v3/catalog/crafty-testing. (method :post). Please see the server logs for details.
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: /puppet/v3/report/crafty-testing. (method :put). Please see the server logs for details.

Would you mind help me fix whatever I'm missing? I checked auth.conf and it seems alright.

Here is some debug info

found those errors on the log:

puppet-1       | 2024-09-03 21:12:20,219 ERROR [p.t.a.rules] Forbidden request: crafty-testing.(172.20.0.7) access to /puppet/v3/catalog/crafty-testing. (method :post) (authenticated: true) denied by rule 'puppetlabs v3 catalog from agents'.
puppet-1       | 172.20.0.7 - - - 03/Sep/2024:21:12:20 +0000 "POST /puppet/v3/catalog/crafty-testing.?environment=production HTTP/1.1" 403 109 172.20.0.7 172.20.0.7 8140 6
puppet-1       | 2024-09-03 21:12:20,279 ERROR [p.t.a.rules] Forbidden request: crafty-testing.(172.20.0.7) access to /puppet/v3/report/crafty-testing. (method :put) (authenticated: true) denied by rule 'puppetlabs report'.

I also tried to run this command on the puppet container:

sudo docker container exec puppet-puppet-1  puppet agent -t

and I got those errors:

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from puppet:8140 (172.20.0.5)
Notice: Catalog compiled by puppet.
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: /puppet/v3/catalog/puppet. (method :post). Please see the server logs for details.
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: /puppet/v3/report/puppet. (method :put). Please see the server logs for details.

and the logs:

puppet-1       | 2024-09-03 21:15:21,841 ERROR [p.t.a.rules] Forbidden request: puppet.(172.20.0.5) access to /puppet/v3/catalog/puppet. (method :post) (authenticated: true) denied by rule 'puppetlabs v3 catalog from agents'.
puppet-1       | 172.20.0.5 - - - 03/Sep/2024:21:15:21 +0000 "POST /puppet/v3/catalog/puppet.?environment=production HTTP/1.1" 403 101 172.20.0.5 172.20.0.5 8140 9

[rwaffen]

hmm strange. i used the same commands and it succeeded for me. but i also updatet the container images of the server and puppetdb recently. may you retry again please?

did you wait until you got these lines in the log?

puppet-1       | 2024-09-06 14:06:17,915 INFO  [p.s.m.master-service] Puppet Server has successfully started and is now ready to handle requests
...
puppetdb-1     | 2024-09-06 14:06:24,571 INFO  [p.p.pdb-routing] PuppetDB finished starting, disabling maintenance mode
...
puppetboard-1  | INFO:puppetboard.utils:PuppetDB version: 8.7.0
puppetboard-1  | [2024-09-06 14:06:26 +0000] [1] [INFO] Starting gunicorn 21.2.0
puppetboard-1  | [2024-09-06 14:06:26 +0000] [1] [INFO] Listening at: http://0.0.0.0:8088 (1)

and checked with docker compose ps if all containers are healthy?

NAME                IMAGE                                         COMMAND                  SERVICE       CREATED         STATUS                   PORTS
oss-postgres-1      docker.io/postgres:16-alpine                  "docker-entrypoint.s…"   postgres      7 minutes ago   Up 7 minutes (healthy)   5432/tcp
oss-puppet-1        ghcr.io/voxpupuli/puppetserver:8.6.1-latest   "dumb-init /docker-e…"   puppet        7 minutes ago   Up 7 minutes (healthy)   0.0.0.0:8140->8140/tcp
oss-puppetboard-1   ghcr.io/voxpupuli/puppetboard:latest          "/bin/sh -c 'gunicor…"   puppetboard   7 minutes ago   Up 6 minutes (healthy)   80/tcp, 0.0.0.0:8088->8088/tcp
oss-puppetdb-1      ghcr.io/voxpupuli/puppetdb:8.7.0-latest       "dumb-init /docker-e…"   puppetdb      7 minutes ago   Up 7 minutes (healthy)   8080/tcp, 0.0.0.0:8081->8081/tcp

[johnduarte]

I am also experiencing 403 (forbidden) errors for the catalog and report endpoints when running the testing command.

The containers were all in a healthy state prior to running the testing command.

root@ubuntu-s-1vcpu-2gb-sfo3-01:~/crafty/puppet/oss# docker container ps
CONTAINER ID   IMAGE                                         COMMAND                  CREATED         STATUS                    PORTS
                                              NAMES
efef62966719   ghcr.io/voxpupuli/puppetserver:8.6.3-latest   "dumb-init /docker-e…"   4 minutes ago   Up 4 minutes (healthy)    0.0.0.0:
8140->8140/tcp, :::8140->8140/tcp             oss-puppet-1
dcedb4707df0   ghcr.io/voxpupuli/puppetdb:8.7.0-latest       "dumb-init /docker-e…"   4 minutes ago   Up 4 minutes (healthy)    8080/tcp
, 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp   oss-puppetdb-1
1be2517d34cf   ghcr.io/voxpupuli/puppetboard:latest          "/bin/sh -c 'gunicor…"   4 minutes ago   Up 16 seconds (healthy)   80/tcp,
0.0.0.0:8088->8088/tcp, :::8088->8088/tcp     oss-puppetboard-1
c78c3508a8a6   postgres:17-alpine                            "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes (healthy)    5432/tcp
                                              oss-postgres-1
root@ubuntu-s-1vcpu-2gb-sfo3-01:~/crafty/puppet/oss# docker compose --profile test run testing puppet agent -t
[+] Creating 1/0
 ✔ Volume "oss_agent-ssl"  Created                                                                                                 0.0s
[+] Running 9/9
 ✔ testing Pulled                                                                                                                 46.3s
   ✔ 6414378b6477 Already exists                                                                                                   0.0s
   ✔ ee424688b5cb Pull complete                                                                                                    0.3s
   ✔ 9b2d84335313 Pull complete                                                                                                    0.4s
   ✔ 95a17f9f6d59 Pull complete                                                                                                    0.4s
   ✔ 9db99164c2d3 Pull complete                                                                                                   42.7s
   ✔ 220735b9dc96 Pull complete                                                                                                   43.7s
   ✔ 24efd0fb060e Pull complete                                                                                                   45.4s
   ✔ b3e1c9a498fd Pull complete                                                                                                   45.4s
Info: Creating a new RSA SSL key for crafty-testing.
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for crafty-testing.
Info: Certificate Request fingerprint (SHA256): F3:42:49:45:9B:82:F5:48:CD:84:45:46:4D:8E:90:7F:1D:89:C9:D2:30:92:47:72:81:B9:1D:89:3F:28:28:D7
Info: Downloaded certificate for crafty-testing. from https://puppet:8140/puppet-ca/v1
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: /puppet/v3/node/crafty-testing. (method :get).  Please see the server logs for details.
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from puppet:8140 (172.18.0.4)
Notice: Catalog compiled by puppet.
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: /puppet/v3/catalog/crafty-testing. (method :post). Please see the server logs for details.
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: /puppet/v3/report/crafty-testing. (method :put). Please see the server logs for details.

Environment

I am running on a Digital Ocean droplet for the Docker host:

• "size":"s-1vcpu-2gb"
• "image":"ubuntu-24-04-x64"

I am using a crafty clone at SHA 9fa6ee7 .

Docker version 27.3.1, build ce12230

Container versions pulled by docker compose --profile puppet:

root@ubuntu-s-1vcpu-2gb-sfo3-01:~/crafty/puppet/oss# docker image ls
REPOSITORY                       TAG            IMAGE ID       CREATED         SIZE
ghcr.io/betadots/pdc             latest         6a12d656eeff   2 days ago      843MB
ghcr.io/voxpupuli/puppetserver   8.6.3-latest   695634975b25   4 days ago      715MB
postgres                         17-alpine      bb46dc8bfad7   3 weeks ago     248MB
ghcr.io/voxpupuli/puppetdb       8.7.0-latest   fe14b3d33c54   6 weeks ago     589MB
ghcr.io/voxpupuli/puppetboard    latest         3498a78ecea7   7 months ago    253MB

Puppet Server container log

oss-puppet-1.log

[johnduarte]

The condition I documented above appears to be specific to DigitalOcean. I presume this is a side effect of Droplets not having a domain name, causing the puppet container's certname to be puppet. and the testing container's certname to be crafty-testing.. The empty trailing dot may be the cause of this behavior on this platform.

Attempting to reproduce this condition on an AWS EC2 instance allows the catalog and report endpoints for the puppet and crafty-testing container to be delivered as expected.

[rwaffen]

ah okay, now i see. the domain is not set and so an invalid one is generated 🤦

[jaredjadu]

I have the same problem on a local proxmox VM

Ubuntu 24.04
hostname puppet1.local.domain.org

cloned repo at bae9cfe

root@puppet:~/crafty/puppet/oss# docker version
Client: Docker Engine - Community
 Version:           27.4.1
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        b9d17ea
 Built:             Tue Dec 17 15:45:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.4.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       c710b88
  Built:            Tue Dec 17 15:45:46 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

root@puppet:~/crafty/puppet/oss# docker compose --profile test run testing puppet agent -t
[+] Creating 1/0
 ✔ Volume "oss_agent-ssl"  Created                                                                                                                                                                 0.0s
Info: Creating a new RSA SSL key for crafty-testing.
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for crafty-testing.
Info: Certificate Request fingerprint (SHA256): 04:1B:12:CF:74:85:FC:60:F6:A8:68:B2:F9:45:CA:2D:89:72:6E:46:4D:A9:50:CA:ED:52:2A:86:B7:07:8B:B7
Info: Downloaded certificate for crafty-testing. from https://puppet:8140/puppet-ca/v1
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: /puppet/v3/node/crafty-testing. (method :get). Please see the server logs for details.
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from puppet:8140 (172.18.0.2)
Notice: Catalog compiled by puppet
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: /puppet/v3/catalog/crafty-testing. (method :post). Please see the server logs for details.
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: /puppet/v3/report/crafty-testing. (method :put). Please see the server logs for details.

root@puppet:~/crafty/puppet/oss# docker ps
CONTAINER ID   IMAGE                                         COMMAND                  CREATED             STATUS                       PORTS                                                 NAMES
acd14fc6a2cc   postgres:17-alpine                            "docker-entrypoint.s…"   About an hour ago   Up About an hour (healthy)   5432/tcp                                              oss-postgres-1
3293a5f1e80a   ghcr.io/voxpupuli/puppetserver:8.7.0-latest   "dumb-init /docker-e…"   About an hour ago   Up About an hour (healthy)   0.0.0.0:8140->8140/tcp, :::8140->8140/tcp             oss-puppet-1
cc70cefc9503   ghcr.io/voxpupuli/puppetboard:latest          "/bin/sh -c 'gunicor…"   About an hour ago   Up About an hour (healthy)   80/tcp, 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp     oss-puppetboard-1
cdae70a0e931   ghcr.io/voxpupuli/puppetdb:8.8.1-latest       "dumb-init /docker-e…"   About an hour ago   Up About an hour (healthy)   8080/tcp, 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp   oss-puppetdb-1

puppet-1       | 172.18.0.6 - - - 24/Dec/2024:12:51:07 +0000 "GET /puppet-ca/v1/certificate/ca HTTP/1.1" 200 2775 172.18.0.6 172.18.0.6 8140 11
puppet-1       | 172.18.0.6 - - - 24/Dec/2024:12:51:07 +0000 "GET /puppet-ca/v1/certificate_revocation_list/ca HTTP/1.1" 200 1465 172.18.0.6 172.18.0.6 8140 4
puppet-1       | 2024-12-24 12:51:10,019 INFO  [p.p.certificate-authority] Entity CA signed 1 certificate: crafty-testing..
puppet-1       | 172.18.0.6 - - - 24/Dec/2024:12:51:10 +0000 "PUT /puppet-ca/v1/certificate_request/crafty-testing. HTTP/1.1" 200 0 172.18.0.6 172.18.0.6 8140 88
puppet-1       | 127.0.0.1 - - - 24/Dec/2024:12:51:10 +0000 "GET /status/v1/simple HTTP/1.1" 200 7 127.0.0.1 127.0.0.1 8140 5
puppet-1       | 127.0.0.1 - - - 24/Dec/2024:12:51:33 +0000 "GET /status/v1/simple HTTP/1.1" 200 7 127.0.0.1 127.0.0.1 8140 5
puppet-1       | 172.18.0.6 - - - 24/Dec/2024:12:51:40 +0000 "GET /puppet-ca/v1/certificate/crafty-testing. HTTP/1.1" 200 1596 172.18.0.6 172.18.0.6 8140 3
puppet-1       | 2024-12-24 12:51:40,139 ERROR [p.t.a.rules] Forbidden request: crafty-testing.(172.18.0.6) access to /puppet/v3/node/crafty-testing. (method :get) (authenticated: true) denied by rule 'puppetlabs node'.
puppet-1       | 172.18.0.6 - - - 24/Dec/2024:12:51:40 +0000 "GET /puppet/v3/node/crafty-testing.?environment=production&configured_environment=production&transaction_uuid=70e44197-f055-4378-a908-3cef64b16cdc HTTP/1.1" 403 105 172.18.0.6 172.18.0.6 8140 18
puppet-1       | 172.18.0.6 - - - 24/Dec/2024:12:51:40 +0000 "GET /puppet/v3/file_metadatas/plugins?recurse=false&links=manage&checksum_type=sha256&source_permissions=ignore&environment=production HTTP/1.1" 200 182 172.18.0.6 172.18.0.6 8140 462
puppet-1       | 172.18.0.6 - - - 24/Dec/2024:12:51:40 +0000 "GET /puppet/v3/file_metadatas/pluginfacts?recurse=true&max_files=-1&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&links=follow&checksum_type=sha256&source_permissions=use&environment=production HTTP/1.1" 200 181 172.18.0.6 172.18.0.6 8140 21
puppet-1       | 172.18.0.6 - - - 24/Dec/2024:12:51:40 +0000 "GET /puppet/v3/file_metadatas/plugins?recurse=true&max_files=-1&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&links=follow&checksum_type=sha256&source_permissions=ignore&environment=production HTTP/1.1" 200 183 172.18.0.6 172.18.0.6 8140 14
puppet-1       | 2024-12-24 12:51:42,911 ERROR [p.t.a.rules] Forbidden request: crafty-testing.(172.18.0.6) access to /puppet/v3/catalog/crafty-testing. (method :post) (authenticated: true) denied by rule 'puppetlabs v3 catalog from agents'.
puppet-1       | 172.18.0.6 - - - 24/Dec/2024:12:51:42 +0000 "POST /puppet/v3/catalog/crafty-testing.?environment=production HTTP/1.1" 403 109 172.18.0.6 172.18.0.6 8140 6
puppet-1       | 2024-12-24 12:51:42,988 ERROR [p.t.a.rules] Forbidden request: crafty-testing.(172.18.0.6) access to /puppet/v3/report/crafty-testing. (method :put) (authenticated: true) denied by rule 'puppetlabs report'.
puppet-1       | 172.18.0.6 - - - 24/Dec/2024:12:51:42 +0000 "PUT /puppet/v3/report/crafty-testing.?environment=production HTTP/1.1" 403 107 172.18.0.6 172.18.0.6 8140 6