Uploading to S3 fails with connection timeout #24
Description
We are not able to upload the collection to the S3 server (as requested by Frank de Bot)
using the current ceph-collect version (cb1a7fa).
Collecting and uploading inside cephadm shell container:
[ceph: root@ceph-3-1 ceph-collect]# ./ceph-collect --upload --ticket NSC-862
INFO:root:Gathering overall system information
/bin/sh: line 1: lsb_release: command not found
INFO:root:Gathering overall Ceph information
INFO:root:Gathering Health information
INFO:root:Gathering MON information
INFO:root:Gathering OSD information
INFO:root:Gathering PG information
INFO:root:Gathering MDS information
INFO:root:Gathering Orchestrator Information
INFO:root:Outputted Ceph information to /tmp/ceph-collect_7efa00f9-182f-40f4-9136-d51895db1f0b_20250521_160437.tar.gz
INFO:root:Parsing "ceph-collect_7efa00f9-182f-40f4-9136-d51895db1f0b_20250521_160437.tar.gz"
INFO:root:Uploading "/tmp/ceph-collect_7efa00f9-182f-40f4-9136-d51895db1f0b_20250521_160437.tar.gz"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0
curl: (28) Connection timed out after 30000 milliseconds
ERROR:root:Upload failed
Traceback (most recent call last):
File "/root/ceph-collect/./ceph-collect", line 635, in <module>
upload_tarball (ARGS, tarball)
File "/root/ceph-collect/./ceph-collect", line 603, in upload_tarball
result = spawn (
File "/root/ceph-collect/./ceph-collect", line 211, in spawn
raise RuntimeError("Process exited abnormally")
RuntimeError: Process exited abnormally
Trying uploading outside of cephadm shell container (this time with debug output):
[ceph: root@ceph-3-1 ceph-collect]# mv /tmp/ceph-collect_7efa00f9-182f-40f4-9136-d51895db1f0b_20250521_160437.tar.gz .
0|0[root@ceph-3-1 /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/home/ceph-collect]# ./ceph-collect --upload-file ceph-collect_7efa00f9-182f-40f4-9136-d51895db1f0b_20250521_160437.tar.gz --ticket NSC-862 --debug
INFO:root:Parsing "ceph-collect_7efa00f9-182f-40f4-9136-d51895db1f0b_20250521_160437.tar.gz"
INFO:root:Uploading "ceph-collect_7efa00f9-182f-40f4-9136-d51895db1f0b_20250521_160437.tar.gz"
DEBUG:root:Upload endpoint: "https://%(bucket)s.fig-nl-dc1.s3.fairbanks.nl:1443"
DEBUG:root:Upload bucket: "ceph-collect"
DEBUG:root:Upload headers: "['x-amz-acl: bucket-owner-full-control']"
DEBUG:root:Upload timeout: "30"
DEBUG:root:curl --max-time 30 --request PUT --upload-file 'ceph-collect_7efa00f9-182f-40f4-9136-d51895db1f0b_20250521_160437.tar.gz' -H 'x-amz-acl: bucket-owner-full-control' -H 'x-amz-tagging: ticket=NSC-862' https://ceph-collect.fig-nl-dc1.s3.fairbanks.nl:1443/ceph-collect_7efa00f9-182f-40f4-9136-d51895db1f0b_20250521_160437.tar.gz(shell:True)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:30 --:--:-- 0
curl: (28) Connection timed out after 30000 milliseconds
ERROR:root:Upload failed
Traceback (most recent call last):
File "/var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/home/ceph-collect/./ceph-collect", line 635, in <module>
upload_tarball (ARGS, tarball)
File "/var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/home/ceph-collect/./ceph-collect", line 603, in upload_tarball
result = spawn (
File "/var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/home/ceph-collect/./ceph-collect", line 211, in spawn
raise RuntimeError("Process exited abnormally")
RuntimeError: Process exited abnormally
Is the target address correctly assembled?
https://ceph-collect.fig-nl-dc1.s3.fairbanks.nl:1443/ceph-collect_7efa00f9-182f-40f4-9136-d51895db1f0b_20250521_160437.tar.gz
The server ceph-collect.fig-nl-dc1.s3.fairbanks.nl can be reached.
But the port 1443 can't be scanned (could have many reasons, though):
0|0[root@ceph-3-1 ~]# ping -c3 ceph-collect.fig-nl-dc1.s3.fairbanks.nl
PING ceph-collect.fig-nl-dc1.s3.fairbanks.nl (85.17.11.65) 56(84) bytes of data.
64 bytes from s3.onweb.gr (85.17.11.65): icmp_seq=1 ttl=54 time=16.9 ms
64 bytes from s3.onweb.gr (85.17.11.65): icmp_seq=2 ttl=54 time=16.6 ms
64 bytes from s3.onweb.gr (85.17.11.65): icmp_seq=3 ttl=54 time=16.7 ms
--- ceph-collect.fig-nl-dc1.s3.fairbanks.nl ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 16.598/16.727/16.917/0.136 ms
0|0[root@ceph-3-1 ~]# nmap -Pn ceph-collect.fig-nl-dc1.s3.fairbanks.nl
Starting Nmap 7.92 ( https://nmap.org ) at 2025-05-21 18:40 CEST
Nmap scan report for ceph-collect.fig-nl-dc1.s3.fairbanks.nl (85.17.11.65)
Host is up (0.013s latency).
rDNS record for 85.17.11.65: s3.onweb.gr
Not shown: 990 filtered tcp ports (no-response)
PORT STATE SERVICE
20/tcp closed ftp-data
21/tcp open ftp
22/tcp open ssh
53/tcp closed domain
80/tcp closed http
113/tcp closed ident
443/tcp open https
873/tcp closed rsync
2000/tcp open cisco-sccp
5060/tcp open sip
Nmap done: 1 IP address (1 host up) scanned in 11.46 seconds
Double checked our firewall, the port is not blocked by us.