pkgin update reports "No Address Found" if started within a few seconds of rebooting a zone

[puphubv]

While writing scripts to configure a zone, I ran into a strange behavior...

After rebooting a zone, and then running "pkgin update"...

I'll get: pkgin: Could not fetch https://pkgsrc.smartos.org/packages/SmartOS/trunk/x86_64/All/pkg_summary.gz: No address record

if I don't wait at least 4 seconds or so...

During my testing, I found I could actually curl the file, download it completely, and still get the no address found error.

Reboot VM...
Successfully completed reboot for VM 20283614-1e7c-4b08-b773-d4b40ae499e2
curl "https://pkgsrc.smartos.org/packages/SmartOS/trunk/x86_64/All/pkg_summary.gz" | wc c
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5202k 100 5202k 0 0 1515k 0 0:00:03 0:00:03 -:--:-- 1514k

11/25/2024 07:58 UTC:
pkgin: Could not fetch https://pkgsrc.smartos.org/packages/SmartOS/trunk/x86_64/All/pkg_summary.gz: No address record
processing remote summary (https://pkgsrc.smartos.org/packages/SmartOS/trunk/x86_64/All)...

Rejected at illumos: https://www.illumos.org/issues/16914

[danmcd]

Still probably not the best place for it, but I'll give it a quick look now...

The error message and situationlooks similar to this now-closed issue on pksrc

TritonDataCenter/pkgsrc#391

First thing, check your /etc/resolv.conf in the zone you're running pkgsrc in. It's weird that you can curl but you cannot use pkgsrc.

Seconde, you may wish to re-open that bug?

[puphubv]

I really don't think it's the resolv.conf (which is nameserver 8.8.4.4)

I'm happy to reopen the pkgsrc bug, however it seems worth waiting to see what you find first.

It occurred to me to mention that I found the bug in a zone with postfix etc, which I thought could be affecting the timing, however, I just tested on a minimal zone, and it still fails with a sleep 3, and works with a sleep 5 after the reboot.

[danmcd]

I'm assuming this is a native zone? If it isn't, ignore the following...

What does svcs -xv name-service-cache say in your zone?

[jperkin]

libfetch isn't doing anything unusual here:

https://github.com/TritonDataCenter/pkgsrc/blob/trunk/net/libfetch/files/common.c#L285-L294

If getaddrinfo() is failing for pkgin then I would expect it to be failing for everything else too.

[puphubv]

svcs -xv name-service-cache
svc:/system/name-service-cache:default (name service cache)
State: online since Mon Nov 25 15:39:20 2024
See: man -M /usr/share/man -s 8 nscd
See: /var/svc/log/system-name-service-cache:default.log
Impact: None.

I also tried it immediately after rebooting:

Successfully completed reboot for VM 20283614-1e7c-4b08-b773-d4b40ae499e2
svcs: Could not bind to repository server: repository server unavailable. Exiting.

But I don't think it's the smoking gun because I also tested with a sleep 3 after the reboot:

Successfully completed reboot for VM 20283614-1e7c-4b08-b773-d4b40ae499e2
[sleep 3 here]
svc:/system/name-service-cache:default (name service cache)
State: online since Mon Nov 25 17:58:22 2024
See: man -M /usr/share/man -s 8 nscd
See: /var/svc/log/system-name-service-cache:default.log
Impact: None.

pkgin: Could not fetch https://pkgsrc.smartos.org/packages/SmartOS/trunk/x86_64/All/pkg_summary.gz: No address record

[gemelen]

I'd like to add to this that pkgsrc.smartos.org availability to my environments in Hetzner are quite fragile for a few last years and issues are still visible every time I touch pkgin with default repo.

I was doing a fresh setup with a new to me networking side of it in last few days and instability of access gave me a lot to investigate, while the root cause was still the default repo.

(Sorry for such a "pure observation" report, I'd be glad to add more diagnostics if you'd like to go for an investigation):

The problem that it's really erratic - DNS records are visible, URL is curlable most of the times, while pkgin itself reports connection timeout more frequently. If it conencts, it could download the particular packages, but any next of in the list could fail.

BTW: as a solution: I gladly switched to packages.pkgsrc.pub, which is slightly slower bandwidth-wise, but way more stable (and also available via IPv6, which was the reason I found out about it).

PS: by using a random looking glass with US presence it could be seen that pkgsrc.smartos.org (eg ...triton.zone) connectivity is problematic even within different US ISPs.