Bots are down, commit didn't fix

[lizmat]

[02:30:29]  notable6 (~not-able@ns3002604.ip-151-80-20.eu) left IRC (Server closed connection)

@AlexDaniel Any idea what we could do to get them back up again?

[AlexDaniel]

First of all, a commit didn't fix it because it failed to build:

#15 [ 9/11] RUN zef install --force --/test --deps-only .
#15 6.335 ===> Searching for missing dependencies: App::GPTrixie, Config::INI, Cro, Cro::HTTP::Client, Digest::HMAC, Digest::SHA256::Native, HTTP::Server::Async, IRC::Client, IRC::TextColor, Number::Denominate, Pastebin::Gist, ake, Stats, Text::Diff::Sift4
#15 45.32 Failed to open file /root/.zef/store/69991e0bc50d8ede1db29a06827e86c0ce2799df.tar.gz/OpenSSL-0.2.5/resources/libraries.json: No such file or directory
#15 45.33 ===> Searching for missing dependencies: WWW, IO::Socket::Async::SSL:ver<0.8.0+>, Lingua::Conjunction, XML, IO::Socket::Async::SSL, IO::Path::ChildSecure, Base64, Cro::TLS:ver<0.8.10+>:auth<zef:cro>:api<0>, Crypt::Random, JSON::JWT, Log::Timeline, HTTP::Roles
#15 45.33 ===> Searching for missing dependencies: if, OpenSSL, CBOR::Simple
#15 45.33 ===> Searching for missing dependencies: TinyFloats:ver<0.0.3+>, PathTools:ver<0.2.0>:auth<github:ugexe>, JSON::Fast:ver<0.19+>:auth<cpan:TIMOTIMO>
#15 45.33 ===> Failed to find dependencies: JSON::Fast:ver<0.19+>:auth<cpan:TIMOTIMO>
#15 45.33 Failed to resolve missing dependencies, but continuing with --force-resolve
#15 45.33 ===> Building: OpenSSL:ver<0.2.5>:auth<zef:raku-community-modules>
#15 45.33 [OpenSSL] ===SORRY!=== Error while compiling /root/.zef/store/69991e0bc50d8ede1db29a06827e86c0ce2799df.tar.gz/OpenSSL-0.2.5/Build.rakumod
#15 45.33 [OpenSSL] Could not find JSON::Fast:ver<0.19+>:auth<cpan:TIMOTIMO> in:
#15 45.33 [OpenSSL]     /root/.zef/store/Raku-PathTools.git/bf88cbf2084c552f5b22d1a0a9f7d0622c86ee2c
#15 45.33 [OpenSSL]     /root/.raku
#15 45.33 [OpenSSL]     /usr/share/perl6/site
#15 45.33 [OpenSSL]     /usr/share/perl6/vendor
#15 45.33 [OpenSSL]     /usr/share/perl6/core
#15 45.33 [OpenSSL]     CompUnit::Repository::AbsolutePath<3885584576120>
#15 45.33 [OpenSSL]     CompUnit::Repository::NQP<3885506883344>
#15 45.33 [OpenSSL]     CompUnit::Repository::Perl5<3885506883384>
#15 45.33 [OpenSSL] Please note that a 'META6.json' file was found in '../../Raku-PathTools.git/bf88cbf2084c552f5b22d1a0a9f7d0622c86ee2c', of which the 'provides' section was used to determine if a dependency is available or not.  Perhaps you need to add 'JSON::Fast:ver<0.19+>:auth<cpan:TIMOTIMO>' in the <provides> section of that file?  Or need to specify a directory that does *not* have a 'META6.json' file?
#15 45.33 [OpenSSL] at /root/.zef/store/69991e0bc50d8ede1db29a06827e86c0ce2799df.tar.gz/OpenSSL-0.2.5/Build.rakumod:2
#15 45.33 ===> Building [FAIL]: OpenSSL:ver<0.2.5>:auth<zef:raku-community-modules>
#15 45.33 [OpenSSL] Failed to build, but continuing with --force-build
#15 45.33 ===> Building: Digest::SHA256::Native:ver<0.04>
#15 45.33 ===> Building [OK] for Digest::SHA256::Native:ver<0.04>
#15 45.33 ===> Installing: WWW:ver<1.005007>:auth<zef:raku-community-modules>
#15 45.33 ===> Installing: Pastebin::Gist:ver<1.007>:auth<zef:raku-community-modules>
#15 45.33 ===> Installing: Cro::Core:ver<0.8.10>:auth<zef:cro>:api<0>
#15 45.33 ===> Installing: PathTools:ver<0.2.0>:auth<github:ugexe>
#15 45.33 ===> Installing: OpenSSL:ver<0.2.5>:auth<zef:raku-community-modules>
#15 45.33 ===> Install [FAIL] for OpenSSL:ver<0.2.5>:auth<zef:raku-community-modules>: Failed to open file /root/.zef/store/69991e0bc50d8ede1db29a06827e86c0ce2799df.tar.gz/OpenSSL-0.2.5/resources/libraries.json: No such file or directory
#15 ERROR: process "/bin/sh -c zef install --force --/test --deps-only ." did not complete successfully: exit code: 1
------
 > [ 9/11] RUN zef install --force --/test --deps-only .:
45.33 ===> Building [FAIL]: OpenSSL:ver<0.2.5>:auth<zef:raku-community-modules>
45.33 [OpenSSL] Failed to build, but continuing with --force-build
45.33 ===> Building: Digest::SHA256::Native:ver<0.04>
45.33 ===> Building [OK] for Digest::SHA256::Native:ver<0.04>
45.33 ===> Installing: WWW:ver<1.005007>:auth<zef:raku-community-modules>
45.33 ===> Installing: Pastebin::Gist:ver<1.007>:auth<zef:raku-community-modules>
45.33 ===> Installing: Cro::Core:ver<0.8.10>:auth<zef:cro>:api<0>
45.33 ===> Installing: PathTools:ver<0.2.0>:auth<github:ugexe>
45.33 ===> Installing: OpenSSL:ver<0.2.5>:auth<zef:raku-community-modules>
45.33 ===> Install [FAIL] for OpenSSL:ver<0.2.5>:auth<zef:raku-community-modules>: Failed to open file /root/.zef/store/69991e0bc50d8ede1db29a06827e86c0ce2799df.tar.gz/OpenSSL-0.2.5/resources/libraries.json: No such file or directory
------
Dockerfile:12
--------------------
  10 |     COPY META6.json /srv
  11 |     RUN zef install --force --/test HTTP::HPACK # to work around the dependency issue
  12 | >>> RUN zef install --force --/test --deps-only .
  13 |     
  14 |     COPY .git/ /srv/.git/
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c zef install --force --/test --deps-only ." did not complete successfully: exit code: 1
Error: buildx failed with: ERROR: failed to build: failed to solve: process "/bin/sh -c zef install --force --/test --deps-only ." did not complete successfully: exit code: 1

Feel free to fix that. In the meantime, I just restarted the existing images manually.

The reason they're stuck is they get this at startup:

Starting Bisectable…
connection reset by peer

Ideally we need to add some kind of watchdog. I used to have it before when running the bots as systemd services, but not anymore. It can be something simple like:

• “If no IRC messages were seen in the last 200 minutes, then exit”
• Docker will restart the bot automatically

[lizmat]

Thanks for the restart: will look at the dependency issue tomorrow

[lizmat]

I've just uploaded OpenSSL 0.2.6 with a different build dependency.

Still not sure why this failed, as the listed dependencies all look find to me!

[lizmat]

Hmmmm there's something weird going on: on one of my older Debian boxes, JSON::Fast:ver<0.19>:auth<cpan:TIMOTIMO> could not be found as well. Then I did a zef upgrade zef, and after that it did install.

@AlexDaniel could you try doing a zef upgrade zef on that box?

[AlexDaniel]

There's no box for building, it's all in GitHub actions. You can try changing the dockerfile.

…

On Sun, 17 Aug 2025, 13:47 Elizabeth Mattijsen, ***@***.***> wrote: *lizmat* left a comment (Raku/whateverable#399) <#399 (comment)> Hmmmm there's something weird going on: on one of my older Debian boxes, JSON::Fast:ver<0.19>:auth<cpan:TIMOTIMO> could not be found as well. Then I did a zef upgrade zef, and after that it *did* install. @AlexDaniel <https://github.com/AlexDaniel> could you try doing a zef upgrade zef on that box? — Reply to this email directly, view it on GitHub <#399 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKATL6SHUPEVR6BPM5XY3L3OBMTJAVCNFSM6AAAAACEBYRILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCOJUGMYDCNJXGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[lizmat]

So done. How will we see the effects?

[lizmat]

Ok, took a few changes, but it appears setting the Rakudo Star to 2025-04 makes for a clean CI.

@AlexDaniel so all the bots went offline (presumably because of the successful change to the Dockerfile). And took a long time to come back. And some of them still aren't back:

• benchable6
• shareable6
• bisectable6
• quitable6
• linkable6

And oddly enough, one bot re-emerged: greppable6 which already left yesterday.

Is there a way to see whether they're still being built, or that they died trying?

[AlexDaniel]

@lizmat yep, you did everything right, thank you! Committing to the main branch results in restarting all the bots (and they come back with a random delay).

For bots that didn't start up, the output is like this:

Starting Bisectable…
connection reset by peer

Unfortunately, the process doesn't finish with that, and it just keeps hanging. If it did exit, then the service would automatically restart and the bots would be back online (eventually). It's a bit unclear as to where exactly this connection reset by peer is coming from and how to catch that, but if you could add it, it'd really help.

Another option, as I mentioned elsewhere, is to add some kind of watchdog. The logic should be “if there are no messages received in the last 3 hours, then kill the process”. Can be inside Whateverable.pm6 or any other file, as long as it does the trick.

[AlexDaniel]

@lizmat I have restarted the bots manually, and they came back online… but really they need to crash when “connection reset by peer” happens instead of just hanging.

[lizmat]

Thanks. And good to know.

Yes, they all need a workover. There is a "communification" branch where I started working on that. Need more tuits...