Pyramid v1.1.1 using all CPU resources and filling up memory

[FabioBonfiglio]

My pyramid 1.1.1 instance, approx. 10 minutes after startup, starts to use 100% of all CPU cores, and progressively fills all the available memory.

Here are the logs, after 15 minutes:

Mar 21 20:55:04 neuchstr systemd[1]: Started pyramid.service - pyramid relay.
Mar 21 20:55:04 neuchstr pyramid-exe[16320]: 8:55PM INF running pyramid version=v1.1.1
Mar 21 20:55:04 neuchstr pyramid-exe[16320]: 8:55PM DBG calculated free ranges from index scan 10015555=1209 10165498=192 10173298=1363 10187258=952 10193748=278 10266381=249 10268983=383 10299166=1449 10302649=393 1039512=300 10403275=145 10407470=348 1>
Mar 21 20:55:04 neuchstr pyramid-exe[16320]: 8:55PM INF running on https://0.0.0.0:443 and http://0.0.0.0:80
Mar 21 20:55:04 neuchstr pyramid-exe[16320]: 8:55PM INF fetched latest version source= version=v1.1.1
Mar 21 20:57:04 neuchstr pyramid-exe[16320]: 8:57PM INF fetching primary follow lists for members n=27 relay=inbox
Mar 21 20:57:14 neuchstr pyramid-exe[16320]: 8:57PM INF fetching secondary follow lists for follows n=2125 relay=inbox
Mar 21 20:58:04 neuchstr pyramid-exe[16320]: 8:58PM INF checking ots proofs
Mar 21 20:58:04 neuchstr pyramid-exe[16320]: 8:58PM INF upgraded pending OTS proofs errored=0 pending=4 upgraded=0
Mar 21 21:01:27 neuchstr pyramid-exe[16320]: 2026/03/21 21:01:27 NOTICE from wss://relay.gathr.gives: 'error: internal error processing subscription 2236:repl~3'
Mar 21 21:04:50 neuchstr pyramid-exe[16320]: [khatru-relay] 2026/03/21 21:04:50 failed to upgrade websocket: write tcp 10.240.1.3:443->162.158.217.38:9967: write: connection reset by peer
Mar 21 21:04:50 neuchstr pyramid-exe[16320]: [khatru-relay] 2026/03/21 21:04:50 failed to upgrade websocket: write tcp 10.240.1.3:443->162.158.217.24:9506: write: connection reset by peer
Mar 21 21:04:51 neuchstr pyramid-exe[16320]: [khatru-relay] 2026/03/21 21:04:51 failed to upgrade websocket: write tcp 10.240.1.3:443->172.71.141.75:13003: write: connection reset by peer
Mar 21 21:08:16 neuchstr pyramid-exe[16320]: [khatru-relay] 2026/03/21 21:08:16 error writing ping: write tcp 10.240.1.3:443->23.129.64.152:19937: write: broken pipe; closing websocket
Mar 21 21:09:03 neuchstr pyramid-exe[16320]: [khatru-relay] 2026/03/21 21:09:03 error writing ping: write tcp 10.240.1.3:443->45.84.107.17:2461: write: connection reset by peer; closing websocket
Mar 21 21:09:37 neuchstr pyramid-exe[16320]: [khatru-relay] 2026/03/21 21:09:37 error writing ping: write tcp 10.240.1.3:443->185.220.101.35:10879: write: connection reset by peer; closing websocket

At this point, already 3.5 GB of RAM is used, and growing at a pace of ~10 MB per second.

To be able to restart pyramid, I always need to manually delete the mmmm.lock file.

[Sakrecoer]

Same issue, on mine with same version number. RAM usage is toping arround 4.5gb. CPU got wild once, seems to have stabilized after ungraceful restart thus unable to discern a pattern.

[fiatjaf]

The CPU stuff is a bug I've seen but it happens very rarely to me, so it's hard to pinpoint. I think I might have a fix now but who knows. It's weird if it's happening constantly for you though.

Now the memory is even weirder, I have never seen it. My pyramid has a bunch of users and handles tons of requests per minute and memory usage tops at 800MB, but I think that's not the real number even, part of it is the Golang runtime preallocating memory before it's really needed.

We'll have to figure this out. What are the machines you're running pyramid on? A VPS? amd64?

[FabioBonfiglio]

My machine is a VM running on a baremetal server in a tier III datacenter, amd64 architecture, with 4 vCPUs, 8 GB RAM, 1 GB swap, and 500 GB storage (~20% used), the VM is running Ubuntu Server :

Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.4 LTS
Release:	24.04
Codename:	noble

Restarted pyramid this morning and now it seems fine ! 👀
But from last Thursday, until last night (thus ~48h) it was unexplainably doing what I described in the issue.

Also, there is another thing I noticed. This started to appear a couple of weeks ago, but I did not take the time to investigate what it is exactly.
I was monitoring the relay activity from grimoire.rocks with the following command:

req --title "Logs (all public, internal and personal events) on NEstr" -l 420 --since 7d -v compact nestr.nedao.ch nestr.nedao.ch/internal nestr.nedao.ch/personal

And it seems there is a constant flood of events kind 20001:

I don't know if it might be related to our problem.
Right now I can see approx 30% CPU usage and normal RAM (~less than 300 MB used) usage from Pyramid, as these kind 20001 floods are occurring right now.

Nothing special in the pyramid's logs.

I'll monitor more actively in the coming days and post updates here.

[fiatjaf]

30% CPU usage is already a problem. Mine is running in a less powerful VM and it averages 4%. I think you might already hit the bug (partly?) again.

Can you try updating to v1.1.2 and see if it fixes the issue?

But also this 20001 flood is super weird. How did you find about that? Do you know who might be flooding your relay with that? I'm not being flooded, but also I think rate-limits should be catching those attacks, I have to figure out why they aren't.

[FabioBonfiglio]

Not sure since when exactly, but now CPU usage is back down to ~6-7 % (and RAM usage still at less than 300 MB) !!
Here are the logs for today since I restarted it.

Mar 22 11:56:12 neuchstr systemd[1]: Started pyramid.service - pyramid relay.
Mar 22 11:56:12 neuchstr pyramid-exe[4811]: 11:56AM INF running pyramid version=v1.1.1
Mar 22 11:56:12 neuchstr pyramid-exe[4811]: 11:56AM DBG calculated free ranges from index scan 10015555=1209 10165498=192 10173298=1363 10187258=952 10193748=278 10266381=249 10268983=383 10299166=1449 10302649=393 1039512=300 10403275=145 10407470=348 1>
Mar 22 11:56:12 neuchstr pyramid-exe[4811]: 11:56AM INF running on https://0.0.0.0:443 and http://0.0.0.0:80
Mar 22 11:56:12 neuchstr pyramid-exe[4811]: 11:56AM INF fetched latest version source= version=v1.1.1
Mar 22 11:58:12 neuchstr pyramid-exe[4811]: 11:58AM INF fetching primary follow lists for members n=27 relay=inbox
Mar 22 11:58:18 neuchstr pyramid-exe[4811]: 11:58AM INF fetching secondary follow lists for follows n=2126 relay=inbox
Mar 22 11:59:12 neuchstr pyramid-exe[4811]: 11:59AM INF checking ots proofs
Mar 22 11:59:14 neuchstr pyramid-exe[4811]: 11:59AM INF publishing OTS event event="{\"kind\":1040,\"id\":\"6a208777a7f82bce539d90d79449a170222a40e99d8b530f7b9cf1cb15b98147\",\"pubkey\":\"309d9509a645583df2ec6d96506991ead80e72545af1473860729a1e9d18b3e6\">
Mar 22 11:59:14 neuchstr pyramid-exe[4811]: 11:59AM INF upgraded pending OTS proofs errored=0 pending=5 upgraded=1
Mar 22 11:59:55 neuchstr pyramid-exe[4811]: 2026/03/22 11:59:55 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 11:59:56 neuchstr pyramid-exe[4811]: 2026/03/22 11:59:56 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 11:59:59 neuchstr pyramid-exe[4811]: 2026/03/22 11:59:59 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:06 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:06 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:06 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:06 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:07 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:07 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:08 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:08 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:08 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:08 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:09 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:09 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:09 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:09 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:11 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:11 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:12 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:12 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:12 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:12 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:12 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:12 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:16 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:16 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:16 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:16 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:16 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:16 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:20 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:20 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:51 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:51 NOTICE from wss://relay.gathr.gives: 'error: internal error processing subscription 2198:repl~3'
Mar 22 12:00:55 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:55 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:55 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:55 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:56 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:56 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:56 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:56 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:58 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:58 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:59 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:59 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:00:59 neuchstr pyramid-exe[4811]: 2026/03/22 12:00:59 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:01:01 neuchstr pyramid-exe[4811]: 2026/03/22 12:01:01 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:01:01 neuchstr pyramid-exe[4811]: 2026/03/22 12:01:01 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:05 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:05 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:05 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:05 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:05 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:05 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:05 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:05 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:06 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:06 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:09 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:09 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:09 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:09 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:11 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:11 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:25 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:25 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:26 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:26 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:31 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:31 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:31 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:31 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:31 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:31 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:32 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:32 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:32 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:32 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:34 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:34 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:34 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:34 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:34 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:34 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:34 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:34 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:34 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:34 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:35 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:35 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:37 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:37 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:37 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:37 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:37 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:37 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:37 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:37 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:38 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:38 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:40 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:40 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:40 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:40 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:40 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:40 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:47 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:47 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:50 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:50 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:50 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:50 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:50 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:50 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:51 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:51 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:52 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:52 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:52 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:52 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:52 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:52 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:52 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:52 NOTICE from wss://relay.bullishbounty.com: 'rate limited'
Mar 22 12:02:52 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:52 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:02:53 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:53 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:02:53 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:53 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:02:55 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:55 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:02:55 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:55 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:02:55 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:55 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:02:55 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:55 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:02:57 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:57 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:02:58 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:58 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:02:58 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:58 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:02:59 neuchstr pyramid-exe[4811]: 2026/03/22 12:02:59 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
...
Mar 22 12:05:04 neuchstr pyramid-exe[4811]: 2026/03/22 12:05:04 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:05:06 neuchstr pyramid-exe[4811]: 2026/03/22 12:05:06 NOTICE from wss://relay.bullishbounty.com: 'Subscription rejected: Too many subscriptions: Number of subscriptions must be less than or equal to 10'
Mar 22 12:05:09 neuchstr pyramid-exe[4811]: 12:05PM INF finishing wot xor filter n=54996 relay=inbox
Mar 22 12:05:09 neuchstr pyramid-exe[4811]: [nl][info] 2026/03/22 12:05:09 computed aggregated WoT with 54996 entries
Mar 22 14:56:13 neuchstr pyramid-exe[4811]: 2:56PM INF fetched latest version source= version=v1.1.2
Mar 22 15:59:14 neuchstr pyramid-exe[4811]: 3:59PM INF checking ots proofs
Mar 22 15:59:15 neuchstr pyramid-exe[4811]: 3:59PM INF upgraded pending OTS proofs errored=0 pending=4 upgraded=0

About the kind 20001 flood, I noticed it as I use grimoire to both read my feeds and community (group) chat on wss://nestr.nedao.ch , and monitor it as well, with the spell I mentioned in my previous post.
It started a couple of weeks ago, and I have absolutely no idea where it could be coming from. 20001 isn't even in the accepted kinds list, so I thought it might be some kind of internal event of the pyramid relay, as it is in the ephemeral events range.

Not sure what that NOTICE from relay.bullishbounty.com is exactly, but I haven't seen it before and it stopped now (while the 20001 flood is still ongoing, so I don't think it is related to the kind 20001 flood.

Updating to v.1.1.2 right now. I'll monitor it and update here.

[FabioBonfiglio]

Kind 20001 events flood still ongoing after upgrade to v1.1.2.

Here are some examples of their content:

{
  "kind": 20001,
  "id": "fd3595e0244f2705b7b026b38c91eab3656d85f4d7c6e9ff98f3d7a0e5978934",
  "pubkey": "61258cfe71815db356bf98ad6f25d1e667592583e5ee67e045430fc85d429296",
  "created_at": 1774196884,
  "tags": [
    [
      "g",
      "spnpd"
    ]
  ],
  "content": "",
  "sig": "2b52ddf9b96a05e263319115df23c27ebb233cc15fa95cbac2f6f2935b52dbe6ce83ac16c30ddef7277f516ab90afa606d8ff6f219333b7fcd9d5acaf702af2b"
}

{
  "kind": 20001,
  "id": "c465d68dbcba74ba66722e7af50962ab261d24c53128adb10ddc031c0dfb2a91",
  "pubkey": "d57e814fbc4f791dd090ba1ac02be46b6dc7a410295aa5cb23ff9ff865813b9e",
  "created_at": 1774196999,
  "tags": [
    [
      "g",
      "sp"
    ]
  ],
  "content": "",
  "sig": "515e70dab548520987815e99adc5dffb9c1e53601d9d21ff6ec0c2f02cc2cacf800e2092a3e5cfea1a3d6908e9058dad99872b501a8ebde696f4d99c47a49b33"
}

{
  "kind": 20001,
  "id": "bb5798511cbce60b32ef98edd0cbfb41d2f1d7f6b5c173b0be58ee3fc547e2a1",
  "pubkey": "b4f12bbeb770c802c7dc6f5b8e3ee55ad879b86f71d91583a62e8f91834afe07",
  "created_at": 1774197023,
  "tags": [
    [
      "g",
      "snyx"
    ]
  ],
  "content": "",
  "sig": "94642d27936858052d7fb7c03ec1e9dabe0271ccfb0961f0aadc1d8637627816389657db2810286197ce4942a9a1d30b072f106bf417dfa2b5f715b71fb7edef"
}

{
  "kind": 20001,
  "id": "8a865fb0837adb5cb53d3c02a45aafa0acdb9149d341dfbaa7315c82ee00ed90",
  "pubkey": "95cb65b89a3e37cb44a07fdf10859c33c1b8702370e12c47c576453b2c75ea8e",
  "created_at": 1774197062,
  "tags": [
    [
      "g",
      "spnp"
    ]
  ],
  "content": "",
  "sig": "8fbde95b559fe1eba37737cb9b25cff66ac0d3feb978da876191cf660c9b4755f08a43df87122b67634ebf12d7fb7f33d6bcceb90c1b0fabf41dca698eaaba39"
}

As g tags are for geohashes, I decoded some of them, and most of them are pointing to general areas in the western Mediterranean, and some more precise locations in Sardinia, Switzerland, Tunisia,... !! 🤔

In the stats page of the pyramid dashboard, the kinds 20001 aren't listed.
Weird thing is that they are signed by random keys, so if coming from outside of the relay, they should not be processed by the WoT relay, no ?

[fiatjaf]

The bullishbounty notice I see and other pyramiders do too. It's caused by pyramid fetching data from other relays, which doesn't happen much: it only happens when we are building the WoT graph of relay participants and when an auto-curated event has to be fetched for /popular or /uppermost. This specific relay is the only one being way too strict in its limits even though pyramid is very careful when fetching data from other relays and tries to not open more than 15 subscriptions at the same time.

The kind:20001 is completely unrelated. Pyramid doesn't do it by itself, I had never seen it, this seems to be someone else creating these events and flooding you for no known reason other than causing turmoil.

Pyramid by default accepts ephemeral events from non-members, which I guess is something we should be able to toggle off.

It would also be interesting to have a log interface that displayed the IPs of people sending events so those could be blocked manually.

[FabioBonfiglio]

This morning, the relay is still running fine, and using ~7-8% CPU and ~350 MB RAM.

The limiting mechanism (which could also be adjustable within a given range) should indeed capture these events.
Perhaps at the moment it only restricts by public key, and a secondary rate limiting should be added by IP ?
However, some processes sometimes require several events to be sent in quick succession, like NIP-17, from the same IP, so the limitation mechanism must take that into account.

[Sakrecoer]

This afternoon, my pryramid is going full CPU usage again. Used RAM is hovering between 3 and 4gb. I've restarted it, and it calms down. For a moment. Logs aren't saying much. Running 1.1.2.

Mar 23 17:22:45 basspistol1 pyramid[141823]: 5:22PM INF running on http://127.0.0.1:8777
Mar 23 17:22:46 basspistol1 pyramid[141823]: 5:22PM INF fetched latest version source= version=v1.1.2
Mar 23 17:24:45 basspistol1 pyramid[141823]: 5:24PM INF fetching primary follow lists for members n=51 relay=inbox
Mar 23 17:25:07 basspistol1 pyramid[141823]: 5:25PM INF fetching secondary follow lists for follows n=3691 relay=inbox
Mar 23 17:25:14 basspistol1 pyramid[141823]: 2026/03/23 17:25:14 NOTICE from wss://sendit.nosflare.com: 'Denied! This relay does not accept REQs.'
Mar 23 17:25:45 basspistol1 pyramid[141823]: 5:25PM INF checking ots proofs
Mar 23 17:25:50 basspistol1 pyramid[141823]: 5:25PM INF publishing OTS event event="{\"kind\":1040,\"id\":\"cdaa15a21405a6f5b0e5a25071aed933c572053ea6e1ac9a79f6b7ba4a5c36eb\",\"pubkey\":\"afd7da3f794182dc3433c478c7e19735276faf16057a3ed8e58bf10944161041\",\"created_at\":1774276756,\"tags\":[[\"e\",\"6a352733dbc7ee053fa8542b608ce9322715966e2bf8bd299f33faf6b637d3ef\",\"wss://basspistol.org\"],[\"k\",\"1\"],[\"p\",\"16cb4b38fe9a748b6608f6a596913992dfc50906524c8311cd360bc40222ab94\"]],\"content\":\"AE9wZW5UaW1lc3RhbXBzAABQcm9vZgC/ieLohOiSlAEIajUnM9vH7gU/qFQrYIzpMicVlm4r+L0pnzP69rY30+/wCAy0N93+lol3CPAQEeCi+FF3LNS8xu28B36qHgjxIFiFivqeogfKt2BCn03mAYX4TGdVF8IBH2/gP/0I63AFCPEEacFQlPAIyzvFqzXdShII8SB1fN1TQxTgWeTnsWL3GKMv1KkaG0GEyJCvvAzRUp5MFAjxIF0OiiErxgi4KUoLjpGypdcfEUjFs679Kd7a8EsyDq+iCPEgB7O8hRbXhQ5H5PwOVqbL7IJLH6R4B/kCKh9gnJpaRi4I8SCfeaHijWWceRk9mWjAYeAePoUBIvU+i6JP/PQ5dHaN7gjxIDpbxE+4L1jxqU3Kqtm18LDMNJKgMfRnZWr323Cvg2DrCPAgHnyd4J0pJbyFn7IG1vePh7HLxikQJ7L+NI+9JadhC2YI8SDKwJSgZhOUbVNBaNUQ1Jje5BkHpYWicM5sHVT9dkvp5gjwINrU9pvQjzEXhAPyEwA/8cuWjEYjX2A49Ht989du4kfwCPAgKGEdoIuPidAjwoEy5R0rfEr9GuNKEY2nwFT4xzrqnosI8CBpkIvrzWwvxTIjF2MXEfKUxOVct2BvhHz0BB5s04Y86wjwIDun+etdPok+N3daT06/jDwthnvtsbelBAOQP21xSverCPEgeinK0lcZtw3lr2Nb5JZFaE+oRuJGKR3iwixOzs46YHAI8CAODaiCrVH2wwg3X6da8qarCV1cL9BNSrCHXhNNpt0qrwjxWQEAAAABjEzd7PbmvB4b85XgNpSKKwG1hPMPMxf5r4m2I8YZ8awAAAAAAP7///8CSq4BAAAAAAAWABR3Oy/WgcuMbp3jYuJoCCylt+WVdQAAAAAAAAAAImog8AQ2Xw4ACAjwIGPR0ZghH7C+Sr5QMoiWUz6J0g2wt4Oc0z1++RXkRCxNCAjwIH96NOXJJEvU3RXWkFHlEynbpbMSz92pqHjrIvGY1AxPCAjwIL+QRc4wxdp7g1T8FnDOjPnkITL1oGNqttRz5hlgJebTCAjxIIFrkVyWywcQHa/aLEvpeamS+bPp6JQh7WrCzxfUuisoCAjxINg+MglqzZavn69wZ7ze+qzM9jz3YEsLs69C1ClUh5PaCAjwIBHsRbh125zwoHxfdn7m7X1IKTr3lFZkAuGPpo9DnYDoCAjxIGKw7/Ut6IXFOv0mrvlqvftmlXZtPA54vSYcX2B893B+CAjxIEQHELx+3MIvf8m7B9lsIItitSCM8D3rW+xa7NK26G+eCAjxIM+ieZ7f6KEFJkFrTN5isP6ytw5a3sn4Cj26ns12RW0CCAjwIE5+2lRPpxQdWOm7cVKSVPYJqNKE/EjqBi4Juv30cZplCAjxIJcCEacz+nORv809mYOzqoEkekK05X2kGYUwI/xlBue3CAjwIFBj1huX2fyT3vrN7H3uj1ndnXKKTaLTsqg3BXeBuO7bCAgABYiWDXPXGQEDt745\",\"sig\":\"926428c42bbccfab09271269e01e76f8d0472018bcb059d1e8776bc097df6e152317b7c539c1bfcb4112f3b71cf44ba21fc6b5eac3df16176ce8798af5675d4d\"}"
Mar 23 17:25:53 basspistol1 pyramid[141823]: 5:25PM INF upgraded pending OTS proofs errored=0 pending=130 upgraded=1
Mar 23 17:28:07 basspistol1 pyramid[141823]: 5:28PM INF creating OTS proof id=4cb864f6eedbd74c2c6bb6a93e13fac48b9cc0db985ecf81e8c95a925c5dc066 kind=1 server=https://bob.btc.calendar.opentimestamps.org/
Mar 23 17:34:18 basspistol1 pyramid[141823]: 5:34PM INF creating OTS proof id=917b87fc8aab72c5be0e4ca6e6986dedac84a1d46f5e7d6c57c937fe82e660d3 kind=1 server=https://ots.btc.catallaxy.com/
Mar 23 17:36:04 basspistol1 pyramid[141823]: 2026/03/23 17:36:04 NOTICE from wss://sendit.nosflare.com: 'Denied! This relay does not accept REQs.'
Mar 23 17:42:28 basspistol1 pyramid[141823]: 5:42PM INF creating OTS proof id=e6e0305d4ae19601a73351c06c20950913775f46f3f0b1856c974669adb4b70f kind=1111 server=https://finney.calendar.eternitywall.com/
Mar 23 17:44:51 basspistol1 pyramid[141823]: 5:44PM INF creating OTS proof id=0000765e454850ab8592da2adc4d7a405382510e41c256d5b0c4a3c5d542aa13 kind=1 server=https://alice.btc.calendar.opentimestamps.org/
Mar 23 17:45:06 basspistol1 pyramid[141823]: 5:45PM INF finishing wot xor filter n=86701 relay=inbox
Mar 23 17:45:06 basspistol1 pyramid[141823]: [nl][info] 2026/03/23 17:45:06 computed aggregated WoT with 86701 entries
Mar 23 17:58:27 basspistol1 pyramid[141823]: 5:58PM INF creating OTS proof id=20243e0b11ec4541862237d5c0bf1169fc45bdc066b570d9f9774fbdbd1a764d kind=1111 server=https://bob.btc.calendar.opentimestamps.org/

[Sakrecoer]

The strange thing is that, while htop also reports 100% on all CPUs, the server is not acting like any of it is true: all other services on the machine are running and loading smooth...

[FabioBonfiglio]

Yep, 100% again on my instance too !
But not too much RAM usage this time... 🤷‍♂️

Logs for this afternoon:

Mar 23 11:16:38 neuchstr pyramid-exe[4811]: 11:16AM INF fetched latest version source= version=v1.1.2
Mar 23 11:19:10 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 11:19:10 failed to upgrade websocket: write tcp 10.240.1.3:443->162.158.102.175:12361: write: connecti>
Mar 23 11:35:52 neuchstr pyramid-exe[4811]: 11:35AM WRN failed to save to popular layer error="duplicate: event already exists"
Mar 23 11:35:52 neuchstr pyramid-exe[4811]: 11:35AM INF creating OTS proof id=497f44cd8a933a2207dabfa72853b00403dcd02efc77aab6174bea4dc3e3f90c kind=1 server=https://ots.btc>
Mar 23 11:51:29 neuchstr pyramid-exe[4811]: 11:51AM INF creating OTS proof id=00000083e23dfcd1eb713439e7840c7a47065015e6cc93091eb0432fad742e42 kind=1 server=https://finney.>
Mar 23 11:53:01 neuchstr pyramid-exe[4811]: 11:53AM INF creating OTS proof id=0000033dd139b802ca5af0159e6df539c73d5d5bd7478ebfc31f313edccd24a8 kind=1 server=https://alice.b>
Mar 23 11:55:02 neuchstr pyramid-exe[4811]: 11:55AM INF creating OTS proof id=00000286a4b0b13d2394db65792cc2cab2e4ab953ab5ae9ac475f37395de5240 kind=1 server=https://bob.btc>
Mar 23 13:20:19 neuchstr pyramid-exe[4811]: 1:20PM INF checking ots proofs
Mar 23 13:20:22 neuchstr pyramid-exe[4811]: 1:20PM INF publishing OTS event event="{\"kind\":1040,\"id\":\"1ae92826668c1728231421dc0401124b4ce3f83c0210a016f9d8d5d8c873b31b\>
Mar 23 13:20:23 neuchstr pyramid-exe[4811]: 1:20PM INF upgraded pending OTS proofs errored=0 pending=8 upgraded=1
Mar 23 13:33:15 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 13:33:15 failed to upgrade websocket: write tcp 10.240.1.3:443->104.23.217.118:9246: write: connection>
Mar 23 13:33:15 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 13:33:15 failed to upgrade websocket: write tcp 10.240.1.3:443->104.23.221.18:13650: write: connection>
Mar 23 13:33:15 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 13:33:15 failed to upgrade websocket: write tcp 10.240.1.3:443->104.23.221.76:13652: write: connection>
Mar 23 13:33:15 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 13:33:15 failed to upgrade websocket: write tcp 10.240.1.3:443->104.23.221.137:10271: write: connectio>
Mar 23 13:33:15 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 13:33:15 failed to upgrade websocket: write tcp 10.240.1.3:443->104.23.221.107:10612: write: connectio>
Mar 23 14:16:38 neuchstr pyramid-exe[4811]: 2:16PM INF fetched latest version source= version=v1.1.2
Mar 23 14:33:57 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:33:57 error writing ping: write tcp 10.240.1.3:443->109.70.100.7:41616: i/o timeout; closing websoc>
Mar 23 14:34:22 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:34:22 error writing ping: write tcp 10.240.1.3:443->90.9.57.176:31632: i/o timeout; closing websock>
Mar 23 14:38:31 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:38:31 error writing ping: write tcp 10.240.1.3:443->194.26.192.22:58188: i/o timeout; closing webso>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.22.48.140:10032: write: connection>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.22.48.177:10326: write: connection>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.23.216.205:12073: write: connectio>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.22.48.153:11223: write: connection>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.22.48.217:12144: write: connection>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.23.216.59:14077: write: connection>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.22.48.150:10131: write: connection>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.22.60.144:13216: write: connection>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.22.48.176:11375: write: connection>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.22.48.153:11221: write: connection>
Mar 23 14:51:43 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 14:51:43 failed to upgrade websocket: write tcp 10.240.1.3:443->104.23.216.208:11404: write: connectio>
Mar 23 17:16:39 neuchstr pyramid-exe[4811]: 5:16PM INF fetched latest version source= version=v1.1.2
Mar 23 17:20:23 neuchstr pyramid-exe[4811]: 5:20PM INF checking ots proofs
Mar 23 17:20:25 neuchstr pyramid-exe[4811]: 5:20PM INF publishing OTS event event="{\"kind\":1040,\"id\":\"8e27fb6ef7d1357e3cc754c10b7855453d0ced7cb1e002e904bdfe4a7a92a264\>
Mar 23 17:20:26 neuchstr pyramid-exe[4811]: 5:20PM INF upgraded pending OTS proofs errored=0 pending=7 upgraded=1
Mar 23 18:37:57 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 18:37:57 error writing ping: write tcp 10.240.1.3:443->185.220.101.34:13729: i/o timeout; closing webs>
Mar 23 18:37:58 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 18:37:58 error writing ping: write tcp 10.240.1.3:443->45.148.10.111:36688: i/o timeout; closing webso>
Mar 23 18:37:59 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 18:37:59 error writing ping: write tcp 10.240.1.3:443->193.189.100.198:64039: i/o timeout; closing web>
Mar 23 18:38:06 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 18:38:06 error writing ping: write tcp 10.240.1.3:443->45.141.215.114:36342: i/o timeout; closing webs>
Mar 23 18:38:11 neuchstr pyramid-exe[4811]: [khatru-relay] 2026/03/23 18:38:11 error writing ping: write tcp 10.240.1.3:443->185.220.101.163:32529: i/o timeout; closing web>

[FabioBonfiglio]

Been trying to restart it for the past hour but it still ends up at 100% CPU each time...

Mar 23 20:28:42 neuchstr systemd[1]: Started pyramid.service - pyramid relay.
Mar 23 20:28:42 neuchstr pyramid-exe[15428]: 8:28PM INF running pyramid version=v1.1.2
Mar 23 20:28:42 neuchstr pyramid-exe[15428]: 8:28PM DBG calculated free ranges from index scan 10165498=192 10174410=251 10193748=278 10266381=249 10268983=383 10302649=393>
Mar 23 20:28:42 neuchstr pyramid-exe[15428]: 8:28PM INF running on https://0.0.0.0:443 and http://0.0.0.0:80
Mar 23 20:28:42 neuchstr pyramid-exe[15428]: 8:28PM INF fetched latest version source= version=v1.1.2
Mar 23 20:30:42 neuchstr pyramid-exe[15428]: 8:30PM INF fetching primary follow lists for members n=27 relay=inbox
Mar 23 20:31:00 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/23 20:31:00 error writing ping: write tcp 10.240.1.3:443->185.220.101.179:7155: write: broken pipe; clos>
Mar 23 20:31:06 neuchstr pyramid-exe[15428]: 8:31PM INF fetching secondary follow lists for follows n=2126 relay=inbox
Mar 23 20:31:42 neuchstr pyramid-exe[15428]: 8:31PM INF checking ots proofs
Mar 23 20:31:44 neuchstr pyramid-exe[15428]: 8:31PM INF publishing OTS event event="{\"kind\":1040,\"id\":\"59bc63473ede027938af7dbb31b2a730449a35b6ccd283697000aed3ec61c1e7>
Mar 23 20:31:44 neuchstr pyramid-exe[15428]: 8:31PM INF upgraded pending OTS proofs errored=0 pending=6 upgraded=1
Mar 23 20:31:57 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/23 20:31:57 failed to upgrade websocket: write tcp 10.240.1.3:443->172.70.50.78:11968: write: connection>
Mar 23 20:32:03 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/23 20:32:03 error writing ping: write tcp 10.240.1.3:443->109.70.100.4:58922: write: broken pipe; closin>
Mar 23 20:32:40 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/23 20:32:40 error writing ping: write tcp 10.240.1.3:443->185.220.101.172:40237: write: broken pipe; clo>
Mar 23 20:33:00 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/23 20:33:00 error writing ping: write tcp 10.240.1.3:443->45.128.133.242:41948: write: broken pipe; clos>
Mar 23 20:34:02 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/23 20:34:02 error writing ping: write tcp 10.240.1.3:443->109.70.100.11:57866: write: connection reset b>
Mar 23 20:36:48 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/23 20:36:48 error writing ping: write tcp 10.240.1.3:443->172.71.141.164:9701: write: broken pipe; closi>
Mar 23 20:36:54 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/23 20:36:54 error writing ping: write tcp 10.240.1.3:443->45.84.107.174:54777: write: broken pipe; closi>
Mar 23 20:38:44 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/23 20:38:44 error writing ping: write tcp 10.240.1.3:443->185.220.101.172:24653: write: connection reset>
Mar 23 20:39:18 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/23 20:39:18 failed to upgrade websocket: write tcp 10.240.1.3:443->162.158.103.117:12130: write: connect>

[FabioBonfiglio]

This morning, pyramid was running with 100% CPU usage on two threads for a while, and then at the moment when I took the screenshot, it started to use >60% CPU on 4 - 5 threads :

Here are the logs for the past few hours:

Mar 24 08:28:45 neuchstr pyramid-exe[15428]: 8:28AM INF fetched latest version source= version=v1.1.2
Mar 24 08:31:49 neuchstr pyramid-exe[15428]: 8:31AM INF checking ots proofs
Mar 24 08:35:02 neuchstr pyramid-exe[15428]: 8:35AM INF upgraded pending OTS proofs errored=0 pending=5 upgraded=0
Mar 24 09:00:47 neuchstr pyramid-exe[15428]: 9:00AM INF creating OTS proof id=5eda2de0375ee2bb8e0980ff2e60ce1cdb24c4246b86ee1ac44e165564bd6ddf kind=1 server=https://ots.btc.catallaxy.com/
Mar 24 09:45:39 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 09:45:39 failed to upgrade websocket: write tcp 10.240.1.3:443->146.70.202.189:46028: write: connection reset by peer
Mar 24 09:47:08 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 09:47:08 error writing ping: write tcp 10.240.1.3:443->185.129.61.7:65488: write: connection reset by peer; closing websocket
Mar 24 09:47:24 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 09:47:24 error writing ping: write tcp 10.240.1.3:443->185.129.61.7:65384: write: broken pipe; closing websocket
Mar 24 09:47:55 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 09:47:55 error writing ping: write tcp 10.240.1.3:443->185.129.61.7:64298: write: connection reset by peer; closing websocket
Mar 24 09:48:27 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 09:48:27 error writing ping: write tcp 10.240.1.3:443->185.129.61.7:65280: write: connection reset by peer; closing websocket
Mar 24 09:48:51 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 09:48:51 error writing ping: write tcp 10.240.1.3:443->185.129.61.7:64828: write: broken pipe; closing websocket
Mar 24 09:49:10 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 09:49:10 error writing ping: write tcp 10.240.1.3:443->152.53.253.26:49976: i/o timeout; closing websocket
Mar 24 10:04:32 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:04:32 error writing ping: write tcp 10.240.1.3:443->45.84.107.128:38599: write: broken pipe; closing websocket
Mar 24 10:08:02 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:08:02 error writing ping: write tcp 10.240.1.3:443->74.207.252.32:60906: i/o timeout; closing websocket
Mar 24 10:25:29 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:25:29 error writing ping: write tcp 10.240.1.3:443->45.84.107.17:45063: write: broken pipe; closing websocket
Mar 24 10:25:44 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:25:44 error writing ping: write tcp 10.240.1.3:443->193.26.115.123:57438: write: connection reset by peer; closing websocket
Mar 24 10:26:05 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:26:05 error writing ping: write tcp 10.240.1.3:443->45.84.107.33:58011: write: connection reset by peer; closing websocket
Mar 24 10:29:00 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:29:00 failed to upgrade websocket: write tcp 10.240.1.3:443->172.69.162.203:13815: write: connection reset by peer
Mar 24 10:32:04 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:32:04 error writing ping: write tcp 10.240.1.3:443->185.220.101.61:54185: i/o timeout; closing websocket
Mar 24 10:35:02 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:35:02 error writing ping: write tcp 10.240.1.3:443->171.25.193.80:11414: write: broken pipe; closing websocket
Mar 24 10:50:19 neuchstr pyramid-exe[15428]: 2026/03/24 10:50:19 timeout waiting for SETTINGS frames from 45.76.253.154:43228
Mar 24 10:53:39 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:53:39 failed to upgrade websocket: write tcp 10.240.1.3:443->172.69.59.193:12755: write: connection reset by peer
Mar 24 10:54:43 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:54:43 failed to upgrade websocket: write tcp 10.240.1.3:443->172.69.59.194:11564: write: connection reset by peer
Mar 24 10:54:43 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:54:43 failed to upgrade websocket: write tcp 10.240.1.3:443->162.158.107.16:11920: write: connection reset by peer
Mar 24 10:54:46 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:54:46 error writing ping: write tcp 10.240.1.3:443->109.70.100.6:38504: i/o timeout; closing websocket
Mar 24 10:55:47 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 10:55:47 failed to upgrade websocket: write tcp 10.240.1.3:443->172.69.59.194:11106: write: connection reset by peer
Mar 24 11:01:25 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 11:01:25 failed to upgrade websocket: write tcp 10.240.1.3:443->8.227.214.226:63201: write: connection reset by peer
Mar 24 11:04:02 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 11:04:02 failed to upgrade websocket: write tcp 10.240.1.3:443->172.71.203.11:13907: write: connection reset by peer
Mar 24 11:15:15 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 11:15:15 error writing ping: write tcp 10.240.1.3:443->45.84.107.101:14719: write: connection reset by peer; closing websocket
Mar 24 11:17:53 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 11:17:53 error writing ping: write tcp 10.240.1.3:443->185.220.101.15:23112: i/o timeout; closing websocket
Mar 24 11:18:34 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 11:18:34 error writing ping: write tcp 10.240.1.3:443->45.84.107.97:11311: write: broken pipe; closing websocket
Mar 24 11:22:49 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 11:22:49 failed to upgrade websocket: write tcp 10.240.1.3:443->45.84.107.198:24735: write: broken pipe
Mar 24 11:29:15 neuchstr pyramid-exe[15428]: 11:29AM ERR failed to fetch latest release error="failed to fetch github release: Get \"https://api.github.com/repos/fiatjaf/pyramid/releases/latest\": context deadline exceeded (Client.Timeout exceeded while >
Mar 24 11:30:54 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 11:30:54 error writing ping: write tcp 10.240.1.3:443->45.84.107.198:7927: i/o timeout; closing websocket
Mar 24 11:31:35 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 11:31:35 error writing ping: write tcp 10.240.1.3:443->45.84.107.172:22941: i/o timeout; closing websocket
Mar 24 11:33:16 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 11:33:16 failed to upgrade websocket: write tcp 10.240.1.3:443->8.227.214.226:57415: write: connection reset by peer
Mar 24 12:30:42 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 12:30:42 error writing ping: write tcp 10.240.1.3:443->162.158.217.36:14265: write: broken pipe; closing websocket
Mar 24 12:31:05 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 12:31:05 error writing ping: write tcp 10.240.1.3:443->185.220.100.240:55112: write: broken pipe; closing websocket
Mar 24 12:31:27 neuchstr pyramid-exe[15428]: [khatru-relay] 2026/03/24 12:31:27 error writing ping: write tcp 10.240.1.3:443->162.158.217.93:9389: i/o timeout; closing websocket

[fiatjaf]

@FabioBonfiglio so it starts to ramp up CPU usage as soon as you turn it on? This may be different from the 100% CPU bug I've seen, because mine was so rare.

What happens if you run Pyramid locally? If you leave it running locally (with no activity) for a day does it also start to use a lot of CPU?

These logs don't really tell me anything unfortunately.

[FabioBonfiglio]

Yes, most of the time, pyramid ramps up CPU usage as soon as the service starts. But the RAM filling up is more rare, and happened only 2 or 3 times so far.
Most of the time, I can see 4 pyramid threads with 100% usage, filling up the 4 vCPUs of the machine, but the machine remains strangely responsive and quick, with even pyramid's web interface being very quick.
Some other times there are 5 to ~7 pyramid threads at ~50% CPU usage.
When the RAM fills up, then the pyramid service ends up crashing after ~20 - 30 hours, and I have to manually delete mmmm.lock to be able to restart it.

I have monitored I/O counters for a while, but nothing relevant was shown.