bug: Missing RecvPacket / ack for subset of Osmosis → Celestia packets (channel-6994 ↔ channel-2)

[jcstein]

Summary of Bug

Hey team, we're investigating an issue where both the go-relayer and hermes fail to acknowledge certain packets. These are all incoming packets from other chains to celestia (e.g. Osmosis has >275 packet commitments on the Celestia channel), we have tracked all packets and all have been received on Celestia, but none have been acknowledged on the Osmosis side.

Both hermes and the go relayer don't "see the packets"

Packets pending on source chain (osmosis-1):
  Unreceived packets:
  Unreceived acks:
Packets pending on destination chain (celestia):
  Unreceived packets:
  Unreceived acks:

Another weird characteristic is that these "unacknowledgable" packets aren't sequential, some other packets have been acknowledged successfully in between.

Does anyone have an idea why this is happening?

pending commitments on the Osmosis side: https://rest.cosmos.directory/osmosis/ibc/core/channel/v1/channels/channel-6994/ports/transfer/packet_commitments

hermes debug logs: (tx packet-ack) https://pastebin.com/aEbM62bP

FLUP:

unfortunately i need to revert on the "all packets have been received", it seems like we can't track 27 receives. Will keep digging

============================================================
Sequence #1455772
============================================================
(osmosis-1) send packet tx hash: 3DCF932F571605454898B258B019ADC7E28366C63A6342C93C1E0CBAC27DE8B2 (height: 48728279)
No receive-packet found on celestia for seq 1455772.
No acknowledgement found on osmosis-1 for seq 1455772.
Searching for timeout-packet…
No timeout-packet found either. Something’s up!

all packets: https://pastebin.com/XDMxgBpS

Version

Steps to Reproduce

https://www.mintscan.io/celestia/relayers/channel-2/osmosis/channel-6994

[damiannolan]

I'm writing to share what I already did in the discord thread here for completeness.

All of the txs I ended up triaging were packet forward middleware transactions originating from osmosis and going through celestia with a PFM memo intended to spawn a packet to a third-party chain.

Packet forward middleware IBC txs spawn another IBC packet. And require the usage of an asynchronous acknowledgement. This means the original packet which came from the source can not be ackwnowledged until the new packet spawned for the third-party chain completes its lifecycle and is acknowledged on Celestia.

1. For example, packet forward middleware tx intended for astria.

{
  "amount": "150000000",
  "denom": "transfer/channel-6994/utia",
  "memo": {
    "forward": {
      "receiver": "astria13vptdafyttpmlwppt0s844efey2cpc0mevy92p",
      "port": "transfer",
      "channel": "channel-48",
      "timeout": "10m",
      "retries": 2,
      "next": {
        "rollupDepositAddress": "0xce0f9257f77296afd46c319cbb1e666d403617dd"
      }
    }
  },
  "receiver": "celestia1rwe8rnw0a7n3sgwwu8jqvu0jhmz75jcj4k438ye",
  "sender": "osmo1rwe8rnw0a7n3sgwwu8jqvu0jhmz75jc4oyn8gx"
}

2. Another example of a more recent tx, intended for pryzm with a wasm contract call.

{
  "amount": "20000",
  "denom": "transfer/channel-6994/utia",
  "memo": {
    "forward": {
      "channel": "channel-34",
      "next": {
        "wasm": {
          "contract": "pryzm1tugep3navnjwlyjtnz7qzm2csnhpvg7y8gn9hffjqe37ymq5c76sz2e2xq",
          "msg": {
            "swap_and_action": {
              "affiliates": [
                {
                  "address": "pryzm1my4tk420gjmhggqwvvha6ey9330gqwfrfjwkaa",
                  "basis_points_fee": "57"
                },
                {
                  "address": "pryzm14gf6xslwe9phn2z965t4dcu7vchhthfgupmqv9",
                  "basis_points_fee": "18"
                }
              ],
              "min_asset": {
                "native": {
                  "amount": "182543",
                  "denom": "ibc/13B2C536BB057AC79D5616B8EA1B9540EC1F2170718CAF66F0083C966FFFED0B"
                }
              },
              "post_swap_action": {
                "ibc_transfer": {
                  "ibc_info": {
                    "memo": "",
                    "receiver": "osmo1dcxcprrmwrqm7lhwhsc2uys0ur998slr5hkd9z",
                    "receiver_address": "pryzm1dcxcprrmwrqm7lhwhsc2uys0ur998slryuj6dr",
                    "source_channel": "channel-2"
                  }
                }
              },
              "timeout_timestamp": 1763169828352930000,
              "user_swap": {
                "swap_exact_asset_in": {
                  "operations": [
                    {
                      "denom_in": "ibc/BF28D9C17E0306B194D50F51C3B2590BEAD15E04E03ADD34C3A26E62D85C9676",
                      "denom_out": "ibc/BFAAB7870A9AAABF6A7366DAAA0B8E5065EAA1FCE762F45677DC24BE796EF65",
                      "pool": "amm:21"
                    },
                    {
                      "denom_in": "ibc/BFAAB7870A9AAABF6A7366DAAA0B8E5065EAA1FCE762F45677DC24BE796EF65",
                      "denom_out": "ibc/13B2C536BB057AC79D5616B8EA1B9540EC1F2170718CAF66F0083C966FFFED0B",
                      "pool": "amm:11"
                    }
                  ],
                  "swap_venue_name": "pryzm-native"
                }
              }
            }
          }
        }
      }
    },
    "port": "transfer",
    "receiver": "pryzm1tugep3navnjwlyjtnz7qzm2csnhpvg7y8gn9hffjqe37ymq5c76sz2e2xq",
    "retries": 2,
    "timeout": 1763169828352914054
  },
  "receiver": "celestia1dcxcprrmwrqm7lhwhsc2uys0ur998slrdx5dfa",
  "sender": "osmo1dcxcprrmwrqm7lhwhsc2uys0ur998slr5hkd9z"
}

We can dig into the event logs from the MsgRecvPacket transaction on Celestia for example 2 above.
https://www.mintscan.io/celestia/tx/649F0DE56584FD975D74206F88614297CFFCB7D6E1E9066DE15BE9CB21B8173E?sector=logs

Here we can identify the PFM memo and it committing to a new packet intended for pryzm1tugep3navnjwlyjtnz7qzm2csnhpvg7y8gn9hffjqe37ymq5c76sz2e2xq.

We can query celestia to see if this packet has been acknowledged.

curl https://celestia-rest.publicnode.com/ibc/core/channel/v1/channels/channel-34/ports/transfer/packet_commitments/11828/unreceived_acks | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    84  100    84    0     0    625      0 --:--:-- --:--:-- --:--:--   622
{
  "sequences": [
    "11828"
  ],
  "height": {
    "revision_number": "0",
    "revision_height": "8616124"
  }
}

This confirms the second packet was never acknowledged on Celestia, and thus the original packet cannot be acknowledged on Osmosis because there has still not been an async ack written for it.
My suspicion is that this packet never reached przym and this is likely a dead channel with no active relayers.
If there was an active relayer they could timeout the second packet allowing the original one to also complete its lifecycle.