want humility command for reading out packrat ereport buffer

[hawkw]

at least for eliza's personal use.

it would be cool to be able to inspect what's in the buffer, so i can test ereport recording on bench systems without a switch. presently, ereport testing requires a management network if you want to see if the CBOR actually looks okay.

[hawkw]

(this might not be worth the effort if it turns out to be more than an afternoon of work...)

[mkeeter]

This came up at the Hubris Huddle today from a different perspective: if the net task doesn't come up in manufacturing, we don't have a great way to tell the system! We can't evacuate ereports along the usual network path, but if the net task reported an ereport and humility could read them, it would be helpful!

[hawkw]

Yeah, I think the needs of the manufacturing use-case is more motivating than my "it would be nice to test on systems without a management network"

[hawkw]

Thinking about this for a couple minutes, there are basically two ways to read the contents of the ereport buffer without using the management network:

1. Humility could reach directly into packrat's memory and read the contents of the ring buffer itself.

   This has the advantage of not requiring any cooperation from Hubris, and works as long as we can access memory via a probe, even if Hubris is completely non-responsive. On the other hand , it means we have to reimplement a decent chunk of the ring buffer algorithm in Humility, and it breaks if the in memory format for the ereport ringbuf changes pretty much at all.

2. Humility makes a hiffy call to the Packrat.read_ereports IPC, and gets back a 1024B buffer that's formatted identically to the UDP packet sent in response to an ereport request over the management network.

   This has the advantage of giving us back basically exactly what MGS gets, allowing us to reuse some of the ereport wire message definitions and stuff. It's also probably a lot less work in Humility, and is more resilient to changes to the internal implementation of the ereport ring buffer --- as long as the IPC interface doesn't change. And, we could easily ask the SP to flush ereports as part of this command, in addition to just reading them, which may be useful for the manufacturing use-case if the SP ends up generating a bunch of ereports and we want to get more out of it. On the other hand, this requires the SP's active cooperation: there has to be a hiffy in the target image, and both hiffy and packrat have to be functioning for it to work.

Currently, I think approach (2) seems like the better option. It's easier to implement in Humility, is less likely to break from internal changes in packrat, and gets us more or less exactly the same interface as reading ereports over the management network, including discarding old ones. I think the disadvantage of not working if Hubris is out to lunch is not a huge deal here, since the intent behind this is mostly not to debug issues with the SP firmware itself, but instead to check for hardware faults in manufacturing. We have other mechanisms for debugging a totally unresponsive SP (e.g. ringbufs) and I suspect that any data we could get from reading ereports will also be able to be accessed through other means --- in particular, we often include anything that goes in an ereport in a ringbuf entry, too.

If the manufacturing folks (cc @adamlouis, I think you were one of the people asking about this) disagree and think that the ability to get ereports from a stuck SP is really important for your use-case, I'm happy to reconsider.