Contributing RtSmith to P4Tools

[fruffy]

As part of a paper I wrote RtSmith: https://github.com/nyu-systems/rtsmith

This tool generates random control-plane configurations based on the API defined by the P4 program and the P4Runtime specification. The purpose was to test some tools that ingest control-plane configurations. Right now, it mostly generates Protos.

Now with agentic tools, it could be easy to contribute to the existing tools framework.

@smolkaj @qobilidop Does this have any use for you? I wonder what else you could do here. Also, happy to hear any suggestions on the name.

[smolkaj]

Generally yes, this could be useful! Would love to learn more about the details.

1. Are the entries random or are they synthesized to meet some objective?
2. On complex programs like SAI-P4, random entries tend to just drop and not exhibit interesting dataplane behavior. How does RtSmith address this if at all?
3. How does this compare to p4-fuzzer (which generates entries randomly, respecting validity -- including p4-constraints and @refers_to -- but not doing anything to address (2))?
4. How does this compare to p4-symbolic's table synthesis (which generates entries using Z3 in a targeted way to achieve specific packet fate and coverage goals).

In the spirit of agentic AI, I should probably pose these question along with your paper and GitHub repo to an AI first :) I have not yet done my homework. But sharing here for transparency.

[kfcripps]

@AlexeyAliev @alexseibe @dparker01 @eyg1331 FYI

[fruffy]

1. Are the entries random or are they synthesized to meet some objective?

Totally random, works similar to how P4Smith does: https://github.com/p4lang/p4c/tree/main/backends/p4tools/modules/smith

The goal is to check whether a particular control-plane API can handle all kinds of entries, sizes, and combinations of those without performance issues or crashes. It's less about generating random packet paths and more about checking for performance or robustness of the actual API layer.

We developed this tool as means to check for pathological cases or generate large amounts of table updates for a tool that ingests control-plane configurations. It was for Flay: https://dl.acm.org/doi/pdf/10.1145/3696348.3696870

2. On complex programs like SAI-P4, random entries tend to just drop and not exhibit interesting dataplane behavior.  How does RtSmith address this if at all?

Not in scope for RtSmith. So far this is a pure stochastic fuzzer, but I think there is an interesting direction there. Basically generating broad categories of control-plane entries that still make sense.

3. How does this compare to p4-fuzzer (which generates entries randomly, respecting validity -- including p4-constraints and @refers_to -- but not doing anything to address (2))?

p4-constraints and @refers_to annotations are not supported. But we should try to respect them down the line to make sure we generate valid configurations. You could also generate entries that explicitly violated the constraints in some form.

5. How does this compare to p4-symbolic's table synthesis (which generates entries using Z3 in a targeted way to achieve specific packet fate and coverage goals).> 

Generating particular entry sequences is something more in scope of tools such as P4Testgen, since I assume you'd need a constraint solver for it.

[smolkaj]

Thanks for the details, that’s very informative.