Consider rewriting the Bash in Python

[dhuseby]

Hiya everybody, I know this may cause a bit of whiplash but I thought I should bring this up in light of recent developments on multiple fronts.

The choice to use Bash to write the scripts for running the tests was primary made for three reasons:

1. Bash runs on all systems with minimal/no setup
2. Bash scripts do not cause dependency maintenance hell like JS/TS/Node seems to.
3. Debugging and modifying the scripts only takes a text editor; there is no complicated dev environment setup required (e.g. Python venv, npm, etc).

I did a deep dive into the Cloudlab platform we're using for scaling experiments and their primary programming interface is a Python API. That made me start thinking...could we rewrite the Bash in Python while still meeting the three goals outlined above.

After doing some investigation, I think there is a path forward: PyInstaller. If we adopted PyInstaller, we could have the Python scripts in the repo for shared functions as well as test-specific functions and produce self-contained executables (e.g. ./run) in each of the test subfolders that run without any Python environment setup. If we design the transport/run.py application correctly, we can add in support for a --local-edits command line switch that checks the hash of each file included in the PyInstaller-built executable against the hash of the source python file and then use importlib.util to pull in the locally edited Python script to execute instead of the .pyc bytecode included in the executable. This would allow for a developer to clone the repo, make some edits to the Python scripts (i.e. fix a bug in transport/lib/run-single-test.py) and then run the test with ./run --local-edits ... and the ./run executable would pull in the locally edited transport/lib/run-single-test.py and use that code instead of the checked-in version that was built into the ./run executable. This would facilitate debugging and fixing of the framework itself without requiring a full Python development setup with virtualenv/conda/uv/etc.

This doesn't eliminate the dependency maintenance hell though. We would have to work hard to minimize the dependency tree as much as possible to minimize the number of external dependencies.

What this does buy us is this:

1. The framework is easier to understand and maintain. Most developers know Python enough and most are not Bash experts.
2. No Python dev environment nor dependency installation needed to run tests locally and debug/modify the test framework.
3. We gain a native integration with the Cloudlab platform while maintaining Docker integration

As it stands right now, we can use Cloudlab with the existing Bash scripts since most/all interactions are over SSH with pre-authorized keys similar to how Ansible works. I'm worried that the Bash would get even more complex than it already is and that will make the debug and maintenance of it a burden. Right now it's manageable but supporting Docker Swarm and Cloudlab at the same time might get super complicated with Bash.

I am decidedly NOT married to the Bash solution and if Python can replace it while reducing maintenance overhead and lowering the barrier to entry, I think we should consider it. I wanted to articulate this Python rewrite idea and see what everybody thinks.

Cheers! 🍻

[seetadev]

@dhuseby : This is fantastic, Dave. Appreciate it.

CCing key contributors from py-libp2p, lean-libp2p and dotnet-libp2p who will work with you on this initiative. Organized the 3 different teams, who will collaborate to enable achieving at the earliest:

1. Transport interop: @sumanjeet0012, @acul71, @yashksaini-coder, @Keerthivasan-Venkitajalam

2. Perf Module: @acul71, @itsmoh, @bismuth01, (@K-21 will contribute here). @lla-dane & @sumanjeet0012 will join this effort soon.

3. Hole punching interop: @Winter-Soren, @iron_jam , @parth-soni, @bismuth01

[paschal533]

Thank you @dhuseby for starting this discussion. I think the move to Python makes a lot of sense, especially with Cloudlab's Python API in the picture because trying to shell out to Python from Bash for 800+ machine orchestration would get messy fast. The Bash framework is running daily and producing results. I'd suggest we start by building the Cloudlab integration layer in Python, then progressively port the shared libraries (filter engine, image building, caching, etc.) as we touch them. That way we're never blocked on a big-bang rewrite. On PyInstaller, the idea of self-contained executables is appealing, but we might not need it. Python 3.8+ is available on pretty much every target system already. A simple requirements.txt with minimal dependencies could give us the same "clone and run" experience without the packaging overhead. We could always add PyInstaller later if distribution becomes a real pain point. The --local-edits flag is a great idea, being able to iterate on Python scripts without re-packaging is important for contributor DX. For testing the migration, we could run both implementations in parallel for a cycle to validate the Python version produces identical results before cutting over. I'll be working through the open issues this week.

[dhuseby]

I really want to avoid requiring developers to install any dependency libraries before they can run a test. 1. It causes dependency maintenance overhead and 2. it creates drag related to polluting development machines...do I use a venv? do I need to?

The PyInstaller packages everything up into an executable that saves developers from worrying about any dependencies.

[paschal533]

That makes total sense... keeping the barrier to entry as low as possible is important.

[bismuth01]

I think the bash setup is already pretty good and works really well for local development & testing as well as CI/CD testing.
When I personally worked on the leanp2p integration, due to gcc version constraints on my host Ubuntu 22.04 LTS, I had to do my development on an Ubuntu 24.04 Docker container and I made changes there and did the bash & docker based testing on my host machine.

If we are going for Cloudlabs for scaling experiments, why don't we just create another section of scripts for executing the experiments ? To the best of my knowledge, we want to interface with the Cloudlab's python API to create profiles of experiments and run them which can be kept completely separately with respect to local development.

[dhuseby]

we want to interface with the Cloudlab's python API to create profiles of experiments and run them

Yes. I am now thinking that we may just want to make a small PyInstaller executable cli tool that is a cloudlab interface tool similar to how the gh cli tool is an interface tool to GitHub. I'm envisioning a tool that can do things likes:

$ clab experiment init gossip-scaling-101
Experiment "gossip-scaling-101" initialized

# edit the experiment configuration file to define the machines, the topology, the test applications, etc

$ clab experiment start gossip-scaling-101
Experiment "gossip-scaling-101" started...
... provisioning machines...10...20...30...40...50 machines provisioned
... deploying Docker images
... starting metrics server... METRICS_ENDPOINT="https://172.24.0.64:1234"
... starting Docker images
... health check...10...20...30...40...50 machines reported healthy
... executing test
... "gossip-scaling-101" test completed in 10m 34s

[Winter-Soren]

Thank you, @dhuseby.

Switching to Python seems like the right move, especially since Cloudlab already provides a Python API. Trying to coordinate 800+ machines from Bash while constantly invoking Python scripts would become unwieldy pretty quickly. Given that the existing Bash framework is stable and running daily, I’d recommend migrating incrementally: start by implementing the Cloudlab integration layer in Python, then gradually transition shared components such as the filter engine, image builder, and caching, whenever we touch them. This approach minimizes the risk compared to a full rewrite all at once.

Regarding PyInstaller, although standalone executables are attractive, they don’t feel essential at this stage. Python 3.8+ is already present on most of our target systems, and keeping dependencies light with a simple requirements.txt should provide a comparable “clone and run” workflow without adding packaging complexity. If distribution later becomes burdensome, we can revisit PyInstaller then.

[dhuseby]

keeping dependencies light with a simple requirements.txt should provide a comparable “clone and run” workflow without adding packaging complexity.

I disagree on this point. I really hate it when I have to set up virtual environments for Python, Node, etc just to run an application. I strongly prefer AppImage packaged applications or statically linked binaries. Those are truly clone and run.

[Keerthivasan-Venkitajalam]

Hiya everybody, I know this may cause a bit of whiplash but I thought I should bring this up in light of recent developments on multiple fronts.

The choice to use Bash to write the scripts for running the tests was primary made for three reasons:

1. Bash runs on all systems with minimal/no setup
2. Bash scripts do not cause dependency maintenance hell like JS/TS/Node seems to.
3. Debugging and modifying the scripts only takes a text editor; there is no complicated dev environment setup required (e.g. Python venv, npm, etc).

I did a deep dive into the Cloudlab platform we're using for scaling experiments and their primary programming interface is a Python API. That made me start thinking...could we rewrite the Bash in Python while still meeting the three goals outlined above.

After doing some investigation, I think there is a path forward: PyInstaller. If we adopted PyInstaller, we could have the Python scripts in the repo for shared functions as well as test-specific functions and produce self-contained executables (e.g. ./run) in each of the test subfolders that run without any Python environment setup. If we design the transport/run.py application correctly, we can add in support for a --local-edits command line switch that checks the hash of each file included in the PyInstaller-built executable against the hash of the source python file and then use importlib.util to pull in the locally edited Python script to execute instead of the .pyc bytecode included in the executable. This would allow for a developer to clone the repo, make some edits to the Python scripts (i.e. fix a bug in transport/lib/run-single-test.py) and then run the test with ./run --local-edits ... and the ./run executable would pull in the locally edited transport/lib/run-single-test.py and use that code instead of the checked-in version that was built into the ./run executable. This would facilitate debugging and fixing of the framework itself without requiring a full Python development setup with virtualenv/conda/uv/etc.

This doesn't eliminate the dependency maintenance hell though. We would have to work hard to minimize the dependency tree as much as possible to minimize the number of external dependencies.

What this does buy us is this:

1. The framework is easier to understand and maintain. Most developers know Python enough and most are not Bash experts.
2. No Python dev environment nor dependency installation needed to run tests locally and debug/modify the test framework.
3. We gain a native integration with the Cloudlab platform while maintaining Docker integration

As it stands right now, we can use Cloudlab with the existing Bash scripts since most/all interactions are over SSH with pre-authorized keys similar to how Ansible works. I'm worried that the Bash would get even more complex than it already is and that will make the debug and maintenance of it a burden. Right now it's manageable but supporting Docker Swarm and Cloudlab at the same time might get super complicated with Bash.

I am decidedly NOT married to the Bash solution and if Python can replace it while reducing maintenance overhead and lowering the barrier to entry, I think we should consider it. I wanted to articulate this Python rewrite idea and see what everybody thinks.

Cheers! 🍻

@dhuseby from my side working on transport and misc, i can see how the bash scripts are starting to get more complex. they work well now, but adding Cloudlab and scaling logic could make them harder to maintain. agree that keeping things clone and run is important. if PyInstaller lets us ship a single executable without requiring venv or extra setup, that keeps the barrier low for contributors. an incremental approach sounds safest. maybe we can start with a small py tool for Cloudlab and see how that feels before migrating more of the framework. happy to help test or adapt transport and misc if we move in that direction.

[dhuseby]

the bash scripts are starting to get more complex.

true

maybe we can start with a small py tool for Cloudlab and see how that feels before migrating more of the framework.

i think this is the smartest way forward.

does anybody want to take a crack at learning the cloudlab Python API/SDK (https://docs.cloudlab.us) and writing an initial cloudlab cli tool that compiles to a PyInstaller executable?

[yashksaini-coder]

I want to take a crack, at it.