Python integration into WPILibSuite via gradle

[thenetworkgrinch]

How?

1. vendor-template modification to publish whls, maybe add an option into the vendordep json.
   • vendor-template would have a src/main/python folder for python files to be combined into the whl.
   • src/main/python/semiwrap would have the semiwrap yml files which will try to be generated via scraping the src/main/native LOCALLY if option is set to true in gradle.
   • semiwrap would run AFTER driver and native are built using compiled shared object libraries
   • semiwrap would generate a gradle file to build it that would then be loaded via apply in a doLast block.
   • (OPTIONAL) semiwrap would package the driver and native shared object files into the whl with a generated .pc file that points the include dir to be /usr/local/include/wpilib/vendors and lib to be /usr/local/lib/wpilib/vendors where the whl setup could place the vendordep .so's into their respective locations and the .pc into its respective location on the robot.
2. build.gradle for python projects which "builds"
   • downloads pip packages from requirements.txt THEN vendordep python packages from maven repo's in the vendordep json.
   • (OPTIONAL) downloads vendordep driver+native .so to .robotpy/lib/ and vendordep native headers to .robot/include for easier deployment.
   • Desktop whl's stored in venv
   • RoboRIO/SC whl's stored in .robotpy/venv
3. deploy.gradle for python projects
   • Checks if python is installed and up to date with latest python version from .robotpy/python
   • Installs python and startup scripts if not present on the robot
   • "Deploys" .robotpy/packages to python packages on the robot
   • Copies the robot project over to the robot.
4. WPILib Example projects for python using the new gradle files.
5. WPILib extension modifications to support intellisense and file recognition, as well as a thin wrapper around pip to install packages for robot arch and desktop arch.
6. Included python distribution in the WPILib installer

Why?

It is no secret that vendor support for Python is lacking. RobotPy development feels very different from traditional development within modern FRC since it is very disconnected from standard tooling. SystemCore will be much more capable of handling python and students will want to take advantage of that.

Python development shouldn't feel isolated, more difficult, or more limited than Java or C++ development. Vendors have already sunken in so much time into created their projects for existing languages that it would be considered a "waste" to focus on python when very few teams use it, the trade off isn't there. So we will have to make it easier on them to link and build inside of their existing infrastructure.

It is assumed that vendors update their vendordep gradle's annually so this would line up with those updates if done in the offseason and will especially benefit vendors who develop everything in C++ anyways.

Problems

Unfortunately those who don't use C++ as their native code will have to make their own semiwrap yml files for now, it may be possible to read and parse symbols to get classes/functions however that will not resolve parameter parsing unless done with real RE software like Ghidra or a similar tool. We could provide a Ghidra script to generate semiwrap yml files as well, possibly even a headless one that can run in CI; however the parameter resolution will not be perfect, unless done on a compiled object with debug symbols, it will be a decent start.

This is a HUGE undertaking and likely won't be ready until 2027.

[auscompgeek]

Unfortunately those who don't use C++ as their native code will have to make their own semiwrap yml files for now

semiwrap only supports reading C/C++ header files. My 2c: if a vendor is writing native code in a language other than C or C++, their tooling should be outputting a C or C++ header file to build against anyway for C++ teams.

Rust is a good example here, where cbindgen will generate C or C++ headers from Rust code with #[no_mangle] pub extern fn symbols.

[virtuald]

If a vendor does not support c/c++, a semiwrap yaml file does nothing. It only exists to wrap things, not create things. You could write c++, and just have those as part of the build though.

A vendor could write their own pybind11 bindings but if they're not going to do C++ they probably would be better off just doing pure python in 2027.

[nobody5050]

I support this. I'm of the opinion that a lot of the reason C++ is so well supported by vendors is the fact that the vendordep template makes it so easy. Finding a way to make python just as easy would be a great way to get vendor support for it.

[AustinSchuh]

There's more going on than just that. WPILib will continue to support C/C++ because that's the language needed to talk to the hardware, and forms the basis for language agnostic libraries used by JNI, rust support, python support, etc. Vendors work the same way, it is easier to write core, performant, and critical code in C/C++ and then call it from the various languages than to write it once per language.

Ultimately, market penetration drives resources. If lots of teams use Python, then it will be well supported.

[nobody5050]

I think we have the same opinion.

Vendors work the same way, it is easier to write core, performant, and critical code in C/C++ and then call it from the various languages than to write it once per language.

Let's make it easy to add python support and vendors will do so

[virtuald]

There's a lot of different things to unpack here.

1. vendor-template update for Python ... I think this sort of thing could be doable, but it probably would be best developed using an actual vendor codebase to make sure it's actually usable. There's a lot of bazel related work happening too, so it might be better to just tell vendors to do that instead. One advantage of building the python extension in the vendor template is you wouldn't need to have a vendor shared library -- just statically link the c++ code with the python extension and that would be simpler.

2/3/4. build.gradle and deploy.gradle and examples: I'm not sure anyone would use it -- it seems like it would be a lot more annoying to use than just updating the vscode extension to do things the robotpy way when a python project is involved? But maybe students would prefer a unified experience? You could try to prototype a simple version of this that just calls the existing robotpy tooling to do things and that would likely work.

5. VSCode extension updates to support Python - yes, someone needs to take time to do this, whether it be gradle or otherwise. Nobody has done it, so it hasn't been done. I'm not going to do it.
6. We've discussed this internally for awhile, would eventually like to do it. Need to complete something like Create offline bundle robotpy/robotpy-installer#120 to make it more doable I think.

[thenetworkgrinch]

2/3/4 would be used by teams familiar with java/c++ bc they would just use the templates/example code. Think how the current create project page has the language project template/example then which you would like to use, that is what I am proposing for the vscode.

[virtuald]

Sure. But vscode doesn't have to generate a build.gradle/deploy.gradle, and the various vscode actions dont have to invoke gradle. They just do so because that's what's required. One could just add some simple logic to detect what project type is present (which is probably already there) and then invoke the correct action whether it be gradle or a python script.

With that said, it should be easy to create a demo of this approach, and then you could give it to some users and get opinions.

[auscompgeek]

The unfinished WPILib Python VSCode extension even did exactly that; it knew you're in a Python project so it should run the relevant RobotPy commands with the active Python interpreter.