Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -258,3 +258,6 @@ bazel_auth.rc

# Meson
.meson-subproject*

# Copybara user config
shared/bazel/copybara/.copybara.json
7 changes: 7 additions & 0 deletions BUILD.bazel
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
load("@aspect_bazel_lib//lib:write_source_files.bzl", "write_source_files")
load("@rules_java//java:java_binary.bzl", "java_binary")
load("@rules_pkg//:mappings.bzl", "pkg_files")
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
load("//shared/bazel/rules:publishing.bzl", "publish_all")
Expand Down Expand Up @@ -41,6 +42,12 @@ alias(
visibility = ["//visibility:public"],
)

java_binary(
name = "copybara",
main_class = "com.google.copybara.Main",
runtime_deps = ["@com_github_google_copybara//jar"],
)

# This is a helper to run all of the pregeneration scripts at once.
write_source_files(
name = "write_pregenerated_files",
Expand Down
36 changes: 34 additions & 2 deletions README-RobotPy.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,42 @@ The upstream RobotPy repository uses toml configuration files and semiwrap to pr
Building the robotpy software on top of the standard C++/Java software can result in more than doubling the amount of time it takes to compile. To skip building the robotpy tooling you can add `--config=skip_robotpy` to the command line or to your `user.bazelrc`

# Syncing with robotpy
NOTE: This process is currently unlanded while robotpy gets the 2027 branch stable

[Copybara](https://github.com/google/copybara) is used to maintin synchronization between the upstream robotpy repositories and the allwpilib mirror. Github actions can be manually run which will create pull requests that will update all of the robotpy files between the two repositories. The ideal process is that the allwpilib mirror is always building in CI, and once a release is created the RobotPy team can run the `wpilib -> robotpy` copybara task, make any fine tuned adjustements and create their release. In the event that additional changes are made on the robotpy side, they can run the `robotpy -> wpilib` task to push the updates back to the mirror. However the goal of the mirroring the software here is to be able to more rapidly test changes and will hopefully overwhelmingly eliminate the need for syncs this direction.

## Creating a user config
The copybara scripts needs to know information about what repositories it will be pushing the sync'd changes. These can be specified on the command line, or you can create a `shared/bazel/copybara/.copybara.json` config file to save your personalized settings to avoid having to type things out every time. To run the full suite of migrations, you need a fork of [allwpilib](https://github.com/wpilibsuite/allwpilib), a fork of [mostrobotpy](https://github.com/robotpy/mostrobotpy), and a fork of robotpy's [commands-v2](https://github.com/robotpy/robotpy-commands-v2). If you only wish to run a subset of commands (i.e. not sync the commands project), you do not need to include that in your user config.

Example config:
```
{
"mostrobotpy_local_repo_path": "/home/<username>/git/robotpy/robotpy_monorepo/mostrobotpy",

"mostrobotpy_fork_repo": "https://github.com/<username>/mostrobotpy.git",
"allwpilib_fork_repo": "https://github.com/<username>/allwpilib.git",
"robotpy_commandsv2_fork_repo": "https://github.com/<username>/robotpy-commands-v2.git"
}
```

## Running syncs
- **Pulling changes from mostrobotpy**:

`python3 shared/bazel/copybara/run_copybara.py mostrobotpy_to_allwpilib`


- **Pulling changes from the commands library**:

`python3 shared/bazel/copybara/run_copybara.py commandsv2_to_allwpilib`

- **Pushing changes to the commands library**:

`python3 shared/bazel/copybara/run_copybara.py allwpilib_to_commandsv2`

- **Pushing changes to mostrobotpy**:

This process is slightly more complicated, because you will almost certainly also need to update the maven artifacts that mostrobopy is using. Because of this, you must also specify the version number that has been published to wpilibs maven repository. If you are trying to get an early, non-released development build pushed over, you can also add the `--development_build` flag

`python3 shared/bazel/copybara/run_copybara.py allwpilib_to_mostrobotpy --wpilib_bin_version=2027.0.0-alpha-3-86-g418b381 --development_build -y`


# Debugging Build Errors
The build process is highly automated and automatically parses C++ header files to generate pybind11 bindings. Some of these steps here are considered "pregeneration" steps, and the bazel build system will update build files as necessary. If a new header is added, or if the contents of a header file has changed, some of the pregeneration scripts might need to be run. If you encounter an error building `robotpy` code, it is recommended that you go through these steps to make sure everything is set up correctly. The examples are for `wpilibc`, but similar build tasks and tests exist for each wrapped project
Expand Down
8 changes: 7 additions & 1 deletion WORKSPACE
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_file")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_file", "http_jar")
load("//thirdparty/ceres:repositories.bzl", "ceres_repositories")

ceres_repositories()
Expand Down Expand Up @@ -440,3 +440,9 @@ doxygen_repository(
"1.15.0",
],
)

http_jar(
name = "com_github_google_copybara",
integrity = "sha256-IHW6y6WXJFjX9RYD+IwVAMwAbEo36fLqonIKR+FaqpQ=",
urls = ["https://github.com/google/copybara/releases/download/v20251027/copybara_deploy.jar"],
)
249 changes: 249 additions & 0 deletions copy.bara.sky
Original file line number Diff line number Diff line change
@@ -0,0 +1,249 @@
MOSTROBOTPY_PROJECTS = [
struct(
wpilib_name = "apriltag",
robotpy_name = "robotpy-apriltag",
native_robotpy_name = "robotpy-native-apriltag",
has_tests = True,
),
struct(
wpilib_name = "datalog",
robotpy_name = "robotpy-wpilog",
native_robotpy_name = "robotpy-native-datalog",
has_tests = True,
),
struct(
wpilib_name = "hal",
robotpy_name = "robotpy-hal",
native_robotpy_name = "robotpy-native-wpihal",
has_tests = True,
),
struct(
wpilib_name = "ntcore",
robotpy_name = "pyntcore",
native_robotpy_name = "robotpy-native-ntcore",
has_tests = True,
),
struct(
wpilib_name = "romiVendordep",
robotpy_name = "robotpy-romi",
native_robotpy_name = "robotpy-native-romi",
has_tests = True,
),
struct(
wpilib_name = "wpilibc",
robotpy_name = "robotpy-wpilib",
native_robotpy_name = "robotpy-native-wpilib",
has_tests = True,
),
struct(
wpilib_name = "wpimath",
robotpy_name = "robotpy-wpimath",
native_robotpy_name = "robotpy-native-wpimath",
has_tests = True,
),
struct(
wpilib_name = "wpinet",
robotpy_name = "robotpy-wpinet",
native_robotpy_name = "robotpy-native-wpinet",
has_tests = True,
),
struct(
wpilib_name = "wpiutil",
robotpy_name = "robotpy-wpiutil",
native_robotpy_name = "robotpy-native-wpiutil",
has_tests = True,
),
struct(
wpilib_name = "xrpVendordep",
robotpy_name = "robotpy-xrp",
native_robotpy_name = "robotpy-native-xrp",
has_tests = True,
),
]

IGNORED_MOSTROBOTPY_PROJECTS = [
"subprojects/robotpy-cscore",
"subprojects/robotpy-halsim-ds-socket",
"subprojects/robotpy-halsim-gui",
"subprojects/robotpy-halsim-ws",
]

def define_mostrobotpy_to_allwpilib():
origin_files = []
destination_files = []
transformations = []

rename_transforms = []
rename_transforms.append(core.replace(
before = 'version = "${version}"',
after = 'version = "0.0.0"',
regex_groups = {"version": ".*"},
paths = glob(["**/*.toml"]),
))

EXCLUDES = ["**/meson.build", "**/.gitignore", "**/requirements.txt", "**/.gittrack", "**/.gittrackexclude", "**/run_tests.py"]

for project_info in MOSTROBOTPY_PROJECTS:
origin_files += glob([
"subprojects/" + project_info.robotpy_name + "/**",
], exclude = EXCLUDES)

rename_transforms.append(core.replace(
before = '"' + project_info.robotpy_name + '==${version}"',
after = '"{}==0.0.0"'.format(project_info.robotpy_name),
regex_groups = {"version": ".*"},
paths = glob(["**/*.toml"]),
))

if project_info.native_robotpy_name:
origin_files += glob([
"subprojects/" + project_info.native_robotpy_name + "/pyproject.toml",
], exclude = EXCLUDES)

rename_transforms.append(core.replace(
before = '"' + project_info.native_robotpy_name + '==${version}"',
after = '"{}==0.0.0"'.format(project_info.native_robotpy_name),
regex_groups = {"version": ".*"},
paths = glob(["**/*.toml"]),
))

destination_files += glob([
project_info.wpilib_name + "/src/main/python/**",
project_info.wpilib_name + "/src/test/python/**",
], exclude = [])

if project_info.has_tests:
transformations.append(core.move("subprojects/" + project_info.robotpy_name + "/tests", project_info.wpilib_name + "/src/test/python"))
if project_info.native_robotpy_name:
transformations.append(core.move("subprojects/" + project_info.native_robotpy_name + "/pyproject.toml", "subprojects/" + project_info.native_robotpy_name + "/native-pyproject.toml"))
transformations.append(core.move("subprojects/" + project_info.robotpy_name, project_info.wpilib_name + "/src/main/python"))
if project_info.native_robotpy_name:
transformations.append(core.move("subprojects/" + project_info.native_robotpy_name + "/native-pyproject.toml", project_info.wpilib_name + "/src/main/python/native-pyproject.toml"))

rename_transforms.append(core.replace(
before = '"wpilib==${version}"',
after = '"wpilib==0.0.0"',
regex_groups = {"version": ".*"},
paths = glob(["**/*.toml"]),
))

rename_transforms.append(core.replace(
before = 'version = "0.0.0"',
after = 'version = "0.0.1"',
paths = ["subprojects/robotpy-wpiutil/tests/cpp/pyproject.toml"],
))
rename_transforms.append(core.replace(
before = 'version = "0.0.0"',
after = 'version = "0.1"',
paths = ["subprojects/robotpy-wpimath/tests/cpp/pyproject.toml"],
))

transformations = [core.transform(rename_transforms, noop_behavior = "IGNORE_NOOP", reversal = [])] + transformations

core.workflow(
name = "mostrobotpy_to_allwpilib",
origin = git.origin(
url = "https://github.com/robotpy/mostrobotpy.git",
ref = "2027",
),
destination = git.destination(
url = "https://github.com/OVERRIDE_ME/OVERRIDE_ME",
fetch = "2027",
push = "copybara_mostrobotpy_to_allwpilib",
),
destination_files = destination_files,
origin_files = origin_files,
authoring = authoring.pass_thru("Default email <[email protected]>"),
transformations = transformations,
)

def define_allwpilib_to_mostrobotpy():
ignored_project_exclude = [p + "/**" for p in IGNORED_MOSTROBOTPY_PROJECTS]
origin_files = []
destination_files = glob(["**"], exclude = ["*", ".github/**", "docs/**", "**/.gitignore", "**/meson.build", "**/requirements.txt", "devtools/**", "examples/**", "**/run_tests.py"] + ignored_project_exclude)
transformations = []

for project_info in MOSTROBOTPY_PROJECTS:
origin_files += glob([
project_info.wpilib_name + "/src/main/python/**",
project_info.wpilib_name + "/src/test/python/**",
], exclude = [])

if project_info.has_tests:
transformations.append(core.move(project_info.wpilib_name + "/src/test/python", "subprojects/" + project_info.robotpy_name + "/tests"))
transformations.append(core.move(project_info.wpilib_name + "/src/main/python", "subprojects/" + project_info.robotpy_name))
transformations.append(core.move("subprojects/" + project_info.robotpy_name + "/native-pyproject.toml", "subprojects/" + project_info.native_robotpy_name + "/pyproject.toml"))

core.workflow(
name = "allwpilib_to_mostrobotpy",
origin = git.origin(
url = "https://github.com/wpilibsuite/allwpilib.git",
ref = "2027",
),
destination = git.github_destination(
url = "https://github.com/OVERRIDE_ME/OVERRIDE_ME",
fetch = "2027",
push = "copybara_allwpilib_to_mostrobotpy",
),
destination_files = destination_files,
origin_files = origin_files,
authoring = authoring.pass_thru("Default email <[email protected]>"),
transformations = transformations,
)

def define_robotpy_commandsv2_to_allwpilib():
origin_files = glob(["commands2/**", "tests/**"], exclude = ["tests/run_tests.py", "tests/requirements.txt"])
destination_files = glob(["commandsv2/src/main/python/**", "commandsv2/src/test/python/**"])
transformations = []

transformations.append(core.move("commands2/", "commandsv2/src/main/python/commands2/"))
transformations.append(core.move("tests/", "commandsv2/src/test/python/"))

core.workflow(
name = "commandsv2_to_allwpilib",
origin = git.origin(
url = "https://github.com/robotpy/robotpy-commands-v2.git",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a good reason to maintain the robotpy-commands-v2 repo in the first place? It's pure-Python, so we could just build and publish it from here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that is a good question. IT shouldn't be hard to add the wheel creation and figure out how to get it pushed to pypi. I think that should be a @virtuald decision though

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's two separate problems with pushing it from here.

  • One is that @PeterJohnson often slides the tag, and PyPI won't allow you to overwrite a PyPI artifact. That part of the current allwpilib workflow would need to change.
  • Historically it's been useful to be able to fix python issues out-of-band from normal wpilib releases. Perhaps this is less important if everything is being built/maintained along with everything else.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking we could make it a manually triggered workflow (workflow_dispatch). Alternatively we could have that workflow trigger on a different tag naming convention. Either way seems like a solvable problem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either way I say punt kill the source of truth for commandsv-2 while the dust settles on this mirroring effort

ref = "2027",
),
destination = git.destination(
url = "https://github.com/OVERRIDE_ME/OVERRIDE_ME",
fetch = "2027",
push = "copybara_commandsv2_to_allwpilib",
),
destination_files = destination_files,
origin_files = origin_files,
authoring = authoring.pass_thru("Default email <[email protected]>"),
transformations = transformations,
)

def define_allwpilib_to_robotpy_commandsv2():
ignored_project_exclude = [p + "/**" for p in IGNORED_MOSTROBOTPY_PROJECTS]
origin_files = glob(["commandsv2/src/main/python/**", "commandsv2/src/test/python/**"])
destination_files = glob(["**"], exclude = ["*", ".github/**", "**/run_tests.py", "docs/**", "tests/requirements.txt"])
transformations = []

transformations.append(core.move("commandsv2/src/main/python/", ""))
transformations.append(core.move("commandsv2/src/test/python", "tests"))

core.workflow(
name = "allwpilib_to_commandsv2",
origin = git.origin(
url = "https://github.com/wpilibsuite/allwpilib.git",
ref = "2027",
),
destination = git.github_destination(
url = "https://github.com/OVERRIDE_ME/OVERRIDE_ME",
fetch = "2027",
push = "copybara_allwpilib_to_commandsv2",
),
destination_files = destination_files,
origin_files = origin_files,
authoring = authoring.pass_thru("Default email <[email protected]>"),
transformations = transformations,
)

define_mostrobotpy_to_allwpilib()
define_allwpilib_to_mostrobotpy()

define_robotpy_commandsv2_to_allwpilib()
define_allwpilib_to_robotpy_commandsv2()
Loading
Loading