Team2890/allwpilib
4
0
Fork 0
mirror of https://github.com/wpilibsuite/allwpilib synced 2026-06-19 00:41:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

[py] Add copybara scripts (#8368)

Browse Source 
These are the scripts I've been using to sync between mostrobotpy and
here. I debated putting it in the "source of truth" that is
`mostrobotpy` , but I think it makes more sense here since it already
has bazel set up, and I've also recently added the ability to sync the
`commands-v2` repository, so having it all in one copybara script makes
sense.

This includes a helper python script to make it a little bit easier to
run.
This commit is contained in:
PJ Reiniger
2025-12-12 23:06:19 -05:00
committed by GitHub
parent 13abb3d332
commit 0049c6f23f
6 changed files with 557 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@@ -258,3 +258,6 @@ bazel_auth.rc
# Meson
.meson-subproject*
# Copybara user config
shared/bazel/copybara/.copybara.json

7
BUILD.bazel
View File

@@ -1,4 +1,5 @@
load("@aspect_bazel_lib//lib:write_source_files.bzl", "write_source_files")
load("@rules_java//java:java_binary.bzl", "java_binary")
load("@rules_pkg//:mappings.bzl", "pkg_files")
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
load("//shared/bazel/rules:publishing.bzl", "publish_all")
@@ -41,6 +42,12 @@ alias(
visibility = ["//visibility:public"],
)
java_binary(
name = "copybara",
main_class = "com.google.copybara.Main",
runtime_deps = ["@com_github_google_copybara//jar"],
)
# This is a helper to run all of the pregeneration scripts at once.
write_source_files(
name = "write_pregenerated_files",

36
README-RobotPy.md
View File

@@ -8,10 +8,42 @@ The upstream RobotPy repository uses toml configuration files and semiwrap to pr
Building the robotpy software on top of the standard C++/Java software can result in more than doubling the amount of time it takes to compile. To skip building the robotpy tooling you can add `--config=skip_robotpy` to the command line or to your `user.bazelrc`
# Syncing with robotpy
NOTE: This process is currently unlanded while robotpy gets the 2027 branch stable
[Copybara](https://github.com/google/copybara) is used to maintin synchronization between the upstream robotpy repositories and the allwpilib mirror. Github actions can be manually run which will create pull requests that will update all of the robotpy files between the two repositories. The ideal process is that the allwpilib mirror is always building in CI, and once a release is created the RobotPy team can run the `wpilib -> robotpy` copybara task, make any fine tuned adjustements and create their release. In the event that additional changes are made on the robotpy side, they can run the `robotpy -> wpilib` task to push the updates back to the mirror. However the goal of the mirroring the software here is to be able to more rapidly test changes and will hopefully overwhelmingly eliminate the need for syncs this direction.
## Creating a user config
The copybara scripts needs to know information about what repositories it will be pushing the sync'd changes. These can be specified on the command line, or you can create a `shared/bazel/copybara/.copybara.json` config file to save your personalized settings to avoid having to type things out every time. To run the full suite of migrations, you need a fork of [allwpilib](https://github.com/wpilibsuite/allwpilib), a fork of [mostrobotpy](https://github.com/robotpy/mostrobotpy), and a fork of robotpy's [commands-v2](https://github.com/robotpy/robotpy-commands-v2). If you only wish to run a subset of commands (i.e. not sync the commands project), you do not need to include that in your user config.
Example config:
```
{
"mostrobotpy_local_repo_path": "/home/<username>/git/robotpy/robotpy_monorepo/mostrobotpy",
"mostrobotpy_fork_repo": "https://github.com/<username>/mostrobotpy.git",
"allwpilib_fork_repo": "https://github.com/<username>/allwpilib.git",
"robotpy_commandsv2_fork_repo": "https://github.com/<username>/robotpy-commands-v2.git"
}
```
## Running syncs
- **Pulling changes from mostrobotpy**:
`python3 shared/bazel/copybara/run_copybara.py mostrobotpy_to_allwpilib`
- **Pulling changes from the commands library**:
`python3 shared/bazel/copybara/run_copybara.py commandsv2_to_allwpilib`
- **Pushing changes to the commands library**:
`python3 shared/bazel/copybara/run_copybara.py allwpilib_to_commandsv2`
- **Pushing changes to mostrobotpy**:
This process is slightly more complicated, because you will almost certainly also need to update the maven artifacts that mostrobopy is using. Because of this, you must also specify the version number that has been published to wpilibs maven repository. If you are trying to get an early, non-released development build pushed over, you can also add the `--development_build` flag
`python3 shared/bazel/copybara/run_copybara.py allwpilib_to_mostrobotpy --wpilib_bin_version=2027.0.0-alpha-3-86-g418b381 --development_build -y`
# Debugging Build Errors
The build process is highly automated and automatically parses C++ header files to generate pybind11 bindings. Some of these steps here are considered "pregeneration" steps, and the bazel build system will update build files as necessary. If a new header is added, or if the contents of a header file has changed, some of the pregeneration scripts might need to be run. If you encounter an error building `robotpy` code, it is recommended that you go through these steps to make sure everything is set up correctly. The examples are for `wpilibc`, but similar build tasks and tests exist for each wrapped project

8
WORKSPACE
View File

@@ -1,4 +1,4 @@
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_file")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_file", "http_jar")
load("//thirdparty/ceres:repositories.bzl", "ceres_repositories")
ceres_repositories()
@@ -413,3 +413,9 @@ doxygen_repository(
"1.15.0",
],
)
http_jar(
name = "com_github_google_copybara",
integrity = "sha256-IHW6y6WXJFjX9RYD+IwVAMwAbEo36fLqonIKR+FaqpQ=",
urls = ["https://github.com/google/copybara/releases/download/v20251027/copybara_deploy.jar"],
)

249
copy.bara.sky Normal file
View File

@@ -0,0 +1,249 @@
MOSTROBOTPY_PROJECTS = [
struct(
wpilib_name = "apriltag",
robotpy_name = "robotpy-apriltag",
native_robotpy_name = "robotpy-native-apriltag",
has_tests = True,
),
struct(
wpilib_name = "datalog",
robotpy_name = "robotpy-wpilog",
native_robotpy_name = "robotpy-native-datalog",
has_tests = True,
),
struct(
wpilib_name = "hal",
robotpy_name = "robotpy-hal",
native_robotpy_name = "robotpy-native-wpihal",
has_tests = True,
),
struct(
wpilib_name = "ntcore",
robotpy_name = "pyntcore",
native_robotpy_name = "robotpy-native-ntcore",
has_tests = True,
),
struct(
wpilib_name = "romiVendordep",
robotpy_name = "robotpy-romi",
native_robotpy_name = "robotpy-native-romi",
has_tests = True,
),
struct(
wpilib_name = "wpilibc",
robotpy_name = "robotpy-wpilib",
native_robotpy_name = "robotpy-native-wpilib",
has_tests = True,
),
struct(
wpilib_name = "wpimath",
robotpy_name = "robotpy-wpimath",
native_robotpy_name = "robotpy-native-wpimath",
has_tests = True,
),
struct(
wpilib_name = "wpinet",
robotpy_name = "robotpy-wpinet",
native_robotpy_name = "robotpy-native-wpinet",
has_tests = True,
),
struct(
wpilib_name = "wpiutil",
robotpy_name = "robotpy-wpiutil",
native_robotpy_name = "robotpy-native-wpiutil",
has_tests = True,
),
struct(
wpilib_name = "xrpVendordep",
robotpy_name = "robotpy-xrp",
native_robotpy_name = "robotpy-native-xrp",
has_tests = True,
),
]
IGNORED_MOSTROBOTPY_PROJECTS = [
"subprojects/robotpy-cscore",
"subprojects/robotpy-halsim-ds-socket",
"subprojects/robotpy-halsim-gui",
"subprojects/robotpy-halsim-ws",
]
def define_mostrobotpy_to_allwpilib():
origin_files = []
destination_files = []
transformations = []
rename_transforms = []
rename_transforms.append(core.replace(
before = 'version = "${version}"',
after = 'version = "0.0.0"',
regex_groups = {"version": ".*"},
paths = glob(["**/*.toml"]),
))
EXCLUDES = ["**/meson.build", "**/.gitignore", "**/requirements.txt", "**/.gittrack", "**/.gittrackexclude", "**/run_tests.py"]
for project_info in MOSTROBOTPY_PROJECTS:
origin_files += glob([
"subprojects/" + project_info.robotpy_name + "/**",
], exclude = EXCLUDES)
rename_transforms.append(core.replace(
before = '"' + project_info.robotpy_name + '==${version}"',
after = '"{}==0.0.0"'.format(project_info.robotpy_name),
regex_groups = {"version": ".*"},
paths = glob(["**/*.toml"]),
))
if project_info.native_robotpy_name:
origin_files += glob([
"subprojects/" + project_info.native_robotpy_name + "/pyproject.toml",
], exclude = EXCLUDES)
rename_transforms.append(core.replace(
before = '"' + project_info.native_robotpy_name + '==${version}"',
after = '"{}==0.0.0"'.format(project_info.native_robotpy_name),
regex_groups = {"version": ".*"},
paths = glob(["**/*.toml"]),
))
destination_files += glob([
project_info.wpilib_name + "/src/main/python/**",
project_info.wpilib_name + "/src/test/python/**",
], exclude = [])
if project_info.has_tests:
transformations.append(core.move("subprojects/" + project_info.robotpy_name + "/tests", project_info.wpilib_name + "/src/test/python"))
if project_info.native_robotpy_name:
transformations.append(core.move("subprojects/" + project_info.native_robotpy_name + "/pyproject.toml", "subprojects/" + project_info.native_robotpy_name + "/native-pyproject.toml"))
transformations.append(core.move("subprojects/" + project_info.robotpy_name, project_info.wpilib_name + "/src/main/python"))
if project_info.native_robotpy_name:
transformations.append(core.move("subprojects/" + project_info.native_robotpy_name + "/native-pyproject.toml", project_info.wpilib_name + "/src/main/python/native-pyproject.toml"))
rename_transforms.append(core.replace(
before = '"wpilib==${version}"',
after = '"wpilib==0.0.0"',
regex_groups = {"version": ".*"},
paths = glob(["**/*.toml"]),
))
rename_transforms.append(core.replace(
before = 'version = "0.0.0"',
after = 'version = "0.0.1"',
paths = ["subprojects/robotpy-wpiutil/tests/cpp/pyproject.toml"],
))
rename_transforms.append(core.replace(
before = 'version = "0.0.0"',
after = 'version = "0.1"',
paths = ["subprojects/robotpy-wpimath/tests/cpp/pyproject.toml"],
))
transformations = [core.transform(rename_transforms, noop_behavior = "IGNORE_NOOP", reversal = [])] + transformations
core.workflow(
name = "mostrobotpy_to_allwpilib",
origin = git.origin(
url = "https://github.com/robotpy/mostrobotpy.git",
ref = "2027",
),
destination = git.destination(
url = "https://github.com/OVERRIDE_ME/OVERRIDE_ME",
fetch = "2027",
push = "copybara_mostrobotpy_to_allwpilib",
),
destination_files = destination_files,
origin_files = origin_files,
authoring = authoring.pass_thru("Default email <default@default.com>"),
transformations = transformations,
)
def define_allwpilib_to_mostrobotpy():
ignored_project_exclude = [p + "/**" for p in IGNORED_MOSTROBOTPY_PROJECTS]
origin_files = []
destination_files = glob(["**"], exclude = ["*", ".github/**", "docs/**", "**/.gitignore", "**/meson.build", "**/requirements.txt", "devtools/**", "examples/**", "**/run_tests.py"] + ignored_project_exclude)
transformations = []
for project_info in MOSTROBOTPY_PROJECTS:
origin_files += glob([
project_info.wpilib_name + "/src/main/python/**",
project_info.wpilib_name + "/src/test/python/**",
], exclude = [])
if project_info.has_tests:
transformations.append(core.move(project_info.wpilib_name + "/src/test/python", "subprojects/" + project_info.robotpy_name + "/tests"))
transformations.append(core.move(project_info.wpilib_name + "/src/main/python", "subprojects/" + project_info.robotpy_name))
transformations.append(core.move("subprojects/" + project_info.robotpy_name + "/native-pyproject.toml", "subprojects/" + project_info.native_robotpy_name + "/pyproject.toml"))
core.workflow(
name = "allwpilib_to_mostrobotpy",
origin = git.origin(
url = "https://github.com/wpilibsuite/allwpilib.git",
ref = "2027",
),
destination = git.github_destination(
url = "https://github.com/OVERRIDE_ME/OVERRIDE_ME",
fetch = "2027",
push = "copybara_allwpilib_to_mostrobotpy",
),
destination_files = destination_files,
origin_files = origin_files,
authoring = authoring.pass_thru("Default email <default@default.com>"),
transformations = transformations,
)
def define_robotpy_commandsv2_to_allwpilib():
origin_files = glob(["commands2/**", "tests/**"], exclude = ["tests/run_tests.py", "tests/requirements.txt"])
destination_files = glob(["commandsv2/src/main/python/**", "commandsv2/src/test/python/**"])
transformations = []
transformations.append(core.move("commands2/", "commandsv2/src/main/python/commands2/"))
transformations.append(core.move("tests/", "commandsv2/src/test/python/"))
core.workflow(
name = "commandsv2_to_allwpilib",
origin = git.origin(
url = "https://github.com/robotpy/robotpy-commands-v2.git",
ref = "2027",
),
destination = git.destination(
url = "https://github.com/OVERRIDE_ME/OVERRIDE_ME",
fetch = "2027",
push = "copybara_commandsv2_to_allwpilib",
),
destination_files = destination_files,
origin_files = origin_files,
authoring = authoring.pass_thru("Default email <default@default.com>"),
transformations = transformations,
)
def define_allwpilib_to_robotpy_commandsv2():
ignored_project_exclude = [p + "/**" for p in IGNORED_MOSTROBOTPY_PROJECTS]
origin_files = glob(["commandsv2/src/main/python/**", "commandsv2/src/test/python/**"])
destination_files = glob(["**"], exclude = ["*", ".github/**", "**/run_tests.py", "docs/**", "tests/requirements.txt"])
transformations = []
transformations.append(core.move("commandsv2/src/main/python/", ""))
transformations.append(core.move("commandsv2/src/test/python", "tests"))
core.workflow(
name = "allwpilib_to_commandsv2",
origin = git.origin(
url = "https://github.com/wpilibsuite/allwpilib.git",
ref = "2027",
),
destination = git.github_destination(
url = "https://github.com/OVERRIDE_ME/OVERRIDE_ME",
fetch = "2027",
push = "copybara_allwpilib_to_commandsv2",
),
destination_files = destination_files,
origin_files = origin_files,
authoring = authoring.pass_thru("Default email <default@default.com>"),
transformations = transformations,
)
define_mostrobotpy_to_allwpilib()
define_allwpilib_to_mostrobotpy()
define_robotpy_commandsv2_to_allwpilib()
define_allwpilib_to_robotpy_commandsv2()

257
shared/bazel/copybara/run_copybara.py Executable file
View File

@@ -0,0 +1,257 @@
import argparse
import dataclasses
import json
import os
import pathlib
import re
import subprocess
from typing import Optional
@dataclasses.dataclass
class CopybaraConfig:
# Needed to run the additional updates for updating the rdev file
mostrobotpy_local_repo_path: Optional[str] = None
# Settings for migrating to a fork that you own
mostrobotpy_fork_repo: Optional[str] = None
allwpilib_fork_repo: Optional[str] = None
robotpy_commandsv2_fork_repo: Optional[str] = None
# Settings for truth repositories
mostrobotpy_truth_repo: str = "https://github.com/robotpy/mostrobotpy.git"
mostrobotpy_truth_branch: str = "2027"
allwpilib_truth_repo: str = "https://github.com/wpilibsuite/allwpilib.git"
allwpilib_truth_branch: str = "2027"
def run_copybara(copybara_file: pathlib.Path, migration: str, destination_url: str):
args = [
"bazel",
"run",
"//:copybara",
"--",
"migrate",
str(copybara_file),
migration,
"--force",
"--git-destination-url",
destination_url,
"--git-destination-non-fast-forward",
]
subprocess.check_call(args)
def checkout_branch(auto_delete_branch: bool, branch_name: str):
"""
This will attempt run a fetch on the repository in the cwd and checkout the given branch.
If the branch currently exists locally, it will be deleted before the fetch happens.
"""
# Check if the magic branch exists locally, and if so attempt to delete it
ret = subprocess.call(["git", "rev-parse", "--verify", branch_name])
branch_exists = ret == 0
if branch_exists:
if not auto_delete_branch:
ans = input(f"Delete local branch {branch_name}?")
if ans.lower() != "y":
raise Exception(
f"You must delete your local copy of {branch_name} before the script can finish"
)
subprocess.check_call(["git", "branch", "-D", branch_name])
subprocess.check_call(["git", "fetch", "--all"])
subprocess.check_call(["git", "checkout", branch_name])
def update_mostrobotpy_rdev(wpilib_bin_version: str, is_development_build: bool):
with open("rdev.toml") as f:
contents = f.read()
artifactory_path = "https://frcmaven.wpi.edu/artifactory/"
if is_development_build:
artifactory_path += "development-2027"
else:
artifactory_path += "release-2027"
contents = re.sub(
'wpilib_bin_version = ".*"',
f'wpilib_bin_version = "{wpilib_bin_version}"',
contents,
)
contents = re.sub(
'wpilib_bin_url = ".*"', f'wpilib_bin_url = "{artifactory_path}"', contents
)
with open("rdev.toml", "w") as f:
f.write(contents)
subprocess.check_call(["./rdev.sh", "update-pyproject", "--commit"])
def allwpilib_to_mostrobotpy(
copybara_file: pathlib.Path,
mostrobotpy_local_repository: str,
mostrobotpy_fork_repo: str,
wpilib_bin_version: str,
is_development_build: bool,
auto_delete_branch: bool,
):
run_copybara(copybara_file, "allwpilib_to_mostrobotpy", mostrobotpy_fork_repo)
os.chdir(mostrobotpy_local_repository)
checkout_branch(auto_delete_branch, "copybara_allwpilib_to_mostrobotpy")
update_mostrobotpy_rdev(wpilib_bin_version, is_development_build)
subprocess.check_call(["git", "push", "-f"])
def mostrobotpy_to_allwpilib(copybara_file: pathlib.Path, allwpilib_fork):
run_copybara(copybara_file, "mostrobotpy_to_allwpilib", allwpilib_fork)
def commandsv2_to_allwpilib(copybara_file: pathlib.Path, allwpilib_fork):
run_copybara(copybara_file, "commandsv2_to_allwpilib", allwpilib_fork)
def allwpilib_to_commandsv2(copybara_file: pathlib.Path, allwpilib_fork):
run_copybara(copybara_file, "allwpilib_to_commandsv2", allwpilib_fork)
def load_user_config() -> CopybaraConfig:
script_dir = pathlib.Path(__file__).parent
user_config_file = script_dir / ".copybara.json"
if os.path.exists(user_config_file):
print(f"Loading user config from '{user_config_file}'")
with open(user_config_file, "r") as f:
json_config = json.load(f)
else:
print(
f"No user config present at '{user_config_file}', no defaults will be loaded"
)
json_config = {}
return CopybaraConfig(**json_config)
def main():
user_config = load_user_config()
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(
dest="migration", required=True, help="Available commands"
)
def add_allwpilib_fork_arg(subparser):
subparser.add_argument(
"--allwpilib_fork_repo",
default=user_config.allwpilib_fork_repo,
help="URL to your github fork of allwpilib that you have write permissions for",
)
# allwpilib -> mostrobotpy
allwpilib_to_mostrobotpy_parser = subparsers.add_parser(
"allwpilib_to_mostrobotpy",
help="Pushes changes from the allwpilib mirror to mostrobotpy",
)
allwpilib_to_mostrobotpy_parser.add_argument(
"--wpilib_bin_version",
required=True,
type=str,
help="The wpilib release version as hosted on artifactory",
)
allwpilib_to_mostrobotpy_parser.add_argument(
"--development_build",
action="store_true",
help="True if you are upgrading to a development build instead of a release build. Affects where artifacts will be downloaded from.",
)
allwpilib_to_mostrobotpy_parser.add_argument(
"--mostrobotpy_local_repo_path",
default=user_config.mostrobotpy_local_repo_path,
help="Path on your local computer to a mostrobotpy clone",
)
allwpilib_to_mostrobotpy_parser.add_argument(
"--mostrobotpy_fork_repo",
default=user_config.mostrobotpy_fork_repo,
help="URL to your github fork of mostrobotpy that you have write permissions for",
)
allwpilib_to_mostrobotpy_parser.add_argument(
"-y",
"--auto_delete_branch",
action="store_true",
help="If present, will automatically delete the local version of the copybara branch if it exists. Otherwise you will be prompted if it is ok to delete",
)
# mostrobotpy -> allwpilib
mostrobotpy_to_allwpilib_parser = subparsers.add_parser(
"mostrobotpy_to_allwpilib",
help="Pulls changes from the mostrobotpy source of truth to this mirror",
)
add_allwpilib_fork_arg(mostrobotpy_to_allwpilib_parser)
# allwpilib -> commands-v2
allwpilib_to_commandsv2_parser = subparsers.add_parser(
"allwpilib_to_commandsv2",
help="Pushes changes from the allwpilib mirror to the robotpy commands-v2 repo",
)
allwpilib_to_commandsv2_parser.add_argument(
"--robotpy_commandsv2_fork_repo",
default=user_config.robotpy_commandsv2_fork_repo,
help="URL to your github fork of mostrobotpy that you have write permissions for",
)
# commands-v2 -> allwpilib
commandsv2_to_allwpilib_parser = subparsers.add_parser(
"commandsv2_to_allwpilib",
help="Pulls changes from the robotpy commands-v2 source of truth into this mirror",
)
add_allwpilib_fork_arg(commandsv2_to_allwpilib_parser)
script_dir = pathlib.Path(__file__).parent
copybara_file = script_dir / "../../../copy.bara.sky"
args = parser.parse_args()
if args.migration == "allwpilib_to_mostrobotpy":
if args.mostrobotpy_local_repo_path is None:
raise Exception(
"You mist specify mostrobotpy_local_repo_path, either on the command line or in your user config"
)
if args.mostrobotpy_fork_repo is None:
raise Exception(
"You mist specify mostrobotpy_fork_repo, either on the command line or in your user config"
)
allwpilib_to_mostrobotpy(
copybara_file,
args.mostrobotpy_local_repo_path,
args.mostrobotpy_fork_repo,
args.wpilib_bin_version,
args.development_build,
args.auto_delete_branch,
)
elif args.migration == "mostrobotpy_to_allwpilib":
if args.allwpilib_fork_repo is None:
raise Exception(
"You mist specify allwpilib_fork_repo, either on the command line or in your user config"
)
mostrobotpy_to_allwpilib(copybara_file, args.allwpilib_fork_repo)
elif args.migration == "allwpilib_to_commandsv2":
if args.robotpy_commandsv2_fork_repo is None:
raise Exception(
"You mist specify robotpy_commandsv2_fork_repo, either on the command line or in your user config"
)
allwpilib_to_commandsv2(copybara_file, args.robotpy_commandsv2_fork_repo)
elif args.migration == "commandsv2_to_allwpilib":
if args.allwpilib_fork_repo is None:
raise Exception(
"You mist specify allwpilib_fork_repo, either on the command line or in your user config"
)
commandsv2_to_allwpilib(copybara_file, args.allwpilib_fork_repo)
else:
raise Exception(f"Unexpected migration {args.migration}")
if __name__ == "__main__":
main()