a9ac5b8e24
Reading exported data from shared objects on windows is broken. It requires __declspec(dllimport). However, this is problematic, as we use the same static libraries both from a shared and static context. So we can't just blindly apply dllimport. The linker should have caught this, as data members are exported in a different way. However, due to a bug in native-utils, data member symbols were exposed directly. However, interacting with those data member was completely broken. The only way we can really solve this is to just not use static data members. We're pretty good about this in WPILib itself. However, protobuf is absolutely terrible at this. There are a ton of inline functions that access global data. For the protobuf library itself, we can solve this easily enough. However, for the generated protobuf code, this is much more problematic. The member needed to bypass the global data is private. This means using just the stock protobuf code, this problem is not solvable. But, protobuf generated code has insertion points. Those insertion points let us add our own code into the generated code via a protoc plugin. And it just so happens that an insertion point exists to add extra public methodsto the generated protobuf header. There is also an insertion point to let us add to the cpp file. The methods we need are the getters, for unpacking protobufs. For any protobuf that has a message as a member, we generate a new wpi_x() getter (the existing one is just x(), where x is the field name). We then implement this in the cpp file. A trick we can use is in the cpp file, we can safely call the x() function, as the cpp file is in the same library as the global. Thus we can call that inline method, and not actually need to directly access any internal private state of the protobuf object. TL;DR, all protobuf classes that have messages as fields now have a wpi_x() accessor that must be used instead of x() if you want the code to work on windows. After wpilibsuite/native-utils#212, the bad code will fail to link, rather then just fail at runtime.
Upstream utils
Layout
Each thirdparty library has a Python script for updating it. They generally:
- Check out a thirdparty Git repository to a specific commit or tag
- Apply patch files to the thirdparty repo to fix things specific to our build
- Copy a subset of the thirdparty files into our repo
- Comment out any header includes that were invalidated, if needed
upstream_utils.py contains utilities common to these update scripts.
Patches are generated in the thirdparty repo with git's format-patch command so
they can be applied as individual commits and easily rebased onto newer
versions. Each library has its own patch directory (e.g., lib_patches).
Updating thirdparty library version
The example below will update a hypothetical library called lib to the tag
2.0.
Start in the upstream_utils folder. Make sure a clone of the upstream repo exists.
./<lib>.py clone
Rebase the clone of the upstream repo.
./<lib>.py rebase 2.0
Update the upstream_utils patch files and the tag in the script.
./<lib>.py format-patch
Copy the updated upstream files into the thirdparty files within allwpilib.
./<lib>.py copy-src
Adding patch to thirdparty library
The example below will add a new patch file to a hypothetical library called
lib (Replace <lib> with llvm, fmt, eigen, ... in the following steps).
Start in the upstream_utils folder. Make sure a clone of the upstream repo exists.
./<lib>.py clone
Update the clone of the upstream repo.
./<lib>.py reset
Navigate to the repo. If you can't find it, the directory of the clone is printed at the start of the clone command.
cd /tmp/<lib>
Make a commit with the desired changes.
git add ...
git commit -m "..."
Navigate back to upstream_utils.
cd allwpilib/upstream_utils
Update the upstream_utils patch files.
./<lib>.py format-patch
Rerun <lib>.py to reimport the thirdparty files.
./<lib>.py copy-src