Due to how submessages are encoded (with a length prefix), nanopb currently does the encoding twice. It encodes once to get the length to write, then writes the length, then reencodes the entire message a 2nd time.
This results in a requirement that each encode always encodes the same. Generally, this is fine, but it'd be nice to not make this a requirement.
The double encode also requires going through the entire set of fields again, which has the possibility to be slow.
Instead of doing this, write to a temporary SmallVector. Then we can just encode the length of that buffer, and do a memcpy into primary stream. Theoretically in most cases, this should be much faster.
The Google C++ protobuf implementation has issues with dynamic linkage across DLL boundaries because it uses global variables. It also has a compile-time dependency because the protoc version must exactly match the libprotobuf version. Using nanopb with a customized generator fixes both of these issues.
Co-authored-by: Gold856 <117957790+Gold856@users.noreply.github.com>
This refactors Alert in both c++ and java to fix the issues with the current c++ implementation and improve performance.
Currently, constructing an Alert adds it to a list of Alerts with the same group and type. Activating an alert sets a flag on the alert. When the SendableAlerts is polled (GetStrings), the entire list is iterated over, filtered, and the filtered list is sorted by timestamp. This leads to a worst case O(m + nlog(n)) time complexity for GetStrings, where m and n are the count of total constructed alerts active alerts respectively. It also allocates intermediate data structures to hold the active alert strings for sorting.
This changes the implementation to improve the performance of GetStrings, by shifting the sorting overhead to Alert.Set
Constructing the Alert only initializes the alert's initial state, and initializes the SendableAlerts for the group if it is not already initialized.
Activating or deactivating an alert sets an internal flag for state tracking, and inserts or removes a structure containing the timestamp and text into a self-sorting data structure (std::set, TreeSet) containing other active alerts with the same group and type. (worst case O(log(n))
Now, SendableAlerts.GetStrings only has to iterate over the structure and copy the strings to the returned array. (amortized O(n))
This also fixes the c++ implementation by removing the need for SendableAlerts to directly access the Alert.
This also adds a helper method to SendableRegistry to force initialization of the instance to prevent static initialization ordering issues.
As string_view operations on std::map<std::string> won't be integrated
until C++26, placeholder implementations are used which are less efficient
in a couple of situations (e.g. insert with hint).
An empty path isn't valid on it's own, so fs::space always returns an error. This results in UINT_MAX bytes being used instead of the actual free space, which means a default constructed DataLogBackgroundWriter won't stop for low space.
Using "." instead makes the directory path the current working directory, which is the desired behavior
Update() checks/updates the last value and appends only if changed.
GetLastValue() gets the last value.
Also add OutputStream support to Java DataLogWriter.
Reverts #6609 since that fix didn't Just Work(tm) on Windows. (edit: or Ubuntu. Seems to have broken everything except macOS.) This PR configures CMake to try and find protobuf-config.cmake first, which allows protobuf to pull in abseil for us. If protobuf-config.cmake is not available (coprocessors which don't have a new enough protobuf installed are a common case), it will fallback to CMake's built-in FindProtobuf module, which is what we were using before.
Add wpi::CreateMessage, a wrapper with an ifdef to switch between Arena::CreateMessage and Arena::Create, since the former is deprecated in newer versions of protobuf. This allows forward compatibility with newer versions of protobuf.
Currently in the entire C API of WPILib we have ~8 different ways of handling strings. The C API actually isn't built for pure C callers (We don't actually have any of those). Instead, they're built for interop between languages like LabVIEW and C# which can talk to C API's directly.
For output parameters, the choice was fairly obvious. An output struct containing a const string pointer and a length makes the most sense. Its easy to use these from most other languages, and doesn't require special null termination handling. Freeing these is also easy, as if you ever receive one of these string structures, theres just a single function call to free it.
Input parameters are a bit more complex. To be used from pure C, and from LabVIEW, a null terminated string is the best in most cases. However, null terminated strings in general have a lot of downsides. Additionally, from LabVIEW there are other considerations around encoding that having a wrapper struct helps make a bit easier. From a language like C#, a wrapper struct is by far the easiest, as custom marshalling can make it trivial to marshal both UTF8 and UTF16 strings down.
The final consideration is its nice to have an identical concept for both input and output. It makes the rules fairly easy to understand.
WPILib will not have any APIs that manipulate a string allocated externally. This means WPI_String can be const, as across the boundary it is always const.
If a WPILib API takes a const WPI_String*, WPILib will not manipulate or attempt to free that string, and that string is treated as an input. It is up to the caller to handle that memory, WPILib will never hold onto that memory longer than the call.
If a WPILib API takes a WPI_String*, that string is an output. WPILib will allocate that API with WPI_AllocateString(), fill in the string, and return to the caller. When the caller is done with the string, they must free it with WPI_FreeString().
If an output struct contains a WPI_String member, that member is considered read only, and should not be explicitly freed. The caller should call the free function for that struct.
If an array of WPI_Strings are returned, each individual string is considered read only, and should not be explicitly freed. The free function for that array should be called by the caller.
If an input struct containing a WPI_String, or an input array of WPI_Strings is passed to WPILib, the individual strings will not be manipulated or freed by WPILib, and the caller owns and should free that memory.
Callbacks also follow these rules. The most common is a callback either getting passed a const WPI_String* or a struct containing a WPI_String. In both of these cases, the callback target should consider these strings read only, and not attempt to free them or manipulate them.
DataLog is now a base class, with DataLogBackgroundWriter being the
background thread version and DataLogWriter being a non-threaded version.
Also split the C header into a separate file to make it more wpiformat friendly.
We now use a wrapper (wpi::print) to catch exceptions since we can't patch
std::print() to not throw when we ultimately migrate to it.
fmtlib and std format/print throw the same exceptions and always have. We previously patched fmt::print() to not throw a write failure exception, but we can't do that for std::print(); wpi::print() is the migration plan.
Uses getUsableSpace in Java, matching how C++ determines available space (C++ calls it available, but they mean the same thing.) This fixes a bug where logs wouldn't get deleted due to incorrect available space detection.
The DataLog thread now also checks if the state was marked as stopped after a call to StartLogFile.
When low on space, a log file won't be created. This is detected as a "deletion", and the DataLog thread will continously try to create a log, fail to do so because of low space, detect it as a "deletion", and do so in a loop.
If there's not enough space, the DataLog will be marked as stopped, preventing this infinite loop. Calls to start() will hit this code path and mark it as stopped again.
Dynamic structs had a few major issues.
In C++, if the string was the last definition in the schema, attempting to set a string would trigger an assertion. This has been fixed
Setting a string value could truncate the string actually stored in the struct, if the definition was shorter than the string to set.
There was no way to detect if this case occurred. The set string function now returns a bool if the string was fully written or not.
Reading a string that had a value shorter than the schema definition would result in embedded trailing nulls in the string. This would make comparing string equality basically impossible, as those embedded nulls count for the length of the string.
The above truncating didn't take into account UTF8 code points. This means a truncation could happen in the middle of a unicode character. Depending on the language this had different behavior, but unpaired code points are problematic to detect in any case. On the decoding side, detect if a split UTF8 code point has occurred by the writer, and if so just ignore it and treat it as not part of the string. Doing this on the receive side means a newer receive side is all that is needed to fix this, which is generally a better option then requiring all senders to update.
Actual DynamicStruct instances have 0 units tests for them. Added a bunch of unit tests around strings to ensure things work properly.
Restarting a stopped log results in creating a new log file with fresh copies of the same start records and schema data records.
Also check to see if the file has been deleted or if the log file exceeds 1.8 GB, and start a new one.
This adds support for two serialization formats for complex data types:
- Protobuf for complex objects with variable length internals that need forward and backward wire compatibility (lower speed, more flexible)
- Raw struct (ByteBuffer-style) for fixed-length objects (higher speed, less flexible)
Deserialization can be done either by creating a new object (for immutable objects) or overwriting the contents of an existing object (for mutable objects).
Implementing classes should provide inner classes that implement the Protobuf or Struct interface (in Java) or specialize the wpi::Protobuf or wpi::Struct struct (in C++). It is possible for classes to implement both. If the class itself does not implement serialization, it's possible for third parties/users to provide an implementation instead.
Uses the Google protobuf implementation for C++ and the QuickBuffers alternative protobuf implementation for Java.
