Fixes#1472. TTF fonts were included as part of the FontAwesome kit, but
I opted to not add them because basically every browser won't use them
and it just pollutes the repo.
Everything under the `photon-lib` directory is intended to all be
licensed under MIT:
e40c8fbca0/photon-lib/.styleguide-license (L2)
Presumably we don't want to force teams to release their robot code
under GPL.
Somebody wanted a description, so here I am.
This PR fixes an error which caused the discoverModels function to be
rerun after each upload of a new model, but without clearing the list of
available models. This causes any models that were on the list prior to
the import to be duplicated. This PR also makes it so that uploading a
model automatically updates the list of available models.
---------
Co-authored-by: Matt Morley <matthew.morley.ca@gmail.com>
Co-authored-by: Chris Gerth <gerth2@users.noreply.github.com>
The following deprecation warnings have been fixed:
- `SwerveModuleState.optimize(desiredState, currentRotation);`, which is
now an instance method
- `AprilTagFields.kDefaultField.loadAprilTagLayoutField();`, which is
now `AprilTagFieldLayout.loadField(AprilTagFields.kDefaultField);`
WIP:
- [x] C++
- [x] Python
This PR implements several refinements to the way that NetworkManager
controls the network interface.
- The monitor detects and logs changes to the network address
- The monitor detects and logs changes to the connection and will
reinitialize the connection if needed
- Remove NetworkInterface.java class, which wasn't used anywhere
- Use java.net.NetworkInterface to get IP addresses for any interface
(device)
- Adds a metric for the current IP address (address on the currently
selected interface)
This PR changes the text input fields to allow a user to type in a value and press enter (or click away) before the UI registers the updated value. It makes text input behave in a more expected manner. The spin buttons are no longer shown for the text box. Instead, there are up and down buttons on either side of the slider.
The only thing that I can't figure out is a way to make the up and down buttons continuously increment (decrement) the value if you click and hold. I'm not sure that this is required, but if someone wants to propose a solution, please add it to this PR.
In softprops/action-gh-release, when using an glob pattern to exclude files, all of the files to exclude must be specified on one line. Splitting them to multiple lines causes all files to match one of the two lines.
To combine on one line, use `|` between the patterns.
Selective builds breaks my ability to require that checks pass before
merging. It's not worth the bite for very few docs-only PRs, given
github limitations on needing to skip INDIVIDUAL JOB STEPS on required
CI jobs.
Maybe we should move to gitlab 💀
This fixes a bug introduced in #1592 that caused the Manage Device
Networking toggle to be disabled for systems where PhotonVision is
managing the network.
There is still a problem with the toggle defaulting to "off" and not
staying in the "on" position after settings are saved. I need help from
someone who understands the frontend to figure out why it keeps getting
set back to "off".
---------
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
Co-authored-by: Cameron (3539) <theforgelover@gmail.com>
If the exposure property for the generic USB camera settable was not
valid for one camera in the list, then the thread would crash/hang and
no cameras would show up in the list
It wasn't hard to not notice or skip over the recommendation, or just
not take it seriously. So I added some banners to direct people to read
the quick start guide, and a banner to heavily suggest using an
industrial SD card, since they're just the best thing available right
now.
Occasionally, the sim projects are capable of simulating current draw of
over 600A, which triggers a condition in
`BatterySim::calculateDefaultBatteryLoadedVoltage` that limits the
minimum measured battery voltage to 0V (to prevent it from going
negative).
When battery voltage measures 0, this causes NaN values to propagate
through the drivetrain model, making sim inoperable. Specifically, [this
is the
line](https://github.com/PhotonVision/photonvision/blob/master/photonlib-java-examples/aimandrange/src/main/java/frc/robot/subsystems/drivetrain/SwerveDriveSim.java#L452)
that causes the initial NaN values in simulation.
This PR is posed as a patch to ensure that simulation doesn't break.
This fixes mostly formatting issues so there is no longer any diff with
things after rerunning the generation script. Yes, most of this can be
fixed by running wpiformat but that doesn't exist as a pre-commit hook
atm, so I think this is nicer.
It also removes any reference to the java encode/decode within the
python message gen
WAS: we used raw hash-maps to encode program state
S/B: we use Jackson to do this encoding for us for free. We have
Objects, and we should use them to represent structured data.
---------
Co-authored-by: Craig Schardt <crschardt@fastem.com>
Fixes#1564. Also copies vendordep JSONs to the examples as advised by
Thad. Removes unused shared/javacpp/setupBuild.gradle. Also removes
unnecessary `chmod +x gradlew` from CI workflows.
Provide instructions for installing the PhotonVision version that is
compatible with WPILibPi 2023.4.2, which is the newest version available
for the Romi.
The older text is hidden in a comment so that it can be restored when
there is a newer version of WPILibPi that is compatible with newer
versions of PhotonVision.
What it says on the tin. This is all stuff from our initial effort to
port the sim things. Right now it is coupled to #1557 because this fixes
things up in that. Lets merge that one before dealing with this one
This uses the version of libcamera-gl-driver that was built using our
image. This assumes
the correct update path of the pi image version to libcamera to
photonvision.
I tasked my team with updating our upcoming Reefscape codebase to target
2025 packages, only to realise we have created an unsolvable dependency
nightmare with these things being neglected...
Signal/noise ratio is too low with this enabled. When dealing with #1567
I got ~46000 lines of errors going around in a circle rather than just
an import failure at the scope of each failed test
Instead of writing the kernel logs to the photonvision logs, this will
download them in the same zip file as the photonvision logs. Only includes dmesg logs from the current boot, which is fine since we should capture most of them in our logs now.
ARM-based machines populate the device model into Device Tree. We can
use this information to automatically detect and report the hardware
model for most Single Board Computers (SBCs). Vendors who want to
override this can still do so via the value in the configuration
database.
I made a mistake when cherry-picking things into #1534. Fixing it also
prompted me to regenerate message things without thinking even though it
wasn't needed here but it helped me catch an issue with a bad shim. I
must not have saved it properly on my computer and missed it before
review.
I found these with a quick find-and-replace and checked against the inbuilt Python type checking. I am away from my robot and can't really
confirm there are no flow-on effects. There are no other active usages of the bad casing in the Python code, so we should be good. The generated serde messages already use this casing, so we don't need to update there.
List types should never be optional if sent to NT because an empty list conveys the same
thing.
The equivalent C++ struct takes the same approach with empty vectors rather than an optional vector.
Fixes GPL violation, the license has been missing since 2024.
This also puts licenses in as many JARs and native library archives as possible (for good measure.)
We've moved the install script to photon-image-modifier. This updates
the install script in photonvision to just download and run the
install.sh from photon-image-modifier.
This is quite an odd issue/fix.
So this is what happened... Photonvision booted with the camera
connected and the camera was working...
After a short time the camera stopped working (for some reason maybe
static, maybe temp, maybe wiring, idk).
During this time pv showed
Jul 04 06:25:18 BackLeft java[643]: [2024-07-04 06:25:18] [CSCore -
PvCSCoreLogger] [ERROR] CS: ERROR 40: ioctl VIDIOC_QBUF failed at
UsbCameraImpl.cpp:723: Invalid argument (UsbUtil.cpp:156)
Jul 04 06:25:18 BackLeft java[643]: [2024-07-04 06:25:18] [CSCore -
PvCSCoreLogger] [WARN] CS: WARNING 30: BackLeft: could not queue buffer
0 (UsbCameraImpl.cpp:724)
I went over and played with the wire. The camera fully disconnected but
it ended up "reconnecting"
When the camera was "reconnected" photonvision detected a "new camera"
except this time with no otherpaths (aka no usb path, or by id path).
That resulted in pv creating a new camera configuration for a camera
with no otherpaths
Cscore then started to report errors that look like it attempted to
connect to the same camera twice
This fixes it by filtering out USB cameras that have no otherpath on
linux.
# Overview
Previously if the coproc came up later, getProperty would return the
string literal "null", which made us print the BFW. Add tests to make
sure that we don't do that anymore by rebooting a sim coproc +
robot in a combination of different orders.
There is a weird edge case at least with arducam/broken arducams/used
arducams where cscore will see it when pv starts but not be able to
connect to it. If we always read out the "current" video mode instead of
null when it is disconnected things will work. If the camera is
disconnected while we try to change the video mode when we get the
current video mode it will tell us what we wanted to set it to. Then
when the camera reconnects it will be in that video mode.
This prevents spamming of the logs by the network interface device
monitor by:
- checking to make sure the device file exists before starting the
monitoring task
- only logging once if it throws an exception, but keep trying in case
the exception is transient
Also reworks OV9782 defaults. Probably doesn't work on windows. We should hide these sliders probably.
Co-authored-by: Cameron (3539) <theforgelover@gmail.com>
This bug would only appear when there are cameras with the same naming.
Old config matching would also match using the by-id this was
problematic. When one camera is disconnected it would assign the by-id
path to the other camera with the
same name. When the camera is replugged in it would not be reassigned
the by-id path and would fail the camerainfo equals check.
Sliders for exposure and brightness would spam messages on the backend.
This used to cause crashes and can cause it to get quite laggy /
delayed. This will add a 20ms debounce which won't send the value to the
backend until the value hasn't changed for 20ms.
---------
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
This PR changes the way that photonvision interacts with nmcli to control networking on the coprocessor. Instead of modifying an existing connection, Photonvision adds new connections for DHCP and Static IP configurations. It then activiates the proper one at startup and any time that the network configuration is changed.
It also now uses the interface name and not the connection name and checks that the interface is available before making any changes. If the saved interface is not found, it updates the stored interface name and applies the network settings to the current interface. This should minimize the failure to control the network if the network interface wasn't available when PhotonVision first booted.
One other benefit of not altering the default configuration is that, if PhotonVision fails to run for any reason, the device can be accessed using the original networking configuration.
The code has been tested on an OrangePi5 and and a Raspberry Pi 4.
Addresses: #1261
Rotate camera calibration coefficients based on camera rotation. Probably. Seems to work. Maybe.
---------
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
We accidentally copied more settings then we wanted. This adds an
annotation that we can mark variables with that will prevent them from
being copied when we switch pipeline types.
The target list in networktables is limited to 127 items. When you
capture more than 127 calibration images it breaks this limit and errors
out and dies. Do not publish calibration targets to nt. And also move cal images into their own folder
There are no settings for solvePNPEnabled while in driver mode but the
UI tries to set it. Let us not do that.
Fixes#1377
Co-authored-by: Chris Gerth <gerth2@users.noreply.github.com>
Chrome prints an error to the console when you have `<img src="null" />`
The path `//:0` can be used for an empty image and Chrome will not raise
an error.
This PR is for part 1 of #1354. It focuses on adding a model selection
interface for models that exist in `photonvision_config/models/`. Upon
completion we can ship more than 1 model and users could upload their
own through `ssh` without deleting the shipped model. This PR also adds
the abstractions need to support more DNN backends (say OpenCV, or RPI
AI Kit)
Up next is adding a CRUD interface for managing models through the UI.
Fixes the following issues with the client log viewer:
- Inconsistent and excessive spacing between log entries
- Lack of responsiveness to window size or scaling
Adds the following features to the log viewer:
- Auto-scroll if scrolled to the bottom
- Ability to clear logs on button click
- Search function to filter logs
- Displays the time the frontend captured a log and displays that timestamp in hh::mm::ss in the log viewer
- Allows logs to be filtered to be after a certain time
- General styling refinements to increase usability
---------
Co-authored-by: Sriman Achanta <68172138+srimanachanta@users.noreply.github.com>
You know... I made those charuco changes now I need to document how it
works... basic stuff.
---------
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
Uses logic in
https://github.com/PhotonVision/photon-libcamera-gl-driver/pull/16 to
push the ov9281 down to its true minimum exposure.
Updates UI to list the exposure settings in ~~microseconds.~~ Native
units - not everyone works in microseconds.
Does its darndest to actually try to set the exposure in
~~microseconds.~~ Native Units. To do this...
Lifecam is funky when doing this - [cscore limits the exposure settings
to certain quantized
values](https://github.com/wpilibsuite/allwpilib/blob/main/cscore/src/main/native/linux/UsbCameraImpl.cpp#L129).
Add a new camera quirk to allow that.
~~Updated camera quirks to re-evaluate every camera load (rather than
recalling from settings - this shouldn't be necessary)~~ This should be
rolled back, needed for arducam type selection.
Updated camera quirk matching logic to make PID/VID optional, and
basename optional (and only match trailing characters). This enables
mirroring CSCore's logic for identifying lifecams by name.
Updated the USBCamera to primarily use cscore's exposed property names.
Since camera manufacturers use a potpourri of names for the same
thing....
For nice-to-have settings: new soft-set logic to try all possible names,
but gracefully pass if the property isn't there.
For required settings: Search a list for the first setting that's
supported, fail if none are supported.
More logging of camera properties to help debug.
Note: most of this work is because cscore doesn't directly expose a
massaged exposure-setting-absolute API (and, given what we've seen,
probably _shouldn't_, this struggle is not for the faint of heart).
---------
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
This updates the install script to work correctly on Ubuntu 24.04
versions of the Orange Pi 5 images.
Changes include:
- installing libatomic1
- disabling networkd-wait-online if using Network Manager
- using systemctl instead of service to detect if photonvision is
running
- detecting if this is a RK3588 cpu and enabling all cores
Reverts PhotonVision/photonvision#1375
Causes white screen UI Bug, "the way we currently strap everything with
vue2 and vuetify has a lot of footguns in it, and using a newer package
manager where each subdependency gets its own version of node is causing
incorrect dependency resolution which also means we can't fix this
without either updating node or patching those dependencies id say just
revert the PR for now until I or someone else can do the vue3 update"
Pnpm is like npm except instead of keeping multiple copies of
dependencies, it shares a single copy for multiple dependencies
significantly reducing build time and the space needed to hold all the
dependencies. Read [here](https://pnpm.io/motivation) for more info.
This changes our CI to use pnpm and allows developers to choose to use
pnpm instead of npm. Also, pnpm has a built-in node version manager so
devs no longer need to use nvm to work on photonvision. All npm
functionality (including photon-server gradle tasks) still functions
using npm so this isn't breaking. We should make a docs change to
suggest to use pnpm.
The OV9782 camera has a specific exposure range, so a camera quirk for
it needs to exist. The default white balance is also pretty bad, so it
must be adjusted.
Closes#1204
---------
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
Co-authored-by: Cameron (3539) <theforgelover@gmail.com>
Windows users will have to add
`"-Dorg.gradle.java.home=C:\Users\Public\wpilib\2024\jdk"` to gradle
invocations, ie `./gradlew run
"-Dorg.gradle.java.home=C:\Users\Public\wpilib\2024\jdk"`, due to MSVC
ABI breakages and other stupidity
Add support for the old opencv charuco board like calibio.
Add support for other tag families while calibrating.
Fix calibration issue index out of range with charuco missing points.
* change verifyversion to use member variable
* Revert "change verifyversion to use member variable"
This reverts commit 4439839c8f.
* Removed inline specifier for versioncheck variable
---------
Co-authored-by: Drew Williams <DrewW@iARx.com>
Previously reported itself which was confusing. New print:
```
Error at org.photonvision.PhotonCamera.verifyVersion(PhotonCamera.java:378): Found the following PhotonVision cameras active on NetworkTables:
==> HD_Pro_Webcam_C920
==> Arducam_OV9281_USB_Camera
```
Add charuco calibration to photonvision. Currently does not support generating custom charuco boards. This does not support https://calib.io/pages/camera-calibration-pattern-generator. Currently only supports the 4X4_50 family. Also removes all dotboard calibration. Fixes using the lowest possible fps while doing calibration (now uses the highest fps available for each resolution).
Fixes bug where switching tabs/etc causes buildup of connected mjpg streams in network, eventually slowing down streams and causing stream failure until refresh. Accomplishes this by directly setting the source of stream elements to null before unmount, allowing chrome/edge to close the connection.
Fixes#1106
With the latest dev opi image, i saw this stack trace when object detection stopped working (threads hanging forever on detect(). The stack pointed me to somewhere inside the RGA. Based on this i moved resize into CPU (as our [native code already is lazy](6934abb26c/src/main/native/cpp/yolo_common.cpp (L227))), and was not able to see more crashes
[message.txt](https://github.com/PhotonVision/photonvision/files/14630158/message.txt)
Includes also a quick hack to add a shutdown hook that releases pipelines at exit.
* Disable camera orientation option when camera is calibrated.
* Flip logic on if camera is calibrated when disabling camera orientation rotation
* Add comment on why orientation is disabled when camera is calibrated
* Add v banner warning regarding rotating calibrated camera bug
* Run lint
---------
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
- Fix CameraInfo equality check (which prevents the same camera on a new usb port from being enumerated by us)
- Fix warning prints
- Make matchCamerasOnlyByPath apply to Windows
- Add unit tests
Our current code matches cameras in this order (which I think is objectively wrong and stupid)
- by-id (/dev/v4l/by-id/product-string)
- by path (/dev/videoN)
- product string/name, but ascii only
- asks cscore to reconnect to cameras using `path`, which on linux is actually /dev/videoN. This isn't guaranteed to stick to a camera if you replug them weirdly at runtime.
This is silly and does not consider the actual physical usb port. I propose instead, in this order:
- By physical usb port path and base name
- by physical usb port path and USB VID/PID
- By base name only (with a toggle switch to disable this, and create a new VisionModule instead)
- Give cscore /dev/video/by-path on Linux systems, pinning Photon USBCameras to a particular usb port once created.
This changes lots of things so stay paranoid!
* Sort object detection results and reduce code dup.
* Filter objdet results by ratio and area
* Address code review
---------
Co-authored-by: Mohammad Durrani <46766905+mdurrani808@users.noreply.github.com>
* Fixed cpp sim apriltag layout and cleaned up cpp sim example
* changed layout for photoncamerasim
---------
Co-authored-by: Drew Williams <DrewW@iARx.com>
fixed the unpacking order to match the current pipelineresult data layout.
* fix positioning of multitarget struct in pipelineresult unpack
* fix encode order in PhotonPipelineResult.cpp
* commented controls that should depend on networkingIsDisabled
* add the thing
* fix Manage Device Networking showing disabled
* commented controls that should depend on networkingIsDisabled
* add the thing
* fix Manage Device Networking showing disabled
* Hide the settings that aren't available when networking is disabled
* Update NetworkingCard.vue
* Update NetworkingCard.vue
---------
Co-authored-by: Sriman Achanta <68172138+srimanachanta@users.noreply.github.com>
Tested on Orange Pi 5 and Cool Pi 4B. Merge with parts of the OpenCV DNN PR.
Adds support for YOLOv5s models for Rockchip CPUs with a NPU. Right now hard coded to a note model from alex_idk. Very much still incubating and largely untested.
* [photon-lib java] Fix classes with protobuf support not "announcing it"
Since they didn't implement `ProtobufSerializable` this meant that most other software didn't even know protobufs were even implemented.
In AdvantageKit for example this would cause it to not work it all and crash.
* Run `spotlessJavaApply`
Add the following args to the install script:
Syntax: sudo ./install.sh [-h|m|n|q]
options:
-h Display this help message.
-m Install and configure NetworkManager (Ubuntu only).
-n Disable networking. This will also prevent installation of NetworkManager.
-q Silent install, automatically accepts all defaults. For non-interactive use.
Previously, if someone were changing network or camera settings while the backend sent an update request, the frontend wouldn't update the UI until the HTTP request was sent, likely leading to an error or confusion, now, values will be reset whenever new settings are sent. It also checks that settings were changed before allowing the user to click the save button.
* data updates to capture multiple rawBytes packets associated with serde updates from late this past month
---------
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
Adds logic that will reset the CSI camera in the case that it doesn't receive any new frames from the camera in 3 seconds. This is very helpful for cases where the camera cable was bumped enough to cause a temporary disconnect. Most of the time (if not all the time) the camera needs to be recreated for it to start sending frames again.
Goes with PhotonVision/photon-libcamera-gl-driver#13
Upgrades to Debian 12 as the base for our Pi images and adds an orange pi 5 build.
This uses the latest stable libcamera, which has weird AWB bugs with the OV5647/pi camera v1.
* Serialize all calibration data
* Run lint
* typing nit
* fix code
* move these tables around some
* Add cool formatting
* add request to get snapshots by resolution and camera
* re-enable all resolutions
* add wip so i can change computers (SQUASH ME AND KILL ME AHHHH)
* Get everything working but viewing snapshots
* Update RequestHandler.java
* Update CameraCalibrationInfoCard.vue
* Update CameraCalibrationInfoCard.vue
* add observation viewer
* round
* fix illiegal import
* Swap to PNG and serialize insolution
* move import/export buttons TO THE TOP
* Update WebsocketDataTypes.ts
* Add snapshotname to observation
* Refactor to serialize snapshot image itself
* Run lint
* Use new base64 image data in info card
* Update SettingTypes.ts
* Create calibration json -> mrcal converter script
* Update calibrationUtils.py
* Fix calibrate NPEs in teest
* Run lint
* Always run cornersubpix
* Update CameraCalibrationInfoCard.vue
Update CameraCalibrationInfoCard.vue
* Update OpenCVHelp.java
* Update OpenCVHelp.java
* Replace test mode camera JSONs
* Run wpiformat
* Revert intrinsics but keep other data
* Remove misc comments
* Rename JsonMat->JsonImageMat and add calobject_warp
* Update Server.java
* Rename cameraExtrinsics to distCoeffs
* fix typing issues
* use util methods
* Formatting fixes
* fix styling
* move to devTools
* remove unneeded or unused imports
* Remove fixed-right css
If its really that big of a deal, we can add it back later, kind of a drag to fix rn.
* Create util method
* Remove extra legacy calibration things
---------
Co-authored-by: Sriman Achanta <68172138+srimanachanta@users.noreply.github.com>
Fix grayscale passthrough with libcamera. Additionally fixes issue #1091.
Must go with PhotonVision/photon-libcamera-gl-driver#11.
When grayscale passthrough is used currently the frames that are returned do not have the type grayscale so calculations that need grayscale to not run. With these changes pipelines that need grayscale will now run and properly display fps.
UI would say that every camera on a Pi device was a CSI camera basically. It would not let you rotate usb cameras 90 degrees or 270 degrees because it thought that they were CSI cameras. Fixes: #1098
Allows logging software and live data view to see results. Also removes the requirement for AScope to keep up with the packet serde schema and instead just use the Protobuf descriptor.
Resolved race condition between saveGlobal and saveOneFile modifying settings on shutdown. Previously, saveGlobal would overwrite the action of saveOneFile on a clean shutdown.
Continuation of #802
Support RGB status LED to indicate:
Running/no-running
NT connected
At least one target visible
Configured by manually uploading hardware config JSON
Follows a similar system to the current Protobuf implementation that helps make code more readable and expandable. Also wraps the NT topic to be more useful. Impl stuff is hidden so it can't be extended. Optimizes AT-specific classes by only serializing data when needed, won't save on size but will on time.
closes#1003
* Record standard deviations for multi-tag pose
* Add XYZ translation and angle to stdev uI
* create multitag result buffer in store
allows results to be stored in the background
* simplify logic in targeting tab
also adds a reset button
* Formatting fixes
* convert rad angles to deg
---------
Co-authored-by: Sriman Achanta <68172138+srimanachanta@users.noreply.github.com>
add aliases for older enum names
* two more aliases for 6.5" 36h11 tags
* added unit test for missing @JsonAlias
* use proper tempfiles
* check proper TargetModel enum
Deals with new otherpaths on pi 5 CSI cameras and bumps libcamera driver to latest from the pi 5 PR
---------
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
Allows multiple cameras of the same model to be used while ensuring they stay tied to the physical camera and not the port. Matching occurs when first connecting cameras so swapping the ports while PV is running will swap the virtual cameras until a restart. Currently only tested on Linux.
Fixes a bug where multiple cameras would receive identical stream indexes and would cause at least one camera to not function at all and the other to not display a stream.
One of the reasons the that usbcamerasource equals method used to incorrectly determine equality was because the quirkycamera object didn't have a basename. When a camera that had a quirk was found it would set equal to a predefined quirkycamera without changing the name first. Add unit test that will better test the equality of two usbcamerasources. This required a few changes to allow creating a fake usbcamerasource.
Cleanup project-wide gradle configuration.
removes native dependencies from java only projects
increases readability
Pass generated headers in setup instead of modifying model
* add classes to targeting and update gradle
* rename me
* Finish cleanup
* Formatting fixes
* just use common.gradle
* Update build.gradle
* Update config.gradle
* remove typo
* simplify
* Add Packet Headers
* move simulation classes into simulation folder
* draw in dependency
* fix
* Everything working minus tests cause im lazy
* formatting fixes
REMEMBER TO REMOVE UNUSED IMPORTS, IM JUST TOO LAZY TO CHECK RN
* move packet test to targeting
* Formatting fixes
* remove TargetCorner from c++
im not 100% sure but just doing std::pair<double, double> is sufficient and shouldnt conflict with protobuf
* think i added packet
* fix namespace issue
* organize imports in photon-targeting
* Formatting fixes
* remove ctors
* fix typo
* Add PNP and Multitag packet tests
* revert TargetCorner class
* Add Test placeholders
* remove annoying print
* Reorganize build and publish process
channeling inner Thad
* add targeting as flag
* Update config.gradle
* fix issue with platform binaries not building
* Update photonlib.json.in
casing still needs to be checked
* add minimum level back
* add back UTF-8 encoding of javadoc
* simplify publish model for photon-lib
* fix windows symbol generation
* formatting fixes
* move task from main gradle to config
* Update config.gradle
* Make MultiTagPNPResult and PNPResult singular
* add java tests
* Formatting fixes
* bring in the rest of the little stuff
* final things
* Formatting fixes
* add multisubscriber back
* Formatting fixes
* make comments better about x and y relationship
Removes websocket-based camera streaming functionality.
Fixes#975. This was caused by destroying the camera streams and recreating them on nickname change. Even when directly using `MJPGFrameConsumer` and the streams were exactly the same, the freeze would occur when creating a new `MjpegServer` and require a refresh. I think this is simply how cscore works?
- Aruco pipeline now infers tag width from tag family like the AprilTag pipeline
- Removes unused Aruco and 200mm AprilTag models
- `VisionEstimation.estimateCamPosePNP()` now requires a target model instead of assuming 16h5
- Multitarget pipeline similarly infers target model of tag family now
- `PhotonPoseEstimator` can have target model set for on-rio multitarget
---------
Co-authored-by: amquake <noleetarrr@gmail.com>
- Fixes ArUco on picam
- Adds `ArucoPoseEstimatorPipe` for single-tag pose estimation
- Previously, `Aruco.estimatePoseSingleMarkers()` was used for tag pose estimation. This uses the default `SOLVEPNP_ITERATIVE` solver and I believe the method is removed in opencv 4.8. The `SOLVEPNP_IPPE_SQUARE` solver implemented is more appropriate for markers.
- Pipeline architecture cleanup
- Re-enables ArUco pipeline in UI
- Multi-tag support
ArUco detector support is still considered experimental at this time. This should enable a baseline of support for initial testing, but expect some quirks to remain across platforms.
closes#965
Had to disable the eslint rule that was causing the bug.
Disallows disabling driver mode when there are no other pipelines to swap to
Also adds icons for saving and canceling name edits
Adds the option to create a new pipeline in driver mode if there are no other pipelines
Adds disable prop to the driver mode switch on the camera settings page
> there is a problem if the found tags are a valid one plus an invalid tag, so eg it sees a tag but finds noise on the wall, and so lists tags 2 and 15. Lines 90-99 will ignore tag 15, and put 4 corners into the list. It will get past the test on line 100 (there are 4 corners). BUT then at line 106, there are 2 entries in visTags, so it will go to the "else" and try to use solvePNP_SQPNP().
* fix an issue where the fov isnt reset on error
* Fix issue with incorrectly reading fov on update
* Properly handle NPE in case of error
* Fix issue with vuetify comps not converting strings to numbers
* Formatting fixes
- `PNPResults` can now be empty (`isPresent` = false)
- solvePNP methods actually handle errors and return empty `PNPResults`
- This reveals an odd error where some inputs to `solvePNP_SQUARE()` resulted in an estimated transform with NaN values, and attempts to handle it
- Overwrites java changes from #817 since #742 had duplicate fixes
- Minor bugfixes
This reverts commit 013ff5e, which caused crashes with libcamera cameras.
More testing required to root-cause and fix this PR, but rolling back for testing for now.
### What does this do?
- Deprecates previous sim classes
- Has a `CameraProperties` class for describing a camera's basic/calibration info, and performance values for simulation. Calibration values can be loaded from the `config.json` in the settings exported by photonvision.
- `OpenCVHelp` provides convenience functions for using opencv methods with wpilib/photonvision classes, mainly to project 3d points to a camera's 2d image and perform solvePnP with the above camera calibration info.
- `TargetModel`s describe the 3d shape of a target, both for projecting into the camera's 2d image and use in solvePnP.
- `PhotonCameraSim` uses camera properties to simulate how 3d targets would appear in its view, and has simulated noise, latency, and FPS. For apriltags, the best/alternate camera-to-target transform is also estimated with solvePnP.
- `VideoSimUtil` has helper functions for drawing apriltags to a simulated raw and processed MJPEG stream for each camera using the projected tag corners.
- `VisionSystemSim` stores `VisionTargetSim`s and `PhotonCameraSim`s, and is periodically updated with the robot's simulated pose. When updating, camera sims are automatically processed and published with their visible targets from their respective poses with proper latency.
### What's still not working?
- Mac Arm builds are broken
- More examples
- Update website/docs
Serializes settings using a sqlite database instead of just putting them on the filesystem. Ideally since sqlite deals with filesystem robustness stuff this should work a lot better
Merging this now so we have lots of time to stabilize pre-beta
Fixes leftover bugs from #767 and #856
- Because new URLs aren't based on hash URL schemas, all the routes need to be updated to final paths instead of relative ones that were previously caught by the hash URL
- Fixed misc Vue warnings pointed out by @mcm001
closes#855
Only issues I see is that only HTML5 History API supported browsers support this which is fine given that the only thing that doesn't is probably the laptop they use to flash routers at competitions.
The 404 page can be up for debate lol.
Currently, there is a difficult-to-reproduce bug where the backend reports that camera calibration was successful in logs via the logger but then throws an exception causing the backend to return a 500 error code with no request body which causes the frontend to interpret this as a failed calibration attempt. This ultimately leads to the entire instance of photonvision crashing and requiring the entire pi to be restarted. It is believed this issue resides inside the ConfigManager's saving action following the calibration update but is not confirmed.
This adds a regex that ignores cameras if they match it, for if you have another piece of software running that needs a second camera, or if you have a webcam in your laptop that cscore hates.
- Made alternate constructor for ```PhotonPoseEstimator``` that doesn't need ```PhotonCamera```
- Changed ```update``` function to take in missing cameraMatrixData and coeffsData that were previously received from PhotonCamera.
- Changed the internal ```update``` and ```multiTagPNP``` function to take in cameraMatrixData & coeffsData
- If not needed for the specified strategy then the parameters are simply not used. Also if PhotonCamera is used in constructor it instead backs up to that.
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
This should reduce loading time as browsers show a blank screen while they try and load the inaccessible Google Fonts. Fontsource simply provides an easy way to integrate the font without much work on our end.
* Add pose caching to Java
* Refactor strategy fallthrough
* Hopefully add pose caching to C++
* Make Java switch same order as enum and C++ switch
* C++ absolute value in timestamp check
* Fix Java NPE
* Use `units::second_t` in timestamp
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
* Expand Java unit test
* Copy comments into C++
* Add tests to C++
* Run format
* Update PhotonCamera.cpp
* Probably fix bad access exception
* a
* init timestamp
* Remove prints
---------
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
Co-authored-by: Joseph Eng <joseng2358@gmail.com>
There are a few references to PhotonPipelineResult in this header. I noticed this was being indirectly included when trying to remove the PhotonCamera dependency.
Allows teams with a mechanism that moves the camera's position (eg, a pan and tilt mechanism) to update the location of their camera for their pose calculations.
---------
Co-authored-by: Matthew Morley <matthew.morley.ca@gmail.com>
* Switch network management to networkmanager
* Run style
* Fix command formatting
* Add curst Pi 5 second sleep
* Run formatter
* Also bring up/down on other linux
* Switch to nmcli down/up
* Remove sleep in nmcli down/up
* Address review
* Add and use a function in install.sh to determine if package is installed.
Move the "is a package installed" code into a function.
* Install libopencv-core4.5 on aarch64, which is likely raspberry pi.
The libphotonvision.so on Raspberry pi depends on libopencv-core4.5.
The code here installs that package on all aarch64 systems, as
there was not an obvious way to install on only Raspberry pi systems.
Fixes#748.
Co-authored-by: Scott Moser <smoser@brickies.net>
* update documentation
* add suggested changes
* rename April Tag to AprilTag
* Update RobotPoseEstimator.java
Co-authored-by: Mohammad Durrani <46766905+mdurrani808@users.noreply.github.com>
* Reordered ov video modes to be lowest-to-highest res
* Save off sensor model on init. Guard against low, crashy exposures.
* Pulled in matt's fixups from https://github.com/PhotonVision/photon-libcamera-gl-driver/suites/10144555465/artifacts/495489276
* Further autoexposure tweaks for picam v1
* Allow undercores in camera rename
* Additional guarding against output images being empty
* lock out auto-exposure on ov9281's
* Guarding stream pipelines against empty frames from cameras. Rearranged driver stream to resize first, then draw crosshairs (matchces with other pipelines now).
* NT Priority fixup - if client is sending commands on NT, its nt value should win over anything done from the UI
* Synchronous pipline adjustmet fix, method cleanup
* lint
* circle pipe and data publish bugfixes
* lint
* Pulled in matt's latest .so and re-enabled auto exposure on 9281's
* Use List for RobotPoseEstimator constructor
* Update `RobotPoseEstimator` constructor to accept wpilib `AprilTagFieldLayout` java
* Initial cpp changes
* Java return optional from update
* Fix java test
* Clean up strategy switch
* small lint
* Actually link to vision_shared
* Fix auto optimized imports
* format
* report error
* small method changes
* format and clean up
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
* Make install script auto-detect arch #679
Tested on linux x64 and aarch64
* Fix arm32 uname string
Co-authored-by: Chris Gerth <gerth2@users.noreply.github.com>
* change to 64 bit image generation
* Generate LL and Pi images in workflow
* Update main.yml
* Update main.yml
* Update main.yml
* REVERTME yeet publish
* Update main.yml
* Add archive suffix to generator
* Bump base images to beta 3
* Add more error prints to image gen
* Fix image base URL
* Bump pi/LL base images
* Update main.yml
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
Co-authored-by: Chris Gerth <gerth2@users.noreply.github.com>
* clean up front end ui
* address changes
* Further tweaks to camera default gains to help make sure users get a good first impression
* even more saner defaults
* Even even more camera sane defaults
* lint
* lint pt 2
* unit test fixup
Co-authored-by: Chris <chrisgerth010592@gmail.com>
RobotPoseEstimator can pick the most likely pose for the robot given a number of possible poses, using a number of different strategies. Examples are still WIP.
Bumps to a wpilib dev version, until they cut a new release. Should help address the random NPEs from the old JNI.
Co-authored-by: Chris Gerth <gerth2@users.noreply.github.com>
* Add nice value to service file, and give example CPU selection for those who need it.
* Use cpufrequtils package to set CPUs into performance mode
* Add comment about nice value
* Need to say "yes" to installing cpufrequtils, and safer to do so for all installs.
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
* Revised isRaspian() call to look in multiple spots to check if we're a Pi or not
* wpiformat
* linefeed fixup
* whoops
* WIP updating platform
* More platform fixups WIP
* Condensed metrics classes, but expanded the configuration to default to file, but fall back on hardcoded commands for certain platforms
* Migrate unixSupported to isLinux
* applied spotless
* wpiformat
* Linux metrics (#641)
* Move generic commands from PiCmds to LinuxCmds; have PiCmds inherit from LinuxCmds
* Better names for variables to save the total memory values
* Remove "Bionic" from the architecture; that is not actually determined.
* Trigger PhotonVision CI
* Dummy change to trigger CI
* Run format
Update index.html
Co-authored-by: Mohammad Durrani <46766905+mdurrani808@users.noreply.github.com>
Co-authored-by: Paul Rensing <prensing@gmail.com>
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
* Reverted to front end using MJPEG streams. Added FPS limiting to the stream.
* formatting
* fixup - got handlers getting called on error to reload
* revised architecture to let a click open a new tab
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
* logging for NTDataPublisher
* logging name along with index
* formatting lol
* resolution logging
* Removed pipelineManager object from data publisher
* wip support for a stats overlay
* WIP adding stream stats. But.... eeeh. Corporate.
* kbits over mbytes
* A ton more tweaks:
- Increased thread priority for streaming to reduce "stutter/slow" issues
- revised client side URL creation order to prevent the possibility of repeat-identical URL's
- Improved overlay to only be visible on mouseover, and fully centered in the screen
* wpiformat on js
* Fix sim versionEntry NT table path
* Fix compile issue that mainTable is not accessible form SimPhotonCamera
* Fix format
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
Co-authored-by: Chris Gerth <gerth2@users.noreply.github.com>
* WIP updating sim stuff for 2023 and pose3d's
* vision system build fixups, but test not yet passing.
* WIP Sim fixups and working on testcases
* Still doesn't work, but closer
* tests pass
* removed C++ sim support
* formatting update
* adjusted target height above ground per review
* Turns out its unused
* missed example removal
* WIP adding second websocket handling for cameras
* just more WIP
* even more wip. Most java-side framework completed, but not yet debugged
* IT LIVES. Still needs lots of cleanup. But we're transferring and displaying data!
* moved down an architecture layer. Improved multiple-camera handling
* Additional WIP to help improve smoothness and performance, though not yet tested
* bugfixes galore
* tweak compression
* spotless
* more tweaks for handling slow/intermittent streams
* wpilibformat maybe?
* clang-format maybe?
* WIP - adding thinclient. I don't like it yet, it should be more auto-generated than it is.
* thinclient formatting fixups
* Reduced amount of empty send data by limiting to only one stream per client (which is all we really need). Framerate is up slightly, overhead is down.
* bugfixes, faster streaming, better mjpeg compression settings, thinclient working
* spotless and formatting
* cmon wpiformat....
* re-added mjpg streams
* added a loading GIF to imporve the feeling of responsiveness
* formatting
* urlparams and built-in thinclient
* wpiformat
* prevent wpiformat complaints
* Removed uint8 array and base64 conversion from client side
* Synced up js implementations for ws streaming
* formatting/spotless
* Fix Start Calibration button requiring a page refresh
* Fix camera resolution selection
* Fix camera resolution selection so it works with the default selection
* Added throttling reasons and cpu uptime
* spotless
* adding tooltips for the acronyms used
* Added icon for suggesting folks should attempt a hover-over for tooltip
* wip making the implementaiton more platform independent
* spotless
* wpiformat
* wpilibformat pt 2
* Correct image capture time
`Timer.getFPGATimesptamp()` returns the current time in _seconds_, but `res.getLatencyMillis()` is in _miliseconds_.
* Correct image capture time (correctly)
* Change double literal to not use suffix
Co-authored-by: shueja-personal <32416547+shueja-personal@users.noreply.github.com>
Revised stream and target draw logic to divide the streams by "Raw" and "Processed" and only draw the results on the "Processed" stream.
Should allow for input sterams to be recorded for raw camera input, and output for debug info.
* Add UI-side changes for invertable hue slider
* Add hue inverted range
* Add new slider backgrounds to threshold tab
* Run spotless
* Updated libpicam.so to artifact built from commit c458bab87740 in that repo on gerth2's pi.
* undo the java-side hack since isVCSMSupported is fixed
* Hook up hue inversion frontend to backend and UI tweaks
* Remove unused .flipped class
* Fix hueInverted name in Vue.js store
Co-authored-by: Declan Freeman-Gleason <declanfreemangleason@gmail.com>
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
Co-authored-by: Chris Gerth <chrisgerth010592@gmail.com>
This approach is quite brittle, but it's easy to get working and can ship in an initial 2022 release. It's necessary to prevent GPU acceleration from happening on Pi 4s though. Let's try to put together something better for future releases.
* Addresses null pointer crash in logging when log files are not writable
* One of these days, I'll be able to type code without spotless complaining.
But today is not that day.
* Move test images out of resources folder
* Limit workers in CI
* Fix image area filtering bug in colored shape
* Add missing picam settings
* Swap to make blank/empty Mat when a picam doesn't supply a color image.
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
* Fixup colored shape backend code
* More colored shape stuff
* Start adding shape change to drawing
* Mostly works!
* Add powercell image for shape test mode
* Make super-duper-sure to release stuff
Fixes colored shape leak
* Move approx poly dp into Contour
* Adjust epsilon threshold
* Add dialog to change pipeline type
* Run spotless
* Make yes red :>
* Move "no" button
* Fix duplication/deletion name logic and switching
* Fix compilation errors from rebase
* Update VisionSourceManager.java
* Update type dialog, remove duplicate popup
The dropdown still switches even if the user says "no" tho
Co-authored-by: Banks Troutman <btrout.dhrs@gmail.com>
* Add libpicam with gain slider bugfix
* Patches to get zero-copy working with Pi3.
-- Success/Failure mistmatch assumptions - lots of functions in the JNI return true on failure (not true on success)
-- isVSCMSupported() is currently implemented to be "isVSCMNotSupported()"
Likely, we'll want at least some .so changes
Co-authored-by: Chris Gerth <chrisgerth010592@gmail.com>
Updated Java examples to fix radians/degrees mismatch.
Inspected c++ examples - they create a units::degree_t from the NT value, which should be handled properly inside the PhotonUtils methods.
* WIP adding sim pose example
* WIP making examples buildable like WPI. Not quite there yet....
* Make examples runnable
* remove lock
* add lock
* WIP Adding a simpler example for simulation
* Spotless Apply
* Added simulation-supporting aim and range example
* Spotless, revised hand usage to be consistent across examples, and propagated required -1.0's to non-sim examples.
Co-authored-by: Prateek Machiraju <prateek.machiraju@gmail.com>
This uses the same time source as CSCore does for image captures, and will make latency measurements more accurate.
Co-authored-by: Banks T <btrout.dhrs@gmail.com>
Modified VisionModule.java to cause any camera the either has a vendor-defined FOV or the PiCam quirks to control the LED state.
This is a bit of a patch, and directly assumes that piCam's are the devices primarily used for vision processing.
This fixes a tiny bit of pain experienced with a pi3b + picam v1 + usb Cam where not having the USB cam causes the LED's to be working normally, but plugging in the USB cam causes them to be always off.
Parallel issue entered in documentation to add this to known limitations of photon - if you had a set of all usb cameras, we wouldn't know which one is being used for vision processing, and would not be able to control the LED's accordingly. The ultimate solution is just to have teams control the NT entry themselves, which is easy enough. But, docs shouild be updated to reflect that.
* Incorporated new images from gloworm beta unit from Declan into unit test suite.
* Missed one unit test case - looks like the 1280x720 image set also returns wonky center values.
* WIP monkeying around with adding new targets.
* Added testcase to replicate large extrinsics result from calibration.
* Tweaked calibration pipline to return image with chessboard corner detection overlaid.
* Removed "bad" images from the cal, but that didn't seem to help...
* Added test logic to output the undistorted Mat for evaluating the quality of the calibration.
* Tweaked generation of chessboard points to be in a square pattern, not parallelogram.
* Spotless and removed bespoke test target in prep for PR.
* Revised to a double-for loop for less complex-looking logic
* #thanks spotless
al;ksgfjh akljghf ;lakdfdhg ksadfgh klasdfjhg kasdfjghj aklsjg two spaces
* Boop
* Updated findBoardCorners to operate on the image at a fixed, small resolution.
This means the image is scaled down before passing to openCV for processing, then the returned point cloud is scaled back up by that same factor.
Added additional flags to findBoardCorners to "fail fast".
Revised subpixel optimization parameters to use a dynamically-sized window (based on the distance observed between board corners).
* Added additional unit testing on calibration at different resolutions
* Spotless cleanup
* Add Mat release calls
* One more missing release.
Also, tried to make spotless a bit happier
* Additional formatting and WIP tracking down t he memory leak
* Cleaned up `Point` allocation in utility functions that iterate over mats of points
* Maybe fixing this bugger????
* Indeed! We can now get through the image capture stage of calibration at full resolution and save off images. Still crashing on out of memory as expected.
Added additional unit sim pass/fail criteria to catch unreleased mat's and a new set of images for testing high-res cal.
* Letting spotless do its thing
* Undistort debug seems to be acting a bit wonky in CI, so just commenting out for now
* Guard against testcases bleeding state into each other.
In particular, at least in CI, it appears not all mats are getting released in previous testcases, which borks up a hard check against "0 mats allocated" in our calibration tests.
* Removed obsolete tests
* One of these days, I will indeed learn to run spotless before I push.
But today is (still) not that day.
Today we push. FOR ROHAN!
* Cherry-pick the extra debug info from Bank's patches.
* Updated Calibrate3d pipeline to release all unnedded mat's prior to returning.
Update a few raw mat operations to use CVMat for better traceability.
* spotless cleanup
* Added check to shouldPrint for optimization on cvmat deallocate
* Reworked stack trace printing to use lambdas for efficiency
* Missed an unneeded logger.trace
* Formatting improvements
* WIP monkeying around with adding new targets.
* Added testcase to replicate large extrinsics result from calibration.
* Tweaked calibration pipline to return image with chessboard corner detection overlaid.
* Removed "bad" images from the cal, but that didn't seem to help...
* Added test logic to output the undistorted Mat for evaluating the quality of the calibration.
* Tweaked generation of chessboard points to be in a square pattern, not parallelogram.
* Spotless and removed bespoke test target in prep for PR.
* Revised to a double-for loop for less complex-looking logic
* #thanks spotless
al;ksgfjh akljghf ;lakdfdhg ksadfgh klasdfjhg kasdfjghj aklsjg two spaces
* Boop
* Reverting my changes in Calibrate3dPipeline.java to make this merge better with other PR's
* Derp changed the wrong one
Removes duplicate members, try-catches streaming processing, and prevents drawing empty contours. The latter likely stems from some sort of use-after-free condition.
* Add native stuff
* use runtimeloader
* add more native methods
* more stuff
* Switch JNI methods to static
* Remove non-java classes from the picam jni
* Add gradle task for JNI generation
* Migrate my previous GPU accel work
* Initial work on defining JNI interface
* Change libpicam to a symlink for now
* Initial work on adding no-copy OMX GPU accel on the pi
* Make DIRECT_OMX GPU accel mode not crash
* Clean up OMX changes (still not getting valid data back)
* Re-add GPU unit test
* A couple debugging tweaks/notes
* Add temporary special cases to get RGB out of ProcessingMode.NONE
* Code clarity improvements; fix possible VBO bug
* Get DIRECT_OMX working
* Remove some debugging switches in GPUAccelerator
* Pipe in VCSM stuff to read out pixels FAST
* Apply Spotless
* Revert versioningHelper changes
* Add missing import
* Convert to MMAL and move everything to native
* Re-add shared object
* Rework to use MMAL and do everything natively
* Condense pipeline settings classes
* Add OutputStreamPipeline
* Apply spotless
* Fix duplicate variable inits and add more video modes
* Integrate color frames and latency measurements for GPU
* Fix camera detection on pi and other platforms
* Add proper color copy disabling and camera settings calls
* Fix things that were broken by rebase
* Fix spotless issues and remove uneeded prints
* Remove libpicam symlink
* Fix stream resolution limiting
* Remove testing code in GPUAcceleratedHSVPipe
* Make profiling options general to all computers
* Make PicamJNI load from resources
* run spotlessApply
* Address review comments
* Update Maven repo for JOGL
* Fix release race condition
* Only run GPU accel test on the pi
* Lint fix and merge conflict accident fixes
* Make Jackson ignore extra fields when unmarshalling HardwareConfig
* Fix Mat releasing data race
* Spotless apply
* Remove broken header generation task
* Fix shared library loading typo
* Add a ZeroCopyPicam quirk to allow setting gain with the MMAL backend
* Make sure that exposure/brightness/gain get set after res changes
* Make rawInputMat properly local
* Remove bogus set of shouldRun flag
* Clean up small GPUHSVPipe print
* Add in some things that missed the ZeroCopyPiCameraSource rename
* Fix incorrect scoping introduced in past rebase
* Don't filter out too-low resolutions
* Only show latency when GPU accel is enabled
* Don't free Mats in stream thread before we use them
* Fix use-after-free and latency caluclation bugs on USB camera source
* Update libpicam
* Remove unwanted print
* Add libpicam forceLoad in unit test
* Fix streaming during camera calibration
* Fix zerocopy Picam calculation
* Use logger trace method instead of raw prints
* Fix calibration and driver mode pipes with the Picam
Co-authored-by: Matt <matthew.morley.ca@gmail.com>
Co-authored-by: Banks Troutman <btrout.dhrs@gmail.com>
Added disk usage percentage hardware stat
Updated Settings-General UI to have table-layout for metrics and other info
Added a `synchronized` modifier to the command runner due to some InterruptedThreadExceptions that were spuriously showing up while getting stats. Added additional stack trace logging when issues arise.
Changes exposure setters to accept a floating point input (rather than integer)
Updates the UI to change exposure in increments of 0.1 (rather than 1.0)
Updates quirky PI camera logic to use the raw_ interface with scaling/offset logic matching the GPU-accelerated pi3 camera logic from Declan.
Adds logic to disable auto-white-balance in the PI camera, which should yield more consistent vision processing results.
Searches for an appropriate remote target or uses an address specified by -PtgtIP="X.X.X.X" on the build command.
Stops photonvision, copies the new jar to the correct location, and restarts photonvision.
* Added input and output frame file save routines
* First pass at review items
* Revised logic to not crash on start and pass tests
* Updated build.gradle to force line endings. Spotless passes now.
* Reverted lineEndings to not force Unix.
Gerth needs to fix up his dev pc.
Co-authored-by: Banks T <btrout.dhrs@gmail.com>
* Rename MJPEG streams when camera name changes
* Change camera name to HTTP request
This allows us to wait for it to for sure be done
* Fix reload logic
* whee lnt
* Reload on backend connect too
* Update CameraAndPipelineSelect.vue
* Change chessboard size to be squares not interior corners
This reduces ambiguity
* Force users to select resolution
This forces the correct video mode index to be selected. Otherwise the 0th camera videomode index will be used, as it's zero-inited. This is undesirable.
* Make target model an enum
This will allow the UI to remember the currently selected target.
* Fix DriverMode settings
* Update FileVisionSource.java
* Sort modes by resolution
* Filter duplicated modes
* run spotless
* Fix calibration bug
* run format
* aaaaa
* Add hardware and platform support
* decrease timing sensitivity
* Better handle jvm exitg
* Make reboot happen immediately
* Cleanup restart
* Remove debug print
* Fix Jackson exploding when deserializing old versions of configs
* Add unit test for old config versions
* Run format
* Add a comment
* remove isvendorcam from pipeline manager
* oops
* Update QuirkyCamera to do name-based matching
* Add pi-cam exposure set
* Refactor QuirkyCamera
* Add PiCam quirk test, fix tests for no-name quirky cameras
* Apply Spotless
* Add pinhole model unit test
A clear and concise description of what the bug is and what the expected behavior should have been.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Screenshots / Videos**
If applicable, add screenshots to help explain your problem. Additionally, provide journalctl logs and settings zip export. If your issue is regarding the web dashboard, please provide screenshots and the output of the browser console.
**Platform:**
- Hardware Platform (ex. Raspberry Pi 4, Windows x64):
- Network Configuration (Connection between the Radio and any devices in between, such as a Network Switch):
<!-- What changed? Why? (the code + comments should speak for itself on the "how") -->
<!-- Fun screenshots or a cool video or something are super helpful as well. If this touches platform-specific behavior, this is where test evidence should be collected. -->
<!-- Any issues this pull request closes or pull requests this supersedes should be linked with `Closes #issuenumber`. -->
## Meta
Merge checklist:
- [ ] Pull Request title is [short, imperative summary](https://cbea.ms/git-commit/) of proposed changes
- [ ] The description documents the _what_ and _why_
- [ ] If this PR changes behavior or adds a feature, user documentation is updated
- [ ] If this PR touches photon-serde, all messages have been regenerated and hashes have not changed unexpectedly
- [ ] If this PR touches configuration, this is backwards compatible with settings back to v2024.3.1
- [ ] If this PR addresses a bug, a regression test for it is added
A copy of the latest development release is available [here](https://github.com/PhotonVision/photonvision/releases/tag/Dev).
PhotonVision is the free, fast, and easy-to-use computer vision solution for the *FIRST* Robotics Competition. You can read an overview of our features [on our website](https://photonvision.org). You can find our comprehensive documentation [here](https://docs.photonvision.org).
PhotonVision is a fork of [Chameleon Vision](https://github.com/Chameleon-Vision/chameleon-vision/), a free open-source software for FRC teams to use for vision processing on their robots. Thank you to everyone who worked on the original project.
The latest release of platform-specific jars and images is found [here](https://github.com/PhotonVision/photonvision/releases).
For information on contributing or running PhotonVision, please read our documentation on ReadTheDocs.
# Roadmap
Our roadmap is publicly available on [Trello](https://trello.com/photonvision).
If you are interested in contributing code or documentation to the project, please [read our getting started page for contributors](https://docs.photonvision.org/en/latest/docs/contributing/index.html) and **[join the Discord](https://discord.gg/wYxTwym) to introduce yourself!** We hope to provide a welcoming community to anyone who is interested in helping.
## Authors
A list of contributors is available in our documentation on ReadTheDocs.
- C++ Doxygen [cppdocs.photonvision.org](https://cppdocs.photonvision.org) (or [manual link](https://photonvision.github.io/photonvision/built-docs/doxygen/html/))
## Building
Gradle is used for all C++ and Java code, and NPM is used for the web UI. Instructions to compile PhotonVision yourself can be found [in our docs](https://docs.photonvision.org/en/latest/docs/contributing/building-photon.html#compiling-instructions).
You can run one of the many built in examples straight from the command line, too! They contain a fully featured robot project, and some include simulation support. The projects can be found inside the [`photonlib-java-examples`](photonlib-java-examples) and [`photonlib-cpp-examples`](photonlib-cpp-examples) subdirectories, respectively. Instructions for running these examples directly from the repo are found [in the docs](https://docs.photonvision.org/en/latest/docs/contributing/building-photon.html#running-examples).
## Gradle Arguments
Note that these are case sensitive!
*`-PArchOverride=foobar`: builds for a target system other than your current architecture. [Valid overrides](https://github.com/wpilibsuite/wpilib-tool-plugin/blob/main/src/main/java/edu/wpi/first/tools/NativePlatforms.java) are:
* winx32
* winx64
* winarm64
* macx64
* macarm64
* linuxx64
* linuxarm64
* linuxathena
-`-PtgtIP`: Specifies where `./gradlew deploy` should try to copy the fat JAR to
-`-Pprofile`: enables JVM profiling
If you're cross-compiling, you'll need the wpilib toolchain installed. This can be done via Gradle: for example `./gradlew installArm64Toolchain` or `./gradlew installRoboRioToolchain`
## Out-of-Source Dependencies
PhotonVision uses the following additional out-of-source repositories for building code.
- Base system images for Raspberry Pi & Orange Pi: https://github.com/PhotonVision/photon-image-modifier
- C++ driver for Raspberry Pi CSI cameras: https://github.com/PhotonVision/photon-libcamera-gl-driver
- JNI code for [mrcal](https://mrcal.secretsauce.net/): https://github.com/PhotonVision/mrcal-java
- Custom build of OpenCV with GStreamer/Protobuf/other custom flags: https://github.com/PhotonVision/thirdparty-opencv
- JNI code for aruco-nano: https://github.com/PhotonVision/aruconano-jni
## Additional packages
For now, using mrcal requires installing these additional packages on Linux systems:
* [WPILib](https://github.com/wpilibsuite) - Specifically [cscore](https://github.com/wpilibsuite/allwpilib/tree/master/cscore), [CameraServer](https://github.com/wpilibsuite/allwpilib/tree/master/cameraserver), [NTCore](https://github.com/wpilibsuite/allwpilib/tree/master/ntcore), and [OpenCV](https://github.com/wpilibsuite/thirdparty-opencv).
PhotonVision was forked from [Chameleon Vision](https://github.com/Chameleon-Vision/chameleon-vision/). Thank you to everyone who worked on the original project.
* [WPILib](https://github.com/wpilibsuite) - Specifically [cscore](https://github.com/wpilibsuite/allwpilib/tree/main/cscore), [CameraServer](https://github.com/wpilibsuite/allwpilib/tree/main/cameraserver), [NTCore](https://github.com/wpilibsuite/allwpilib/tree/main/ntcore), and [OpenCV](https://github.com/wpilibsuite/thirdparty-opencv).
* [Apache Commons](https://commons.apache.org/) - Specifically [Commons Math](https://commons.apache.org/proper/commons-math/), and [Commons Lang](https://commons.apache.org/proper/commons-lang/)
@@ -29,5 +77,10 @@ A list of contributors is available in our documentation on ReadTheDocs.
This page you were looking for was not found. If you think this is a mistake, [file an issue on our GitHub.](https://github.com/PhotonVision/photonvision-docs/issues)
- Ensure you have spares of the relevant electronics if you can afford it (switch, coprocessor, cameras, etc.).
- Stay on the latest version of PhotonVision until you have tested your full robot system to be functional.
- Some time before the competition, lock down the version you are using and do not upgrade unless you encounter a critical bug.
- Have a copy of the installation image for the version you are using on your programming laptop, in case re-imaging (without internet) is needed.
- Extensively test at your home setup. Practice tuning from scratch under different lighting conditions.
- Use SmartDashboard / Shuffleboard to view your camera streams during practice.
- Confirm you have followed all the recommendations under the Networking section in installation (network switch and static IP).
- Only use high quality ethernet cables that have been rigorously tested.
- Set up RIO USB port forwarding using the guide in the Networking section in installation.
## During the Competition
- Use the field calibration time given at the start of the event:
- Bring your robot to the field at the allotted time.
- Make sure the field has match-accurate lighting conditions active.
- Turn on your robot and pull up the dashboard on your driver station.
- Point your robot at the targets and ensure you get a consistent tracking (you hold one targets consistently, the ceiling lights aren't detected, etc.).
- If you have problems with your pipeline, go to the pipeline tuning section and retune the pipeline using the guide there.
- Move the robot close, far, angled, and around the field to ensure no extra targets are found.
- Monitor camera feeds during a practice match to ensure everything is working correctly.
- After field calibration, use the "Export Settings" button in the "Settings" page to create a backup.
- Do this for each coprocessor on your robot that runs PhotonVision, and name your exports with meaningful names.
- This will contain camera information/calibration, pipeline information, network settings, etc.
- In the event of software/hardware failures (IE lost SD Card, broken device), you can then use the "Import Settings" button and select "All Settings" to restore your settings.
- This effectively works as a snapshot of your PhotonVision data that can be restored at any point.
- Before every match:
- Check the ethernet and USB connectors are seated fully.
- Close streaming dashboards when you don't need them to reduce bandwidth.
- Stream at as low of a resolution as possible while still detecting AprilTags to stay within field bandwidth limits.
PhotonVision stores and loads settings in the {code}`photonvision_config` directory, in the same folder as the PhotonVision JAR is stored. On supported hardware, this is in the {code}`/opt/photonvision` directory. The contents of this directory can be exported as a zip archive from the settings page of the interface, under "export settings". This export will contain everything detailed below. These settings can later be uploaded using "import settings", to restore configurations from previous backups.
## Directory Structure
The directory structure is outlined below.
```{image} images/configDir.png
:alt: Config directory structure
:width: 600
```
- calibImgs
- Images saved from the last run of the calibration routine
- cameras
- Contains a subfolder for each camera. This folder contains the following files:
- pipelines folder, which contains a {code}`json` file for each user-created pipeline.
- config.json, which contains all camera-specific configuration. This includes FOV, pitch, current pipeline index, and calibration data
- drivermode.json, which contains settings for the driver mode pipeline
- imgSaves
- Contains images saved with the input/output save commands.
- logs
- Contains timestamped logs in the format {code}`photonvision-YYYY-MM-D_HH-MM-SS.log`. These timestamps will likely be significantly behind the real time. Coprocessors on the robot have no way to get current time.
- hardwareSettings.json
- Contains hardware settings. Currently this includes only the LED brightness.
- networkSettings.json
- Contains network settings, including team number (or remote network tables address), static/dynamic settings, and hostname.
## Importing and Exporting Settings
The entire settings directory can be exported as a ZIP archive from the settings page.
PhotonVision interfaces with PhotonLib, our vendor dependency, using NetworkTables. If you are running PhotonVision on a robot (ie. with a RoboRIO), you should **turn the NetworkTables server switch (in the settings tab) off** in order to get PhotonLib to work. Also ensure that you set your team number. The NetworkTables server should only be enabled if you know what you're doing!
:::
## API
:::{warning}
NetworkTables is not a supported setup/viable option when using PhotonVision as we only send one target at a time (this is problematic when using AprilTags, which will return data from multiple tags at once). We recommend using PhotonLib.
:::
The tables below contain the the name of the key for each entry that PhotonVision sends over the network and a short description of the key. The entries should be extracted from a subtable with your camera's nickname (visible in the PhotonVision UI) under the main `photonvision` table.
| `ledMode` | `int` | Sets the LED Mode (-1: default, 0: off, 1: on, 2: blink) |
:::{warning}
Setting the LED mode to -1 (default) when `multiple` cameras are connected may result in unexpected behavior. {ref}`This is a known limitation of PhotonVision. <docs/troubleshooting/common-errors:LED Control>`
Single camera operation should work without issue.
This page will help you install PhotonVision on non-supported coprocessor.
## Step 1: Software Install
This section will walk you through how to install PhotonVision on your coprocessor. Your coprocessor is the device that has the camera and you are using to detect targets (ex. if you are using a Limelight / Raspberry Pi, that is your coprocessor and you should follow those instructions).
:::{warning}
You only need to install PhotonVision on the coprocessor/device that is being used to detect targets, you do NOT need to install it on the device you use to view the webdashboard. All you need to view the webdashboard is for a device to be on the same network as your vision coprocessor and an internet browser.
Pre-release/development version of PhotonVision can be tested by installing/downloading artifacts from Github Actions (see below), which are built automatically on commits to open pull requests and to PhotonVision's `main` branch, or by {ref}`compiling PhotonVision locally <docs/contributing/building-photon:Build Instructions>`.
:::{warning}
If testing a pre-release version of PhotonVision with a robot, PhotonLib must be updated to match the version downloaded! If not, packet schema definitions may not match and unexpected things will occur. To update PhotonLib, refer to {ref}`installing specific version of PhotonLib<docs/programming/photonlib/adding-vendordep:Install Specific Version - Java/C++>`.
:::
GitHub Actions builds pre-release version of PhotonVision automatically on PRs and on each commit merged to main. To test a particular commit to main, navigate to the [PhotonVision commit list](https://github.com/PhotonVision/photonvision/commits/main/) and click on the check mark (below). Scroll to "Build / Build fat JAR - PLATFORM", click details, and then summary. From here, JAR and image files can be downloaded to be flashed or uploaded using "Offline Update".
```{image} images/gh_actions_1.png
:alt: Github Actions Badge
```
```{image} images/gh_actions_2.png
:alt: Github Actions artifact list
```
Built JAR files (but not image files) can also be downloaded from PRs before they are merged. Navigate to the PR in GitHub, and select Checks at the top. Click on "Build" to display the same artifact list as above.
PhotonVision exposes some command line options which may be useful for customizing execution on Debian-based installations.
## Running a JAR File
Assuming `java` has been installed, and the appropriate environment variables have been set upon installation (a package manager like `apt` should automatically set these), you can use `java -jar` to run a JAR file. If you downloaded the latest stable JAR of PhotonVision from the [GitHub releases page](https://github.com/PhotonVision/photonvision/releases), you can run the following to start the program:
```bash
java -jar /path/to/photonvision/photonvision.jar
```
## Updating a JAR File
When you need to update your JAR file, run the following:
```bash
wget https://git.io/JqkQ9 -O update.sh
sudo chmod +x update.sh
sudo ./update.sh
sudo reboot now
```
## Creating a `systemd` Service
You can also create a systemd service that will automatically run on startup. To do so, first navigate to `/lib/systemd/system`. Create a file called `photonvision.service` (or name it whatever you want) using `touch photonvision.service`. Then open this file in the editor of your choice and paste the following text:
```
[Unit]
Description=Service that runs PhotonVision
[Service]
WorkingDirectory=/path/to/photonvision
# Optional: run photonvision at "nice" -10, which is higher priority than standard
Then copy the `.service` file to `/etc/systemd/system/` using `cp photonvision.service /etc/systemd/system/photonvision.service`. Then modify the file to have `644` permissions using `chmod 644 /etc/systemd/system/photonvision.service`.
:::{note}
Many ARM processors have a big.LITTLE architecture where some of the CPU cores are more powerful than others. On this type of architecture, you may get more consistent performance by limiting which cores PhotonVision can use. To do this, add the parameter `AllowedCPUs` to the systemd service file in the `[Service]` section.
For instance, for an Orange Pi 5, cores 4 through 7 are the fast ones, and you can target those cores with the line `AllowedCPUs=4-7`.
:::
## Installing the `systemd` Service
To install the service, simply run `systemctl enable photonvision.service`.
:::{note}
It is recommended to reload configurations by running `systemctl daemon-reload`.
PhotonVision may be run on a Debian-based Linux Desktop PC for basic testing and evaluation.
:::{note}
You do not need to install PhotonVision on a Windows PC in order to access the webdashboard (assuming you are using an external coprocessor like a Raspberry Pi).
:::
## Installing Java
PhotonVision requires a JDK installed and on the system path. JDK 17 is needed (different versions will not work). If you don't have JDK 17 already, run the following to install it:
```
$ sudo apt-get install openjdk-17-jdk
```
:::{warning}
Using a JDK other than JDK17 will cause issues when running PhotonVision and is not supported.
:::
## Downloading the Latest Stable Release of PhotonVision
Go to the [GitHub releases page](https://github.com/PhotonVision/photonvision/releases) and download the relevant .jar file for your coprocessor.
:::{note}
If your coprocessor has a 64 bit ARM based CPU architecture (OrangePi, Raspberry Pi, etc.), download the LinuxArm64.jar file.
If your coprocessor has an 64 bit x86 based CPU architecture (Mini PC, laptop, etc.), download the Linuxx64.jar file.
:::
:::{warning}
Be careful to pick the latest stable release. "Draft" or "Pre-Release" versions are not stable and often have bugs.
:::
## Running PhotonVision
To run PhotonVision, open a terminal window of your choice and run the following command:
If your computer has a compatible webcam connected, PhotonVision should startup without any error messages. If there are error messages, your webcam isn't supported or another issue has occurred. If it is the latter, please open an issue on the [PhotonVision issues page](https://github.com/PhotonVision/photonvision/issues).
## Accessing the PhotonVision Interface
Once the Java backend is up and running, you can access the main vision interface by navigating to `localhost:5800` inside your browser.
Due to current [cscore](https://github.com/wpilibsuite/allwpilib/tree/main/cscore) restrictions, the PhotonVision server backend may have issues running macOS.
:::
:::{note}
You do not need to install PhotonVision on a Mac in order to access the webdashboard (assuming you are using an external coprocessor like a Raspberry Pi).
:::
VERY Limited macOS support is available.
## Installing Java
PhotonVision requires a JDK installed and on the system path. JDK 17 is needed (different versions will not work). You may already have this if you have installed WPILib 2025+. If not, [download and install it from here](https://adoptium.net/temurin/releases?version=17).
:::{warning}
Using a JDK other than JDK17 will cause issues when running PhotonVision and is not supported.
:::
## Downloading the Latest Stable Release of PhotonVision
Go to the [GitHub releases page](https://github.com/PhotonVision/photonvision/releases) and download the relevant .jar file for your coprocessor.
:::{note}
If you have an M1/M2 Mac, download the macarm64.jar file.
If you have an Intel based Mac, download the macx64.jar file.
:::
:::{warning}
Be careful to pick the latest stable release. "Draft" or "Pre-Release" versions are not stable and often have bugs.
:::
## Running PhotonVision
To run PhotonVision, open a terminal window of your choice and run the following command:
Due to current [cscore](https://github.com/wpilibsuite/allwpilib/tree/main/cscore) restrictions, the PhotonVision using test mode is all that is known to work currently.
:::
## Accessing the PhotonVision Interface
Once the Java backend is up and running, you can access the main vision interface by navigating to `localhost:5800` inside your browser.
:::{warning}
Due to current [cscore](https://github.com/wpilibsuite/allwpilib/tree/main/cscore) restrictions, it is unlikely any streams will open from real webcams.
Working with unsupported coprocessors requires some level of "know how" of your system. The install script has only been tested on Debian/Raspberry Pi OS Buster and Ubuntu Bionic. If any issues arise with your specific OS, please open an issue on our [issues page](https://github.com/PhotonVision/photonvision/issues).
:::
:::{note}
We'd love to have your input! If you get PhotonVision working on another coprocessor, consider documenting your steps and submitting a [docs issue](https://github.com/PhotonVision/photonvision-docs/issues)., [pull request](https://github.com/PhotonVision/photonvision-docs/pulls) , or [ping us on Discord](https://discord.com/invite/wYxTwym). For example, Limelight and Romi install instructions came about because someone spent the time to figure it out, and did a writeup.
:::
## Installing PhotonVision
We provide an [install script](https://git.io/JJrEP) for other Debian-based systems (with `apt`) that will automatically install PhotonVision and make sure that it runs on startup.
```bash
$ wget https://git.io/JJrEP -O install.sh
$ sudo chmod +x install.sh
$ sudo ./install.sh
$ sudo reboot now
```
:::{note}
Your co-processor will require an Internet connection for this process to work correctly.
:::
For installation on any other co-processors, we recommend reading the {ref}`advanced command line documentation <docs/advanced-installation/sw_install/advanced-cmd:Advanced Command Line Usage>`.
## Updating PhotonVision
PhotonVision can be updated by downloading the latest jar file, copying it onto the processor, and restarting the service.
For example, from another computer, run the following commands. Substitute the correct username for "\[user\]" ( Provided images use username "pi")
The [Romi](https://docs.wpilib.org/en/latest/docs/romi-robot/index.html) is a small robot that can be controlled with the WPILib software. The main controller is a Raspberry Pi that must be imaged with [WPILibPi](https://docs.wpilib.org/en/latest/docs/romi-robot/imaging-romi.html) .
## Installation
The WPILibPi image includes FRCVision, which reserves USB cameras; to use PhotonVision, we need to edit the `/home/pi/runCamera` script to disable it. First we will need to make the file system writeable; the easiest way to do this is to go to `10.0.0.2` and choose "Writable" at the top.
SSH into the Raspberry Pi (using Windows command line, or a tool like [Putty](https://www.chiark.greenend.org.uk/~sgtatham/putty/) ) at the Romi's default address `10.0.0.2`. The default user is `pi`, and the password is `raspberry`.
:::.. The following paragraph can be restored when WPILibPi becomes compatible with the current version of PhotonVision.
:::.. Follow the process for installing PhotonVision on {ref}`"Other Debian-Based Co-Processor Installation" <docs/advanced-installation/sw_install/other-coprocessors:Other Debian-Based Co-Processor Installation>`. As it mentions, this will require an internet connection so connecting the Raspberry Pi to an internet-connected router via an Ethernet cable will be the easiest solution. The pi must remain writable while you are following these steps!
:::..Temporary instructions explaining how to install the older version of PhotonVision on a Romi. Remove when no longer needed.
:::{attention}
The version of WPILibPi for the Romi is 2023.2.1, which is not compatible with the current version of PhotonVision. **If you are using WPILibPi 2023.2.1 on your Romi, you must install PhotonVision v2023.4.2 or earlier!**
To install a compatible version of PhotonVision, enter these commands in the SSH terminal connected to the Raspberry Pi. This will download and run the install script, which will install PhotonVision on your Raspberry Pi and configure it to run at startup.
```bash
$ wget https://git.io/JJrEP -O install.sh
$ sudo chmod +x install.sh
$ sudo ./install.sh -v v2023.4.2
```
The install script requires an internet connection, so connecting the Raspberry Pi to an internet-connected router via an Ethernet cable will be the easiest solution. The pi must remain writable while you are following these steps!
:::
:::..End of temporary instructions.
Next, from the SSH terminal, run `sudo nano /home/pi/runCamera` then arrow down to the start of the exec line and press "Enter" to add a new line. Then add `#` before the exec command to comment it out. Then, arrow up to the new line and type `sleep 10000`. Hit "Ctrl + O" and then "Enter" to save the file. Finally press "Ctrl + X" to exit nano. Now, reboot the Romi by typing `sudo reboot now`.
```{image} images/nano.png
```
After the Romi reboots, you should be able to open the PhotonVision UI at: [`http://10.0.0.2:5800/`](http://10.0.0.2:5800/). From here, you can adjust settings and configure {ref}`Pipelines <docs/pipelines/index:Pipelines>`.
:::{warning}
In order for settings, logs, etc. to be saved / take effect, ensure that PhotonVision is in writable mode.
:::
:::{attention}
When using an older version of PhotonVision, the user interface and features may be different than what appears in the online documentation. The [Documentation](http://10.0.0.2:5800/#/docs) link in the User Interface will open a bundled version of the documentation that matches the PhotonVision version running on your coprocessor.
PhotonVision may be run on a Windows Desktop PC for basic testing and evaluation.
:::{note}
You do not need to install PhotonVision on a Windows PC in order to access the webdashboard (assuming you are using an external coprocessor like a Raspberry Pi).
:::
## Install Bonjour
Bonjour provides more stable networking when using Windows PCs. Install [Bonjour here](https://support.apple.com/downloads/DL999/en_US/BonjourPSSetup.exe) before continuing to ensure a stable experience while using PhotonVision.
## Installing Java
PhotonVision requires a JDK installed and on the system path. **JDK 17 is needed. Windows Users must use the JDK that ships with WPILib.** [Download and install it from here.](https://github.com/wpilibsuite/allwpilib/releases/tag/v2025.3.1) Either ensure the only Java on your PATH is the WPILIB Java or specify it to gradle with `-Dorg.gradle.java.home=C:\Users\Public\wpilib\2025\jdk`:
```
> ./gradlew run "-Dorg.gradle.java.home=C:\Users\Public\wpilib\2025\jdk"
```
:::{warning}
Using a JDK other than WPILIB's JDK17 will cause issues when running PhotonVision and is not supported.
:::
## Downloading the Latest Stable Release of PhotonVision
Go to the [GitHub releases page](https://github.com/PhotonVision/photonvision/releases) and download the winx64.jar file.
## Running PhotonVision
To run PhotonVision, open a terminal window of your choice and run the following command:
```
> java -jar C:\path\to\photonvision\NAME OF JAR FILE GOES HERE.jar
```
If your computer has a compatible webcam connected, PhotonVision should startup without any error messages. If there are error messages, your webcam isn't supported or another issue has occurred. If it is the latter, please open an issue on the [PhotonVision issues page](https://github.com/PhotonVision/photonvision/issues).
:::{warning}
Using an integrated laptop camera may cause issues when trying to run PhotonVision. If you are unable to run PhotonVision on a laptop with an integrated camera, try disabling the camera's driver in Windows Device Manager.
:::
## Accessing the PhotonVision Interface
Once the Java backend is up and running, you can access the main vision interface by navigating to `localhost:5800` inside your browser.
Before you get started tracking AprilTags, ensure that you have followed the previous sections on installation, wiring and networking. Next, open the Web UI, go to the top right card, and switch to the "AprilTag" or "Aruco" type. You should see a screen similar to the one below.
```{image} images/apriltag.png
:align: center
```
You are now able to detect and track AprilTags in 2D (yaw, pitch, roll, etc.). In order to get 3D data from your AprilTags, please see {ref}`here. <docs/apriltag-pipelines/3D-tracking:3D Tracking>`
## Tuning AprilTags
AprilTag pipelines come with reasonable defaults to get you up and running with tracking. However, in order to optimize your performance and accuracy, you must tune your AprilTag pipeline using the settings below. Note that the settings below are different between the AprilTag and Aruco detectors but the concepts are the same.
```{image} images/apriltag-tune.png
:align: center
:scale: 45 %
```
### Target Family
Target families are defined by two numbers (before and after the h). The first number is the number of bits the tag is able to encode (which means more tags are available in the respective family) and the second is the hamming distance. Hamming distance describes the ability for error correction while identifying tag ids. A high hamming distance generally means that it will be easier for a tag to be identified even if there are errors. However, as hamming distance increases, the number of available tags decreases.
The 2025 FRC game will be using 36h11 tags, which can be found [here](https://github.com/AprilRobotics/apriltag-imgs/tree/main/tag36h11).
### Decimate
Decimation (also known as down-sampling) is the process of reducing the sampling frequency of a signal (in our case, the image). Increasing decimate will lead to an increased detection rate while decreasing detection distance. We recommend keeping this at the default value.
### Blur
This controls the sigma of Gaussian blur for tag detection. In clearer terms, increasing blur will make the image blurrier, decreasing it will make it closer to the original image. We strongly recommend that you keep blur to a minimum (0) due to it's high performance intensity unless you have an extremely noisy image.
### Threads
Threads refers to the threads within your coprocessor's CPU. The theoretical maximum is device dependent, but we recommend that users to stick to one less than the amount of CPU threads that your coprocessor has. Increasing threads will increase performance at the cost of increased CPU load, temperature increase, etc. It may take some experimentation to find the most optimal value for your system.
### Refine Edges
The edges of the each polygon are adjusted to "snap to" high color differences surrounding it. It is recommended to use this in tandem with decimate as it can increase the quality of the initial estimate.
### Pose Iterations
Pose iterations represents the amount of iterations done in order for the AprilTag algorithm to converge on its pose solution(s). A smaller number between 0-100 is recommended. A smaller amount of iterations cause a more noisy set of poses when looking at the tag straight on, while higher values much more consistently stick to a (potentially wrong) pair of poses. WPILib contains many useful filter classes in order to account for a noisy tag reading.
### Max Error Bits
Max error bits, also known as hamming distance, is the number of positions at which corresponding pieces of data / tag are different. Put more generally, this is the number of bits (think of these as squares in the tag) that need to be changed / corrected in the tag to correctly detect it. A higher value means that more tags will be detected while a lower value cuts out tags that could be "questionable" in terms of detection.
We recommend a value of 0 for the 16h5 and at most 3 for the 36h11 family.
### Decision Margin Cutoff
The decision margin cutoff is how much “margin” the detector has left before it rejects a tag; increasing this rejects poorer tags. We recommend you keep this value around a 30.
3D AprilTag tracking will allow you to track the real-world position and rotation of a tag relative to the camera's image sensor. This is useful for robot pose estimation and other applications like autonomous scoring. In order to use 3D tracking, you must first {ref}`calibrate your camera <docs/calibration/calibration:Calibrating Your Camera>`. Once you have, you need to enable 3D mode in the UI and you will now be able to get 3D pose information from the tag! For information on getting and using this information in your code, see {ref}`the programming reference. <docs/programming/index:Programming Reference>`.
## Ambiguity
Translating from 2D to 3D using data from the calibration and the four tag corners can lead to "pose ambiguity", where it appears that the AprilTag pose is flipping between two different poses. You can read more about this issue `here. <https://docs.wpilib.org/en/stable/docs/software/vision-processing/apriltag/apriltag-intro.html#d-to-3d-ambiguity>` Ambiguity is calculated as the ratio of reprojection errors between two pose solutions (if they exist), where reprojection error is the error corresponding to the image distance between where the apriltag's corners are detected vs where we expect to see them based on the tag's estimated camera relative pose.
There are a few steps you can take to resolve/mitigate this issue:
1. Mount cameras at oblique angles so it is less likely that the tag will be seen straight on.
2. Use the {ref}`MultiTag system <docs/apriltag-pipelines/multitag:MultiTag Localization>` in order to combine the corners from multiple tags to get a more accurate and unambiguous pose.
3. Reject all tag poses where the ambiguity ratio (available via PhotonLib) is greater than 0.2.
AprilTags are a common type of visual fiducial marker. Visual fiducial markers are artificial landmarks added to a scene to allow "localization" (finding your current position) via images. In simpler terms, tags mark known points of reference that you can use to find your current location. They are similar to QR codes in which they encode information, however, they hold only a single number. By placing AprilTags in known locations around the field and detecting them using PhotonVision, you can easily get full field localization / pose estimation. Alternatively, you can use AprilTags the same way you used retroreflective tape, simply using them to turn to goal without any pose estimation.
A more technical explanation can be found in the [WPILib documentation](https://docs.wpilib.org/en/latest/docs/software/vision-processing/apriltag/apriltag-intro.html).
:::{note}
You can get FIRST's [official PDF of the targets used in 2025 here](https://firstfrc.blob.core.windows.net/frc2025/FieldAssets/Apriltag_Images_and_User_Guide.pdf).
PhotonVision follows the WPILib conventions for the robot and field coordinate systems, as defined [here](https://docs.wpilib.org/en/stable/docs/software/advanced-controls/geometry/coordinate-systems.html).
You define the camera to robot transform in the robot coordinate frame.
## Camera Coordinate Frame
OpenCV by default uses x-left/y-down/z-out for camera transforms. PhotonVision applies a base rotation to this transformation to make robot to tag transforms more in line with the WPILib coordinate system. The x, y, and z axes are also shown in red, green, and blue in the 3D mini-map and targeting overlay in the UI.
- The origin is the focal point of the camera lens
- The x-axis points out of the camera
- The y-axis points to the left
- The z-axis points upwards
```{image} images/camera-coord.png
:align: center
:scale: 45 %
```
```{image} images/multiple-tags.png
:align: center
:scale: 45 %
```
## AprilTag Coordinate Frame
The AprilTag coordinate system is defined as follows, relative to the center of the AprilTag itself, and when viewing the tag as a robot would. Again, PhotonVision changes this coordinate system to be more in line with WPILib. This means that a robot facing a tag head-on would see a robot-to-tag transform with a translation only in x, and a rotation of 180 degrees about z. The tag coordinate system is also shown with x/y/z in red/green/blue in the UI target overlay and mini-map.
- The origin is the center of the tag
- The x-axis is normal to the plane the tag is printed on, pointing outward from the visible side of the tag.
PhotonVision offers two different AprilTag pipeline types based on different implementations of the underlying algorithm. Each one has its advantages / disadvantages, which are detailed below.
:::{note}
Note that both of these pipeline types detect AprilTag markers and are just two different algorithms for doing so.
:::
## AprilTag
The AprilTag pipeline type is based on the [AprilTag](https://april.eecs.umich.edu/software/apriltag.html) library from the University of Michigan and we recommend it for most use cases. It is (to our understanding) most accurate pipeline type, but is also ~2x slower than AruCo. This was the pipeline type used by teams in the 2023 season and is well tested.
## AruCo
The AruCo pipeline is based on the [AruCo](https://docs.opencv.org/4.8.0/d9/d6a/group__aruco.html) library implementation from OpenCV. It is ~2x higher fps and ~2x lower latency than the AprilTag pipeline type, but is less accurate. We recommend this pipeline type for teams that need to run at a higher framerate or have a lower powered device. This pipeline type was new for the 2024 season.
PhotonVision can combine AprilTag detections from multiple simultaneously observed AprilTags from a particular camera with information about where tags are expected to be located on the field to produce a better estimate of where the camera (and therefore robot) is located on the field. PhotonVision can calculate this multi-target result on your coprocessor, reducing CPU usage on your RoboRio. This result is sent over NetworkTables along with other detected targets as part of the `PhotonPipelineResult` provided by PhotonLib.
:::{warning}
MultiTag requires an accurate field layout JSON to be uploaded! Differences between this layout and the tags' physical location will drive error in the estimated pose output.
:::
:::{warning}
For the 2025 Reefscape Season, there are two different field layouts. The first is the [welded field layout](https://github.com/wpilibsuite/allwpilib/blob/main/apriltag/src/main/native/resources/edu/wpi/first/apriltag/2025-reefscape-welded.json), which photonvision ships with. The second is the [Andymark field layout](https://github.com/wpilibsuite/allwpilib/blob/main/apriltag/src/main/native/resources/edu/wpi/first/apriltag/2025-reefscape-andymark.json). It is very important to ensure that you use the correct field layout, both in the [PhotonPoseEstimator](https://docs.photonvision.org/en/latest/docs/programming/photonlib/robot-pose-estimator.html#apriltags-and-photonposeestimator) and on the [coprocessor](https://docs.photonvision.org/en/latest/docs/apriltag-pipelines/multitag.html#updating-the-field-layout).
:::
## Enabling MultiTag
Ensure that your camera is calibrated and 3D mode is enabled. Navigate to the Output tab and enable "Do Multi-Target Estimation". This enables MultiTag to use the uploaded field layout JSON to calculate your camera's pose in the field. This 3D transform will be shown as an additional table in the "targets" tab, along with the IDs of AprilTags used to compute this transform.
```{image} images/multitag-ui.png
:alt: Multitarget enabled and running in the PhotonVision UI
:width: 600
```
:::{note}
By default, enabling multi-target will disable calculating camera-to-target transforms for each observed AprilTag target to increase performance; the X/Y/angle numbers shown in the target table of the UI are instead calculated using the tag's expected location (per the field layout JSON) and the field-to-camera transform calculated using MultiTag. If you additionally want the individual camera-to-target transform calculated using SolvePNP for each target, enable "Always Do Single-Target Estimation".
:::
This multi-target pose estimate can be accessed using PhotonLib. We suggest using {ref}`the PhotonPoseEstimator class <docs/programming/photonlib/robot-pose-estimator:AprilTags and PhotonPoseEstimator>` with the `MULTI_TAG_PNP_ON_COPROCESSOR` strategy to simplify code, but the transform can be directly accessed using `getMultiTagResult`/`MultiTagResult()` (Java/C++).
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
var result = camera.getLatestResult();
if (result.getMultiTagResult().estimatedPose.isPresent) {
The returned field to camera transform is a transform from the fixed field origin to the camera's coordinate system. This does not change based on alliance color, and by convention is on the BLUE ALLIANCE wall.
:::
## Updating the Field Layout
PhotonVision ships by default with the [2025 welded field layout JSON](https://github.com/wpilibsuite/allwpilib/blob/main/apriltag/src/main/native/resources/edu/wpi/first/apriltag/2025-reefscape-welded.json). The layout can be inspected by navigating to the settings tab and scrolling down to the "AprilTag Field Layout" card, as shown below.
```{image} images/field-layout.png
:alt: The currently saved field layout in the Photon UI
:width: 600
```
An updated field layout can be uploaded by navigating to the "Device Control" card of the Settings tab and clicking "Import Settings". In the pop-up dialog, select the "AprilTag Layout" type and choose an updated layout JSON (in the same format as the WPILib field layout JSON linked above) using the paperclip icon, and select "Import Settings". The AprilTag layout in the "AprilTag Field Layout" card below should be updated to reflect the new layout.
:::{note}
Currently, there is no way to update this layout using PhotonLib, although this feature is under consideration.
In order to detect AprilTags and use 3D mode, your camera must be calibrated at the desired resolution! Inaccurate calibration will lead to poor performance.
:::
To calibrate a camera, images of a Charuco board (or chessboard) are taken. By comparing where the grid corners should be in object space (for example, a corner once every inch in an 8x6 grid) with where they appear in the camera image, we can find a least-squares estimate for intrinsic camera properties like focal lengths, center point, and distortion coefficients. For more on camera calibration, please review the [OpenCV documentation](https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html).
:::{warning}
While any resolution can be calibrated, higher resolutions may be too performance-intensive for some coprocessors to handle. Therefore, we recommend experimenting to see what works best for your coprocessor.
:::
:::{note}
The calibration data collected during calibration is specific to each physical camera, as well as each individual resolution.
:::
## Calibration Tips
Accurate camera calibration is required in order to get accurate pose measurements when using AprilTags and 3D mode. The tips below should help ensure success:
01. Ensure your the images you take have the target in different positions and angles, with as big of a difference between angles as possible. It is important to make sure the target overlay still lines up with the board while doing this. Tilt no more than 45 degrees.
02. Use as big of a calibration target as your printer can print.
03. Ensure that your printed pattern has enough white border around it.
04. Ensure your camera stays in one position during the duration of the calibration.
05. Make sure you get all 12 images from varying distances and angles.
06. Take at least one image that covers the total image area, and generally ensure that you get even coverage of the lens with your image set.
07. Have good lighting, having a diffusely lit target would be best (light specifically shining on the target without shadows).
08. Ensure the calibration target is completely flat and does not bend or fold in any way. It should be mounted/taped down to something flat and then used for calibration, do not just hold it up.
09. Avoid having targets that are parallel to the lens of the camera / straight on towards the camera as much as possible. You want angles and variations within your calibration images.
Following the ideas above should help in getting an accurate calibration.
## Calibrating using PhotonVision
### 1. Navigate to the calibration section in the UI.
The Cameras tab of the UI houses PhotonVision's camera calibration tooling. It assists users with calibrating their cameras, as well as allows them to view previously calibrated resolutions. We support both charuco and chessboard calibrations.
### 2. Print out the calibration target.
In the Camera Calibration tab, we'll print out the calibration target using the "Download" button. This should be printed on 8.5x11 printer paper. This page shows using an 8x8 charuco board (or chessboard depending on the selected calibration type).
:::{warning}
Ensure that there is no scaling applied during printing (it should be at 100%) and that the PDF is printed as is on regular printer paper. Check the square size with calipers or an accurate measuring device after printing to ensure squares are sized properly, and enter the true size of the square in the UI text box. For optimal results, various resources are available online to calibrate your specific printer if needed.
:::
### 3. Select calibration resolution and fill in appropriate target data.
We'll next select a resolution to calibrate and populate our pattern spacing, marker size, and board size. The provided chessboard and charuco board are an 8x8 grid of 1 inch square. The provided charuco board uses the 4x4 dictionary with a marker size of 0.75 inches (this board does not need the old OpenCV pattern selector selected). Printers are not perfect, and you need to measure your calibration target and enter the correct marker size (size of the aruco marker) and pattern spacing (aka size of the black square) using calipers or similar. Finally, once our entered data is correct, we'll click "start calibration."
:::{warning} Old OpenCV Pattern selector. This should be used in the case that the calibration image is generated from a version of OpenCV before version 4.6.0. This would include targets created by calib.io. If this selector is not set correctly the calibration will be completely invalid. For more info view [this GitHub issue](https://github.com/opencv/opencv_contrib/issues/3291).
:::
:::{note}
If you have a [calib.io](https://calib.io/) CharuCo Target you will have to enter the paramaters of your target. For example if your taget says "9x12 | Chceker Size: 30 mm | Marker Size: 22 mm | Dictionary: AruCo DICT 5x5", you would have to set the board type to Dict_5x5_1000, the pattern spacing to 1.1811 in (30 mm converted to inches), the marker size 0.866142 in (22 mm converted to inches), the board width to 12 and the board height to 9. If you chose the wrong tag family the baord wont be detected duting calibration. If you swap the width and height your calibration will have a very high error.
:::
### 4. Take at calibration images from various angles.
Now, we'll capture images of our board from various angles. It's important to check that the board overlay matches the board in your image. The further the overdrawn points are from the true position of the chessboard corners, the less accurate the final calibration will be. We'll want to capture enough images to cover the whole camera's FOV (with a minimum of 12). Once we've got our images, we'll click "Finish calibration" and wait for the calibration process to complete. If all goes well, the mean error and FOVs will be shown in the table on the right. The FOV should be close to the camera's specified FOV (usually found in a datasheet) usually within + or - 10 degrees. The mean error should also be low, usually less than 1 pixel.
Details about a particular calibration can be viewed by clicking on that resolution in the calibrations tab. This tab allows you to download raw calibration data, upload a previous calibration, and inspect details about calculated camera intrinsic.
```{image} images/cal-details.png
:alt: Captured calibration images
:width: 600
```
:::{note}
More info on what these parameters mean can be found in [OpenCV's docs](https://docs.opencv.org/4.8.0/d4/d94/tutorial_camera_calibration.html)
:::
- Fx/Fy: Estimated camera focal length, in mm
- Fx/Cy: Estimated camera optical center, in pixels. This should be at about the center of the image
- Distortion: OpenCV camera model distortion coefficients
- FOV: calculated using estimated focal length and image size. Useful for gut-checking calibration results
- Mean Err: Mean reprojection error, or distance between expected and observed chessboard cameras for the full calibration dataset
Below these outputs are the snapshots collected for calibration, along with a per-snapshot mean reprojection error. A snapshot with a larger reprojection error might indicate a bad snapshot, due to effects such as motion blur or misidentified chessboard corners.
Calibration images can also be extracted from the downloaded JSON file using [this Python script](https://raw.githubusercontent.com/PhotonVision/photonvision/main/devTools/calibrationUtils.py). This script will unpack calibration images, and also generate a VNL file for use [with mrcal](https://mrcal.secretsauce.net/).
[mrcal](https://mrcal.secretsauce.net/tour.html) is a command-line tool for camera calibration and visualization. PhotonVision has the option to use the mrcal backend during camera calibration to estimate intrinsics. mrcal can also be used post-calibration to inspect snapshots and provide feedback. These steps will closely follow the [mrcal tour](https://mrcal.secretsauce.net/tour-initial-calibration.html) -- I'm aggregating commands and notes here, but the mrcal documentation is much more thorough.
Start by [Installing mrcal](https://mrcal.secretsauce.net/install.html). Note that while mrcal *calibration* using PhotonVision is supported on all platforms, but investigation right now only works on Linux. Some users have also reported luck using [WSL 2 on Windows](https://learn.microsoft.com/en-us/windows/wsl/tutorials/gui-apps) as well. You may also need to install `feedgnuplot`. On Ubuntu systems, these commands should be run from a standalone terminal and *not* the one [built into vscode](https://github.com/ros2/ros2/issues/1406).
Let's run `calibrationUtils.py` as described above, and then cd into the output folder. From here, you can follow the mrcal tour, just replacing the VNL filename and camera imager size as necessary. My camera calibration was at 1280x720, so I've set the XY limits to that below.
```
$ cd /path/to/output/folder
$ ls
matt@photonvision:~/Documents/Downloads/2024-01-02_lifecam_1280$ ls
:alt: A diagram showing the locations of all detected chessboard corners.
```
As you can see, we didn't do a fantastic job of covering our whole camera sensor -- there's a big gap across the whole right side, for example. We also only have 14 calibration images. We've also got our "cameramodel" file, which can be used by mrcal to display additional debug info.
Let's inspect our reprojection error residuals. We expect their magnitudes and directions to be random -- if there's patterns in the colors shown, then our calibration probably doesn't fully explain our physical camera sensor.
Clearly we don't have anywhere near enough data to draw any meaningful conclusions (yet). But for fun, let's dig into [camera uncertainty estimation](https://mrcal.secretsauce.net/tour-uncertainty.html). This diagram shows how expected projection error changes due to noise in calibration inputs. Lower projection error across a larger area of the sensor imply a better calibration that more fully covers the whole sensor. For my calibration data, you can tell the projection error isolines (lines of constant expected projection error) are skewed to the left, following my dataset (which was also skewed left).
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.