Team2890/PhotonVision
4
0
Fork 0
mirror of https://github.com/PhotonVision/photonvision synced 2026-06-26 01:51:40 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into py-docs

Browse Source
This commit is contained in:
Sam Freund
2025-11-25 15:25:17 -06:00
committed by GitHub
parent 99ca8228a1 a8d825919e
commit ce0b00ee03
102 changed files with 1757 additions and 1765 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.clang-format
View File

@@ -68,19 +68,26 @@ ForEachMacros:
- BOOST_FOREACH
IncludeBlocks: Regroup
IncludeCategories:
- Regex: '^<ext/.*\.h>'
Priority: 2
SortPriority: 0
- Regex: '^<.*\.h>'
Priority: 1
SortPriority: 0
- Regex: '^<.*'
Priority: 2
SortPriority: 0
- Regex: '.*'
Priority: 3
SortPriority: 0
IncludeIsMainRegex: '([-_](test|unittest))?$'
# C standard library headers
#
# https://en.cppreference.com/w/cpp/header:
# * C compatibility headers
# * Special C compatibility headers
# * Empty C headers
# * Meaningless C headers
# * Unsupported C headers
- Regex: '^<(assert\.h|ctype\.h|errno\.h|fenv\.h|float\.h|inttypes\.h|limits\.h|locale\.h|math\.h|setjmp\.h|signal\.h|stdarg\.h|stddef\.h|stdint\.h|stdio\.h|stdlib\.h|string\.h|time\.h|uchar\.h|wchar\.h|wctype\.h|stdatomic\.h|ccomplex|complex\.h|ctgmath|tgmath\.h|ciso646|cstdalign|cstdbool|iso646\.h|stdalign\.h|stdbool\.h|stdatomic\.h|stdnoreturn\.h|threads\.h)>'
Priority: 1
# C++ standard library headers (lowercase and underscores with no .h suffix)
- Regex: '^<[a-z_]+>'
Priority: 2
# Other library headers (angle brackets)
- Regex: '^<.*'
Priority: 3
# Project headers (double quotes)
- Regex: '^".*'
Priority: 4
IncludeIsMainRegex: '(Test|_test)?$'
IncludeIsMainSourceRegex: ''
IndentCaseLabels: true
IndentGotoLabels: true
@@ -136,7 +143,7 @@ RawStringFormats:
CanonicalDelimiter: ''
BasedOnStyle: google
ReflowComments: true
SortIncludes: false
SortIncludes: true
SortUsingDeclarations: true
SpaceAfterCStyleCast: false
SpaceAfterLogicalNot: false

1
.github/pull_request_template.md vendored
View File

@@ -11,6 +11,7 @@
Merge checklist:
- [ ] Pull Request title is [short, imperative summary](https://cbea.ms/git-commit/) of proposed changes
- [ ] The description documents the _what_ and _why_
- [ ] This PR has been [linted](https://docs.photonvision.org/en/latest/docs/contributing/linting.html).
- [ ] If this PR changes behavior or adds a feature, user documentation is updated
- [ ] If this PR touches photon-serde, all messages have been regenerated and hashes have not changed unexpectedly
- [ ] If this PR touches configuration, this is backwards compatible with settings back to v2025.3.2

6
.github/workflows/build.yml vendored
View File

@@ -88,10 +88,8 @@ jobs:
run: sudo apt-get update && sudo apt-get install -y libcholmod3 liblapack3 libsuitesparseconfig5
- name: Gradle Build
run: ./gradlew photon-targeting:build photon-core:build photon-server:build -x check
- name: Gradle Tests
run: ./gradlew testHeadless --stacktrace
- name: Gradle Coverage
run: ./gradlew jacocoTestReport
- name: Gradle Tests and Coverage
run: ./gradlew test jacocoTestReport --stacktrace
build-offline-docs:
name: "Build Offline Docs"
runs-on: ubuntu-22.04

2
.github/workflows/lint-format.yml vendored
View File

@@ -31,7 +31,7 @@ jobs:
with:
python-version: 3.11
- name: Install wpiformat
run: pip3 install wpiformat==2025.34
run: pip3 install wpiformat==2025.75
- name: Run
run: wpiformat
- name: Check output

38
.styleguide → .wpiformat
View File

@@ -1,41 +1,25 @@
cppHeaderFileInclude {
\.h$
\.hpp$
\.inc$
\.inl$
}
cppSrcFileInclude {
\.cpp$
}
modifiableFileExclude {
\.jpg$
\.jpeg$
\.png$
\.gif$
\.so$
\.dll$
\.webp$
\.gif$
\.ico$
\.rknn$
\.tflite$
\.jpeg$
\.jpg$
\.mp4$
\.pdf$
\.png$
\.rknn$
\.so$
\.svg$
\.tflite$
\.ttf$
\.webp$
\.woff2$
gradlew
photon-lib/py/photonlibpy/generated/
photon-targeting/src/main/native/cpp/photon/constrained_solvepnp/generate/
photon-targeting/src/generated/
}
includeProject {
^photonLib/
}
includeOtherLibs {
^frc/
^networktables/
^units/
^wpi/
photon-targeting/src/main/native/cpp/photon/constrained_solvepnp/generate/
}

0
.styleguide-license → .wpiformat-license
View File

4
build.gradle
View File

@@ -39,7 +39,7 @@ ext {
javalinVersion = "6.7.0"
libcameraDriverVersion = "v2025.0.4"
rknnVersion = "dev-v2025.0.0-5-g666c0c6"
rubikVersion = "dev-v2025.1.0-8-g067a316"
rubikVersion = "dev-v2025.1.0-6-g4a5e508"
frcYear = "2025"
mrcalVersion = "v2025.0.0";
@@ -92,7 +92,7 @@ spotless {
format 'misc', {
target fileTree('.') {
include '**/*.md', '**/.gitignore'
exclude '**/build/**', '**/build-*/**'
exclude '**/build/**', '**/build-*/**', '**/node_modules/**'
}
trimTrailingWhitespace()
indentWithSpaces(2)

18
docs/.styleguide
View File

@@ -1,18 +0,0 @@
modifiableFileExclude {
\.jpg$
\.jpeg$
\.png$
\.gif$
\.so$
\.pdf$
\.mp4$
\.dll$
\.webp$
\.ico$
\.rknn$
\.tflite$
\.svg$
\.woff2$
gradlew
}

25
docs/source/conf.py
View File

@@ -21,6 +21,29 @@ project = "PhotonVision"
copyright = "2024, PhotonVision"
author = "Banks Troutman, Matt Morley"
# -- Git configuration -----------------------------------------------------
import subprocess
try:
# Use closest tag
git_tag_ref = (
subprocess.check_output(
[
"git",
"describe",
"--tags",
],
stderr=subprocess.DEVNULL,
)
.strip()
.decode()
)
except subprocess.CalledProcessError:
# Couldn't find closest tag, fallback to main
git_tag_ref = "main"
myst_substitutions = {"git_tag_ref": git_tag_ref}
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
@@ -158,4 +181,4 @@ if token:
linkcheck_auth = [(R"https://github.com/.+", token)]
# MyST configuration (https://myst-parser.readthedocs.io/en/latest/configuration.html)
myst_enable_extensions = ["colon_fence"]
myst_enable_extensions = ["colon_fence", "substitution"]

4
docs/source/docs/apriltag-pipelines/2D-tracking-tuning.md
View File

@@ -2,7 +2,7 @@
## Tracking AprilTags
Before you get started tracking AprilTags, ensure that you have followed the previous sections on installation, wiring and networking. Next, open the Web UI, go to the top right card, and switch to the "AprilTag" or "Aruco" type. You should see a screen similar to the one below.
Before you get started tracking AprilTags, ensure that you have followed the previous sections on installation, wiring and networking. Next, open the Web UI, go to the top right card, and switch to the "AprilTag" or "ArUco" type. You should see a screen similar to the one below.
```{image} images/apriltag.png
:align: center
@@ -12,7 +12,7 @@ You are now able to detect and track AprilTags in 2D (yaw, pitch, roll, etc.). I
## Tuning AprilTags
AprilTag pipelines come with reasonable defaults to get you up and running with tracking. However, in order to optimize your performance and accuracy, you must tune your AprilTag pipeline using the settings below. Note that the settings below are different between the AprilTag and Aruco detectors but the concepts are the same.
AprilTag pipelines come with reasonable defaults to get you up and running with tracking. However, in order to optimize your performance and accuracy, you must tune your AprilTag pipeline using the settings below. Note that the settings below are different between the AprilTag and ArUco detectors but the concepts are the same.
```{image} images/apriltag-tune.png
:align: center

6
docs/source/docs/apriltag-pipelines/detector-types.md
View File

@@ -8,8 +8,8 @@ Note that both of these pipeline types detect AprilTag markers and are just two
## AprilTag
The AprilTag pipeline type is based on the [AprilTag](https://april.eecs.umich.edu/software/apriltag.html) library from the University of Michigan and we recommend it for most use cases. It is (to our understanding) most accurate pipeline type, but is also ~2x slower than AruCo. This was the pipeline type used by teams in the 2023 season and is well tested.
The AprilTag pipeline type is based on the [AprilTag](https://april.eecs.umich.edu/software/apriltag.html) library from the University of Michigan and we recommend it for most use cases. It is (to our understanding) most accurate pipeline type, but is also ~2x slower than ArUco. This was the pipeline type used by teams in the 2023 season and is well tested.
## AruCo
## ArUco
The AruCo pipeline is based on the [AruCo](https://docs.opencv.org/4.8.0/d9/d6a/group__aruco.html) library implementation from OpenCV. It is ~2x higher fps and ~2x lower latency than the AprilTag pipeline type, but is less accurate. We recommend this pipeline type for teams that need to run at a higher framerate or have a lower powered device. This pipeline type was new for the 2024 season.
The ArUco pipeline is based on the [ArUco](https://docs.opencv.org/4.8.0/d9/d6a/group__aruco.html) library implementation from OpenCV. It is ~2x higher fps and ~2x lower latency than the AprilTag pipeline type, but is less accurate. We recommend this pipeline type for teams that need to run at a higher framerate or have a lower powered device. This pipeline type was new for the 2024 season.

14
docs/source/docs/calibration/calibration.md
View File

@@ -4,7 +4,7 @@
In order to detect AprilTags and use 3D mode, your camera must be calibrated at the desired resolution! Inaccurate calibration will lead to poor performance.
:::
To calibrate a camera, images of a Charuco board (or chessboard) are taken. By comparing where the grid corners should be in object space (for example, a corner once every inch in an 8x6 grid) with where they appear in the camera image, we can find a least-squares estimate for intrinsic camera properties like focal lengths, center point, and distortion coefficients. For more on camera calibration, please review the [OpenCV documentation](https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html).
To calibrate a camera, images of a ChArUco board (or chessboard) are taken. By comparing where the grid corners should be in object space (for example, a corner once every inch in an 8x6 grid) with where they appear in the camera image, we can find a least-squares estimate for intrinsic camera properties like focal lengths, center point, and distortion coefficients. For more on camera calibration, please review the [OpenCV documentation](https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html).
:::{warning}
While any resolution can be calibrated, higher resolutions may be too performance-intensive for some coprocessors to handle. Therefore, we recommend experimenting to see what works best for your coprocessor.
@@ -16,6 +16,10 @@ The calibration data collected during calibration is specific to each physical c
## Calibration Tips
:::{warning}
The usage of chessboards can result in bad calibration results if multiple similar images are taken. We strongly recommend that teams use ChArUco boards instead!
:::
Accurate camera calibration is required in order to get accurate pose measurements when using AprilTags and 3D mode. The tips below should help ensure success:
01. Ensure the images you take have the target in different positions and angles, with as big of a difference between angles as possible. It is important to make sure the target overlay still lines up with the board while doing this. Tilt no more than 45 degrees.
@@ -34,11 +38,11 @@ Following the ideas above should help in getting an accurate calibration.
### 1. Navigate to the calibration section in the UI.
The Cameras tab of the UI houses PhotonVision's camera calibration tooling. It assists users with calibrating their cameras, as well as allows them to view previously calibrated resolutions. We support both charuco and chessboard calibrations.
The Cameras tab of the UI houses PhotonVision's camera calibration tooling. It assists users with calibrating their cameras, as well as allows them to view previously calibrated resolutions. We support both ChArUco and chessboard calibrations.
### 2. Print out the calibration target.
In the Camera Calibration tab, we'll print out the calibration target using the "Download" button. This should be printed on 8.5x11 printer paper. This page shows using an 8x8 charuco board (or chessboard depending on the selected calibration type).
In the Camera Calibration tab, we'll print out the calibration target using the "Download" button. This should be printed on 8.5x11 printer paper. This page shows using an 8x8 ChArUco board (or chessboard depending on the selected calibration type).
:::{warning}
Ensure that there is no scaling applied during printing (it should be at 100%) and that the PDF is printed as is on regular printer paper. Check the square size with calipers or an accurate measuring device after printing to ensure squares are sized properly, and enter the true size of the square in the UI text box. For optimal results, various resources are available online to calibrate your specific printer if needed.
@@ -46,13 +50,13 @@ Ensure that there is no scaling applied during printing (it should be at 100%) a
### 3. Select calibration resolution and fill in appropriate target data.
We'll next select a resolution to calibrate and populate our pattern spacing, marker size, and board size. The provided chessboard and charuco board are an 8x8 grid of 1 inch square. The provided charuco board uses the 4x4 dictionary with a marker size of 0.75 inches (this board does not need the old OpenCV pattern selector selected). Printers are not perfect, and you need to measure your calibration target and enter the correct marker size (size of the aruco marker) and pattern spacing (aka size of the black square) using calipers or similar. Finally, once our entered data is correct, we'll click "start calibration."
We'll next select a resolution to calibrate and populate our pattern spacing, marker size, and board size. The provided chessboard and ChArUco board are an 8x8 grid of 1 inch square. The provided ChArUco board uses the 4x4 dictionary with a marker size of 0.75 inches (this board does not need the old OpenCV pattern selector selected). Printers are not perfect, and you need to measure your calibration target and enter the correct marker size (size of the ArUco marker) and pattern spacing (aka size of the black square) using calipers or similar. Finally, once our entered data is correct, we'll click "start calibration."
:::{warning} Old OpenCV Pattern selector. This should be used in the case that the calibration image is generated from a version of OpenCV before version 4.6.0. This would include targets created by calib.io. If this selector is not set correctly the calibration will be completely invalid. For more info view [this GitHub issue](https://github.com/opencv/opencv_contrib/issues/3291).
:::
:::{note}
If you have a [calib.io](https://calib.io/) CharuCo Target you will have to enter the paramaters of your target. For example if your target says "9x12 | Checker Size: 30 mm | Marker Size: 22 mm | Dictionary: AruCo DICT 5x5", you would have to set the board type to Dict_5x5_1000, the pattern spacing to 1.1811 in (30 mm converted to inches), the marker size 0.866142 in (22 mm converted to inches), the board width to 12 and the board height to 9. If you chose the wrong tag family the board wont be detected during calibration. If you swap the width and height your calibration will have a very high error.
If you have a [calib.io](https://calib.io/) ChArUco Target you will have to enter the paramaters of your target. For example if your target says "9x12 | Checker Size: 30 mm | Marker Size: 22 mm | Dictionary: ArUco DICT 5x5", you would have to set the board type to Dict_5x5_1000, the pattern spacing to 1.1811 in (30 mm converted to inches), the marker size 0.866142 in (22 mm converted to inches), the board width to 12 and the board height to 9. If you chose the wrong tag family the board wont be detected during calibration. If you swap the width and height your calibration will have a very high error.
:::
### 4. Take at calibration images from various angles.

4
docs/source/docs/camera-specific-configuration/arducam-cameras.md
View File

@@ -1,5 +1,9 @@
# Arducam Cameras
:::{warning}
Arducam Pivariety cameras are **incompatible** with PhotonVision as they require a custom camera library not compatible with PhotonVision.
:::
Arducam cameras are supported for setups with multiple devices. This is possible because Arducam provides software that allows you to assign truly different device names to each camera. This feature is particularly useful in complex setups where multiple cameras are used simultaneously.
## Setting Up Arducam Cameras

27
docs/source/docs/contributing/building-photon.md
View File

@@ -18,6 +18,10 @@ This section contains the build instructions from the source code available at [
[pnpm](https://pnpm.io/) is the package manager used to download dependencies for the UI. To install pnpm, follow [the instructions on the official pnpm website](https://pnpm.io/installation).
**Cross-Compilation Toolchains (Optional):**
If you plan to deploy PhotonVision to a coprocessor like a Raspberry Pi, you will need to install the appropriate cross-compilation toolchain for your platform. For `linuxarm64` devices, this can be accomplished by running `./gradlew installArm64Toolchain` in the root folder of the project.
## Compiling Instructions
### Getting the Source Code
@@ -173,6 +177,29 @@ With the VSCode [Extension Pack for Java](https://marketplace.visualstudio.com/i
To correctly run PhotonVision tests this way, you must [delegate the tests to Gradle](https://code.visualstudio.com/docs/java/java-build#_delegate-tests-to-gradle). Debugging tests like this will [**not** currently](https://github.com/microsoft/build-server-for-gradle/issues/119) collect outputs.
### Running Tests With UI
By default, tests are run with UI disabled so they are not obtrusive during a build. All tests should be useful when the UI is disabled. However, if a particular test would benefit from having UI access (i.e. for debugging info), the UI can be enabled by passing the `enableTestUi` project property to Gradle. This will run all tests by default, but the Gradle `--tests` option can be used to [filter for specific tests](https://docs.gradle.org/current/userguide/java_testing.html#test_filtering).
```{eval-rst}
.. tab-set::
.. tab-item:: Linux
:sync: linux
``./gradlew test -PenableTestUi``
.. tab-item:: macOS
:sync: macos
``./gradlew test -PenableTestUi``
.. tab-item:: Windows (cmd)
:sync: windows
``gradlew test -PenableTestUi``
```
### Debugging PhotonVision Running Locally
Unit tests can instead be debugged through the ``test`` Gradle task for a specific subproject in VSCode, found in the Gradle tab:

1
docs/source/docs/contributing/index.md
View File

@@ -3,6 +3,7 @@
```{toctree}
building-photon
building-docs
linting
developer-docs/index
design-descriptions/index
```

43
docs/source/docs/contributing/linting.md Normal file
View File

@@ -0,0 +1,43 @@
# Linting the PhotonVision Codebase
## Versions
:::{note}
If you work on other projects that use different versions of the same linters as PhotonVision, you may find it beneficial to use a [venv](https://docs.python.org/3/library/venv.html) instead of installing the linters globally. This will allow you to have different versions of the same linter installed for different projects.
:::
The correct versions for each linter can be found under the linting workflow located [here](https://github.com/PhotonVision/photonvision/tree/main/.github/workflows). For *doc8*, the version can be found in `docs/requirements.txt`. If you've linted, and are still unable to pass CI, please check the versions of your linters.
## Frontend
### Linting the frontend
In order to lint the frontend, run `pnpm -C photon-client lint && pnpm -C photon-client format`. This should be done from the base level of the repo.
## Backend
### wpiformat installation
To lint the backend, PhotonVision uses *wpiformat* and *spotless*. Spotless is included with gradle, which means installation is not needed. To install wpiformat, run `pipx install wpiformat`. To install a specific version, run `pipx install wpiformat==<version>`.
### Linting the backend
To lint, run `./gradlew spotlessApply` and `wpiformat`.
## Documentation
### doc8 installation
To install *doc8*, the python tool we use to lint our documentation, run `pipx install doc8`. To install a specific version, run `pipx install doc8==<version>`.
### Linting the documentation
To lint the documentation, run `doc8 docs` from the root level of the docs.
## Alias
The following [alias](https://www.computerworld.com/article/1373210/how-to-use-aliases-in-linux-shell-commands.html) can be added to your shell config, which will allow you to lint the entirety of the PhotonVision project by running `pvLint`. The alias will work on Linux, macOS, Git Bash on Windows, and WSL.
```sh
alias pvLint='wpiformat -v && ./gradlew spotlessApply && pnpm -C photon-client lint && pnpm -C photon-client format && doc8 docs'
```

2
docs/source/docs/examples/aimingatatarget.md
View File

@@ -6,7 +6,7 @@ The following example is from the PhotonLib example repository ([Java](https://g
- A Robot
- A camera mounted rigidly to the robot's frame, centered and pointed forward.
- A coprocessor running PhotonVision with an AprilTag or Aruco 2D Pipeline.
- A coprocessor running PhotonVision with an AprilTag or ArUco 2D Pipeline.
- [A printout of AprilTag 7](https://firstfrc.blob.core.windows.net/frc2025/FieldAssets/Apriltag_Images_and_User_Guide.pdf), mounted on a rigid and flat surface.
## Code

2
docs/source/docs/objectDetection/opi.md
View File

@@ -14,6 +14,6 @@ PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv5, YOLOv8, and YOL
Only quantized models are supported, so take care when exporting to select the option for quantization.
:::
PhotonVision now ships with a [Python Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/rknn-convert-tool/rknn_conversion.ipynb) that you can use in [Google Colab](https://colab.research.google.com) or in a local environment. In Google Colab, you can simply paste the PhotonVision GitHub URL into the "GitHub" tab and select the `rknn_conversion.ipynb` notebook without needing to manually download anything.
PhotonVision now ships with a {{ '[Python Notebook](https://github.com/PhotonVision/photonvision/blob/{}/scripts/rknn_conversion.ipynb)'.format(git_tag_ref) }} that you can use in [Google Colab](https://colab.research.google.com) or in a local **Linux** environment (since `rknn-toolkit2` only supports Linux). In Google Colab, you can simply paste the PhotonVision GitHub URL into the "GitHub" tab and select the `rknn_conversion.ipynb` notebook without needing to manually download anything.
Please ensure that the model you are attempting to convert is among the {ref}`supported models <docs/objectDetection/opi:Supported Models>` and using the PyTorch format.

2
docs/source/docs/objectDetection/rubik.md
View File

@@ -14,7 +14,7 @@ PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv8 and YOLOv11 mode
Only quantized models are supported, so take care when exporting to select the option for quantization.
:::
PhotonVision now ships with a [Python Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/rubik_conversion.ipynb) that you can use in [Google Colab](https://colab.research.google.com) or in a local environment. In Google Colab, you can simply paste the PhotonVision GitHub URL into the "GitHub" tab and select the `rubik_conversion.ipynb` notebook without needing to manually download anything.
PhotonVision now ships with a {{ '[Python Notebook](https://github.com/PhotonVision/photonvision/blob/{}/scripts/rubik_conversion.ipynb)'.format(git_tag_ref) }} that you can use in [Google Colab](https://colab.research.google.com), [Kaggle](https://kaggle.com/code), or in a local environment. In Google Colab, you can simply paste the PhotonVision GitHub URL into the "GitHub" tab and select the `rubik_conversion.ipynb` notebook without needing to manually download anything.
Please ensure that the model you are attempting to convert is among the {ref}`supported models <docs/objectDetection/rubik:Supported Models>` and using the PyTorch format.

4
docs/source/docs/pipelines/about-pipelines.md
View File

@@ -10,7 +10,7 @@ A vision pipeline represents a series of steps that are used to acquire an image
## Types of Pipelines
### AprilTag / AruCo
### AprilTag / ArUco
This pipeline type is based on detecting AprilTag fiducial markers. More information about AprilTags can be found in the [WPILib documentation](https://docs.wpilib.org/en/stable/docs/software/vision-processing/apriltag/apriltag-intro.html). This pipeline provides easy to use 3D pose information which allows localization.
@@ -56,7 +56,7 @@ Each pipeline has a set of tabs that are used to configure the pipeline. All pip
Pipielines also have additional tabs that are specific to the pipeline type. Listed below are the tabs for each pipeline type.
### AprilTag / AruCo Pipelines
### AprilTag / ArUco Pipelines
- AprilTag: This tab includes AprilTag specific tuning parameters, such as decimate, blur, threads, pose iterations, and more.

4
docs/source/docs/quick-start/camera-calibration.md
View File

@@ -9,8 +9,8 @@ If you’re not using cameras in 3D mode, calibration is optional, but it can st
## Print the Calibration Target
- Downloaded from our [demo site](http://photonvision.global/#/cameras), or directly from your coprocessors cameras tab.
- Use the Charuco calibration board:
- Board Type: Charuco
- Use the ChArUco calibration board:
- Board Type: ChAruCo
- Tag Family: 4x4
- Pattern Spacing: 1.00in
- Marker Size: 0.75in

19
docs/source/docs/quick-start/camera-focusing.md Normal file
View File

@@ -0,0 +1,19 @@
# Camera Focusing
## Prepare Camera
:::{warning}
Refocusing your camera **will** make your calibration inaccurate, make sure to recalibrate after focusing.
:::
To ensure that your camera is focused properly, mount it to a secure surface and ensure it does not move drastically. Point your camera at a detailed surface like a calibration board, and make sure that it not too close to the camera.
## Using Focus Mode
:::{important}
When you enable Focus Mode, it will assign a *Score* to the current focus, this score depends on your environment and the lighting. This score cannot be compared to a focus score collected from other environments.
:::
- In the Cameras tab, turn on Focus Mode.
- Rotate the lens on your camera to try and get the focus score as high as possible.
- Once you cannot get a higher score, this indicates that your camera is fully focused and can be set in place using glue if desired.
```{image} images/focusModeExample.png
:scale: 50%
```

BIN
docs/source/docs/quick-start/images/focusModeExample.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 313 KiB

1
docs/source/docs/quick-start/index.md
View File

@@ -9,5 +9,6 @@ wiring
networking
camera-matching
camera-calibration
camera-focusing
quick-configure
```

2
photon-client/package.json
View File

@@ -10,7 +10,7 @@
"build-demo": "vite build --mode demo",
"lint": "eslint . --ext .vue,.js,.jsx,.cjs,.mjs,.ts,.tsx,.cts,.mts --fix",
"format": "prettier --write src/",
"lint-ci": "eslint . --max-warnings 0 --ext .vue,.js,.jsx,.cjs,.mjs,.ts,.tsx,.cts,.mts --fix",
"lint-ci": "eslint . --max-warnings 0 --ext .vue,.js,.jsx,.cjs,.mjs,.ts,.tsx,.cts,.mts",
"format-ci": "prettier --check src/"
},
"dependencies": {

10
photon-client/src/components/app/photon-error-snackbar.vue
View File

@@ -13,5 +13,15 @@ import { useStateStore } from "@/stores/StateStore";
<p style="padding: 0; margin: 0; text-align: center">
{{ useStateStore().snackbarData.message }}
</p>
<v-progress-linear
v-if="useStateStore().snackbarData.progressBar != -1"
v-model="useStateStore().snackbarData.progressBar"
height="15"
:color="useStateStore().snackbarData.progressBarColor"
>
<template #default="{ value }">
<strong> {{ Math.ceil(value) }}% </strong>
</template>
</v-progress-linear>
</v-snackbar>
</template>

43
photon-client/src/components/cameras/CameraCalibrationCard.vue
View File

@@ -1,5 +1,5 @@
<script setup lang="ts">
import { computed, ref } from "vue";
import { computed, ref, watchEffect } from "vue";
import { useCameraSettingsStore } from "@/stores/settings/CameraSettingsStore";
import { CalibrationBoardTypes, CalibrationTagFamilies, type VideoFormat } from "@/types/SettingTypes";
import MonoLogo from "@/assets/images/logoMono.png";
@@ -79,6 +79,18 @@ const calibrationDivisors = computed(() =>
})
);
const uniqueVideoResolutionString = ref("");
// Use a watchEffect so the value is populated/reacts when the stores become available or update.
// This avoids trying to index into an array that may be empty during page reload.
watchEffect(() => {
const currentIndex = useCameraSettingsStore().currentVideoFormat.index ?? 0;
useStateStore().calibrationData.videoFormatIndex = currentIndex;
const names = useCameraSettingsStore().currentCameraSettings.validVideoFormats.map((f) =>
getResolutionString(f.resolution)
);
uniqueVideoResolutionString.value = names[currentIndex] ?? names[0] ?? "";
});
const squareSizeIn = ref(1);
const markerSizeIn = ref(0.75);
const patternWidth = ref(8);
@@ -129,7 +141,7 @@ const downloadCalibBoard = async () => {
break;
case CalibrationBoardTypes.Charuco:
// Add pregenerated charuco
// Add pregenerated ChArUco
const charucoImage = new Image();
charucoImage.src = CharucoImage;
doc.addImage(charucoImage, "PNG", 0.25, 1.5, 8, 8);
@@ -279,23 +291,36 @@ const setSelectedVideoFormat = (format: VideoFormat) => {
: 'MrCal failed to load, check journalctl logs for details.'
"
/>
<!-- TODO: the default videoFormatIndex is 0, but the list of unique video mode indexes might not include 0. getUniqueVideoResolutionStrings indexing is also different from the normal video mode indexing -->
<pv-select
v-model="useStateStore().calibrationData.videoFormatIndex"
v-model="uniqueVideoResolutionString"
label="Resolution"
:select-cols="8"
:disabled="isCalibrating"
tooltip="Resolution to calibrate at (you will have to calibrate every resolution you use 3D mode on)"
:items="getUniqueVideoResolutionStrings()"
@update:model-value="
useStateStore().calibrationData.videoFormatIndex =
getUniqueVideoResolutionStrings().find((v) => v.value === $event)?.value || 0
"
/>
<pv-select
v-model="boardType"
label="Board Type"
tooltip="Calibration board pattern to use"
:select-cols="8"
:items="['Chessboard', 'Charuco']"
:items="['Chessboard', 'ChArUco']"
:disabled="isCalibrating"
/>
<v-alert
v-if="boardType !== CalibrationBoardTypes.Charuco"
closable
density="compact"
variant="tonal"
color="warning"
icon="mdi-alert-box"
text="The usage of chessboards can result in bad calibration results if multiple
similar images are taken. We strongly recommend that teams use ChArUco boards instead!"
/>
<pv-select
v-if="boardType !== CalibrationBoardTypes.Charuco"
v-model="useCameraSettingsStore().currentPipelineSettings.streamingFrameDivisor"
@@ -311,7 +336,7 @@ const setSelectedVideoFormat = (format: VideoFormat) => {
v-if="boardType === CalibrationBoardTypes.Charuco"
v-model="tagFamily"
label="Tag Family"
tooltip="Dictionary of aruco markers on the charuco board"
tooltip="Dictionary of ArUco markers on the ChArUco board"
:select-cols="8"
:items="['Dict_4X4_1000', 'Dict_5X5_1000', 'Dict_6X6_1000', 'Dict_7X7_1000']"
:disabled="isCalibrating"
@@ -527,9 +552,9 @@ const setSelectedVideoFormat = (format: VideoFormat) => {
<v-card-text>
Camera has been successfully calibrated for
{{
getUniqueVideoResolutionStrings().find(
(v) => v.value === useStateStore().calibrationData.videoFormatIndex
)?.name
useCameraSettingsStore().currentCameraSettings.validVideoFormats.map((f) =>
getResolutionString(f.resolution)
)[useStateStore().calibrationData.videoFormatIndex]
}}!
</v-card-text>
</template>

45
photon-client/src/components/cameras/CameraCalibrationInfoCard.vue
View File

@@ -3,8 +3,9 @@ import type { CameraCalibrationResult, VideoFormat } from "@/types/SettingTypes"
import { useCameraSettingsStore } from "@/stores/settings/CameraSettingsStore";
import { useStateStore } from "@/stores/StateStore";
import { computed, inject, ref } from "vue";
import { getResolutionString, parseJsonFile } from "@/lib/PhotonUtils";
import { axiosPost, getResolutionString, parseJsonFile } from "@/lib/PhotonUtils";
import { useTheme } from "vuetify";
import PvDeleteModal from "@/components/common/pv-delete-modal.vue";
const theme = useTheme();
@@ -12,6 +13,16 @@ const props = defineProps<{
videoFormat: VideoFormat;
}>();
const confirmRemoveDialog = ref({ show: false, vf: props.videoFormat as VideoFormat });
const removeCalibration = (vf: VideoFormat) => {
axiosPost("/calibration/remove", "delete a camera calibration", {
cameraUniqueName: useCameraSettingsStore().currentCameraSettings.uniqueName,
width: vf.resolution.width,
height: vf.resolution.height
});
};
const exportCalibration = ref();
const openExportCalibrationPrompt = () => {
exportCalibration.value.click();
@@ -93,18 +104,19 @@ const calibrationImageURL = (index: number) =>
</script>
<template>
<v-card color="surface" dark>
<div class="d-flex flex-wrap pt-2 pl-2 pr-2">
<div class="d-flex flex-wrap pt-2 pl-2 pr-2 align-center">
<v-col cols="12" md="6">
<v-card-title class="pa-0"> Calibration Details </v-card-title>
</v-col>
<v-col cols="6" md="3" class="d-flex align-center pt-0 pt-md-3 pl-6 pl-md-3">
<v-col cols="12" md="6" class="d-flex align-center pt-0 pt-md-3">
<v-btn
color="buttonPassive"
style="width: 100%"
class="mr-2"
style="flex: 1"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="openUploadPhotonCalibJsonPrompt"
>
<v-icon start size="large"> mdi-import</v-icon>
<v-icon start size="large">mdi-import</v-icon>
<span>Import</span>
</v-btn>
<input
@@ -114,12 +126,11 @@ const calibrationImageURL = (index: number) =>
style="display: none"
@change="importCalibration"
/>
</v-col>
<v-col cols="6" md="3" class="d-flex align-center pt-0 pt-md-3 pr-6 pr-md-3">
<v-btn
color="buttonPassive"
class="mr-2"
:disabled="!currentCalibrationCoeffs"
style="width: 100%"
style="flex: 1"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="openExportCalibrationPrompt"
>
@@ -132,6 +143,16 @@ const calibrationImageURL = (index: number) =>
:href="exportCalibrationURL"
target="_blank"
/>
<v-btn
color="error"
:disabled="!currentCalibrationCoeffs"
style="flex: 1"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="() => (confirmRemoveDialog = { show: true, vf: props.videoFormat })"
>
<v-icon start size="large">mdi-delete</v-icon>
<span>Delete</span>
</v-btn>
</v-col>
</div>
<v-card-title class="pt-0 pb-0"
@@ -289,6 +310,14 @@ const calibrationImageURL = (index: number) =>
</v-data-table>
</v-card-text>
</v-card>
<pv-delete-modal
v-model="confirmRemoveDialog.show"
:width="500"
:title="'Delete Calibration'"
:description="`Are you sure you want to delete the calibration for '${confirmRemoveDialog.vf.resolution.width}x${confirmRemoveDialog.vf.resolution.height}'? This action cannot be undone.`"
:on-confirm="() => removeCalibration(confirmRemoveDialog.vf)"
/>
</template>
<style scoped>

105
photon-client/src/components/cameras/CameraSettingsCard.vue
View File

@@ -1,13 +1,14 @@
<script setup lang="ts">
import PvSelect, { type SelectItem } from "@/components/common/pv-select.vue";
import PvInput from "@/components/common/pv-input.vue";
import PvDeleteModal from "@/components/common/pv-delete-modal.vue";
import PvNumberInput from "@/components/common/pv-number-input.vue";
import PvSwitch from "@/components/common/pv-switch.vue";
import { useCameraSettingsStore } from "@/stores/settings/CameraSettingsStore";
import { useStateStore } from "@/stores/StateStore";
import { computed, ref, watchEffect } from "vue";
import { type CameraSettingsChangeRequest, ValidQuirks } from "@/types/SettingTypes";
import axios from "axios";
import { useTheme } from "vuetify";
import { axiosPost } from "@/lib/PhotonUtils";
const theme = useTheme();
@@ -15,7 +16,14 @@ const tempSettingsStruct = ref<CameraSettingsChangeRequest>({
fov: useCameraSettingsStore().currentCameraSettings.fov.value,
quirksToChange: Object.assign({}, useCameraSettingsStore().currentCameraSettings.cameraQuirks.quirks)
});
const focusMode = computed<boolean>({
get: () => useCameraSettingsStore().isFocusMode,
set: (v) =>
useCameraSettingsStore().changeCurrentPipelineIndex(
v ? -3 : useCameraSettingsStore().currentCameraSettings.lastPipelineIndex || 0,
true
)
});
const arducamSelectWrapper = computed<number>({
get: () => {
if (tempSettingsStruct.value.quirksToChange.ArduOV9281Controls) return 1;
@@ -112,44 +120,10 @@ watchEffect(() => {
});
const showDeleteCamera = ref(false);
const yesDeleteMySettingsText = ref("");
const deletingCamera = ref(false);
const deleteThisCamera = () => {
if (deletingCamera.value) return;
deletingCamera.value = true;
const payload = { cameraUniqueName: useStateStore().currentCameraUniqueName };
axios
.post("/utils/nukeOneCamera", payload)
.then(() => {
useStateStore().showSnackbarMessage({
message: "Successfully dispatched the delete command. Waiting for backend to start back up",
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
message: "The backend is unable to fulfil the request to delete this camera.",
color: "error"
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
message: "Error while trying to process the request! The backend didn't respond.",
color: "error"
});
} else {
useStateStore().showSnackbarMessage({
message: "An error occurred while trying to process the request.",
color: "error"
});
}
})
.finally(() => {
deletingCamera.value = false;
showDeleteCamera.value = false;
});
axiosPost("/utils/nukeOneCamera", "delete this camera", {
cameraUniqueName: useStateStore().currentCameraUniqueName
});
};
const wrappedCameras = computed<SelectItem[]>(() =>
Object.keys(useCameraSettingsStore().cameras).map((cameraUniqueName) => ({
@@ -192,6 +166,11 @@ const wrappedCameras = computed<SelectItem[]>(() =>
]"
:select-cols="8"
/>
<pv-switch
v-model="focusMode"
tooltip="Enable Focus Mode to start focusing the lens on your camera"
label="Focus Mode"
></pv-switch>
</v-card-text>
<v-card-text class="d-flex pt-0">
<v-col cols="6" class="pa-0 pr-2">
@@ -221,45 +200,13 @@ const wrappedCameras = computed<SelectItem[]>(() =>
</v-col>
</v-card-text>
<v-dialog v-model="showDeleteCamera" width="800">
<v-card color="surface" flat>
<v-card-title> Delete {{ useCameraSettingsStore().currentCameraSettings.nickname }}? </v-card-title>
<v-card-text class="pt-0 pb-10px">
Are you sure you want to delete "{{ useCameraSettingsStore().currentCameraSettings.nickname }}"? This cannot
be undone.
</v-card-text>
<v-card-text class="pt-0 pb-10px">
<pv-input
v-model="yesDeleteMySettingsText"
:label="'Type &quot;' + useCameraSettingsStore().currentCameraName + '&quot;:'"
:label-cols="6"
:input-cols="6"
/>
</v-card-text>
<v-card-actions class="pa-5 pt-0">
<v-btn
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
color="primary"
class="text-black"
@click="showDeleteCamera = false"
>
Cancel
</v-btn>
<v-btn
color="error"
:disabled="
yesDeleteMySettingsText.toLowerCase() !== useCameraSettingsStore().currentCameraName.toLowerCase()
"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
:loading="deletingCamera"
@click="deleteThisCamera"
>
<v-icon start class="open-icon" size="large"> mdi-trash-can-outline </v-icon>
<span class="open-label">Delete</span>
</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
<pv-delete-modal
v-model="showDeleteCamera"
title="Delete Camera"
:description="`Are you sure you want to delete the camera '${useCameraSettingsStore().currentCameraSettings.nickname}'? This action cannot be undone.`"
:expected-confirmation-text="useCameraSettingsStore().currentCameraSettings.nickname"
:on-confirm="deleteThisCamera"
/>
</v-card>
</template>

21
photon-client/src/components/cameras/CamerasView.vue
View File

@@ -58,6 +58,15 @@ const fpsTooLow = computed<boolean>(() => {
<v-chip v-else label color="red" variant="text" style="font-size: 1rem; padding: 0; margin: 0">
<span class="pr-1">Camera not connected</span>
</v-chip>
<v-chip
v-if="useCameraSettingsStore().isFocusMode"
label
color="primary"
variant="text"
style="font-size: 1rem; padding: 0; margin: auto"
>
<span class="pr-1"> Focus: {{ Math.round(useStateStore().currentPipelineResults?.focus || 0) }} </span>
</v-chip>
<v-switch
v-model="driverMode"
:disabled="useCameraSettingsStore().isCalibrationMode || useCameraSettingsStore().pipelineNames.length === 0"
@@ -95,7 +104,11 @@ const fpsTooLow = computed<boolean>(() => {
color="buttonPassive"
class="fill"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
:disabled="useCameraSettingsStore().isDriverMode || useCameraSettingsStore().isCalibrationMode"
:disabled="
useCameraSettingsStore().isDriverMode ||
useCameraSettingsStore().isCalibrationMode ||
useCameraSettingsStore().isFocusMode
"
>
<v-icon start class="mode-btn-icon" size="large">mdi-import</v-icon>
<span class="mode-btn-label">Raw</span>
@@ -104,7 +117,11 @@ const fpsTooLow = computed<boolean>(() => {
color="buttonPassive"
class="fill"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
:disabled="useCameraSettingsStore().isDriverMode || useCameraSettingsStore().isCalibrationMode"
:disabled="
useCameraSettingsStore().isDriverMode ||
useCameraSettingsStore().isCalibrationMode ||
useCameraSettingsStore().isFocusMode
"
>
<v-icon start class="mode-btn-icon" size="large">mdi-export</v-icon>
<span class="mode-btn-label">Processed</span>

85
photon-client/src/components/common/pv-delete-modal.vue Normal file
View File

@@ -0,0 +1,85 @@
<script setup lang="ts">
import { ref } from "vue";
import { useTheme } from "vuetify";
import pvInput from "./pv-input.vue";
const theme = useTheme();
const value = defineModel<boolean | undefined>({ required: true });
const props = withDefaults(
defineProps<{
expectedConfirmationText?: string;
onBackup?: () => void;
onConfirm: () => void;
title: string;
description?: string;
deleteText?: string;
width?: number;
}>(),
{
width: 700
}
);
const confirmationText = ref("");
</script>
<template>
<v-dialog v-model="value" :width="props.width" dark>
<v-card color="surface" flat>
<v-card-title style="display: flex; justify-content: center">
{{ title }}
</v-card-title>
<v-card-text class="pt-0 pb-10px">
<span> {{ description }} </span>
</v-card-text>
<v-card-text v-if="expectedConfirmationText" class="pt-0 pb-0">
<pv-input
v-model="confirmationText"
:label="'Type &quot;' + expectedConfirmationText + '&quot;:'"
:label-cols="6"
:input-cols="6"
/>
</v-card-text>
<v-card-text class="pt-10px">
<v-row class="align-center text-white">
<v-col v-if="onBackup" cols="6">
<v-btn
color="buttonActive"
style="float: right"
width="100%"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="onBackup"
>
<v-icon start class="open-icon" size="large"> mdi-export </v-icon>
<span class="open-label">Backup Data</span>
</v-btn>
</v-col>
<v-col v-if="description" :cols="onBackup ? '6' : '12'">
<v-btn
color="error"
width="100%"
:disabled="
expectedConfirmationText
? confirmationText.toLowerCase() !== expectedConfirmationText.toLowerCase()
: false
"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="
onConfirm();
confirmationText = '';
value = false;
"
>
<v-icon start class="open-icon" size="large"> mdi-trash-can-outline </v-icon>
<span class="open-label">
{{ deleteText ?? title }}
</span>
</v-btn>
</v-col>
</v-row>
</v-card-text>
</v-card>
</v-dialog>
</template>

48
photon-client/src/components/dashboard/CameraAndPipelineSelectCard.vue
View File

@@ -9,6 +9,7 @@ import PvInput from "@/components/common/pv-input.vue";
import { PipelineType } from "@/types/PipelineTypes";
import { useSettingsStore } from "@/stores/settings/GeneralSettingsStore";
import { useTheme } from "vuetify";
import PvDeleteModal from "@/components/common/pv-delete-modal.vue";
const theme = useTheme();
@@ -92,6 +93,9 @@ const pipelineNamesWrapper = computed<SelectItem[]>(() => {
if (useCameraSettingsStore().isDriverMode) {
pipelineNames.push({ name: "Driver Mode", value: WebsocketPipelineType.DriverMode });
}
if (useCameraSettingsStore().isFocusMode) {
pipelineNames.push({ name: "Focus Mode", value: WebsocketPipelineType.FocusCamera });
}
if (useCameraSettingsStore().isCalibrationMode) {
pipelineNames.push({ name: "3D Calibration Mode", value: WebsocketPipelineType.Calib3d });
}
@@ -130,7 +134,7 @@ const validNewPipelineTypes = computed(() => {
{ name: "Reflective", value: WebsocketPipelineType.Reflective },
{ name: "Colored Shape", value: WebsocketPipelineType.ColoredShape },
{ name: "AprilTag", value: WebsocketPipelineType.AprilTag },
{ name: "Aruco", value: WebsocketPipelineType.Aruco }
{ name: "ArUco", value: WebsocketPipelineType.Aruco }
];
if (useSettingsStore().general.supportedBackends.length > 0) {
pipelineTypes.push({ name: "Object Detection", value: WebsocketPipelineType.ObjectDetection });
@@ -168,7 +172,7 @@ const pipelineTypesWrapper = computed<{ name: string; value: number }[]>(() => {
{ name: "Reflective", value: WebsocketPipelineType.Reflective },
{ name: "Colored Shape", value: WebsocketPipelineType.ColoredShape },
{ name: "AprilTag", value: WebsocketPipelineType.AprilTag },
{ name: "Aruco", value: WebsocketPipelineType.Aruco }
{ name: "ArUco", value: WebsocketPipelineType.Aruco }
];
if (useSettingsStore().general.supportedBackends.length > 0) {
pipelineTypes.push({ name: "Object Detection", value: WebsocketPipelineType.ObjectDetection });
@@ -177,6 +181,9 @@ const pipelineTypesWrapper = computed<{ name: string; value: number }[]>(() => {
if (useCameraSettingsStore().isDriverMode) {
pipelineTypes.push({ name: "Driver Mode", value: WebsocketPipelineType.DriverMode });
}
if (useCameraSettingsStore().isFocusMode) {
pipelineTypes.push({ name: "Focus Mode", value: WebsocketPipelineType.FocusCamera });
}
if (useCameraSettingsStore().isCalibrationMode) {
pipelineTypes.push({ name: "3D Calibration Mode", value: WebsocketPipelineType.Calib3d });
}
@@ -187,6 +194,7 @@ const pipelineType = ref<WebsocketPipelineType>(useCameraSettingsStore().current
const currentPipelineType = computed<WebsocketPipelineType>({
get: () => {
if (useCameraSettingsStore().isDriverMode) return WebsocketPipelineType.DriverMode;
if (useCameraSettingsStore().isFocusMode) return WebsocketPipelineType.FocusCamera;
if (useCameraSettingsStore().isCalibrationMode) return WebsocketPipelineType.Calib3d;
return pipelineType.value;
},
@@ -290,6 +298,7 @@ const wrappedCameras = computed<SelectItem[]>(() =>
tooltip="Each pipeline runs on a camera output and stores a unique set of processing settings"
:disabled="
useCameraSettingsStore().isDriverMode ||
useCameraSettingsStore().isFocusMode ||
useCameraSettingsStore().isCalibrationMode ||
!useCameraSettingsStore().hasConnected
"
@@ -366,6 +375,7 @@ const wrappedCameras = computed<SelectItem[]>(() =>
tooltip="Changes the pipeline type, which changes the type of processing that will happen on input frames"
:disabled="
useCameraSettingsStore().isDriverMode ||
useCameraSettingsStore().isFocusMode ||
useCameraSettingsStore().isCalibrationMode ||
!useCameraSettingsStore().hasConnected
"
@@ -413,33 +423,13 @@ const wrappedCameras = computed<SelectItem[]>(() =>
</v-card-actions>
</v-card>
</v-dialog>
<v-dialog v-model="showPipelineDeletionConfirmationDialog" width="500">
<v-card color="surface">
<v-card-title class="pb-0">Delete Pipeline</v-card-title>
<v-card-text>
Are you sure you want to delete
<span style="color: white">"{{ useCameraSettingsStore().currentPipelineSettings.pipelineNickname }}"</span>?
This cannot be undone.
</v-card-text>
<v-card-actions class="pa-5 pt-0">
<v-btn
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
color="primary"
class="text-black"
@click="showPipelineDeletionConfirmationDialog = false"
>
Cancel
</v-btn>
<v-btn
color="error"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="confirmDeleteCurrentPipeline"
>
Delete
</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
<pv-delete-modal
v-model="showPipelineDeletionConfirmationDialog"
:width="500"
title="Delete Pipeline"
description="Are you sure you want to delete the current pipeline? This action cannot be undone."
:on-confirm="confirmDeleteCurrentPipeline"
/>
<v-dialog v-model="showPipelineTypeChangeDialog" persistent width="600">
<v-card color="surface" dark>
<v-card-title class="pb-0">Change Pipeline Type</v-card-title>

6
photon-client/src/components/dashboard/ConfigOptions.vue
View File

@@ -29,7 +29,7 @@ const allTabs = Object.freeze({
thresholdTab: { tabName: "Threshold", component: ThresholdTab },
contoursTab: { tabName: "Contours", component: ContoursTab },
apriltagTab: { tabName: "AprilTag", component: AprilTagTab },
arucoTab: { tabName: "Aruco", component: ArucoTab },
arucoTab: { tabName: "ArUco", component: ArucoTab },
objectDetectionTab: { tabName: "Object Detection", component: ObjectDetectionTab },
outputTab: { tabName: "Output", component: OutputTab },
targetsTab: { tabName: "Targets", component: TargetsTab },
@@ -99,8 +99,8 @@ const tabGroups = computed<ConfigOption[][]>(() => {
!((isAprilTag || isAruco || isObjectDetection) && tabConfig.tabName === "Threshold") && //Filter out threshold tab if we're doing AprilTags
!((isAprilTag || isAruco || isObjectDetection) && tabConfig.tabName === "Contours") && //Filter out contours if we're doing AprilTags
!(!isAprilTag && tabConfig.tabName === "AprilTag") && //Filter out apriltag unless we actually are doing AprilTags
!(!isAruco && tabConfig.tabName === "Aruco") &&
!(!isObjectDetection && tabConfig.tabName === "Object Detection") //Filter out aruco unless we actually are doing Aruco
!(!isAruco && tabConfig.tabName === "ArUco") &&
!(!isObjectDetection && tabConfig.tabName === "Object Detection") //Filter out ArUco unless we actually are doing ArUco
)
)
.filter((it) => it.length); // Remove empty tab groups

4
photon-client/src/components/dashboard/StreamConfigCard.vue
View File

@@ -21,7 +21,9 @@ const processingMode = computed<number>({
<template>
<v-card
:disabled="useCameraSettingsStore().isDriverMode || useStateStore().colorPickingMode"
:disabled="
useCameraSettingsStore().isDriverMode || useCameraSettingsStore().isFocusMode || useStateStore().colorPickingMode
"
class="mt-3 rounded-12"
color="surface"
style="flex-grow: 1; display: flex; flex-direction: column"

223
photon-client/src/components/settings/DeviceControlCard.vue
View File

@@ -2,60 +2,17 @@
import { inject, ref } from "vue";
import { useStateStore } from "@/stores/StateStore";
import PvSelect from "@/components/common/pv-select.vue";
import PvInput from "@/components/common/pv-input.vue";
import axios from "axios";
import PvDeleteModal from "@/components/common/pv-delete-modal.vue";
import { useTheme } from "vuetify";
import { axiosPost } from "@/lib/PhotonUtils";
const theme = useTheme();
const restartProgram = () => {
axios
.post("/utils/restartProgram")
.then(() => {
useStateStore().showSnackbarMessage({ message: "Successfully sent program restart request", color: "success" });
})
.catch((error) => {
// This endpoint always return 204 regardless of outcome
if (error.request) {
useStateStore().showSnackbarMessage({
message: "Error while trying to process the request! The backend didn't respond.",
color: "error"
});
} else {
useStateStore().showSnackbarMessage({
message: "An error occurred while trying to process the request.",
color: "error"
});
}
});
axiosPost("/utils/restartProgram", "restart PhotonVision");
};
const restartDevice = () => {
axios
.post("/utils/restartDevice")
.then(() => {
useStateStore().showSnackbarMessage({
message: "Successfully dispatched the restart command. It isn't confirmed if a device restart will occur.",
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
message: "The backend is unable to fulfil the request to restart the device.",
color: "error"
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
message: "Error while trying to process the request! The backend didn't respond.",
color: "error"
});
} else {
useStateStore().showSnackbarMessage({
message: "An error occurred while trying to process the request.",
color: "error"
});
}
});
axiosPost("/utils/restartDevice", "restart the device");
};
const address = inject<string>("backendHost");
@@ -77,47 +34,27 @@ const handleOfflineUpdate = () => {
timeout: -1
});
axios
.post("/utils/offlineUpdate", formData, {
headers: { "Content-Type": "multipart/form-data" },
onUploadProgress: ({ progress }) => {
const uploadPercentage = (progress || 0) * 100.0;
if (uploadPercentage < 99.5) {
useStateStore().showSnackbarMessage({
message: "New Software Upload in Process, " + uploadPercentage.toFixed(2) + "% complete",
color: "secondary",
timeout: -1
});
} else {
useStateStore().showSnackbarMessage({
message: "Installing uploaded software...",
color: "secondary",
timeout: -1
});
}
}
})
.then((response) => {
useStateStore().showSnackbarMessage({ message: response.data.text || response.data, color: "success" });
})
.catch((error) => {
if (error.response) {
axiosPost("/utils/offlineUpdate", "upload new software", formData, {
headers: { "Content-Type": "multipart/form-data" },
onUploadProgress: ({ progress }) => {
const uploadPercentage = (progress || 0) * 100.0;
if (uploadPercentage < 99.5) {
useStateStore().showSnackbarMessage({
color: "error",
message: error.response.data.text || error.response.data
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
color: "error",
message: "Error while trying to process the request! The backend didn't respond."
message: "New Software Upload in Progress",
color: "secondary",
timeout: -1,
progressBar: uploadPercentage,
progressBarColor: "primary"
});
} else {
useStateStore().showSnackbarMessage({
color: "error",
message: "An error occurred while trying to process the request."
message: "Installing uploaded software...",
color: "secondary",
timeout: -1
});
}
});
}
});
};
const exportLogFile = ref();
@@ -166,29 +103,9 @@ const handleSettingsImport = () => {
break;
}
axios
.post(`/settings${settingsEndpoint}`, formData, { headers: { "Content-Type": "multipart/form-data" } })
.then((response) => {
useStateStore().showSnackbarMessage({ message: response.data.text || response.data, color: "success" });
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
color: "error",
message: error.response.data.text || error.response.data
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
color: "error",
message: "Error while trying to process the request! The backend didn't respond."
});
} else {
useStateStore().showSnackbarMessage({
color: "error",
message: "An error occurred while trying to process the request."
});
}
});
axiosPost(`/settings${settingsEndpoint}`, "import settings", formData, {
headers: { "Content-Type": "multipart/form-data" }
});
showImportDialog.value = false;
importType.value = undefined;
@@ -196,36 +113,8 @@ const handleSettingsImport = () => {
};
const showFactoryReset = ref(false);
const expected = "Delete Everything";
const yesDeleteMySettingsText = ref("");
const nukePhotonConfigDirectory = () => {
axios
.post("/utils/nukeConfigDirectory")
.then(() => {
useStateStore().showSnackbarMessage({
message: "Successfully dispatched the reset command. Waiting for backend to start back up",
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
message: "The backend is unable to fulfill the request to reset the device.",
color: "error"
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
message: "Error while trying to process the request! The backend didn't respond.",
color: "error"
});
} else {
useStateStore().showSnackbarMessage({
message: "An error occurred while trying to process the request.",
color: "error"
});
}
});
showFactoryReset.value = false;
axiosPost("/utils/nukeConfigDirectory", "delete the config directory");
};
</script>
@@ -387,63 +276,15 @@ const nukePhotonConfigDirectory = () => {
</v-col>
</v-row>
</div>
<v-dialog v-model="showFactoryReset" width="800" dark>
<v-card color="surface" flat>
<v-card-title style="display: flex; justify-content: center">
<span class="open-label">
<v-icon end color="red" class="open-icon ma-1" size="large">mdi-alert-outline</v-icon>
Factory Reset PhotonVision
<v-icon end color="red" class="open-icon ma-1" size="large">mdi-alert-outline</v-icon>
</span>
</v-card-title>
<v-card-text class="pt-0 pb-10px">
<v-row class="align-center text-white">
<v-col cols="12" md="6">
<span> This will delete ALL OF YOUR SETTINGS and restart PhotonVision. </span>
</v-col>
<v-col cols="12" md="6">
<v-btn
color="primary"
style="float: right"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="openExportSettingsPrompt"
>
<v-icon start class="open-icon" size="large"> mdi-export </v-icon>
<span class="open-label">Backup Settings</span>
<a
ref="exportSettings"
style="color: black; text-decoration: none; display: none"
:href="`http://${address}/api/settings/photonvision_config.zip`"
download="photonvision-settings.zip"
target="_blank"
/>
</v-btn>
</v-col>
</v-row>
</v-card-text>
<v-card-text class="pt-0 pb-0">
<pv-input
v-model="yesDeleteMySettingsText"
:label="'Type &quot;' + expected + '&quot;:'"
:label-cols="6"
:input-cols="6"
/>
</v-card-text>
<v-card-text class="pt-10px">
<v-btn
color="error"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
:disabled="yesDeleteMySettingsText.toLowerCase() !== expected.toLowerCase()"
@click="nukePhotonConfigDirectory"
>
<v-icon start class="open-icon" size="large"> mdi-trash-can-outline </v-icon>
<span class="open-label">
{{ $vuetify.display.mdAndUp ? "Delete everything, I have backed up what I need" : "Delete Everything" }}
</span>
</v-btn>
</v-card-text>
</v-card>
</v-dialog>
<pv-delete-modal
v-model="showFactoryReset"
title="Factory Reset PhotonVision"
description="This will delete all settings and configurations stored on this device, including network settings. This action cannot be undone."
expected-confirmation-text="Delete Everything"
:on-confirm="nukePhotonConfigDirectory"
:on-backup="openExportSettingsPrompt"
delete-text="Factory reset"
/>
</v-card>
</template>

16
photon-client/src/components/settings/NetworkingCard.vue
View File

@@ -315,23 +315,23 @@ watchEffect(() => {
<v-col class="text-center">
Background
<v-color-picker
v-model:model-value="backgroundColor"
class="ma-auto pt-3"
elevation="0"
mode="hex"
:modes="['hex']"
v-model:model-value="backgroundColor"
v-on:update:model-value="(hex) => setThemeColor(theme, 'background', hex)"
@update:model-value="(hex) => setThemeColor(theme, 'background', hex)"
></v-color-picker>
</v-col>
<v-col class="text-center">
Surface
<v-color-picker
v-model:model-value="surfaceColor"
class="ma-auto pt-3"
elevation="0"
mode="hex"
:modes="['hex']"
v-model:model-value="surfaceColor"
v-on:update:model-value="(hex) => setThemeColor(theme, 'surface', hex)"
@update:model-value="(hex) => setThemeColor(theme, 'surface', hex)"
></v-color-picker>
</v-col>
</v-row>
@@ -339,23 +339,23 @@ watchEffect(() => {
<v-col class="text-center">
Primary
<v-color-picker
v-model:model-value="primaryColor"
class="ma-auto pt-3"
elevation="0"
mode="hex"
:modes="['hex']"
v-model:model-value="primaryColor"
v-on:update:model-value="(hex) => setThemeColor(theme, 'primary', hex)"
@update:model-value="(hex) => setThemeColor(theme, 'primary', hex)"
></v-color-picker>
</v-col>
<v-col class="text-center">
Secondary
<v-color-picker
v-model:model-value="secondaryColor"
class="ma-auto pt-3"
elevation="0"
mode="hex"
:modes="['hex']"
v-model:model-value="secondaryColor"
v-on:update:model-value="(hex) => setThemeColor(theme, 'secondary', hex)"
@update:model-value="(hex) => setThemeColor(theme, 'secondary', hex)"
></v-color-picker>
</v-col>
</v-row>

296
photon-client/src/components/settings/ObjectDetectionCard.vue
View File

@@ -1,11 +1,11 @@
<script setup lang="ts">
import { ref, computed, inject } from "vue";
import axios from "axios";
import { useStateStore } from "@/stores/StateStore";
import { useSettingsStore } from "@/stores/settings/GeneralSettingsStore";
import type { ObjectDetectionModelProperties } from "@/types/SettingTypes";
import pvInput from "@/components/common/pv-input.vue";
import { type ObjectDetectionModelProperties } from "@/types/SettingTypes";
import PvDeleteModal from "@/components/common/pv-delete-modal.vue";
import { useTheme } from "vuetify";
import { axiosPost } from "@/lib/PhotonUtils";
const theme = useTheme();
const showImportDialog = ref(false);
@@ -43,34 +43,25 @@ const handleImport = async () => {
timeout: -1
});
axios
.post("/objectdetection/import", formData, {
headers: { "Content-Type": "multipart/form-data" }
})
.then((response) => {
useStateStore().showSnackbarMessage({
message: response.data.text || response.data,
color: "success"
});
})
.catch((error) => {
if (error.response) {
axiosPost("/objectdetection/import", "import an object detection model", formData, {
headers: { "Content-Type": "multipart/form-data" },
onUploadProgress: ({ progress }) => {
const uploadPercentage = (progress || 0) * 100.0;
if (uploadPercentage < 99.5) {
useStateStore().showSnackbarMessage({
color: "error",
message: error.response.data.text || error.response.data
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
color: "error",
message: "Error while trying to process the request! The backend didn't respond."
message: "Object Detection Model Upload in Process, " + uploadPercentage.toFixed(2) + "% complete",
color: "secondary",
timeout: -1
});
} else {
useStateStore().showSnackbarMessage({
color: "error",
message: "An error occurred while trying to process the request."
message: "Processing uploaded Object Detection Model...",
color: "secondary",
timeout: -1
});
}
});
}
});
showImportDialog.value = false;
@@ -82,41 +73,9 @@ const handleImport = async () => {
};
const deleteModel = async (model: ObjectDetectionModelProperties) => {
useStateStore().showSnackbarMessage({
message: "Deleting Object Detection Model...",
color: "secondary",
timeout: -1
axiosPost("/objectdetection/delete", "delete an object detection model", {
modelPath: model.modelPath
});
axios
.post("/objectdetection/delete", {
modelPath: model.modelPath
})
.then((response) => {
useStateStore().showSnackbarMessage({
message: response.data.text || response.data,
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
color: "error",
message: error.response.data.text || error.response.data
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
color: "error",
message: "Error while trying to process the request! The backend didn't respond."
});
} else {
useStateStore().showSnackbarMessage({
color: "error",
message: "An error occurred while trying to process the request."
});
}
});
confirmDeleteDialog.value.show = false;
};
const renameModel = async (model: ObjectDetectionModelProperties, newName: string) => {
@@ -126,35 +85,10 @@ const renameModel = async (model: ObjectDetectionModelProperties, newName: strin
timeout: -1
});
axios
.post("/objectdetection/rename", {
modelPath: model.modelPath.replace("file:", ""),
newName: newName
})
.then((response) => {
useStateStore().showSnackbarMessage({
message: response.data.text || response.data,
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
color: "error",
message: error.response.data.text || error.response.data
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
color: "error",
message: "Error while trying to process the request! The backend didn't respond."
});
} else {
useStateStore().showSnackbarMessage({
color: "error",
message: "An error occurred while trying to process the request."
});
}
});
axiosPost("/objectdetection/rename", "rename an object detection model", {
modelPath: model.modelPath.replace("file:", ""),
newName: newName
});
showRenameDialog.value.show = false;
};
@@ -181,36 +115,8 @@ const openExportIndividualModelPrompt = () => {
};
const showNukeDialog = ref(false);
const expected = "Delete Models";
const yesDeleteMyModelsText = ref("");
const nukeModels = () => {
axios
.post("/objectdetection/nuke")
.then(() => {
useStateStore().showSnackbarMessage({
message: "Successfully dispatched the clear models command.",
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
message: "The backend is unable to fulfill the request to clear the models.",
color: "error"
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
message: "Error while trying to process the request! The backend didn't respond.",
color: "error"
});
} else {
useStateStore().showSnackbarMessage({
message: "An error occurred while trying to process the request.",
color: "error"
});
}
});
showNukeDialog.value = false;
axiosPost("/objectdetection/nuke", "clear and reset object detection models");
};
const showBulkImportDialog = ref(false);
@@ -221,51 +127,27 @@ const handleBulkImport = () => {
const formData = new FormData();
formData.append("data", importFile.value);
axios
.post("/objectdetection/bulkimport", formData, {
headers: { "Content-Type": "multipart/form-data" },
onUploadProgress: ({ progress }) => {
const uploadPercentage = (progress || 0) * 100.0;
if (uploadPercentage < 99.5) {
useStateStore().showSnackbarMessage({
message: "Object Detection Models Upload in Process, " + uploadPercentage.toFixed(2) + "% complete",
color: "secondary",
timeout: -1
});
} else {
useStateStore().showSnackbarMessage({
message: "Importing New Object Detection Models...",
color: "secondary",
timeout: -1
});
}
}
})
.then((response) => {
useStateStore().showSnackbarMessage({
message: response.data.text || response.data,
color: "success"
});
})
.catch((error) => {
if (error.response) {
axiosPost("/objectdetection/bulkimport", "import object detection models", formData, {
headers: { "Content-Type": "multipart/form-data" },
onUploadProgress: ({ progress }) => {
const uploadPercentage = (progress || 0) * 100.0;
if (uploadPercentage < 99.5) {
useStateStore().showSnackbarMessage({
color: "error",
message: error.response.data.text || error.response.data
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
color: "error",
message: "Error while trying to process the request! The backend didn't respond."
message: "Object Detection Models Upload in Progress",
color: "secondary",
timeout: -1,
progressBar: uploadPercentage,
progressBarColor: "primary"
});
} else {
useStateStore().showSnackbarMessage({
color: "error",
message: "An error occurred while trying to process the request."
message: "Importing New Object Detection Models...",
color: "secondary",
timeout: -1
});
}
});
}
});
showImportDialog.value = false;
importFile.value = null;
};
@@ -485,35 +367,20 @@ const handleBulkImport = () => {
</tbody>
</v-table>
<v-dialog v-model="confirmDeleteDialog.show" width="600">
<v-card color="surface" dark>
<v-card-title>Delete Object Detection Model</v-card-title>
<v-card-text class="pt-0">
Are you sure you want to delete the model {{ confirmDeleteDialog.model.nickname }}?
<v-card-actions class="pt-5 pb-0 pr-0" style="justify-content: flex-end">
<v-btn
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
color="buttonPassive"
@click="confirmDeleteDialog.show = false"
>
Cancel
</v-btn>
<v-btn
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
color="error"
@click="deleteModel(confirmDeleteDialog.model)"
>
Delete
</v-btn>
</v-card-actions>
</v-card-text>
</v-card>
</v-dialog>
<pv-delete-modal
v-model="confirmDeleteDialog.show"
:width="500"
:on-confirm="() => deleteModel(confirmDeleteDialog.model)"
title="Delete Object Detection Model"
:description="`Are you sure you want to delete the model ${confirmDeleteDialog.model.nickname}?`"
delete-text="Delete model"
/>
<v-dialog v-model="showRenameDialog.show" width="600">
<v-card color="surface" dark>
<v-card-title>Rename Object Detection Model</v-card-title>
<v-card-text class="pt-0">
Enter a new name for the model {{ showRenameDialog.model.nickname }}:
Enter a new name for the model "{{ showRenameDialog.model.nickname }}":
<div class="pa-5 pb-0">
<v-text-field v-model="showRenameDialog.newName" hide-details label="New Name" variant="underlined" />
</div>
@@ -569,64 +436,15 @@ const handleBulkImport = () => {
</v-row>
</div>
<v-dialog v-model="showNukeDialog" width="800" dark>
<v-card color="surface" flat>
<v-card-title style="display: flex; justify-content: center">
<span class="open-label">
<v-icon end color="error" class="open-icon ma-1" size="large">mdi-alert-outline</v-icon>
Clear and Reset Object Detection Models
<v-icon end color="error" class="open-icon ma-1" size="large">mdi-alert-outline</v-icon>
</span>
</v-card-title>
<v-card-text class="pt-0 pb-10px">
<v-row class="align-center text-white">
<v-col cols="12" md="6">
<span> This will delete ALL OF YOUR MODELS and re-extract the default models. </span>
</v-col>
<v-col cols="12" md="6">
<v-btn
color="buttonActive"
style="float: right"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="openExportPrompt"
>
<v-icon start class="open-icon" size="large"> mdi-export </v-icon>
<span class="open-label">Backup Models</span>
<a
ref="exportModels"
style="color: black; text-decoration: none; display: none"
:href="`http://${address}/api/objectdetection/export`"
download="photonvision-object-detection-models-export.zip"
target="_blank"
/>
</v-btn>
</v-col>
</v-row>
</v-card-text>
<v-card-text class="pt-0 pb-0">
<pv-input
v-model="yesDeleteMyModelsText"
:label="'Type &quot;' + expected + '&quot;:'"
:label-cols="6"
:input-cols="6"
/>
</v-card-text>
<v-card-text class="pt-10px">
<v-btn
color="error"
width="100%"
:disabled="yesDeleteMyModelsText.toLowerCase() !== expected.toLowerCase()"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="nukeModels"
>
<v-icon start class="open-icon" size="large"> mdi-trash-can-outline </v-icon>
<span class="open-label">
{{ $vuetify.display.mdAndUp ? "Delete models, I have backed up what I need" : "Delete Models" }}
</span>
</v-btn>
</v-card-text>
</v-card>
</v-dialog>
<pv-delete-modal
v-model="showNukeDialog"
:on-backup="openExportPrompt"
:on-confirm="nukeModels"
title="Delete and Reset All Object Detection Models"
:description="'This will delete ALL object detection models and re-extract the default object detection models. This action cannot be undone.'"
:expected-confirmation-text="'Delete Models'"
delete-text="Delete all models"
/>
</v-card>
</template>

40
photon-client/src/lib/PhotonUtils.ts
View File

@@ -1,4 +1,6 @@
import { useStateStore } from "@/stores/StateStore";
import type { Resolution } from "@/types/SettingTypes";
import axios from "axios";
export const resolutionsAreEqual = (a: Resolution, b: Resolution) => {
return a.height === b.height && a.width === b.width;
@@ -18,3 +20,41 @@ export const parseJsonFile = async <T extends Record<string, any>>(file: File):
fileReader.readAsText(file);
});
};
/**
* A helper function to make POST requests using axios with standardized success and error handling.
*
* @param url The endpoint URL to which the POST request is sent
* @param description A brief description of the request for users, e.g., "import object detection models".
* @param data Payload to be sent in the POST request
* @param config Optional axios request configuration
* @returns A promise that resolves when the POST request is complete
*/
export const axiosPost = (url: string, description: string, data?: any, config?: any): Promise<void> => {
return axios
.post(url, data, config)
.then(() => {
useStateStore().showSnackbarMessage({
message: "Successfully dispatched the request to " + description + ". Waiting for backend to respond",
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
message: "The backend is unable to fulfill the request to " + description + ".",
color: "error"
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
message: "Error while trying to process the request to " + description + "! The backend didn't respond.",
color: "error"
});
} else {
useStateStore().showSnackbarMessage({
message: "An error occurred while trying to process the request to " + description + ".",
color: "error"
});
}
});
};

14
photon-client/src/stores/StateStore.ts
View File

@@ -39,6 +39,8 @@ interface StateStore {
snackbarData: {
show: boolean;
progressBar: number;
progressBarColor: string;
message: string;
color: string;
timeout: number;
@@ -86,6 +88,8 @@ export const useStateStore = defineStore("state", {
snackbarData: {
show: false,
progressBar: -1,
progressBarColor: "info",
message: "No Message",
color: "info",
timeout: 2000
@@ -158,11 +162,19 @@ export const useStateStore = defineStore("state", {
updateDiscoveredCameras(data: VsmState) {
this.vsmState = data;
},
showSnackbarMessage(data: { message: string; color: string; timeout?: number }) {
showSnackbarMessage(data: {
message: string;
color: string;
timeout?: number;
progressBar?: number;
progressBarColor?: string;
}) {
this.snackbarData = {
show: true,
progressBar: data.progressBar || -1,
message: data.message,
color: data.color,
progressBarColor: data.progressBarColor || "",
timeout: data.timeout || 2000
};
}

5
photon-client/src/stores/settings/CameraSettingsStore.ts
View File

@@ -45,7 +45,7 @@ export const useCameraSettingsStore = defineStore("cameraSettings", {
},
// This method only exists due to just how lazy I am and my dislike of consolidating the pipeline type enums (which mind you, suck as is)
currentWebsocketPipelineType(): WebsocketPipelineType {
return this.currentPipelineType - 2;
return this.currentPipelineType - 3;
},
currentVideoFormat(): VideoFormat {
return this.currentCameraSettings.validVideoFormats[this.currentPipelineSettings.cameraVideoModeIndex];
@@ -76,6 +76,9 @@ export const useCameraSettingsStore = defineStore("cameraSettings", {
isCalibrationMode(): boolean {
return this.currentCameraSettings.currentPipelineIndex == WebsocketPipelineType.Calib3d;
},
isFocusMode(): boolean {
return this.currentCameraSettings.currentPipelineIndex == WebsocketPipelineType.FocusCamera;
},
isCSICamera(): boolean {
return this.currentCameraSettings.isCSICamera;
},

2
photon-client/src/types/PhotonTrackingTypes.ts
View File

@@ -70,6 +70,8 @@ export interface PipelineResult {
sequenceID: number;
fps: number;
latency: number;
// Focus pipeline
focus?: number;
targets: PhotonTarget[];
// undefined if multitag failed or non-tag pipeline
multitagResult?: MultitagResult;

15
photon-client/src/types/PipelineTypes.ts
View File

@@ -1,13 +1,16 @@
import type { WebsocketNumberPair } from "@/types/WebsocketDataTypes";
import type { ObjectDetectionModelProperties } from "@/types/SettingTypes";
/**
* The on-wire form of PipelineType.java (the enum is serialized with `ordinal()`)
*/
export enum PipelineType {
DriverMode = 1,
Reflective = 2,
ColoredShape = 3,
AprilTag = 4,
Aruco = 5,
ObjectDetection = 6
DriverMode = 2,
Reflective = 3,
ColoredShape = 4,
AprilTag = 5,
Aruco = 6,
ObjectDetection = 7
}
export enum AprilTagFamily {

1
photon-client/src/types/WebsocketDataTypes.ts
View File

@@ -110,6 +110,7 @@ export interface IncomingWebsocketData {
}
export enum WebsocketPipelineType {
FocusCamera = -3,
Calib3d = -2,
DriverMode = -1,
Reflective = 0,

217
photon-client/src/views/CameraMatchingView.vue
View File

@@ -7,16 +7,13 @@ import {
PVCameraInfo,
type PVCSICameraInfo,
type PVFileCameraInfo,
type PVUsbCameraInfo,
type UiCameraConfiguration
type PVUsbCameraInfo
} from "@/types/SettingTypes";
import { getResolutionString } from "@/lib/PhotonUtils";
import { axiosPost, getResolutionString } from "@/lib/PhotonUtils";
import PhotonCameraStream from "@/components/app/photon-camera-stream.vue";
import PvInput from "@/components/common/pv-input.vue";
import PvDeleteModal from "@/components/common/pv-delete-modal.vue";
import PvCameraInfoCard from "@/components/common/pv-camera-info-card.vue";
import axios from "axios";
import PvCameraMatchCard from "@/components/common/pv-camera-match-card.vue";
import type { WebsocketCameraSettingsUpdate } from "@/types/WebsocketDataTypes";
import { useTheme } from "vuetify";
const theme = useTheme();
@@ -28,33 +25,9 @@ const activateModule = (moduleUniqueName: string) => {
if (activatingModule.value) return;
activatingModule.value = true;
axios
.post("/utils/activateMatchedCamera", { cameraUniqueName: moduleUniqueName })
.then(() => {
useStateStore().showSnackbarMessage({
message: "Camera activated successfully",
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
message: "The backend is unable to fulfil the request to activate this camera.",
color: "error"
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
message: "Error while trying to process the request! The backend didn't respond.",
color: "error"
});
} else {
useStateStore().showSnackbarMessage({
message: "An error occurred while trying to process the request.",
color: "error"
});
}
})
.finally(() => (activatingModule.value = false));
axiosPost("/utils/activateMatchedCamera", "activate a matched camera", {
cameraUniqueName: moduleUniqueName
}).finally(() => (activatingModule.value = false));
};
const assigningCamera = ref(false);
@@ -66,106 +39,29 @@ const assignCamera = (cameraInfo: PVCameraInfo) => {
cameraInfo: cameraInfo
};
axios
.post("/utils/assignUnmatchedCamera", payload)
.then(() => {
useStateStore().showSnackbarMessage({
message: "Unmatched camera assigned successfully",
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
message: "The backend is unable to fulfil the request to assign this unmatched camera.",
color: "error"
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
message: "Error while trying to process the request! The backend didn't respond.",
color: "error"
});
} else {
useStateStore().showSnackbarMessage({
message: "An error occurred while trying to process the request.",
color: "error"
});
}
})
.finally(() => (assigningCamera.value = false));
axiosPost("/utils/assignUnmatchedCamera", "assign an unmatched camera", payload).finally(
() => (assigningCamera.value = false)
);
};
const deactivatingModule = ref(false);
const deactivateModule = (cameraUniqueName: string) => {
if (deactivatingModule.value) return;
deactivatingModule.value = true;
axios
.post("/utils/unassignCamera", { cameraUniqueName: cameraUniqueName })
.then(() => {
useStateStore().showSnackbarMessage({
message: "Camera deactivated successfully",
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
message: "The backend is unable to fulfil the request to deactivate this camera.",
color: "error"
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
message: "Error while trying to process the request! The backend didn't respond.",
color: "error"
});
} else {
useStateStore().showSnackbarMessage({
message: "An error occurred while trying to process the request.",
color: "error"
});
}
})
.finally(() => (deactivatingModule.value = false));
axiosPost("/utils/unassignCamera", "unassign a camera", { cameraUniqueName: cameraUniqueName }).finally(
() => (deactivatingModule.value = false)
);
};
const deletingCamera = ref(false);
const deleteThisCamera = (cameraName: string) => {
if (deletingCamera.value) return;
deletingCamera.value = true;
const payload = {
cameraUniqueName: cameraName
};
const confirmDeleteDialog = ref({ show: false, nickname: "", cameraUniqueName: "" });
const deletingCamera = ref<string | null>(null);
axios
.post("/utils/nukeOneCamera", payload)
.then(() => {
useStateStore().showSnackbarMessage({
message: "Camera deleted successfully",
color: "success"
});
})
.catch((error) => {
if (error.response) {
useStateStore().showSnackbarMessage({
message: "The backend is unable to fulfil the request to delete this camera.",
color: "error"
});
} else if (error.request) {
useStateStore().showSnackbarMessage({
message: "Error while trying to process the request! The backend didn't respond.",
color: "error"
});
} else {
useStateStore().showSnackbarMessage({
message: "An error occurred while trying to process the request.",
color: "error"
});
}
})
.finally(() => {
setCameraDeleting(null);
deletingCamera.value = false;
});
const deleteThisCamera = (cameraUniqueName: string) => {
if (deletingCamera.value) return;
deletingCamera.value = cameraUniqueName;
axiosPost("/utils/nukeOneCamera", "delete a camera", { cameraUniqueName: cameraUniqueName }).finally(() => {
deletingCamera.value = null;
});
};
const cameraConnected = (uniquePath: string): boolean => {
@@ -209,15 +105,6 @@ const setCameraView = (camera: PVCameraInfo | null, isConnected: boolean | null)
viewingCamera.value = [camera, isConnected];
};
const viewingDeleteCamera = ref(false);
const cameraToDelete = ref<UiCameraConfiguration | WebsocketCameraSettingsUpdate | null>(null);
const setCameraDeleting = (camera: UiCameraConfiguration | WebsocketCameraSettingsUpdate | null) => {
yesDeleteMySettingsText.value = "";
viewingDeleteCamera.value = camera !== null;
cameraToDelete.value = camera;
};
const yesDeleteMySettingsText = ref("");
/**
* Get the connection-type-specific camera info from the given PVCameraInfo object.
*/
@@ -373,8 +260,16 @@ const getMatchedDevice = (info: PVCameraInfo | undefined): PVCameraInfo => {
class="pa-0"
color="error"
style="width: 100%"
:loading="module.uniqueName === deletingCamera"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="setCameraDeleting(module)"
@click="
() =>
(confirmDeleteDialog = {
show: true,
nickname: module.nickname,
cameraUniqueName: module.uniqueName
})
"
>
<v-icon size="x-large">mdi-trash-can-outline</v-icon>
</v-btn>
@@ -459,8 +354,16 @@ const getMatchedDevice = (info: PVCameraInfo | undefined): PVCameraInfo => {
class="pa-0"
color="error"
style="width: 100%"
:loading="module.uniqueName === deletingCamera"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="setCameraDeleting(module)"
@click="
() =>
(confirmDeleteDialog = {
show: true,
nickname: module.nickname,
cameraUniqueName: module.uniqueName
})
"
>
<v-icon size="x-large">mdi-trash-can-outline</v-icon>
</v-btn>
@@ -564,43 +467,13 @@ const getMatchedDevice = (info: PVCameraInfo | undefined): PVCameraInfo => {
</v-card>
</v-dialog>
<!-- Camera delete modal -->
<v-dialog v-model="viewingDeleteCamera" width="800">
<v-card v-if="cameraToDelete !== null" class="dialog-container" color="surface" flat>
<v-card-title> Delete {{ cameraToDelete.nickname }}? </v-card-title>
<v-card-text class="pb-10px">
Are you sure you want to delete "{{ cameraToDelete.nickname }}"? This cannot be undone.
</v-card-text>
<v-card-text class="pt-0 pb-10px">
<pv-input
v-model="yesDeleteMySettingsText"
:label="'Type &quot;' + cameraToDelete.nickname + '&quot;:'"
:label-cols="6"
:input-cols="6"
/>
</v-card-text>
<v-card-actions class="pa-5 pt-0">
<v-btn
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
color="primary"
class="text-black"
@click="cameraToDelete = null"
>
Cancel
</v-btn>
<v-btn
color="error"
:disabled="yesDeleteMySettingsText.toLowerCase() !== cameraToDelete.nickname.toLowerCase()"
:loading="deletingCamera"
:variant="theme.global.name.value === 'LightTheme' ? 'elevated' : 'outlined'"
@click="deleteThisCamera(cameraToDelete.uniqueName)"
>
<v-icon start class="open-icon" size="large"> mdi-trash-can-outline </v-icon>
<span class="open-label">Delete</span>
</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
<pv-delete-modal
v-model="confirmDeleteDialog.show"
title="Delete Camera"
:description="`Are you sure you want to delete the camera '${useCameraSettingsStore().currentCameraSettings.nickname}'? This action cannot be undone.`"
:expected-confirmation-text="confirmDeleteDialog.nickname"
:on-confirm="() => deleteThisCamera(confirmDeleteDialog.cameraUniqueName)"
/>
</div>
</template>

9
photon-client/src/views/CameraSettingsView.vue
View File

@@ -12,8 +12,13 @@ const cameraViewType = computed<number[]>({
// Only show the input stream in Color Picking Mode
if (useStateStore().colorPickingMode) return [0];
// Only show the output stream in Driver Mode or Calibration Mode
if (useCameraSettingsStore().isDriverMode || useCameraSettingsStore().isCalibrationMode) return [1];
// Only show the output stream in Driver Mode or Calibration Mode or Focus Mode
if (
useCameraSettingsStore().isDriverMode ||
useCameraSettingsStore().isCalibrationMode ||
useCameraSettingsStore().isFocusMode
)
return [1];
const ret: number[] = [];
if (useCameraSettingsStore().currentPipelineSettings.inputShouldShow) {

9
photon-client/src/views/DashboardView.vue
View File

@@ -17,8 +17,13 @@ const cameraViewType = computed<number[]>({
// Only show the input stream in Color Picking Mode
if (useStateStore().colorPickingMode) return [0];
// Only show the output stream in Driver Mode or Calibration Mode
if (useCameraSettingsStore().isDriverMode || useCameraSettingsStore().isCalibrationMode) return [1];
// Only show the output stream in Driver Mode or Calibration Mode or Focus Mode
if (
useCameraSettingsStore().isDriverMode ||
useCameraSettingsStore().isCalibrationMode ||
useCameraSettingsStore().isFocusMode
)
return [1];
const ret: number[] = [];
if (useCameraSettingsStore().currentPipelineSettings.inputShouldShow) {

18
photon-core/src/main/java/org/photonvision/common/configuration/CameraConfiguration.java
View File

@@ -24,6 +24,7 @@ import edu.wpi.first.cscore.UsbCameraInfo;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
import org.opencv.core.Size;
import org.photonvision.common.dataflow.websocket.UICameraConfiguration;
import org.photonvision.common.logging.LogGroup;
import org.photonvision.common.logging.Logger;
@@ -189,6 +190,23 @@ public class CameraConfiguration {
calibrations.add(calibration);
}
/**
* Remove a calibration from our list.
*
* @param calibration The calibration to remove
*/
public void removeCalibration(Size unrotatedImageSize) {
logger.info("deleting calibration " + unrotatedImageSize);
calibrations.stream()
.filter(it -> it.unrotatedImageSize.equals(unrotatedImageSize))
.findAny()
.ifPresent(
(it) -> {
it.release();
calibrations.remove(it);
});
}
/**
* cscore will auto-reconnect to the camera path we give it. v4l does not guarantee that if i swap
* cameras around, the same /dev/videoN ID will be assigned to that camera. So instead default to

5
photon-core/src/main/java/org/photonvision/common/dataflow/websocket/UIDataPublisher.java
View File

@@ -27,6 +27,7 @@ import org.photonvision.common.logging.Logger;
import org.photonvision.common.util.SerializationUtils;
import org.photonvision.vision.pipeline.result.CVPipelineResult;
import org.photonvision.vision.pipeline.result.CalibrationPipelineResult;
import org.photonvision.vision.pipeline.result.FocusPipelineResult;
public class UIDataPublisher implements CVPipelineResultConsumer {
private static final Logger logger = new Logger(UIDataPublisher.class, LogGroup.VisionModule);
@@ -77,6 +78,10 @@ public class UIDataPublisher implements CVPipelineResultConsumer {
var uiMap = new HashMap<String, HashMap<String, Object>>();
uiMap.put(uniqueName, dataMap);
if (result instanceof FocusPipelineResult focusResult) {
dataMap.put("focus", focusResult.focus);
}
DataChangeService.getInstance()
.publishEvent(OutgoingUIEvent.wrappedOf("updatePipelineResult", uiMap));
lastUIResultUpdateTime = now;

2
photon-core/src/main/java/org/photonvision/vision/aruco/PhotonArucoDetector.java
View File

@@ -90,7 +90,7 @@ public class PhotonArucoDetector implements Releasable {
// each detection has a Mat of corners
Mat cornerMat = cornerMats.get(i);
// Aruco detection returns corners (BR, BL, TL, TR).
// ArUco detection returns corners (BR, BL, TL, TR).
// For parity with AprilTags and photonlib, we want (BL, BR, TR, TL).
double[] xCorners = {
cornerMat.get(0, 1)[0],

6
photon-core/src/main/java/org/photonvision/vision/pipe/impl/FindBoardCornersPipe.java
View File

@@ -280,14 +280,14 @@ public class FindBoardCornersPipe
}
board.matchImagePoints(detectedCornersList, detectedIds, objPoints, imgPoints);
// draw the charuco board
// Draw the ChArUco board
Objdetect.drawDetectedCornersCharuco(
outFrame, detectedCorners, detectedIds, new Scalar(0, 0, 255)); // Red Text
imgPoints.copyTo(outBoardCorners);
objPoints.copyTo(objPts);
// Since charuco can still detect without the whole board we need to send "fake" (all
// Since ChaArUco can still detect without the whole board we need to send "fake" (all
// values less than zero) points and then tell it to ignore that corner by setting the
// corresponding level to -1. Calibrate3dPipe deals with piping this into the correct format
// for each backend
@@ -321,7 +321,7 @@ public class FindBoardCornersPipe
detectedCorners.release();
detectedIds.release();
} else { // If not Charuco then do chessboard
} else { // If not ChArUco then do chessboard
// Reduce the image size to be much more manageable
// Note that opencv will copy the frame if no resize is requested; we can skip
// this since we

56
photon-core/src/main/java/org/photonvision/vision/pipe/impl/FocusPipe.java Normal file
View File

@@ -0,0 +1,56 @@
/*
* Copyright (C) Photon Vision.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package org.photonvision.vision.pipe.impl;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.MatOfDouble;
import org.opencv.imgproc.Imgproc;
import org.photonvision.vision.pipe.CVPipe;
public class FocusPipe extends CVPipe<Mat, FocusPipe.FocusResult, FocusPipe.FocusParams> {
private double maxVariance = 0.0;
@Override
protected FocusResult process(Mat in) {
var outputMat = new Mat();
Imgproc.Laplacian(in, outputMat, CvType.CV_64F, 3);
var mean = new MatOfDouble();
var stddev = new MatOfDouble();
Core.meanStdDev(outputMat, mean, stddev);
var sd = stddev.get(0, 0)[0];
var variance = sd * sd;
return new FocusResult(outputMat, variance);
}
public static class FocusResult {
public final Mat frame;
public final double variance;
public FocusResult(Mat frame, double variance) {
this.frame = frame;
this.variance = variance;
}
}
public static class FocusParams {}
}

14
photon-core/src/main/java/org/photonvision/vision/pipeline/AprilTagPipeline.java
View File

@@ -73,13 +73,13 @@ public class AprilTagPipeline extends CVPipeline<CVPipelineResult, AprilTagPipel
settings.threads = Math.max(1, settings.threads);
// for now, hard code tag width based on enum value
// 2023/other: best guess is 6in
double tagWidth = Units.inchesToMeters(6);
TargetModel tagModel = TargetModel.kAprilTag16h5;
if (settings.tagFamily == AprilTagFamily.kTag36h11) {
// 2024 tag, 6.5in
tagWidth = Units.inchesToMeters(6.5);
tagModel = TargetModel.kAprilTag36h11;
// From 2024 best guess is 6.5
double tagWidth = Units.inchesToMeters(6.5);
TargetModel tagModel = TargetModel.kAprilTag36h11;
if (settings.tagFamily == AprilTagFamily.kTag16h5) {
// 2023 tag, 6in
tagWidth = Units.inchesToMeters(6);
tagModel = TargetModel.kAprilTag16h5;
}
var config = new AprilTagDetector.Config();

94
photon-core/src/main/java/org/photonvision/vision/pipeline/FocusPipeline.java Normal file
View File

@@ -0,0 +1,94 @@
/*
* Copyright (C) Photon Vision.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package org.photonvision.vision.pipeline;
import org.opencv.core.Mat;
import org.photonvision.common.util.math.MathUtils;
import org.photonvision.vision.frame.Frame;
import org.photonvision.vision.frame.FrameThresholdType;
import org.photonvision.vision.opencv.CVMat;
import org.photonvision.vision.pipe.impl.CalculateFPSPipe;
import org.photonvision.vision.pipe.impl.FocusPipe;
import org.photonvision.vision.pipe.impl.ResizeImagePipe;
import org.photonvision.vision.pipeline.result.FocusPipelineResult;
public class FocusPipeline extends CVPipeline<FocusPipelineResult, FocusPipelineSettings> {
private final FocusPipe focusPipe = new FocusPipe();
private final CalculateFPSPipe calculateFPSPipe = new CalculateFPSPipe();
private final ResizeImagePipe resizeImagePipe = new ResizeImagePipe();
private static final FrameThresholdType PROCESSING_TYPE = FrameThresholdType.NONE;
public FocusPipeline() {
super(PROCESSING_TYPE);
settings = new FocusPipelineSettings();
}
public FocusPipeline(FocusPipelineSettings settings) {
super(PROCESSING_TYPE);
this.settings = settings;
}
@Override
protected void setPipeParamsImpl() {
resizeImagePipe.setParams(
new ResizeImagePipe.ResizeImageParams(settings.streamingFrameDivisor));
}
@Override
public FocusPipelineResult process(Frame frame, FocusPipelineSettings settings) {
long totalNanos = 0;
var inputMat = frame.colorImage.getMat();
boolean emptyIn = inputMat.empty();
Mat displayMat = new Mat();
double variance = 0.0;
if (!emptyIn) {
totalNanos += resizeImagePipe.run(inputMat).nanosElapsed;
var focusResult = focusPipe.run(inputMat);
totalNanos += focusResult.nanosElapsed;
variance = focusResult.output.variance;
displayMat = focusResult.output.frame;
}
var fpsResult = calculateFPSPipe.run(null);
var fps = fpsResult.output;
var processedCVMat = new CVMat(displayMat);
return new FocusPipelineResult(
frame.sequenceID,
MathUtils.nanosToMillis(totalNanos),
fps,
new Frame(
frame.sequenceID,
frame.colorImage,
processedCVMat,
frame.type,
frame.frameStaticProperties),
variance);
}
@Override
public void release() {
// we never actually need to give resources up since pipelinemanager only makes
// one of us
}
}

33
photon-core/src/main/java/org/photonvision/vision/pipeline/FocusPipelineSettings.java Normal file
View File

@@ -0,0 +1,33 @@
/*
* Copyright (C) Photon Vision.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package org.photonvision.vision.pipeline;
import com.fasterxml.jackson.annotation.JsonTypeName;
import org.photonvision.vision.processes.PipelineManager;
@JsonTypeName("FocusPipelineSettings")
public class FocusPipelineSettings extends CVPipelineSettings {
public FocusPipelineSettings() {
super();
pipelineNickname = "Focus Camera";
pipelineIndex = PipelineManager.FOCUS_INDEX;
pipelineType = PipelineType.FocusCamera;
inputShouldShow = true;
cameraAutoExposure = true;
}
}

1
photon-core/src/main/java/org/photonvision/vision/pipeline/PipelineType.java
View File

@@ -19,6 +19,7 @@ package org.photonvision.vision.pipeline;
@SuppressWarnings("rawtypes")
public enum PipelineType {
FocusCamera(-3, FocusPipeline.class),
Calib3d(-2, Calibrate3dPipeline.class),
DriverMode(-1, DriverModePipeline.class),
Reflective(0, ReflectivePipeline.class),

31
photon-core/src/main/java/org/photonvision/vision/pipeline/result/FocusPipelineResult.java Normal file
View File

@@ -0,0 +1,31 @@
/*
* Copyright (C) Photon Vision.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package org.photonvision.vision.pipeline.result;
import java.util.List;
import org.photonvision.vision.frame.Frame;
public class FocusPipelineResult extends CVPipelineResult {
public final double focus;
public FocusPipelineResult(
long seq, double latencyNanos, double fps, Frame outputFrame, double focus) {
super(seq, latencyNanos, fps, List.of(), outputFrame);
this.focus = focus;
}
}

11
photon-core/src/main/java/org/photonvision/vision/processes/PipelineManager.java
View File

@@ -36,10 +36,12 @@ public class PipelineManager {
private static final Logger logger = new Logger(PipelineManager.class, LogGroup.VisionModule);
public static final int DRIVERMODE_INDEX = -1;
public static final int FOCUS_INDEX = -3;
public static final int CAL_3D_INDEX = -2;
protected final List<CVPipelineSettings> userPipelineSettings;
protected final Calibrate3dPipeline calibration3dPipeline;
protected final FocusPipeline focusPipeline = new FocusPipeline();
protected final DriverModePipeline driverModePipeline = new DriverModePipeline();
/** Index of the currently active pipeline. Defaults to 0. */
@@ -93,6 +95,7 @@ public class PipelineManager {
return switch (index) {
case DRIVERMODE_INDEX -> driverModePipeline.getSettings();
case CAL_3D_INDEX -> calibration3dPipeline.getSettings();
case FOCUS_INDEX -> focusPipeline.getSettings();
default -> {
for (var setting : userPipelineSettings) {
if (setting.pipelineIndex == index) yield setting;
@@ -112,6 +115,7 @@ public class PipelineManager {
return switch (index) {
case DRIVERMODE_INDEX -> driverModePipeline.getSettings().pipelineNickname;
case CAL_3D_INDEX -> calibration3dPipeline.getSettings().pipelineNickname;
case FOCUS_INDEX -> focusPipeline.getSettings().pipelineNickname;
default -> {
for (var setting : userPipelineSettings) {
if (setting.pipelineIndex == index) yield setting.pipelineNickname;
@@ -153,6 +157,7 @@ public class PipelineManager {
return switch (currentPipelineIndex) {
case CAL_3D_INDEX -> calibration3dPipeline;
case DRIVERMODE_INDEX -> driverModePipeline;
case FOCUS_INDEX -> focusPipeline;
// Just return the current user pipeline, we're not on a built-in one
default -> currentUserPipeline;
};
@@ -253,7 +258,7 @@ public class PipelineManager {
new AprilTagPipeline((AprilTagPipelineSettings) desiredPipelineSettings);
}
case Aruco -> {
logger.debug("Creating Aruco Pipeline");
logger.debug("Creating ArUco Pipeline");
currentUserPipeline = new ArucoPipeline((ArucoPipelineSettings) desiredPipelineSettings);
}
case ObjectDetection -> {
@@ -261,7 +266,7 @@ public class PipelineManager {
currentUserPipeline =
new ObjectDetectionPipeline((ObjectDetectionPipelineSettings) desiredPipelineSettings);
}
case Calib3d, DriverMode -> {}
case Calib3d, DriverMode, FocusCamera -> {}
}
}
@@ -335,7 +340,7 @@ public class PipelineManager {
case AprilTag -> new AprilTagPipelineSettings();
case Aruco -> new ArucoPipelineSettings();
case ObjectDetection -> new ObjectDetectionPipelineSettings();
case Calib3d, DriverMode -> {
case Calib3d, DriverMode, FocusCamera -> {
logger.error("Got invalid pipeline type: " + type);
yield null;
}

10
photon-core/src/main/java/org/photonvision/vision/processes/VisionModule.java
View File

@@ -680,6 +680,16 @@ public class VisionModule {
saveAndBroadcastAll();
}
public void removeCalibrationFromConfig(Size unrotatedImageSize) {
if (unrotatedImageSize != null) {
visionSource.getSettables().removeCalibration(unrotatedImageSize);
} else {
logger.error("Got null size?");
}
saveAndBroadcastAll();
}
/**
* Add/remove quirks from the camera we're controlling
*

6
photon-core/src/main/java/org/photonvision/vision/processes/VisionSourceSettables.java
View File

@@ -19,6 +19,7 @@ package org.photonvision.vision.processes;
import edu.wpi.first.cscore.VideoMode;
import java.util.HashMap;
import org.opencv.core.Size;
import org.photonvision.common.configuration.CameraConfiguration;
import org.photonvision.common.logging.LogGroup;
import org.photonvision.common.logging.Logger;
@@ -120,6 +121,11 @@ public abstract class VisionSourceSettables {
calculateFrameStaticProps();
}
public void removeCalibration(Size unrotatedImageSize) {
configuration.removeCalibration(unrotatedImageSize);
calculateFrameStaticProps();
}
protected void calculateFrameStaticProps() {
var videoMode = getCurrentVideoMode();
this.frameStaticProperties =

5
photon-core/src/test/java/org/photonvision/common/configuration/NetworkConfigTest.java
View File

@@ -25,6 +25,7 @@ import java.io.File;
import java.io.IOException;
import java.nio.file.Path;
import org.junit.jupiter.api.Test;
import org.photonvision.common.util.TestUtils;
public class NetworkConfigTest {
@Test
@@ -39,13 +40,13 @@ public class NetworkConfigTest {
@Test
public void testDeserializeTeamNumberOrNtServerAddress() {
{
var folder = Path.of("test-resources/network-old-team-number");
var folder = TestUtils.getResourcesFolderPath(true).resolve("network-old-team-number");
var configMgr = new ConfigManager(folder, new LegacyConfigProvider(folder));
configMgr.load();
assertEquals("9999", configMgr.getConfig().getNetworkConfig().ntServerAddress);
}
{
var folder = Path.of("test-resources/network-new-team-number");
var folder = TestUtils.getResourcesFolderPath(true).resolve("network-new-team-number");
var configMgr = new ConfigManager(folder, new LegacyConfigProvider(folder));
configMgr.load();
assertEquals("9999", configMgr.getConfig().getNetworkConfig().ntServerAddress);

1
photon-core/src/test/java/org/photonvision/vision/pipeline/AprilTagTest.java
View File

@@ -109,7 +109,6 @@ public class AprilTagTest {
pipeline.getSettings().solvePNPEnabled = true;
pipeline.getSettings().cornerDetectionAccuracyPercentage = 4;
pipeline.getSettings().cornerDetectionUseConvexHulls = true;
pipeline.getSettings().targetModel = TargetModel.kAprilTag6p5in_36h11;
pipeline.getSettings().tagFamily = AprilTagFamily.kTag16h5;
var frameProvider =

0
photon-lib/.styleguide-license → photon-lib/.wpiformat-license
View File

5
photon-lib/src/main/native/cpp/photon/PhotonCamera.cpp
View File

@@ -24,9 +24,6 @@
#include "photon/PhotonCamera.h"
#include <hal/FRCUsageReporting.h>
#include <net/TimeSyncServer.h>
#include <stdexcept>
#include <string>
#include <string_view>
@@ -36,6 +33,8 @@
#include <frc/Errors.h>
#include <frc/RobotController.h>
#include <frc/Timer.h>
#include <hal/FRCUsageReporting.h>
#include <net/TimeSyncServer.h>
#include <opencv2/core.hpp>
#include <opencv2/core/mat.hpp>
#include <wpi/json.h>

3
photon-lib/src/main/native/cpp/photon/PhotonPoseEstimator.cpp
View File

@@ -24,8 +24,6 @@
#include "photon/PhotonPoseEstimator.h"
#include <hal/FRCUsageReporting.h>
#include <cmath>
#include <iostream>
#include <limits>
@@ -41,6 +39,7 @@
#include <frc/geometry/Pose3d.h>
#include <frc/geometry/Rotation3d.h>
#include <frc/geometry/Transform3d.h>
#include <hal/FRCUsageReporting.h>
#include <opencv2/calib3d.hpp>
#include <opencv2/core/mat.hpp>
#include <opencv2/core/types.hpp>

19
photon-lib/src/main/native/include/photon/simulation/PhotonCameraSim.h
View File

@@ -24,7 +24,16 @@
#pragma once
#include <algorithm>
#include <limits>
#include <string>
#include <utility>
#include <vector>
#include <cameraserver/CameraServer.h>
#include <frc/Timer.h>
#include <frc/apriltag/AprilTagFieldLayout.h>
#include <frc/apriltag/AprilTagFields.h>
#include <photon/PhotonCamera.h>
#include <photon/PhotonTargetSortMode.h>
#include <photon/estimation/CameraTargetRelation.h>
@@ -33,16 +42,6 @@
#include <photon/simulation/SimCameraProperties.h>
#include <photon/simulation/VideoSimUtil.h>
#include <photon/simulation/VisionTargetSim.h>
#include <algorithm>
#include <limits>
#include <string>
#include <utility>
#include <vector>
#include <frc/Timer.h>
#include <frc/apriltag/AprilTagFieldLayout.h>
#include <frc/apriltag/AprilTagFields.h>
#include <units/math.h>
#include <wpi/timestamp.h>

3
photon-lib/src/main/native/include/photon/simulation/SimCameraProperties.h
View File

@@ -24,8 +24,6 @@
#pragma once
#include <photon/estimation/OpenCVHelp.h>
#include <algorithm>
#include <random>
#include <string>
@@ -37,6 +35,7 @@
#include <frc/MathUtil.h>
#include <frc/geometry/Rotation2d.h>
#include <frc/geometry/Translation3d.h>
#include <photon/estimation/OpenCVHelp.h>
#include <units/frequency.h>
#include <units/time.h>

3
photon-lib/src/main/native/include/photon/simulation/VideoSimUtil.h
View File

@@ -24,8 +24,6 @@
#pragma once
#include <cscore_cv.h>
#include <algorithm>
#include <numeric>
#include <string>
@@ -33,6 +31,7 @@
#include <utility>
#include <vector>
#include <cscore_cv.h>
#include <frc/apriltag/AprilTag.h>
#include <opencv2/core.hpp>
#include <opencv2/imgcodecs.hpp>

11
photon-lib/src/test/native/cpp/PhotonCameraTest.cpp
View File

@@ -22,20 +22,19 @@
* SOFTWARE.
*/
#include <string>
#include <vector>
#include <fmt/ranges.h>
#include <frc/smartdashboard/SmartDashboard.h>
#include <gtest/gtest.h>
#include <hal/HAL.h>
#include <net/TimeSyncClient.h>
#include <net/TimeSyncServer.h>
#include <networktables/NetworkTableInstance.h>
#include <photon/PhotonCamera.h>
#include <photon/simulation/PhotonCameraSim.h>
#include <string>
#include <vector>
#include <frc/smartdashboard/SmartDashboard.h>
#include <networktables/NetworkTableInstance.h>
TEST(TimeSyncProtocolTest, Smoketest) {
using namespace wpi::tsp;
using namespace std::chrono_literals;

5
photon-lib/src/test/native/cpp/PhotonPoseEstimatorTest.cpp
View File

@@ -22,6 +22,8 @@
* SOFTWARE.
*/
#include "photon/PhotonPoseEstimator.h"
#include <map>
#include <utility>
#include <vector>
@@ -30,13 +32,12 @@
#include <frc/geometry/Pose3d.h>
#include <frc/geometry/Rotation3d.h>
#include <frc/geometry/Transform3d.h>
#include <gtest/gtest.h>
#include <units/angle.h>
#include <units/length.h>
#include <wpi/SmallVector.h>
#include "gtest/gtest.h"
#include "photon/PhotonCamera.h"
#include "photon/PhotonPoseEstimator.h"
#include "photon/dataflow/structures/Packet.h"
#include "photon/simulation/PhotonCameraSim.h"
#include "photon/simulation/SimCameraProperties.h"

3
photon-lib/src/test/native/cpp/PhotonUtilsTest.cpp
View File

@@ -22,7 +22,8 @@
* SOFTWARE.
*/
#include "gtest/gtest.h"
#include "photon/PhotonUtils.h"
#include <gtest/gtest.h>
TEST(PhotonUtilsTest, Include) {}

3
photon-lib/src/test/native/cpp/VersionTest.cpp
View File

@@ -24,8 +24,9 @@
#include <iostream>
#include <gtest/gtest.h>
#include "PhotonVersion.h"
#include "gtest/gtest.h"
TEST(VersionTest, PrintVersion) {
std::cout << photon::PhotonVersion::versionString << std::endl;

5
photon-lib/src/test/native/cpp/VisionSystemSimTest.cpp
View File

@@ -22,16 +22,17 @@
* SOFTWARE.
*/
#include "photon/simulation/VisionSystemSim.h"
#include <chrono>
#include <thread>
#include <tuple>
#include <vector>
#include <gtest/gtest.h>
#include <wpi/deprecated.h>
#include "gtest/gtest.h"
#include "photon/PhotonUtils.h"
#include "photon/simulation/VisionSystemSim.h"
// Ignore GetLatestResult warnings
WPI_IGNORE_DEPRECATED

53
photon-server/src/main/java/org/photonvision/server/RequestHandler.java
View File

@@ -35,6 +35,7 @@ import org.apache.commons.io.FileUtils;
import org.opencv.core.Mat;
import org.opencv.core.MatOfByte;
import org.opencv.core.MatOfInt;
import org.opencv.core.Size;
import org.opencv.imgcodecs.Imgcodecs;
import org.photonvision.common.configuration.ConfigManager;
import org.photonvision.common.configuration.NetworkConfig;
@@ -655,15 +656,11 @@ public class RequestHandler {
Files.copy(modelFileStream, modelPath, StandardCopyOption.REPLACE_EXISTING);
}
int idx = modelFile.filename().lastIndexOf('.');
String nickname = modelFile.filename().substring(0, idx);
ModelProperties modelProperties =
new ModelProperties(
modelPath,
modelFile.filename().replaceAll("." + family.extension(), ""),
labels,
width,
height,
family,
version);
new ModelProperties(modelPath, nickname, labels, width, height, family, version);
ObjectDetector objDetector = null;
@@ -1001,6 +998,46 @@ public class RequestHandler {
ctx.status(204);
}
private record CalibrationRemoveRequest(int width, int height, String cameraUniqueName) {}
public static void onCalibrationRemoveRequest(Context ctx) {
try {
CalibrationRemoveRequest request =
kObjectMapper.readValue(ctx.body(), CalibrationRemoveRequest.class);
logger.info(
"Attempting to remove calibration for camera: "
+ request.cameraUniqueName
+ " with a resolution of "
+ request.width
+ "x"
+ request.height);
VisionSourceManager.getInstance()
.vmm
.getModule(request.cameraUniqueName)
.removeCalibrationFromConfig(new Size(request.width, request.height));
ctx.status(200);
ctx.result(
"Successfully removed calibration for resolution: "
+ request.width
+ "x"
+ request.height);
logger.info(
"Successfully removed calibration for resolution: "
+ request.width
+ "x"
+ request.height);
} catch (JsonProcessingException e) {
ctx.status(400).result("Invalid JSON format");
logger.error("Failed to process calibration removed request", e);
} catch (Exception e) {
ctx.status(500).result("Failed to removed calibration");
logger.error("Unexpected error while attempting to remove calibration", e);
}
}
public static void onCalibrationSnapshotRequest(Context ctx) {
String cameraUniqueName = ctx.queryParam("cameraUniqueName");
var width = Integer.parseInt(ctx.queryParam("width"));

1
photon-server/src/main/java/org/photonvision/server/Server.java
View File

@@ -148,6 +148,7 @@ public class Server {
// Calibration
app.post("/api/calibration/end", RequestHandler::onCalibrationEndRequest);
app.post("/api/calibration/importFromData", RequestHandler::onDataCalibrationImportRequest);
app.post("/api/calibration/remove", RequestHandler::onCalibrationRemoveRequest);
// Object detection
app.post("/api/objectdetection/import", RequestHandler::onImportObjectDetectionModelRequest);

5
photon-targeting/src/main/java/org/photonvision/common/hardware/Platform.java
View File

@@ -128,7 +128,10 @@ public enum Platform {
}
public static boolean isRK3588() {
return Platform.isOrangePi() || Platform.isCoolPi4b() || Platform.isRock5C();
return Platform.isOrangePi()
|| Platform.isCoolPi4b()
|| Platform.isRock5C()
|| fileHasText("/proc/device-tree/compatible", "rk3588");
}
public static boolean isQCS6490() {

5
photon-targeting/src/main/native/cpp/net/TimeSyncClient.cpp
View File

@@ -17,9 +17,6 @@
#include "net/TimeSyncClient.h"
#include <wpinet/UDPClient.h>
#include <wpinet/uv/util.h>
#include <atomic>
#include <chrono>
#include <cstdlib>
@@ -33,6 +30,8 @@
#include <wpi/Logger.h>
#include <wpi/print.h>
#include <wpi/struct/Struct.h>
#include <wpinet/UDPClient.h>
#include <wpinet/uv/util.h>
#include "ntcore_cpp.h"

5
photon-targeting/src/main/native/cpp/net/TimeSyncServer.cpp
View File

@@ -17,9 +17,6 @@
#include "net/TimeSyncServer.h"
#include <wpinet/UDPClient.h>
#include <wpinet/uv/util.h>
#include <atomic>
#include <chrono>
#include <cstdlib>
@@ -32,6 +29,8 @@
#include <wpi/Logger.h>
#include <wpi/print.h>
#include <wpi/struct/Struct.h>
#include <wpinet/UDPClient.h>
#include <wpinet/uv/util.h>
#include "ntcore_cpp.h"

3
photon-targeting/src/main/native/cpp/photon/constrained_solvepnp/wrap/casadi_wrapper.cpp
View File

@@ -17,8 +17,6 @@
#include "photon/constrained_solvepnp/wrap/casadi_wrapper.h"
#include <fmt/core.h>
#include <chrono>
#include <cstdio>
#include <iostream>
@@ -28,6 +26,7 @@
#include <Eigen/Cholesky>
#include <Eigen/Core>
#include <Eigen/LU>
#include <fmt/core.h>
#include <frc/fmt/Eigen.h>
#include <wpi/timestamp.h>

11
photon-targeting/src/main/native/include/net/TimeSyncClient.h
View File

@@ -17,12 +17,6 @@
#pragma once
#include <wpinet/EventLoopRunner.h>
#include <wpinet/UDPClient.h>
#include <wpinet/uv/Buffer.h>
#include <wpinet/uv/Timer.h>
#include <wpinet/uv/Udp.h>
#include <atomic>
#include <chrono>
#include <cstdlib>
@@ -40,6 +34,11 @@
#include <wpi/print.h>
#include <wpi/static_circular_buffer.h>
#include <wpi/struct/Struct.h>
#include <wpinet/EventLoopRunner.h>
#include <wpinet/UDPClient.h>
#include <wpinet/uv/Buffer.h>
#include <wpinet/uv/Timer.h>
#include <wpinet/uv/Udp.h>
#include "TimeSyncStructs.h"
#include "ntcore_cpp.h"

11
photon-targeting/src/main/native/include/net/TimeSyncServer.h
View File

@@ -17,12 +17,6 @@
#pragma once
#include <wpinet/EventLoopRunner.h>
#include <wpinet/UDPClient.h>
#include <wpinet/uv/Buffer.h>
#include <wpinet/uv/Timer.h>
#include <wpinet/uv/Udp.h>
#include <atomic>
#include <chrono>
#include <cstdlib>
@@ -38,6 +32,11 @@
#include <wpi/Logger.h>
#include <wpi/print.h>
#include <wpi/struct/Struct.h>
#include <wpinet/EventLoopRunner.h>
#include <wpinet/UDPClient.h>
#include <wpinet/uv/Buffer.h>
#include <wpinet/uv/Timer.h>
#include <wpinet/uv/Udp.h>
#include "TimeSyncStructs.h"
#include "ntcore_cpp.h"

4
photon-targeting/src/main/native/include/photon/estimation/OpenCVHelp.h
View File

@@ -29,9 +29,9 @@
#define OPENCV_DISABLE_EIGEN_TENSOR_SUPPORT
#include <opencv2/core/eigen.hpp>
#include "photon/targeting/PnpResult.h"
#include "photon/targeting/MultiTargetPNPResult.h"
#include "photon/targeting/MultiTargetPNPResult.h"
#include "photon/targeting/PnpResult.h"
#include "photon/targeting/TargetCorner.h"
namespace photon {

6
photon-targeting/src/main/native/jni/ConstrainedSolvepnpJNI.cpp
View File

@@ -15,13 +15,13 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
#include <fmt/core.h>
#include <fmt/ranges.h>
#include <iostream>
#include <span>
#include <vector>
#include <fmt/core.h>
#include <fmt/ranges.h>
#include "org_photonvision_jni_ConstrainedSolvepnpJni.h"
#include "photon/constrained_solvepnp/wrap/casadi_wrapper.h"

4
photon-targeting/src/main/native/jni/TimeSyncClientJNI.cpp
View File

@@ -15,11 +15,11 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
#include <org_photonvision_jni_TimeSyncClient.h>
#include <cstdio>
#include <string>
#include <org_photonvision_jni_TimeSyncClient.h>
#include "jni_utils.h"
#include "net/TimeSyncClient.h"

4
photon-targeting/src/main/native/jni/TimeSyncServerJNI.cpp
View File

@@ -15,11 +15,11 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
#include <cstdio>
#include <org_photonvision_jni_TimeSyncClient.h>
#include <org_photonvision_jni_TimeSyncServer.h>
#include <cstdio>
#include "jni_utils.h"
#include "net/TimeSyncServer.h"

3
photon-targeting/src/test/native/cpp/CasadiWrapperTest.cpp
View File

@@ -15,13 +15,12 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
#include <gtest/gtest.h>
#include <chrono>
#include <cstdio>
#include <iostream>
#include <vector>
#include <gtest/gtest.h>
#include <wpi/timestamp.h>
#include "photon/constrained_solvepnp/wrap/casadi_wrapper.h"

3
photon-targeting/src/test/native/cpp/PacketTest.cpp
View File

@@ -15,6 +15,8 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
#include "photon/dataflow/structures/Packet.h"
#include <chrono>
#include <vector>
@@ -22,7 +24,6 @@
#include <wpi/print.h>
#include "gtest/gtest.h"
#include "photon/dataflow/structures/Packet.h"
#include "photon/targeting/MultiTargetPNPResult.h"
#include "photon/targeting/PhotonPipelineResult.h"
#include "photon/targeting/PhotonTrackedTarget.h"

0
photonlib-cpp-examples/.styleguide-license → photonlib-cpp-examples/.wpiformat-license
View File

3
photonlib-cpp-examples/aimandrange/src/main/cpp/Robot.cpp
View File

@@ -24,12 +24,11 @@
#include "Robot.h"
#include <photon/PhotonUtils.h>
#include <iostream>
#include <frc/simulation/BatterySim.h>
#include <frc/simulation/RoboRioSim.h>
#include <photon/PhotonUtils.h>
void Robot::RobotInit() {}

3
photonlib-cpp-examples/aimandrange/src/main/include/Robot.h
View File

@@ -24,10 +24,9 @@
#pragma once
#include <photon/PhotonCamera.h>
#include <frc/TimedRobot.h>
#include <frc/XboxController.h>
#include <photon/PhotonCamera.h>
#include "Constants.h"
#include "VisionSim.h"

11
photonlib-cpp-examples/aimandrange/src/main/include/VisionSim.h
View File

@@ -24,6 +24,11 @@
#pragma once
#include <limits>
#include <memory>
#include <frc/apriltag/AprilTagFieldLayout.h>
#include <frc/apriltag/AprilTagFields.h>
#include <photon/PhotonCamera.h>
#include <photon/PhotonPoseEstimator.h>
#include <photon/estimation/VisionEstimation.h>
@@ -31,12 +36,6 @@
#include <photon/simulation/VisionTargetSim.h>
#include <photon/targeting/PhotonPipelineResult.h>
#include <limits>
#include <memory>
#include <frc/apriltag/AprilTagFieldLayout.h>
#include <frc/apriltag/AprilTagFields.h>
#include "Constants.h"
class VisionSim {

3
photonlib-cpp-examples/aimattarget/src/main/include/Robot.h
View File

@@ -24,10 +24,9 @@
#pragma once
#include <photon/PhotonCamera.h>
#include <frc/TimedRobot.h>
#include <frc/XboxController.h>
#include <photon/PhotonCamera.h>
#include "Constants.h"
#include "VisionSim.h"

11
photonlib-cpp-examples/aimattarget/src/main/include/VisionSim.h
View File

@@ -24,6 +24,11 @@
#pragma once
#include <limits>
#include <memory>
#include <frc/apriltag/AprilTagFieldLayout.h>
#include <frc/apriltag/AprilTagFields.h>
#include <photon/PhotonCamera.h>
#include <photon/PhotonPoseEstimator.h>
#include <photon/estimation/VisionEstimation.h>
@@ -31,12 +36,6 @@
#include <photon/simulation/VisionTargetSim.h>
#include <photon/targeting/PhotonPipelineResult.h>
#include <limits>
#include <memory>
#include <frc/apriltag/AprilTagFieldLayout.h>
#include <frc/apriltag/AprilTagFields.h>
#include "Constants.h"
class VisionSim {

13
photonlib-cpp-examples/poseest/src/main/include/Vision.h
View File

@@ -24,19 +24,18 @@
#pragma once
#include <photon/PhotonCamera.h>
#include <photon/PhotonPoseEstimator.h>
#include <photon/estimation/VisionEstimation.h>
#include <photon/simulation/VisionSystemSim.h>
#include <photon/simulation/VisionTargetSim.h>
#include <photon/targeting/PhotonPipelineResult.h>
#include <functional>
#include <limits>
#include <memory>
#include <frc/apriltag/AprilTagFieldLayout.h>
#include <frc/apriltag/AprilTagFields.h>
#include <photon/PhotonCamera.h>
#include <photon/PhotonPoseEstimator.h>
#include <photon/estimation/VisionEstimation.h>
#include <photon/simulation/VisionSystemSim.h>
#include <photon/simulation/VisionTargetSim.h>
#include <photon/targeting/PhotonPipelineResult.h>
#include "Constants.h"

0
photonlib-java-examples/.styleguide-license → photonlib-java-examples/.wpiformat-license
View File

7
scripts/rknn-convert-tool/MAINTAINERS.md
View File

@@ -1,7 +0,0 @@
# Notebook
In the first cell of the RKNN conversion notebook, the installation script uses a structured list of dictionaries to define the download URLs and filenames for required scripts. Each dictionary includes a `url` (a permalink to a specific commit) and the corresponding `filename`.
Please ensure that all URLs in this array use permalinks—that is, links pointing to a specific commit hash rather than a branch name (e.g., main). This guarantees that the correct version of each script is always fetched, and prevents unexpected changes if the repository is updated in the future.
You typically won’t need to update these permalinks unless one of the referenced scripts is modified. In that case, update the commit hash in the URLs accordingly.

182
scripts/rknn-convert-tool/create_onnx.py
View File

@@ -1,182 +0,0 @@
import argparse
import os.path
import subprocess
import sys
# This will work for all models that don't use anchors (e.g. all YOLO models except YOLOv5/v7)
# This includes YOLOv5u
yolo_non_anchor_repo = "https://github.com/airockchip/ultralytics_yolo11"
# For original YOLOv5 models
yolov5_repo = "https://github.com/airockchip/yolov5"
valid_yolo_versions = ["yolov5", "yolov8", "yolov11"]
comma_sep_yolo_versions = ", ".join(valid_yolo_versions)
ultralytics_folder_name_yolov5 = "airockchip_yolo_pkg_yolov5"
ultralytics_default_folder_name = "airockchip_yolo_pkg"
bad_model_msg = """
This is usually due to passing in the wrong model version.
Please make sure you have the right model version and try again.
"""
def print_bad_model_msg(cause):
print(f"{cause}{bad_model_msg}")
def run_and_exit_with_error(cmd, error_msg, enable_error_output=True):
try:
if enable_error_output:
subprocess.run(
cmd,
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE,
universal_newlines=True,
).check_returncode()
else:
subprocess.run(cmd).check_returncode()
except subprocess.CalledProcessError as e:
print(error_msg)
if enable_error_output:
print(e.stdout)
sys.exit(1)
def check_git_installed():
run_and_exit_with_error(
["git", "--version"],
"""Git is not installed or not found in your PATH.
Please install Git from https://git-scm.com/downloads and try again.""",
)
def check_or_clone_rockchip_repo(repo_url, repo_name=ultralytics_default_folder_name):
if os.path.exists(repo_name):
print(
f'Existing Rockchip repo "{repo_name}" detected, skipping installation...'
)
else:
print(f'Cloning Rockchip repo to "{repo_name}"')
run_and_exit_with_error(
["git", "clone", repo_url, repo_name],
"Failed to clone Rockchip repo, please see error output",
)
def run_pip_install_or_else_exit(args):
print("Running pip install...")
run_and_exit_with_error(
["pip", "install"] + args,
"Pip install rockchip repo failed, please see error output",
)
def run_onnx_conversion_yolov5(model_path):
check_or_clone_rockchip_repo(yolov5_repo, ultralytics_folder_name_yolov5)
run_pip_install_or_else_exit(
[
"-r",
os.path.join(ultralytics_folder_name_yolov5, "requirements.txt"),
"torch<2.6.0",
"onnx==1.18.0",
"onnxscript",
]
)
model_abspath = os.path.abspath(model_path)
try:
subprocess.run(
[
"python",
f"{ultralytics_folder_name_yolov5}/export.py",
"--weights",
model_abspath,
"--rknpu",
"--include",
"onnx",
],
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE,
universal_newlines=True,
).check_returncode()
except subprocess.CalledProcessError as e:
print("Failed to run YOLOv5 export, please see error output")
if "ModuleNotFoundError" in e.stdout and "ultralytics" in e.stdout:
print_bad_model_msg(
"It seems the YOLOv5 repo could not find an ultralytics installation."
)
elif "AttributeError" in e.stdout and "_register_detect_seperate" in e.stdout:
print_bad_model_msg("It seems that you received a model attribute error.")
else:
print("Unknown Error when converting:")
print(e.stdout)
sys.exit(1)
def run_onnx_conversion_no_anchor(model_path):
check_or_clone_rockchip_repo(yolo_non_anchor_repo)
run_pip_install_or_else_exit(
["-e", ultralytics_default_folder_name, "onnx==1.18.0", "onnxscript"]
)
sys.path.insert(0, os.path.abspath(ultralytics_default_folder_name))
model_abs_path = os.path.abspath(model_path)
from ultralytics import YOLO
try:
model = YOLO(model_abs_path)
model.export(format="rknn")
except TypeError as e:
if "originally trained" in str(e):
print_bad_model_msg(
"Ultralytics has detected that this model is a YOLOv5 model."
)
else:
raise e
sys.exit(1)
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Generate valid ONNX file for yolo model"
)
parser.add_argument(
"-m",
"--model_path",
required=True,
help=(f"Path to YOLO model"),
)
parser.add_argument(
"-v",
"--version",
required=True,
choices=valid_yolo_versions,
help=(f"Model version, must be one of: {comma_sep_yolo_versions}"),
)
args = parser.parse_args()
check_git_installed()
try:
if args.version.lower() == "yolov5":
run_onnx_conversion_yolov5(args.model_path)
else:
run_onnx_conversion_no_anchor(args.model_path)
print(
"Model export finished. Please use the generated ONNX file to convert to RKNN."
)
except SystemExit:
print("Model export failed. Please see output above.")

221
scripts/rknn-convert-tool/create_rknn.py
View File

@@ -1,221 +0,0 @@
import argparse
import os
import random
import sys
from rknn.api import RKNN
image_extensions = (".jpg", ".jpeg", ".png", ".bmp", ".gif", ".tiff", ".webp")
DEFAULT_PLATFORM = "rk3588"
def list_img_dir(img_dir):
return [
os.path.abspath(os.path.join(img_dir, f))
for f in os.listdir(img_dir)
if f.lower().endswith(image_extensions)
]
def sample_imgs(num, img_list):
if len(img_list) < num:
return img_list
else:
return random.sample(img_list, num)
def get_image_list_from_dataset(num_imgs, yaml_dir):
print(f"Dataset detected with {yaml_dir} file")
img_raw_paths = []
with open(yaml_dir, "r") as yaml_file:
for line in yaml_file:
line = line.strip()
if (
line.startswith("train:")
or line.startswith("val:")
or line.startswith("test:")
):
img_raw_paths.append(line.split(":", 1)[1].strip())
no_yaml_dir = yaml_dir.replace(
"data.yaml", "dummy_dir"
) # data.yaml sets dirs one level up
img_set_paths = []
for img_raw_path in img_raw_paths:
p = (
img_raw_path
if os.path.isabs(img_raw_path)
else os.path.realpath(os.path.join(no_yaml_dir, img_raw_path))
)
if os.path.exists(p):
img_set_paths.append(p)
if len(img_set_paths) < 1:
return None
all_imgs = [list_img_dir(path) for path in img_set_paths]
for imgs in all_imgs:
print(len(imgs))
total_imgs = sum(len(group) for group in all_imgs)
sampled_imgs = [
sample_imgs(round((len(group) / total_imgs) * num_imgs), group)
for group in all_imgs
]
return [img for group in sampled_imgs for img in group]
def get_image_list_from_img_dir(num_imgs, img_dir):
return sample_imgs(num_imgs, list_img_dir(img_dir))
def get_image_list(num_imgs, image_dir):
yaml_path = os.path.join(image_dir, "data.yaml")
if os.path.exists(yaml_path):
return get_image_list_from_dataset(num_imgs, yaml_path)
else:
return get_image_list_from_img_dir(num_imgs, image_dir)
def run_rknn_conversion(
img_list_txt, disable_quant, model_path, rknn_output, verbose_logging
):
rknn = RKNN(
verbose=verbose_logging,
verbose_file=("rknn_convert.log" if verbose_logging else None),
)
rknn.config(
mean_values=[[0, 0, 0]],
std_values=[[255, 255, 255]],
target_platform=DEFAULT_PLATFORM,
)
print("Attempted RKNN load")
ret = rknn.load_onnx(model=model_path)
if ret != 0:
print("Loading model failed!")
exit(ret)
print("Attempting RKNN build")
ret = rknn.build(do_quantization=(not disable_quant), dataset=img_list_txt)
if ret != 0:
print("Building model failed!")
exit(ret)
print("Build succeeded! Starting export...")
ret = rknn.export_rknn(rknn_output)
if ret != 0:
print("Exporting model failed!")
exit(ret)
print("Finished export!")
# Release
rknn.release()
print(f'Your model is in "{rknn_output}" and ready to use!')
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Generate valid ONNX file for yolo model"
)
parser.add_argument(
"-ni",
"--num_imgs",
type=int,
default=300,
help="Number of images to use for calibration (default: 300)",
)
parser.add_argument(
"-d",
"--img_dir",
help="Directory where your dataset is located (must have data.yaml), or images are located",
)
parser.add_argument(
"-m",
"--model_path",
required=True,
help=(f"Path to generated ONNX model"),
)
parser.add_argument(
"-dq",
"--disable_quantize",
action="store_true",
help="Whether to skip quantization (default: False)",
)
parser.add_argument(
"-o",
"--rknn_output",
default="out.rknn",
help="Where the rknn model should be outputted (default: ./out.rknn)",
)
parser.add_argument(
"-ds",
"--img_dataset_txt",
default="imgs.txt",
help="Where the list of images used for quantization should be outputted (default: ./imgs.txt)",
)
parser.add_argument(
"-vb",
"--verbose",
action="store_true",
help="Whether to enable verbose logging",
)
args = parser.parse_args()
if not args.rknn_output.endswith(".rknn"):
print("RKNN output path must end in .rknn!")
sys.exit(1)
if not args.disable_quantize:
if args.img_dir == None or len(args.img_dir) < 1:
print(f"Must specify list of images to use with --img_dir")
sys.exit(1)
img_dir_abs = os.path.abspath(args.img_dir)
img_list = get_image_list(args.num_imgs, img_dir_abs)
img_list_len = 0 if img_list is None else len(img_list)
if img_list_len == 0:
print(f"No images found in {img_dir_abs}")
sys.exit(1)
elif img_list_len < args.num_imgs:
print(
f"Not enough images in your dataset/directory, you have {img_list_len} images, but need {args.num_imgs}"
)
sys.exit(1)
if not args.img_dataset_txt.endswith(".txt"):
print(f"Image dataset text file path must end in .txt")
sys.exit(1)
with open(args.img_dataset_txt, "w") as set_file:
set_file.writelines(f"{img}\n" for img in img_list)
try:
run_rknn_conversion(
args.img_dataset_txt,
args.disable_quantize,
args.model_path,
args.rknn_output,
args.verbose,
)
except SystemExit:
print("RKNN Conversion failed, see output above")

310
scripts/rknn-convert-tool/rknn_conversion.ipynb
View File

@@ -1,310 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "bb5367ce",
"metadata": {},
"source": [
"# RKNN Conversion Guide\n",
"\n",
"----------------------------\n",
"\n",
"### Before you start\n",
"\n",
"If you are not using Google Colab, it is recommended to create a separate [Python virtual environment](https://docs.python.org/3/library/venv.html) before you run the scripts or the Python notebook from this project. This ensures that packages installed for the conversion process do not conflict with other packages you may already have setup."
]
},
{
"cell_type": "markdown",
"id": "5f42d0a144caceb6",
"metadata": {},
"source": [
"### Preinstallation\n",
"\n",
"This notebook requires the use of external Python scripts. Please run the installation script below to import these external scripts if you do not have them already."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7903189e",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"\n",
"# Define scripts with URLs and inferred filenames\n",
"# DO NOT modify the filenames\n",
"scripts = [\n",
" {\n",
" \"url\": \"https://raw.githubusercontent.com/PhotonVision/photonvision/2bf166bc3f377fa8af9d9d38ee46e0db978a4036/scripts/rknn-convert-tool/create_onnx.py\",\n",
" \"filename\": \"create_onnx.py\" # CREATE_ONNX_SCRIPT\n",
" },\n",
" {\n",
" \"url\": \"https://raw.githubusercontent.com/PhotonVision/photonvision/2bf166bc3f377fa8af9d9d38ee46e0db978a4036/scripts/rknn-convert-tool/create_rknn.py\",\n",
" \"filename\": \"create_rknn.py\" # CREATE_RKNN_SCRIPT\n",
" }\n",
"]\n",
"\n",
"# Download each script\n",
"for script in scripts:\n",
" try:\n",
" subprocess.run([\"wget\", script[\"url\"], \"-O\", script[\"filename\"]]).check_returncode()\n",
" print(f\"Successfully downloaded: {script['filename']}\")\n",
" except subprocess.CalledProcessError as e:\n",
" print(f\"Failed to download script from URL: {script['url']}\")\n",
" print(e)\n"
]
},
{
"cell_type": "markdown",
"id": "d68be4aba4d3022b",
"metadata": {},
"source": [
"#### *Numpy Fix* - Important for Google Colab Users\n",
"\n",
"Google Colab comes with an incompatible version of Numpy installed. To fix this, please run the following cells below and **restart your session** when prompted."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "de0310a3e4401233",
"metadata": {},
"outputs": [],
"source": [
"%pip uninstall numpy -y\n",
"%pip install \"numpy>=1.23.0,<2.0.0\""
]
},
{
"cell_type": "markdown",
"id": "d498ed79",
"metadata": {},
"source": [
"### Step 1: Convert to ONNX\n",
"\n",
"To convert to ONNX, simply run the `create_onnx.py` script, providing the path to your model weights and specifying the model version, as shown below."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0659e15f",
"metadata": {},
"outputs": [],
"source": [
"# where version is either yolov5, yolov8, or yolov11, and model_path is the path to your weights file (.pt)\n",
"%run create_onnx.py --version yolov8 --model_path myyolov8model.pt"
]
},
{
"cell_type": "markdown",
"id": "86ff07e6",
"metadata": {},
"source": [
"### Step 2: Download RKNN API\n",
"\n",
"You can either use `pip` below to automatically detect and install the correct RKNN API Python library for you, or install it manually.\n",
"\n",
"#### Automatic installation\n",
"\n",
"Please run `pip` below. If it does not work, refer to the instructions for manual installation. You may need to restart your session after running the command below.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7ec11f96",
"metadata": {},
"outputs": [],
"source": [
"%pip install rknn-toolkit2"
]
},
{
"cell_type": "markdown",
"id": "8b57fe4d",
"metadata": {},
"source": [
"#### Manual Installation (If Automatic Installation Fails)\n",
"Visit the [RKNN Toolkit 2](https://github.com/airockchip/rknn-toolkit2) Github repository, then click on rknn-toolkit2, followed by packages.\n",
"If you are running an x86_64 CPU (e.g., most Intel and AMD processors), select that option; otherwise, choose arm64 for ARM-based computers (e.g., M-series Macs or Snapdragon processors). If you're unsure which CPU you're using, check your system settings for processor architecture information.\n",
"\n",
"Once you've selected the correct CPU architecture, you'll see multiple packages. The file names will look something like:\n",
"`rknn_toolkit2-2.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`.\n",
"The numbers after cp correspond to your Python version. For example, if you're using Python 3.10, look for a package with cp310 in the name. For Python 3.8, look for cp38; for Python 3.7, cp37, and so on.\n",
"\n",
"Once you've found the correct package, click the \"Raw\" button to download the .whl file. Then, run the following command in your terminal, replacing rknn_toolkit2.whl with the actual path to the file you downloaded:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7414b120",
"metadata": {},
"outputs": [],
"source": [
"%pip install rknn_toolkit2.whl"
]
},
{
"cell_type": "markdown",
"id": "c1db5ef0",
"metadata": {},
"source": [
"### Step 3: Convert to RKNN\n",
"\n",
"Please review the notes about quantization before running the RKNN conversion."
]
},
{
"cell_type": "markdown",
"id": "5e56b2f64bf6e85f",
"metadata": {},
"source": [
"### Quantization\n",
"\n",
"When performing quantization, it is critical to provide representative images of the objects or scenes you are trying to detect. These images are used to calibrate the model’s internal activations and greatly influence the final performance.\n",
"\n",
"It is recommended to use 300–500 representative images that reflect the real-world input your model will encounter. As the old saying goes, it’s quality over quantity — having a diverse, relevant set matters more than simply having many images.\n",
"\n",
"Quantization will cause some loss in model accuracy. However, if your calibration images are chosen wisely, this accuracy drop should be minimal and acceptable. If the sampled images are too uniform or unrelated, your quantized model's performance may worsen significantly.\n",
"\n",
"The script will automatically sample representative images randomly from the provided dataset. While this usually works well, please verify that the dataset contains diverse and relevant examples of your target objects. As a reminder, the images used to quantize the model are stored in the text file specified by `--img_dataset_txt`.\n"
]
},
{
"cell_type": "markdown",
"id": "93e0d0622df170e",
"metadata": {},
"source": [
"### Optional: Download a dataset from Roboflow for quantization\n",
"\n",
"If you do not already have a dataset or set of images containing the objects you want to detect, follow the steps below to download one from Roboflow Universe.\n",
"\n",
"#### **Step 1: Search for a Dataset**\n",
"\n",
"Go to [Roboflow Universe](https://universe.roboflow.com) and use the search bar to locate a dataset relevant to what you want to detect.\n",
"**Note:** The dataset must include the classes or object types you intend to detect.\n",
"\n",
"#### **Step 2: Access the Dataset Tab**\n",
"\n",
"After selecting a suitable project, navigate to the **Dataset** tab. Click the **\"Download Dataset\"** button. A prompt will appear with several options, including:\n",
"\n",
"- Train a model with this dataset\n",
"- Train from a portion of this dataset\n",
"- Download dataset\n",
"\n",
"Select **Download dataset**.\n",
"\n",
"#### **Step 3: Choose Format and View Download Code**\n",
"\n",
"- Under **Image and Annotation Format**, choose the version of YOLO you are using:\n",
" - For **YOLOv5**, choose `YOLOv5 PyTorch`\n",
" - For **YOLOv8**, choose `YOLOv8`\n",
" - For **YOLOv11**, choose `YOLOv11`\n",
"- If multiple annotation formats are listed for your model, always select the one ending in **\"PyTorch\"**.\n",
"\n",
"Then, under **Download Options**, click **\"Show Download Code\"** and continue.\n",
"\n",
"In the resulting screen, you will see three tabs:\n",
"- **Jupyter**\n",
"- **Terminal**\n",
"- **Raw URL**\n",
"\n",
"Select the **Terminal** tab and copy the provided command.\n",
"\n",
"#### **Step 4: Paste and Run**\n",
"\n",
"Paste the copied command into the notebook cell below and run it. This will download and extract the dataset into your environment, making it ready for use in the quantization process.\n",
"\n",
"Make sure to prefix the command with \"`!`\" so it executes properly in this Jupyter Notebook environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8bf75c9dcb328c84",
"metadata": {},
"outputs": [],
"source": [
"!curl -L \"https://universe.roboflow.com/ds/FaF3HbDmF7?key=iMoJR25O9H\" > roboflow.zip; unzip roboflow.zip; rm roboflow.zip"
]
},
{
"cell_type": "markdown",
"id": "72bad9cac670f1ab",
"metadata": {},
"source": [
"### RKNN Conversion Script\n",
"To get started, run the `create_rknn.py` script below, replacing the arguments with your own values. Refer to the table below for detailed information on each argument’s purpose and usage. The `--model_path` argument should point to your exported ONNX model from Step 1, and `--img_dir` must reference a valid directory containing either a dataset or a set of images to be used for quantization.\n",
"\n",
"#### Overview of the `create_rknn.py` script\n",
"\n",
"This script converts a YOLO ONNX model to RKNN format using a set of calibration images. It's designed to work with either:\n",
"\n",
"- A flat directory of images (e.g. `train/images`), **or**\n",
"- A dataset directory containing a `data.yaml` file that defines `train`, `val`, and/or `test` folders.\n",
"\n",
"##### Arguments\n",
"\n",
"| Argument | Type | Description |\n",
"|----------|------|-----------------------------------------------------------------------------------------------------------------|\n",
"| `--img_dir` (`-d`) | `str` (required) | Path to your image directory. This can either be a folder of images **or** a dataset folder with a `data.yaml`. |\n",
"| `--model_path` (`-m`) | `str` (required) | Path to your YOLO ONNX model, created in Step 1. |\n",
"| `--num_imgs` (`-ni`) | `int` (default: `300`) | Number of images to use for quantization calibration. |\n",
"| `--disable_quantize` (`-dq`) | `bool` (default: `False`) | Set to `True` to skip quantization entirely. Not recommended for performance, and should not be used for deployment on PhotonVision, which requires quantization. |\n",
"| `--rknn_output` (`-o`) | `str` (default: `out.rknn`) | File path where the final RKNN model should be saved. |\n",
"| `--img_dataset_txt` (`-ds`) | `str` (default: `imgs.txt`) | File path to store the list of images used during quantization. |\n",
"| `--verbose` (`-vb`) | `bool` (default: `False`) | Enable detailed logging from the RKNN API during conversion. |\n",
"\n",
"\n",
"##### *Notes*\n",
"\n",
"1. This script is designed for use with [PhotonVision](https://photonvision.org), and by default sets the target platform for RKNN conversion to `RK3588`, a chipset commonly found in many variants of the Orange Pi 5 series (e.g., Orange Pi 5, 5 Pro, 5 Plus, 5 Max, etc.). You may modify the `DEFAULT_PLATFORM` value in the `create_rknn.py` script to match your specific hardware or deployment requirements if necessary.\n",
"\n",
"2. If you followed the Roboflow dataset download instructions from the previous section, the dataset will have been extracted to your **current working directory**. In that case, you can simply set `--img_dir` to \"`.`\" to reference the current directory."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b09656dd",
"metadata": {},
"outputs": [],
"source": [
"%run create_rknn.py --img_dir ./datasets --model_path weights.onnx"
]
},
{
"cell_type": "markdown",
"id": "5b3a6806",
"metadata": {},
"source": [
"And that’s it! Your RKNN model file is now ready for deployment on an Orange Pi."
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

621
scripts/rknn_conversion.ipynb Normal file
View File

@@ -0,0 +1,621 @@
{
"cells": [
{
"metadata": {},
"cell_type": "markdown",
"source": [
"# RKNN Conversion Guide\n",
"\n",
"----------------------------\n",
"\n",
"### Before you start\n",
"\n",
"If you are not using Google Colab, it is recommended to create a separate [Python virtual environment](https://docs.python.org/3/library/venv.html) before you run this project. This ensures that packages installed for the conversion process do not conflict with other packages you may already have set up.\n",
"\n",
"## ⚠️ Linux Only\n",
"This notebook can only by run on **Linux** because the `rknn-toolkit2` Python package only supports Linux builds.\n",
"If you don’t have access to a Linux system, consider using a cloud service like [Google Colab](https://colab.research.google.com).\n"
],
"id": "65e9f457d12dcc6b"
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"### Setup\n",
"\n",
"This notebook requires the use of custom Python code. Please run the installation script below to import these external scripts if you do not have them already.\n",
"\n",
"**You may need to run the `Create ONNX/RKNN` cell multiple times when you restart your session. If you see a `create_onnx`\n",
"or `create_rknn` not found error, rerun the cell below and then retry.**\n",
"\n",
"**Do not modify the cells in this setup section unless you know what you’re doing or have a specific reason to.**"
],
"id": "500d656b7cc0ebd7"
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"#### Create ONNX/RKNN\n",
"\n",
"Please run the cell below to be able to use the `create_onnx` and `create_rknn` functions."
],
"id": "798298b1dbe33d2d"
},
{
"metadata": {},
"cell_type": "code",
"outputs": [],
"execution_count": null,
"source": [
"import os.path\n",
"import subprocess\n",
"import sys\n",
"import random\n",
"\n",
"# This will work for all models that don't use anchors (e.g. all YOLO models except YOLOv5/v7)\n",
"# This includes YOLOv5u\n",
"yolo_non_anchor_repo = \"https://github.com/airockchip/ultralytics_yolo11\"\n",
"\n",
"# For original YOLOv5 models\n",
"yolov5_repo = \"https://github.com/airockchip/yolov5\"\n",
"\n",
"valid_yolo_versions = [\"yolov5\", \"yolov8\", \"yolov11\"]\n",
"\n",
"ultralytics_folder_name_yolov5 = \"airockchip_yolo_pkg_yolov5\"\n",
"ultralytics_default_folder_name = \"airockchip_yolo_pkg\"\n",
"\n",
"bad_model_msg = \"\"\"\n",
"This is usually due to passing in the wrong model version.\n",
"Please make sure you have the right model version and try again.\n",
"\"\"\"\n",
"\n",
"\n",
"def print_bad_model_msg(cause):\n",
" print(f\"{cause}{bad_model_msg}\")\n",
"\n",
"def run_and_exit_with_error(cmd, error_msg, enable_error_output=True):\n",
" try:\n",
" if enable_error_output:\n",
" subprocess.run(\n",
" cmd,\n",
" stderr=subprocess.STDOUT,\n",
" stdout=subprocess.PIPE,\n",
" universal_newlines=True,\n",
" ).check_returncode()\n",
" else:\n",
" subprocess.run(cmd).check_returncode()\n",
" except subprocess.CalledProcessError as e:\n",
" print(error_msg)\n",
"\n",
" if enable_error_output:\n",
" print(e.stdout)\n",
"\n",
" sys.exit(1)\n",
"\n",
"\n",
"def check_git_installed():\n",
" run_and_exit_with_error(\n",
" [\"git\", \"--version\"],\n",
" \"\"\"Git is not installed or not found in your PATH.\n",
"Please install Git from https://git-scm.com/downloads and try again.\"\"\",\n",
" )\n",
"\n",
"\n",
"def check_or_clone_rockchip_repo(repo_url, repo_name=ultralytics_default_folder_name):\n",
" if os.path.exists(repo_name):\n",
" print(\n",
" f'Existing Rockchip repo \"{repo_name}\" detected, skipping installation...'\n",
" )\n",
" else:\n",
" print(f'Cloning Rockchip repo to \"{repo_name}\"')\n",
" run_and_exit_with_error(\n",
" [\"git\", \"clone\", repo_url, repo_name],\n",
" \"Failed to clone Rockchip repo, please see error output\",\n",
" )\n",
"\n",
"\n",
"def run_pip_install_or_else_exit(args):\n",
" print(\"Running pip install...\")\n",
" run_and_exit_with_error(\n",
" [\"pip\", \"install\"] + args,\n",
" \"Pip install rockchip repo failed, please see error output\",\n",
" )\n",
"\n",
"\n",
"def run_onnx_conversion_yolov5(model_path):\n",
" check_or_clone_rockchip_repo(yolov5_repo, ultralytics_folder_name_yolov5)\n",
" run_pip_install_or_else_exit(\n",
" [\n",
" \"-r\",\n",
" os.path.join(ultralytics_folder_name_yolov5, \"requirements.txt\"),\n",
" \"torch<2.6.0\",\n",
" \"onnx==1.18.0\",\n",
" \"onnxscript\",\n",
" ]\n",
" )\n",
"\n",
" model_abspath = os.path.abspath(model_path)\n",
"\n",
" try:\n",
" subprocess.run(\n",
" [\n",
" \"python\",\n",
" f\"{ultralytics_folder_name_yolov5}/export.py\",\n",
" \"--weights\",\n",
" model_abspath,\n",
" \"--rknpu\",\n",
" \"--include\",\n",
" \"onnx\",\n",
" ],\n",
" stderr=subprocess.STDOUT,\n",
" stdout=subprocess.PIPE,\n",
" universal_newlines=True,\n",
" ).check_returncode()\n",
" except subprocess.CalledProcessError as e:\n",
" print(\"Failed to run YOLOv5 export, please see error output\")\n",
"\n",
" if \"ModuleNotFoundError\" in e.stdout and \"ultralytics\" in e.stdout:\n",
" print_bad_model_msg(\n",
" \"It seems the YOLOv5 repo could not find an ultralytics installation.\"\n",
" )\n",
" elif \"AttributeError\" in e.stdout and \"_register_detect_seperate\" in e.stdout:\n",
" print_bad_model_msg(\"It seems that you received a model attribute error.\")\n",
" else:\n",
" print(\"Unknown Error when converting:\")\n",
" print(e.stdout)\n",
"\n",
" sys.exit(1)\n",
"\n",
"\n",
"def run_onnx_conversion_no_anchor(model_path):\n",
" check_or_clone_rockchip_repo(yolo_non_anchor_repo)\n",
" run_pip_install_or_else_exit(\n",
" [\"-e\", ultralytics_default_folder_name, \"onnx==1.18.0\", \"onnxscript\"]\n",
" )\n",
"\n",
" sys.path.insert(0, os.path.abspath(ultralytics_default_folder_name))\n",
" model_abs_path = os.path.abspath(model_path)\n",
"\n",
" from ultralytics import YOLO\n",
"\n",
" try:\n",
" model = YOLO(model_abs_path)\n",
" model.export(format=\"rknn\")\n",
" except TypeError as e:\n",
" if \"originally trained\" in str(e):\n",
" print_bad_model_msg(\n",
" \"Ultralytics has detected that this model is a YOLOv5 model.\"\n",
" )\n",
" else:\n",
" raise e\n",
"\n",
" sys.exit(1)\n",
"\n",
"\n",
"def create_onnx(model_path: str, version: str):\n",
" check_git_installed()\n",
"\n",
" if not version in valid_yolo_versions:\n",
" print(f\"YOLO version \\\"{version}\\\" is not a valid version! Valid versions are: {', '.join(valid_yolo_versions)}\")\n",
"\n",
" try:\n",
" if version.lower() == \"yolov5\":\n",
" run_onnx_conversion_yolov5(model_path)\n",
" else:\n",
" run_onnx_conversion_no_anchor(model_path)\n",
"\n",
" print(\n",
" \"Model export finished. Please use the generated ONNX file to convert to RKNN.\"\n",
" )\n",
" except SystemExit:\n",
" raise RuntimeError(\"Model export failed. Please see output above.\")\n",
"\n",
"# RKNN Conversion code\n",
"\n",
"image_extensions = (\".jpg\", \".jpeg\", \".png\", \".bmp\", \".gif\", \".tiff\", \".webp\")\n",
"DEFAULT_PLATFORM = \"rk3588\"\n",
"\n",
"\n",
"def list_img_dir(img_dir):\n",
" return [\n",
" os.path.abspath(os.path.join(img_dir, f))\n",
" for f in os.listdir(img_dir)\n",
" if f.lower().endswith(image_extensions)\n",
" ]\n",
"\n",
"\n",
"def sample_imgs(num, img_list):\n",
" if len(img_list) < num:\n",
" return img_list\n",
" else:\n",
" return random.sample(img_list, num)\n",
"\n",
"\n",
"def get_image_list_from_dataset(num_imgs, yaml_dir):\n",
" print(f\"Dataset detected with {yaml_dir} file\")\n",
" img_raw_paths = []\n",
"\n",
" with open(yaml_dir, \"r\") as yaml_file:\n",
" for line in yaml_file:\n",
" line = line.strip()\n",
" if (\n",
" line.startswith(\"train:\")\n",
" or line.startswith(\"val:\")\n",
" or line.startswith(\"test:\")\n",
" ):\n",
" img_raw_paths.append(line.split(\":\", 1)[1].strip())\n",
"\n",
" no_yaml_dir = yaml_dir.replace(\n",
" \"data.yaml\", \"dummy_dir\"\n",
" ) # data.yaml sets dirs one level up\n",
" img_set_paths = []\n",
"\n",
" for img_raw_path in img_raw_paths:\n",
" p = (\n",
" img_raw_path\n",
" if os.path.isabs(img_raw_path)\n",
" else os.path.realpath(os.path.join(no_yaml_dir, img_raw_path))\n",
" )\n",
"\n",
" if os.path.exists(p):\n",
" img_set_paths.append(p)\n",
"\n",
" if len(img_set_paths) < 1:\n",
" return None\n",
"\n",
" all_imgs = [list_img_dir(path) for path in img_set_paths]\n",
"\n",
" for imgs in all_imgs:\n",
" print(len(imgs))\n",
"\n",
" total_imgs = sum(len(group) for group in all_imgs)\n",
"\n",
" sampled_imgs = [\n",
" sample_imgs(round((len(group) / total_imgs) * num_imgs), group)\n",
" for group in all_imgs\n",
" ]\n",
"\n",
" return [img for group in sampled_imgs for img in group]\n",
"\n",
"\n",
"def get_image_list_from_img_dir(num_imgs, img_dir):\n",
" return sample_imgs(num_imgs, list_img_dir(img_dir))\n",
"\n",
"\n",
"def get_image_list(num_imgs, image_dir):\n",
" yaml_path = os.path.join(image_dir, \"data.yaml\")\n",
"\n",
" if os.path.exists(yaml_path):\n",
" return get_image_list_from_dataset(num_imgs, yaml_path)\n",
" else:\n",
" return get_image_list_from_img_dir(num_imgs, image_dir)\n",
"\n",
"\n",
"def run_rknn_conversion(\n",
" img_list_txt, disable_quant, model_path, rknn_output, verbose_logging\n",
"):\n",
" from rknn.api import RKNN\n",
"\n",
" rknn = RKNN(\n",
" verbose=verbose_logging,\n",
" verbose_file=(\"rknn_convert.log\" if verbose_logging else None),\n",
" )\n",
"\n",
" rknn.config(\n",
" mean_values=[[0, 0, 0]],\n",
" std_values=[[255, 255, 255]],\n",
" target_platform=DEFAULT_PLATFORM,\n",
" )\n",
"\n",
" print(\"Attempted RKNN load\")\n",
" ret = rknn.load_onnx(model=model_path)\n",
" if ret != 0:\n",
" print(\"Loading model failed!\")\n",
" exit(ret)\n",
"\n",
" print(\"Attempting RKNN build\")\n",
" ret = rknn.build(do_quantization=(not disable_quant), dataset=img_list_txt)\n",
" if ret != 0:\n",
" print(\"Building model failed!\")\n",
" exit(ret)\n",
"\n",
" print(\"Build succeeded! Starting export...\")\n",
" ret = rknn.export_rknn(rknn_output)\n",
" if ret != 0:\n",
" print(\"Exporting model failed!\")\n",
" exit(ret)\n",
" print(\"Finished export!\")\n",
"\n",
" # Release\n",
" rknn.release()\n",
"\n",
" print(f'Your model is in \"{rknn_output}\" and ready to use!')\n",
"\n",
"\n",
"def create_rknn(\n",
" model_path: str,\n",
" rknn_output: str = \"out.rknn\",\n",
" num_imgs: int = 300,\n",
" img_dir: str = None,\n",
" img_dataset_txt: str = \"imgs.txt\",\n",
" disable_quantize: bool = False,\n",
" verbose: bool = False,\n",
"):\n",
" if not rknn_output.endswith(\".rknn\"):\n",
" print(\"RKNN output path must end in .rknn!\")\n",
" return\n",
"\n",
" if not disable_quantize:\n",
" if img_dir is None or len(img_dir) < 1:\n",
" print(f\"Must specify list of images to use with --img_dir\")\n",
" return\n",
"\n",
" img_dir_abs = os.path.abspath(img_dir)\n",
"\n",
" img_list = get_image_list(num_imgs, img_dir_abs)\n",
" img_list_len = 0 if img_list is None else len(img_list)\n",
"\n",
" if img_list_len == 0:\n",
" print(f\"No images found in {img_dir_abs}\")\n",
" return\n",
" elif img_list_len < num_imgs:\n",
" print(\n",
" f\"Not enough images in your dataset/directory, you have {img_list_len} images, but need {num_imgs}\"\n",
" )\n",
" return\n",
"\n",
" if not img_dataset_txt.endswith(\".txt\"):\n",
" print(f\"Image dataset text file path must end in .txt\")\n",
" return\n",
"\n",
" with open(img_dataset_txt, \"w\") as set_file:\n",
" set_file.writelines(f\"{img}\\n\" for img in img_list)\n",
"\n",
" try:\n",
" run_rknn_conversion(\n",
" img_dataset_txt,\n",
" disable_quantize,\n",
" model_path,\n",
" rknn_output,\n",
" verbose,\n",
" )\n",
" except SystemExit:\n",
" print(\"RKNN Conversion failed, see output above\")\n"
],
"id": "ea6869140a61126d"
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"#### *Numpy Fix* - Important for Google Colab Users\n",
"\n",
"Google Colab comes with an incompatible version of Numpy installed. To fix this, please run the following cells below and **restart your session** when prompted."
],
"id": "b3a9e1a334bce144"
},
{
"metadata": {},
"cell_type": "code",
"outputs": [],
"execution_count": null,
"source": [
"%pip uninstall numpy -y\n",
"%pip install \"numpy>=1.23.0,<2.0.0\""
],
"id": "7156e69495f48f49"
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"### Step 1: Convert to ONNX\n",
"\n",
"To convert to ONNX, simply run the `create_onnx` function, providing the path to your model weights and specifying the model version, as shown below."
],
"id": "332942d8582c9bae"
},
{
"metadata": {},
"cell_type": "code",
"outputs": [],
"execution_count": null,
"source": "create_onnx(model_path=\"weights.pt\", version=\"yolov8\") # Valid versions are yolov5, yolov8, and yolov11",
"id": "408440b32de224ed"
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"### Step 2: Download RKNN API\n",
"\n",
"You can either use `pip` below to automatically detect and install the correct RKNN API Python library for you, or install it manually.\n",
"\n",
"#### Automatic installation\n",
"\n",
"Please run `pip` below. If it does not work, refer to the instructions for manual installation. You may need to restart your session after running the command below.\n"
],
"id": "ebd148faaa2b1933"
},
{
"metadata": {},
"cell_type": "code",
"outputs": [],
"execution_count": null,
"source": "%pip install rknn-toolkit2",
"id": "7c7ef3010c663fc2"
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"#### Manual Installation (If Automatic Installation Fails)\n",
"Visit the [RKNN Toolkit 2](https://github.com/airockchip/rknn-toolkit2) Github repository, then click on rknn-toolkit2, followed by packages.\n",
"If you are running an x86_64 CPU (e.g., most Intel and AMD processors), select that option; otherwise, choose arm64 for ARM-based computers (e.g., M-series Macs or Snapdragon processors). If you're unsure which CPU you're using, check your system settings for processor architecture information.\n",
"\n",
"Once you've selected the correct CPU architecture, you'll see multiple packages. The file names will look something like:\n",
"`rknn_toolkit2-2.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`.\n",
"The numbers after cp correspond to your Python version. For example, if you're using Python 3.10, look for a package with cp310 in the name. For Python 3.8, look for cp38; for Python 3.7, cp37, and so on.\n",
"\n",
"Once you've found the correct package, click the \"Raw\" button to download the .whl file. Then, run the following command in your terminal, replacing rknn_toolkit2.whl with the actual path to the file you downloaded:"
],
"id": "f44b71c5a820ab2"
},
{
"metadata": {},
"cell_type": "code",
"outputs": [],
"execution_count": null,
"source": "%pip install rknn_toolkit2.whl",
"id": "c9b9d3da532eb916"
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"### Step 3: Convert to RKNN\n",
"\n",
"Please review the notes about quantization before running the RKNN conversion."
],
"id": "fc74ebc99b596f76"
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"### Quantization\n",
"\n",
"When performing quantization, it is critical to provide representative images of the objects or scenes you are trying to detect. These images are used to calibrate the model’s internal activations and greatly influence the final performance.\n",
"\n",
"It is recommended to use 300–500 representative images that reflect the real-world input your model will encounter. As the old saying goes, it’s quality over quantity — having a diverse, relevant set matters more than simply having many images.\n",
"\n",
"Quantization will cause some loss in model accuracy. However, if your calibration images are chosen wisely, this accuracy drop should be minimal and acceptable. If the sampled images are too uniform or unrelated, your quantized model's performance may worsen significantly.\n",
"\n",
"The script will automatically sample representative images randomly from the provided dataset. While this usually works well, please verify that the dataset contains diverse and relevant examples of your target objects. As a reminder, the images used to quantize the model are stored in the text file specified by `--img_dataset_txt`.\n"
],
"id": "80cf63a7f16f9af"
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"### Optional: Download a dataset from Roboflow for quantization\n",
"\n",
"If you do not already have a dataset or set of images containing the objects you want to detect, follow the steps below to download one from Roboflow Universe.\n",
"\n",
"#### **Step 1: Search for a Dataset**\n",
"\n",
"Go to [Roboflow Universe](https://universe.roboflow.com) and use the search bar to locate a dataset relevant to what you want to detect.\n",
"**Note:** The dataset must include the classes or object types you intend to detect.\n",
"\n",
"#### **Step 2: Access the Dataset Tab**\n",
"\n",
"After selecting a suitable project, navigate to the **Dataset** tab. Click the **\"Download Dataset\"** button. A prompt will appear with several options, including:\n",
"\n",
"- Train a model with this dataset\n",
"- Train from a portion of this dataset\n",
"- Download dataset\n",
"\n",
"Select **Download dataset**.\n",
"\n",
"#### **Step 3: Choose Format and View Download Code**\n",
"\n",
"- Under **Image and Annotation Format**, choose the version of YOLO you are using:\n",
" - For **YOLOv5**, choose `YOLOv5 PyTorch`\n",
" - For **YOLOv8**, choose `YOLOv8`\n",
" - For **YOLOv11**, choose `YOLOv11`\n",
"- If multiple annotation formats are listed for your model, always select the one ending in **\"PyTorch\"**.\n",
"\n",
"Then, under **Download Options**, click **\"Show Download Code\"** and continue.\n",
"\n",
"In the resulting screen, you will see three tabs:\n",
"- **Jupyter**\n",
"- **Terminal**\n",
"- **Raw URL**\n",
"\n",
"Select the **Terminal** tab and copy the provided command.\n",
"\n",
"#### **Step 4: Paste and Run**\n",
"\n",
"Paste the copied command into the notebook cell below and run it. This will download and extract the dataset into your environment, making it ready for use in the quantization process.\n",
"\n",
"Make sure to prefix the command with \"`!`\" so it executes properly in this Jupyter Notebook environment."
],
"id": "ae229bef78e3bc0c"
},
{
"metadata": {},
"cell_type": "code",
"outputs": [],
"execution_count": null,
"source": "!curl -L \"https://universe.roboflow.com/ds/FaF3HbDmF7?key=iMoJR25O9H\" > roboflow.zip; unzip roboflow.zip; rm roboflow.zip",
"id": "e16fb4f928fd956b"
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"### RKNN Conversion Script\n",
"To get started, run the `create_rknn` script below, replacing the arguments with your own values. Refer to the table below for detailed information on each argument’s purpose and usage. The `model_path` argument should point to your exported ONNX model from Step 1, and `img_dir` must reference a valid directory containing either a dataset or a set of images to be used for quantization.\n",
"\n",
"#### Overview of the `create_rknn` function\n",
"\n",
"This script converts a YOLO ONNX model to RKNN format using a set of calibration images. It's designed to work with either:\n",
"\n",
"- A flat directory of images (e.g. `train/images`), **or**\n",
"- A dataset directory containing a `data.yaml` file that defines `train`, `val`, and/or `test` folders.\n",
"\n",
"##### Arguments\n",
"\n",
"| Argument | Type | Description |\n",
"|----------|------|-----------------------------------------------------------------------------------------------------------------|\n",
"| `img_dir` | `str` (required) | Path to your image directory. This can either be a folder of images **or** a dataset folder with a `data.yaml`. |\n",
"| `model_path` | `str` (required) | Path to your YOLO ONNX model, created in Step 1. |\n",
"| `num_imgs` | `int` (default: `300`) | Number of images to use for quantization calibration. |\n",
"| `disable_quantize` | `bool` (default: `False`) | Set to `True` to skip quantization entirely. Not recommended for performance, and should not be used for deployment on PhotonVision, which requires quantization. |\n",
"| `rknn_output` | `str` (default: `out.rknn`) | File path where the final RKNN model should be saved. |\n",
"| `img_dataset_txt` | `str` (default: `imgs.txt`) | File path to store the list of images used during quantization. |\n",
"| `verbose` | `bool` (default: `False`) | Enable detailed logging from the RKNN API during conversion. |\n",
"\n",
"\n",
"##### *Notes*\n",
"\n",
"1. This script is designed for use with [PhotonVision](https://photonvision.org), and by default sets the target platform for RKNN conversion to `RK3588`, a chipset commonly found in many variants of the Orange Pi 5 series (e.g., Orange Pi 5, 5 Pro, 5 Plus, 5 Max, etc.). You may modify the `DEFAULT_PLATFORM` value in the setup cell to match your specific hardware or deployment requirements if necessary.\n",
"\n",
"2. If you followed the Roboflow dataset download instructions from the previous section, the dataset will have been extracted to your **current working directory**. In that case, you can simply set `img_dir` to \"`.`\" to reference the current directory."
],
"id": "f8f48b3139509618"
},
{
"metadata": {},
"cell_type": "code",
"outputs": [],
"execution_count": null,
"source": "create_rknn(img_dir=\"./datasets\", model_path=\"weights.onnx\")",
"id": "2c48b133f5c93c7a"
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

160
scripts/rubik_conversion.ipynb
View File

@@ -1,82 +1,84 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "1tMAqVl4p58r"
},
"source": [
"## YOLO to Rubik TFlite Conversion"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nAbygyUYp58s"
},
"source": [
"#### Requirements\n",
"\n",
"This notebook can be run on Colab. However, Colab has some incompatibility issues that result in needing to restart the notebook in the middle of the run. This is normal, and after restarting you should rerun the below cell.\n",
"\n",
"Prior to running the notebook, it is necessary to make an account on [Qualcomm's AI Hub](https://app.aihub.qualcomm.com/account/), and obtain your API token. Then, replace <YOUR_API_TOKEN> with your API token in the cell below.\n",
"\n",
"Documentation for the Qualcomm AI Hub can be found [here](https://app.aihub.qualcomm.com/docs/index.html).\n",
"\n",
"You should also have a PyTorch model (ending in `.pt`) that's been uploaded to the runtime that you intend to convert. After uploading, copy it's absolute path by right-clicking on the file, and replace /PATH/TO/WEIGHTS.\n",
"\n",
"**NOTE: your API key will be listed in the output, and should therefore be redacted if the output is shared.**\n",
"\n",
"Once the run has finished, open the AI Hub link, and download the tflite model for the job you just ran.\n",
"\n",
"If you want to use this notebook to convert a yolo11 model, you'll need to replace all instances of `yolov8` in the cell below with `yolov11`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"id": "aX3JcSFKp58s",
"outputId": "f2cdadd2-c448-4d8c-c681-c19decef7f3e"
},
"outputs": [],
"source": [
"# This installs Python package\n",
"!pip install qai-hub-models[yolov8_det]\n",
"# sets up AI Hub enviroment\n",
"!qai-hub configure --api_token <YOUR_API_TOKEN>\n",
"# Converts the model to be ran on RB3Gen2\n",
"!yes | python -m qai_hub_models.models.yolov8_det.export --quantize w8a8 --device=\"RB3 Gen 2 (Proxy)\" --ckpt-name /PATH/TO/WEIGHTS --device-os linux --target-runtime tflite --output-dir .\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0I2cXQO4p58s"
},
"source": [
"Modified from https://github.com/ramalamadingdong/yolo-rb3gen2-trainer/blob/main/AI_Hub_Quanitization_RB3Gen2.ipynb"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.11.7"
}
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "1tMAqVl4p58r"
},
"source": [
"## YOLO to Rubik TFlite Conversion"
]
},
"nbformat": 4,
"nbformat_minor": 0
{
"cell_type": "markdown",
"metadata": {
"id": "nAbygyUYp58s"
},
"source": [
"#### Requirements\n",
"\n",
"This notebook can be run on Colab. However, Colab has some incompatibility issues that result in needing to restart the notebook in the middle of the run. This is normal, and after restarting you should rerun the below cell.\n",
"\n",
"If you aren't using Google Colab, we recommend creating a [Python venv](https://docs.python.org/3/library/venv.html) so that the packages installed for conversion do not conflict with your existing setup.\n",
"\n",
"Prior to running the notebook, it is necessary to make an account on [Qualcomm's AI Hub](https://app.aihub.qualcomm.com/account/), and obtain your API token. Then, replace <YOUR_API_TOKEN> with your API token in the cell below.\n",
"\n",
"Documentation for the Qualcomm AI Hub can be found [here](https://app.aihub.qualcomm.com/docs/index.html).\n",
"\n",
"You should also have a PyTorch model (ending in `.pt`) that's been uploaded to the runtime that you intend to convert. After uploading, copy it's absolute path by right-clicking on the file, and replace /PATH/TO/WEIGHTS.\n",
"\n",
"**NOTE: your API key will be listed in the output, and should therefore be redacted if the output is shared.**\n",
"\n",
"Once the run has finished, open the AI Hub link, and download the tflite model for the job you just ran.\n",
"\n",
"If you want to use this notebook to convert a yolo11 model, you'll need to replace all instances of `yolov8` in the cell below with `yolov11`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"id": "aX3JcSFKp58s",
"outputId": "f2cdadd2-c448-4d8c-c681-c19decef7f3e"
},
"outputs": [],
"source": [
"# This installs Python package\n",
"!pip install qai-hub-models[yolov8_det]\n",
"# sets up AI Hub enviroment\n",
"!qai-hub configure --api_token <YOUR_API_TOKEN>\n",
"# Converts the model to be ran on RB3Gen2\n",
"!yes | python -m qai_hub_models.models.yolov8_det.export --quantize w8a8 --device=\"RB3 Gen 2 (Proxy)\" --ckpt-name /PATH/TO/WEIGHTS --device-os linux --target-runtime tflite --output-dir .\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0I2cXQO4p58s"
},
"source": [
"Modified from https://github.com/ramalamadingdong/yolo-rb3gen2-trainer/blob/main/AI_Hub_Quanitization_RB3Gen2.ipynb"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

Some files were not shown because too many files have changed in this diff Show More