Team2890/PhotonVision
4
0
Fork 0
mirror of https://github.com/PhotonVision/photonvision synced 2026-06-23 01:21:40 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into py-docs

Browse Source
This commit is contained in:
Sam Freund
2025-10-23 16:14:46 -05:00
committed by GitHub
parent 46e71703ef 7cb3b7a37b
commit 99ca8228a1
223 changed files with 12559 additions and 11613 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
docs/source/conf.py
View File

@@ -30,7 +30,6 @@ extensions = [
"sphinx_rtd_theme",
"sphinx.ext.autosectionlabel",
"sphinx.ext.todo",
"sphinx_tabs.tabs",
"notfound.extension",
"sphinxext.remoteliteralinclude",
"sphinxext.opengraph",
@@ -67,6 +66,10 @@ html_title = "PhotonVision Docs"
html_theme = "furo"
html_favicon = "assets/RoundLogo.png"
# Specify canonical root
# This tells search engines that this domain is preferred
html_baseurl = "https://docs.photonvision.org/en/latest/"
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
@@ -144,7 +147,11 @@ sphinx_tabs_valid_builders = ["epub", "linkcheck"]
# Excluded links for linkcheck
# These should be periodically checked by hand to ensure that they are still functional
linkcheck_ignore = [R"https://www.raspberrypi.com/software/", R"http://10\..+"]
linkcheck_ignore = [
R"https://www.raspberrypi.com/software/",
R"http://10\..+",
R"https://www.gnu.org/",
]
token = os.environ.get("GITHUB_TOKEN", None)
if token:

6
docs/source/docs/apriltag-pipelines/multitag.md
View File

@@ -28,7 +28,7 @@ This multi-target pose estimate can be accessed using PhotonLib. We suggest usin
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
var results = camera.getAllUnreadResults();
for (var result : results) {
@@ -39,7 +39,7 @@ This multi-target pose estimate can be accessed using PhotonLib. We suggest usin
}
.. code-block:: C++
.. code-block:: c++
auto results = camera.GetAllUnreadResults();
for (auto &result : results)
@@ -51,7 +51,7 @@ This multi-target pose estimate can be accessed using PhotonLib. We suggest usin
}
.. code-block:: Python
.. code-block:: python
results = camera.getAllUnreadResults()
for result in results:

8
docs/source/docs/benchmarks/index.md Normal file
View File

@@ -0,0 +1,8 @@
# Performance Benchmarks
```{toctree}
:maxdepth: 0
:titlesonly: true
rknn-model-benchmarks
```

125
docs/source/docs/benchmarks/rknn-model-benchmarks.md Normal file
View File

@@ -0,0 +1,125 @@
# RKNN Benchmarks
## Description
This benchmark compares the performance of four object detection models: YOLOv5, YOLOv5u, YOLOv8, and YOLOv11 on the [COCO 2017 Validation Set](http://images.cocodataset.org/zips/val2017.zip). The main purpose is to assess and compare the inference speed and detection accuracy of these models when deployed on the Orange Pi devices using the RKNN framework and int8 quantization.
## Methodology
- **Dataset**: [COCO 2017 Validation Set](http://images.cocodataset.org/zips/val2017.zip) (5,000 images)
- **Platform**: Orange Pi 5 with RK3588
- **Quantization**: int8 using 20 randomly selected images from the validation set
- **Framework**: RKNN Toolkit 2
## Operator-Level Benchmark Results
The following tables break down the average CPU time, NPU time, and total execution time (in microseconds) for each operator used by the models. Each value represents the mean ± standard deviation across 5,000 inferences.
### YOLOv5
| OpType | CPU Time (μs) | NPU Time (μs) | Total Time (μs) | Time Ratio (%) | Number of Times Called |
|-----------------|---------------------|----------------------|-----------------------|---------------------|-----------------------|
| ConvExSwish | 0.00 ± 0.00 | 10968.81 ± 1126.00 | 10968.81 ± 1126.00 | 73.06 ± 0.94 | 57 |
| ConvSigmoid | 0.00 ± 0.00 | 1243.49 ± 67.66 | 1243.49 ± 67.66 | 8.33 ± 0.57 | 3 |
| Concat | 0.00 ± 0.00 | 1080.68 ± 259.40 | 1080.68 ± 259.40 | 7.09 ± 0.87 | 13 |
| Conv | 0.00 ± 0.00 | 732.15 ± 29.42 | 732.15 ± 29.42 | 4.92 ± 0.42 | 1 |
| Add | 0.00 ± 0.00 | 473.71 ± 131.48 | 473.71 ± 131.48 | 3.10 ± 0.50 | 7 |
| MaxPool | 0.00 ± 0.00 | 272.40 ± 110.52 | 272.40 ± 110.52 | 1.76 ± 0.51 | 6 |
| Resize | 0.00 ± 0.00 | 147.61 ± 38.89 | 147.61 ± 38.89 | 0.97 ± 0.15 | 2 |
| OutputOperator | 106.60 ± 15.00 | 0.00 ± 0.00 | 106.60 ± 15.00 | 0.72 ± 0.13 | 3 |
| InputOperator | 8.64 ± 1.79 | 0.00 ± 0.00 | 8.64 ± 1.79 | 0.06 ± 0.02 | 1 |
| **Total** | **115.24 ± 16.16** | **14918.85 ± 1735.45**| **15034.09 ± 1734.28**| | **93** |
### YOLOv5u
| OpType | CPU Time (μs) | NPU Time (μs) | Total Time (μs) | Time Ratio (%) | Number of Times Called |
|-----------------|---------------------|----------------------|-----------------------|---------------------|-----------------------|
| ConvExSwish | 0.00 ± 0.00 | 16828.24 ± 1332.73 | 16828.24 ± 1332.73 | 83.04 ± 1.61 | 69 |
| Concat | 0.00 ± 0.00 | 1265.94 ± 250.24 | 1265.94 ± 250.24 | 6.17 ± 0.69 | 13 |
| ConvSigmoid | 0.00 ± 0.00 | 613.88 ± 62.97 | 613.88 ± 62.97 | 3.03 ± 0.15 | 3 |
| Add | 0.00 ± 0.00 | 553.75 ± 131.17 | 553.75 ± 131.17 | 2.69 ± 0.44 | 7 |
| Conv | 0.00 ± 0.00 | 298.61 ± 72.72 | 298.61 ± 72.72 | 1.45 ± 0.25 | 3 |
| ConvClip | 0.00 ± 0.00 | 256.02 ± 64.48 | 256.02 ± 64.48 | 1.24 ± 0.23 | 3 |
| MaxPool | 0.00 ± 0.00 | 178.68 ± 58.72 | 178.68 ± 58.72 | 0.86 ± 0.23 | 3 |
| Resize | 0.00 ± 0.00 | 170.87 ± 40.14 | 170.87 ± 40.14 | 0.83 ± 0.13 | 2 |
| OutputOperator | 126.89 ± 16.53 | 0.00 ± 0.00 | 126.89 ± 16.53 | 0.63 ± 0.10 | 9 |
| InputOperator | 8.69 ± 1.45 | 0.00 ± 0.00 | 8.69 ± 1.45 | 0.04 ± 0.01 | 1 |
| **Total** | **135.57 ± 17.51** | **20165.99 ± 1963.70**| **20301.56 ± 1965.88**| | **113** |
### YOLOv8
| OpType | CPU Time (μs) | NPU Time (μs) | Total Time (μs) | Time Ratio (%) | Number of Times Called |
|-----------------|---------------------|----------------------|-----------------------|---------------------|-----------------------|
| ConvExSwish | 0.00 ± 0.00 | 13017.04 ± 1165.76 | 13017.04 ± 1165.76 | 75.66 ± 1.96 | 57 |
| Concat | 0.00 ± 0.00 | 1489.94 ± 257.22 | 1489.94 ± 257.22 | 8.58 ± 0.53 | 13 |
| Split | 0.00 ± 0.00 | 681.47 ± 166.62 | 681.47 ± 166.62 | 3.89 ± 0.53 | 8 |
| ConvSigmoid | 0.00 ± 0.00 | 596.08 ± 75.01 | 596.08 ± 75.01 | 3.45 ± 0.18 | 3 |
| Add | 0.00 ± 0.00 | 443.60 ± 118.05 | 443.60 ± 118.05 | 2.53 ± 0.41 | 6 |
| Conv | 0.00 ± 0.00 | 269.61 ± 78.65 | 269.61 ± 78.65 | 1.54 ± 0.30 | 3 |
| Resize | 0.00 ± 0.00 | 236.79 ± 37.74 | 236.79 ± 37.74 | 1.37 ± 0.08 | 2 |
| ConvClip | 0.00 ± 0.00 | 231.82 ± 68.44 | 231.82 ± 68.44 | 1.32 ± 0.27 | 3 |
| MaxPool | 0.00 ± 0.00 | 156.85 ± 56.94 | 156.85 ± 56.94 | 0.89 ± 0.23 | 3 |
| OutputOperator | 124.86 ± 20.74 | 0.00 ± 0.00 | 124.86 ± 20.74 | 0.73 ± 0.15 | 9 |
| InputOperator | 8.47 ± 1.66 | 0.00 ± 0.00 | 8.47 ± 1.66 | 0.05 ± 0.01 | 1 |
| **Total** | **133.33 ± 21.95** | **17123.19 ± 1985.72**| **17256.52 ± 1986.77** | | **108** |
---
### YOLOv11
| OpType | CPU Time (μs) | NPU Time (μs) | Total Time (μs) | Time Ratio (%) | Number of Times Called |
|-----------------|---------------------|----------------------|-----------------------|---------------------|-----------------------|
| ConvExSwish | 0.00 ± 0.00 | 16034.00 ± 1331.95 | 16034.00 ± 1331.95 | 69.90 ± 1.55 | 77 |
| Concat | 0.00 ± 0.00 | 1888.89 ± 293.99 | 1888.89 ± 293.99 | 8.17 ± 0.51 | 17 |
| exSDPAttention | 0.00 ± 0.00 | 1210.88 ± 17.73 | 1210.88 ± 17.73 | 5.32 ± 0.52 | 1 |
| Split | 0.00 ± 0.00 | 908.30 ± 183.92 | 908.30 ± 183.92 | 3.91 ± 0.45 | 10 |
| Add | 0.00 ± 0.00 | 871.64 ± 212.79 | 871.64 ± 212.79 | 3.73 ± 0.60 | 12 |
| ConvSigmoid | 0.00 ± 0.00 | 617.61 ± 59.61 | 617.61 ± 59.61 | 2.69 ± 0.16 | 3 |
| Conv | 0.00 ± 0.00 | 419.72 ± 89.88 | 419.72 ± 89.88 | 1.80 ± 0.24 | 5 |
| Resize | 0.00 ± 0.00 | 272.09 ± 49.91 | 272.09 ± 49.91 | 1.18 ± 0.12 | 2 |
| ConvClip | 0.00 ± 0.00 | 260.08 ± 59.12 | 260.08 ± 59.12 | 1.12 ± 0.18 | 3 |
| MaxPool | 0.00 ± 0.00 | 181.93 ± 53.32 | 181.93 ± 53.32 | 0.78 ± 0.18 | 3 |
| OutputOperator | 131.48 ± 22.93 | 0.00 ± 0.00 | 131.48 ± 22.93 | 0.58 ± 0.12 | 9 |
| ConvAdd | 0.00 ± 0.00 | 126.79 ± 35.28 | 126.79 ± 35.28 | 0.54 ± 0.11 | 2 |
| Reshape | 0.00 ± 0.00 | 56.61 ± 18.03 | 56.61 ± 18.03 | 0.24 ± 0.06 | 3 |
| InputOperator | 8.66 ± 1.59 | 0.00 ± 0.00 | 8.66 ± 1.59 | 0.04 ± 0.01 | 1 |
| **Total** | **140.14 ± 24.26** | **22848.54 ± 2351.95**| **22988.68 ± 2355.97**| | **148** |
## Model Summary and Accuracy Metrics
The table below summarizes the mean average precision (mAP) and total inference time for each model. These metrics provide a high-level view of how each model performs in terms of both detection accuracy and runtime efficiency.
### Mean Average Precision (mAP) by Model
| Metric | YOLOv5 | YOLOv5u | YOLOv8 | YOLOv11 |
|--------|------------|------------|------------|------------|
| **mAP** | 0.2243 | 0.2745 | 0.3051 | 0.3251 |
| **mAP50** | 0.3538 | 0.3834 | 0.4145 | 0.4406 |
| **mAP75** | 0.2432 | 0.2997 | 0.3349 | 0.3568 |
| **mAP85** | 0.3054 | 0.3472 | 0.3867 | 0.4068 |
| **mAP95** | 0.3708 | 0.4822 | 0.5483 | 0.5858 |
### Model Execution Time and Call Frequency
| Model | Total Time (μs) | Number of Processing Calls |
|---------|------------------------|----------------------------|
| **YOLOv5** | 15034.09 ± 1734.28 | 93 |
| **YOLOv5u** | 20301.56 ± 1965.88 | 113 |
| **YOLOv8** | 17256.52 ± 1986.77 | 108 |
| **YOLOv11** | 22988.68 ± 2355.97 | 148 |
## Conclusion
The benchmark reveals a clear performance trade-off between inference time and detection accuracy:
- **YOLOv5** is the fastest model with the lowest total inference time, making it well-suited for situations where speed is more important than high detection precision.
- **YOLOv11** achieves the highest accuracy (mAP) across all IoU thresholds but comes with the longest inference time, which may limit its use in real-time applications.
- **YOLOv8** offers a strong balance between speed and accuracy, making it a practical choice when both factors matter.
- **YOLOv5u** improves accuracy compared to YOLOv5 but falls behind YOLOv8 in both speed and detection quality.
When choosing a model for edge devices like the Orange Pi 5, it’s important to weigh how much latency your system can tolerate versus how much accuracy you need. A faster model may give quicker results, while a more accurate one may offer better detection reliability, but at the cost of speed.

2
docs/source/docs/camera-specific-configuration/index.md
View File

@@ -1,4 +1,4 @@
# Camera-Specifc Configuration
# Camera-Specific Configuration
```{toctree}
:maxdepth: 2

52
docs/source/docs/contributing/building-photon.md
View File

@@ -12,7 +12,11 @@ This section contains the build instructions from the source code available at [
**Node JS:**
The UI is written in Node JS. To compile the UI, Node 22.15.0 is required. To install Node JS follow the instructions for your platform [on the official Node JS website](https://nodejs.org/en/download/).
The UI is written in Node JS. To compile the UI, Node 22 or later is required. To install Node JS, follow the instructions for your platform [on the official Node JS website](https://nodejs.org/en/download/).
**pnpm:**
[pnpm](https://pnpm.io/) is the package manager used to download dependencies for the UI. To install pnpm, follow [the instructions on the official pnpm website](https://pnpm.io/installation).
## Compiling Instructions
@@ -36,27 +40,7 @@ or alternatively download the source code from GitHub and extract the zip:
In the photon-client directory:
```bash
npm install
```
### Build and Copy UI to Java Source
In the root directory:
```{eval-rst}
.. tab-set::
.. tab-item:: Linux
``./gradlew buildAndCopyUI``
.. tab-item:: macOS
``./gradlew buildAndCopyUI``
.. tab-item:: Windows (cmd)
``gradlew buildAndCopyUI``
pnpm install
```
### Using hot reload on the UI
@@ -64,7 +48,7 @@ In the root directory:
In the photon-client directory:
```bash
npm run dev
pnpm run dev
```
This allows you to make UI changes quickly without having to spend time rebuilding the jar. Hot reload is enabled, so changes that you make and save are reflected in the UI immediately. Running this command will give you the URL for accessing the UI, which is on a different port than normal. You must use the printed URL to use hot reload.
@@ -77,14 +61,17 @@ To compile and run the project, issue the following command in the root director
.. tab-set::
.. tab-item:: Linux
:sync: linux
``./gradlew run``
.. tab-item:: macOS
:sync: macos
``./gradlew run``
.. tab-item:: Windows (cmd)
:sync: windows
``gradlew run``
```
@@ -95,21 +82,24 @@ Running the following command under the root directory will build the jar under
.. tab-set::
.. tab-item:: Linux
:sync: linux
``./gradlew shadowJar``
.. tab-item:: macOS
:sync: macos
``./gradlew shadowJar``
.. tab-item:: Windows (cmd)
:sync: windows
``gradlew shadowJar``
```
### Build and Run PhotonVision on a Raspberry Pi Coprocessor
As a convenience, the build has a built-in `deploy` command which builds, deploys, and starts the current source code on a coprocessor.
As a convenience, the build has a built-in `deploy` command which builds, deploys, and starts the current source code on a coprocessor. It uses [deploy-utils](https://github.com/wpilibsuite/deploy-utils/blob/main/README.md), so it works very similarly to deploys on robot projects.
An architecture override is required to specify the deploy target's architecture.
@@ -117,18 +107,21 @@ An architecture override is required to specify the deploy target's architecture
.. tab-set::
.. tab-item:: Linux
:sync: linux
``./gradlew clean``
``./gradlew deploy -PArchOverride=linuxarm64``
.. tab-item:: macOS
:sync: macos
``./gradlew clean``
``./gradlew deploy -PArchOverride=linuxarm64``
.. tab-item:: Windows (cmd)
:sync: windows
``gradlew clean``
@@ -147,14 +140,17 @@ The photonlib source can be published to your local maven repository after build
.. tab-set::
.. tab-item:: Linux
:sync: linux
``./gradlew publishToMavenLocal``
.. tab-item:: macOS
:sync: macos
``./gradlew publishToMavenLocal``
.. tab-item:: Windows (cmd)
:sync: windows
``gradlew publishToMavenLocal``
```
@@ -197,7 +193,7 @@ Similarly, a local instance of PhotonVision can be debugged in the same way usin
Set up a VSCode configuration in {code}`launch.json`
```
```json
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
@@ -279,3 +275,9 @@ Using the [GitHub CLI](https://cli.github.com/), we can download artifacts from
MacOS builds are not published to releases as MacOS is not an officially
supported platform. However, MacOS builds are still available from the MacOS
build action, which can be found [here](https://github.com/PhotonVision/photonvision/actions/workflows/build.yml).
#### Forcing Object Detection in the UI
In order to force the Object Detection interface to be visible, it's necessary to hardcode the platform that `Platform.java` returns. This can be done by changing the function that detects the RK3588S/QCS6490 platform to always return true, and changing the `getCurrentPlatform()` function to always return the RK3588S/QCS6490 architecture.
Alternatively, it's possible to modify the frontend code by changing all instances of `useSettingsStore().general.supportedBackends.length > 0` to `true`, which will force the card to render.
Make sure to revert these changes before submitting a Pull Request.

5
docs/source/docs/examples/aimandrange.md
View File

@@ -14,8 +14,10 @@ To do this, we'll use the _pitch_ of the target in the camera image and trigonom
```{eval-rst}
.. tab-set::
:sync-group: code
.. tab-item:: Java
:sync: java
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/aimandrange/src/main/java/frc/robot/Robot.java
:language: java
@@ -24,6 +26,7 @@ To do this, we'll use the _pitch_ of the target in the camera image and trigonom
:lineno-start: 84
.. tab-item:: C++ (Header)
:sync: c++
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/aimandrange/src/main/include/Robot.h
:language: c++
@@ -32,6 +35,7 @@ To do this, we'll use the _pitch_ of the target in the camera image and trigonom
:lineno-start: 25
.. tab-item:: C++ (Source)
:sync: c++
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/aimandrange/src/main/cpp/Robot.cpp
:language: c++
@@ -40,6 +44,7 @@ To do this, we'll use the _pitch_ of the target in the camera image and trigonom
:lineno-start: 58
.. tab-item:: Python
:sync: python
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-python-examples/aimandrange/robot.py
:language: python

5
docs/source/docs/examples/aimingatatarget.md
View File

@@ -19,8 +19,10 @@ In this example, while the operator holds a button down, the robot will turn tow
```{eval-rst}
.. tab-set::
:sync-group: code
.. tab-item:: Java
:sync: java
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/aimattarget/src/main/java/frc/robot/Robot.java
:language: java
@@ -29,6 +31,7 @@ In this example, while the operator holds a button down, the robot will turn tow
:lineno-start: 77
.. tab-item:: C++ (Header)
:sync: c++
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/aimattarget/src/main/include/Robot.h
:language: c++
@@ -37,6 +40,7 @@ In this example, while the operator holds a button down, the robot will turn tow
:lineno-start: 25
.. tab-item:: C++ (Source)
:sync: c++
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/aimattarget/src/main/cpp/Robot.cpp
:language: c++
@@ -45,6 +49,7 @@ In this example, while the operator holds a button down, the robot will turn tow
:lineno-start: 56
.. tab-item:: Python
:sync: python
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-python-examples/aimattarget/robot.py
:language: python

214
docs/source/docs/examples/poseest.md
View File

@@ -21,32 +21,24 @@ Please reference the [WPILib documentation](https://docs.wpilib.org/en/stable/do
We use the 2024 game's AprilTag Locations:
```{eval-rst}
.. tab-set::
.. tab-set-code::
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Vision.java
:language: java
:lines: 68-68
:linenos:
:lineno-start: 68
.. tab-item:: Java
:sync: java
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/include/Constants.h
:language: c++
:lines: 42-43
:linenos:
:lineno-start: 42
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Vision.java
:language: java
:lines: 68-68
:linenos:
:lineno-start: 68
.. tab-item:: C++
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/include/Constants.h
:language: c++
:lines: 42-43
:linenos:
:lineno-start: 42
.. tab-item:: Python
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-python-examples/poseest/robot.py
:language: python
:lines: 46-46
:linenos:
:lineno-start: 46
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-python-examples/poseest/robot.py
:language: python
:lines: 46-46
:linenos:
:lineno-start: 46
```
@@ -56,63 +48,47 @@ To incorporate PhotonVision, we need to create a {code}`PhotonCamera`:
```{eval-rst}
.. tab-set::
.. tab-set-code::
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Vision.java
:language: java
:lines: 57-57
:linenos:
:lineno-start: 57
.. tab-item:: Java
:sync: java
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/include/Vision.h
:language: c++
:lines: 145-145
:linenos:
:lineno-start: 145
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Vision.java
:language: java
:lines: 57-57
:linenos:
:lineno-start: 57
.. tab-item:: C++
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/include/Vision.h
:language: c++
:lines: 145-145
:linenos:
:lineno-start: 145
.. tab-item:: Python
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-python-examples/poseest/robot.py
:language: python
:lines: 44-44
:linenos:
:lineno-start: 44
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-python-examples/poseest/robot.py
:language: python
:lines: 44-44
:linenos:
:lineno-start: 44
```
During periodic execution, we read back camera results. If we see AprilTags in the image, we calculate the camera-measured pose of the robot and pass it to the {code}`Drivetrain`.
```{eval-rst}
.. tab-set::
.. tab-set-code::
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Robot.java
:language: java
:lines: 64-74
:linenos:
:lineno-start: 64
.. tab-item:: Java
:sync: java
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/cpp/Robot.cpp
:language: c++
:lines: 38-46
:linenos:
:lineno-start: 38
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Robot.java
:language: java
:lines: 64-74
:linenos:
:lineno-start: 64
.. tab-item:: C++
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/cpp/Robot.cpp
:language: c++
:lines: 38-46
:linenos:
:lineno-start: 38
.. tab-item:: Python
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-python-examples/poseest/robot.py
:language: python
:lines: 54-56
:linenos:
:lineno-start: 54
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-python-examples/poseest/robot.py
:language: python
:lines: 54-56
:linenos:
:lineno-start: 54
```
@@ -121,56 +97,45 @@ During periodic execution, we read back camera results. If we see AprilTags in t
First, we create a new {code}`VisionSystemSim` to represent our camera and coprocessor running PhotonVision, and moving around our simulated field.
```{eval-rst}
.. tab-set::
.. tab-set-code::
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Vision.java
:language: java
:lines: 65-69
:linenos:
:lineno-start: 65
.. tab-item:: Java
:sync: java
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/include/Vision.h
:language: c++
:lines: 49-52
:linenos:
:lineno-start: 49
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Vision.java
:language: java
:lines: 65-69
:linenos:
:lineno-start: 65
.. code-block:: python
.. tab-item:: C++
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/include/Vision.h
:language: c++
:lines: 49-52
:linenos:
:lineno-start: 49
.. tab-item:: Python
# Coming Soon!
# Coming Soon!
```
Then, we add configure the simulated vision system to match the camera system being simulated.
```{eval-rst}
.. tab-set::
.. tab-set-code::
.. tab-item:: Java
:sync: java
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Vision.java
:language: java
:lines: 69-82
:linenos:
:lineno-start: 69
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Vision.java
:language: java
:lines: 69-82
:linenos:
:lineno-start: 69
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/include/Vision.h
:language: c++
:lines: 53-65
:linenos:
:lineno-start: 53
.. tab-item:: C++
.. code-block:: python
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/include/Vision.h
:language: c++
:lines: 53-65
:linenos:
:lineno-start: 53
.. tab-item:: Python
# Coming Soon!
# Coming Soon!
```
@@ -179,28 +144,23 @@ Then, we add configure the simulated vision system to match the camera system be
During simulation, we periodically update the simulated vision system.
```{eval-rst}
.. tab-set::
.. tab-set-code::
.. tab-item:: Java
:sync: java
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Robot.java
:language: java
:lines: 114-132
:linenos:
:lineno-start: 114
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-java-examples/poseest/src/main/java/frc/robot/Robot.java
:language: java
:lines: 114-132
:linenos:
:lineno-start: 114
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/cpp/Robot.cpp
:language: c++
:lines: 95-109
:linenos:
:lineno-start: 95
.. tab-item:: C++
.. code-block:: python
.. rli:: https://raw.githubusercontent.com/PhotonVision/photonvision/abe95dfaa055bbe3609f72cfcaaba0f96ee7978c/photonlib-cpp-examples/poseest/src/main/cpp/Robot.cpp
:language: c++
:lines: 95-109
:linenos:
:lineno-start: 95
.. tab-item:: Python
# Coming Soon!
# Coming Soon!
```
The rest is done behind the scenes.

2
docs/source/docs/integration/advancedStrategies.md
View File

@@ -43,7 +43,7 @@ A simple way to use a pose estimate is to activate robot functions automatically
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
Pose3d robotPose;
boolean launcherSpinCmd;

36
docs/source/docs/objectDetection/about-object-detection.md
View File

@@ -2,9 +2,9 @@
## How does it work?
PhotonVision supports object detection using neural network accelerator hardware built into Orange Pi 5/5+ coprocessors. Please note that the Orange Pi 5/5+ are the only coprocessors that are currently supported. The Neural Processing Unit, or NPU, is [used by PhotonVision](https://github.com/PhotonVision/rknn_jni/tree/main) to massively accelerate certain math operations like those needed for running ML-based object detection.
PhotonVision supports object detection using neural network accelerator hardware, commonly known as an NPU. The two coprocessors currently supported are the {ref}`Orange Pi 5 <docs/objectDetection/opi:Orange Pi 5 (and variants) Object Detection>` and the {ref}`Rubik Pi 3 <docs/objectDetection/rubik:Rubik Pi 3 Object Detection>`.
For the 2025 season, PhotonVision ships with a pretrained ALGAE model. A model to detect coral is not currently stable, and interested teams should ask in the Photonvision discord.
PhotonVision currently ships with a model trained on the [COCO dataset](https://cocodataset.org/) by [Ultralytics](https://github.com/ultralytics/ultralytics) (this model is licensed under [AGPLv3](https://www.gnu.org/licenses/agpl-3.0.en.html)). This model is meant to be used for testing and other miscellaneous purposes. It is not meant to be used in competition. For the 2025 post-season, PhotonVision also ships with a pretrained ALGAE model. A model to detect coral is available in the PhotonVision discord, but will not be distributed with PhotonVision.
## Tracking Objects
@@ -18,7 +18,7 @@ This model output means that while its fairly easy to say that "this rectangle p
## Tuning and Filtering
Compared to other pipelines, object detection exposes very few tuning handles. The Confidence slider changes the minimum confidence that the model needs to have in a given detection to consider it valid, as a number between 0 and 1 (with 0 meaning completely uncertain and 1 meaning maximally certain).
Compared to other pipelines, object detection exposes very few tuning handles. The Confidence slider changes the minimum confidence that the model needs to have in a given detection to consider it valid, as a number between 0 and 1 (with 0 meaning completely uncertain and 1 meaning maximally certain). The Non-Maximum Suppresion (NMS) Threshold slider is used to filter out overlapping detections. Higher values mean more detections are allowed through, but may result in false positives. It's generally recommended that teams leave this set at the default, unless they find they're unable to get usable results with solely the Confidence slider.
```{raw} html
<video width="85%" controls>
@@ -33,31 +33,19 @@ The same area, aspect ratio, and target orientation/sort parameters from {ref}`r
Photonvision will letterbox your camera frame to 640x640. This means that if you select a resolution that is larger than 640 it will be scaled down to fit inside a 640x640 frame with black bars if needed. Smaller frames will be scaled up with black bars if needed.
## Training Custom Models
It is recommended that you select a resolution that results in the smaller dimension being just greater than, or equal to, 640. Anything above this will not see any increased performance.
:::{warning}
Power users only. This requires some setup, such as obtaining your own dataset and installing various tools. It's additionally advised to have a general knowledge of ML before attempting to train your own model. Additionally, this is not officially supported by Photonvision, and any problems that may arise are not attributable to Photonvision.
:::
## Custom Models
Before beginning, it is necessary to install the [rknn-toolkit2](https://github.com/airockchip/rknn-toolkit2). Then, install the relevant [Ultralytics repository](https://github.com/airockchip?tab=repositories&q=yolo&type=&language=&sort=) from this list. After training your model, export it to `rknn`. This will give you an `onnx` file, formatted for conversion. Copy this file to the relevant folder in [rknn_model_zoo](https://github.com/airockchip/rknn_model_zoo), and use the conversion script located there to convert it. If necessary, modify the script to provide the path to your training database for quantization.
For information regarding converting custom models and supported models for each platform, refer to the page detailing information about your specific coprocessor.
## Uploading Custom Models
- {ref}`Orange Pi 5 <docs/objectDetection/opi:Orange Pi 5 (and variants) Object Detection>`
- {ref}`Rubik Pi 3 <docs/objectDetection/rubik:Rubik Pi 3 Object Detection>`
:::{warning}
PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv5, YOLOv8, and YOLOv11 models trained and converted to `.rknn` format for RK3588 CPUs! Other models require different post-processing code and will NOT work. The model conversion process is also highly particular. Proceed with care.
:::
### Training Custom Models
:::{warning}
Non-quantized models are not supported! If you have the option, make sure quantization is enabled when exporting to .rknn format. This will represent the weights and activations of the model as 8-bit integers, instead of 32-bit floats which PhotonVision doesn't support. Quantized models are also much faster.
:::
PhotonVision does not offer any support for training custom models, only conversion. For information on which models are supported for a given coprocessor, use the links above.
In the settings, under `Device Control`, there's an option to upload a new object detection model. Naming convention
should be `name-verticalResolution-horizontalResolution-yolovXXX`. The
`name` should only include alphanumeric characters, periods, and underscores. Additionally, the labels
file ought to have the same name as the RKNN file, with `-labels` appended to the end. For
example, if the RKNN file is named `Algae_1.03.2025-640-640-yolov5s.rknn`, the labels file should be
named `Algae_1.03.2025-640-640-yolov5s-labels.txt`.
### Managing Custom Models
:::{note}
Currently there is no way to delete custom models in the GUI, though this is a planned feature. To do this, you have to SSH into the coprocessor and delete the files manually from `/opt/photonvision/photonvision_config/models`.
:::
Custom models can now be managed from the Object Detection tab in settings. You can upload a custom model by clicking the "Upload Model" button, selecting your model file, and filling out the property fields. Models can also be exported, both individually and in bulk. Models exported in bulk can be imported using the `import bulk` button. Models exported individually must be re-imported as an individual model, and all the relevant metadata is stored in the filename of the model.

4
docs/source/docs/objectDetection/index.md
View File

@@ -1,8 +1,8 @@
# Object Detection
```{toctree}
:maxdepth: 0
:titlesonly: true
about-object-detection
opi
rubik
```

19
docs/source/docs/objectDetection/opi.md Normal file
View File

@@ -0,0 +1,19 @@
# Orange Pi 5 (and variants) Object Detection
## How it works
PhotonVision runs object detection on the Orange Pi 5 by use of the RKNN model architecture, and [this JNI code](https://github.com/PhotonVision/rknn_jni).
## Supported models
PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv5, YOLOv8, and YOLOv11 models trained and converted to `.rknn` format for RK3588 SOCs! Other models require different post-processing code and will NOT work.
## Converting Custom Models
:::{warning}
Only quantized models are supported, so take care when exporting to select the option for quantization.
:::
PhotonVision now ships with a [Python Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/rknn-convert-tool/rknn_conversion.ipynb) that you can use in [Google Colab](https://colab.research.google.com) or in a local environment. In Google Colab, you can simply paste the PhotonVision GitHub URL into the "GitHub" tab and select the `rknn_conversion.ipynb` notebook without needing to manually download anything.
Please ensure that the model you are attempting to convert is among the {ref}`supported models <docs/objectDetection/opi:Supported Models>` and using the PyTorch format.

25
docs/source/docs/objectDetection/rubik.md Normal file
View File

@@ -0,0 +1,25 @@
# Rubik Pi 3 Object Detection
## How it works
PhotonVision runs object detection on the Rubik Pi 3 by use of [TensorflowLite](https://github.com/tensorflow/tensorflow), and [this JNI code](https://github.com/PhotonVision/rubik_jni).
## Supported models
PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv8 and YOLOv11 models trained and converted to `.tflite` format for QCS6490 SOCs! Other models require different post-processing code and will NOT work.
## Converting Custom Models
:::{warning}
Only quantized models are supported, so take care when exporting to select the option for quantization.
:::
PhotonVision now ships with a [Python Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/rubik_conversion.ipynb) that you can use in [Google Colab](https://colab.research.google.com) or in a local environment. In Google Colab, you can simply paste the PhotonVision GitHub URL into the "GitHub" tab and select the `rubik_conversion.ipynb` notebook without needing to manually download anything.
Please ensure that the model you are attempting to convert is among the {ref}`supported models <docs/objectDetection/rubik:Supported Models>` and using the PyTorch format.
## Benchmarking
Before you can perform benchmarking, it's necessary to install `tensorflow-lite-qcom-apps` with apt.
By SSHing into your Rubik Pi and running this command, replacing `PATH/TO/MODEL` with the path to your model, `benchmark_model --graph=src/test/resources/yolov8nCoco.tflite --external_delegate_path=/usr/lib/libQnnTFLiteDelegate.so --external_delegate_options=backend_type:htp --external_delegate_options=htp_use_conv_hmx:1 --external_delegate_options=htp_performance_mode:2` you can determine how long it takes for inference to be performed with your model.

6
docs/source/docs/programming/photonlib/controlling-led.md
View File

@@ -4,17 +4,17 @@ You can control the vision LEDs of supported hardware via PhotonLib using the `s
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Blink the LEDs.
camera.setLED(VisionLEDMode.kBlink);
.. code-block:: C++
.. code-block:: c++
// Blink the LEDs.
camera.SetLED(photonlib::VisionLEDMode::kBlink);
.. code-block:: Python
.. code-block:: python
# Coming Soon!
```

18
docs/source/docs/programming/photonlib/driver-mode-pipeline-index.md
View File

@@ -9,17 +9,17 @@ You can use the `setDriverMode()`/`SetDriverMode()` (Java and C++ respectively)
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Set driver mode to on.
camera.setDriverMode(true);
.. code-block:: C++
.. code-block:: c++
// Set driver mode to on.
camera.SetDriverMode(true);
.. code-block:: Python
.. code-block:: python
# Coming Soon!
```
@@ -31,17 +31,17 @@ You can use the `setPipelineIndex()`/`SetPipelineIndex()` (Java and C++ respecti
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Change pipeline to 2
camera.setPipelineIndex(2);
.. code-block:: C++
.. code-block:: c++
// Change pipeline to 2
camera.SetPipelineIndex(2);
.. code-block:: Python
.. code-block:: python
# Coming Soon!
```
@@ -52,17 +52,17 @@ You can also get the pipeline latency from a pipeline result using the `getLaten
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Get the pipeline latency.
double latencySeconds = result.getLatencyMillis() / 1000.0;
.. code-block:: C++
.. code-block:: c++
// Get the pipeline latency.
units::second_t latency = result.GetLatency();
.. code-block:: Python
.. code-block:: python
# Coming Soon!
```

40
docs/source/docs/programming/photonlib/getting-target-data.md
View File

@@ -20,7 +20,7 @@ The `PhotonCamera` class has two constructors: one that takes a `NetworkTable` a
:language: c++
:lines: 42-43
.. code-block:: Python
.. code-block:: python
# Change this to match the name of your camera as shown in the web ui
self.camera = PhotonCamera("your_camera_name_here")
@@ -51,7 +51,7 @@ Use the `getLatestResult()`/`GetLatestResult()` (Java and C++ respectively) to o
:language: c++
:lines: 35-36
.. code-block:: Python
.. code-block:: python
# Query the latest result from PhotonVision
result = self.camera.getLatestResult()
@@ -69,17 +69,17 @@ Each pipeline result has a `hasTargets()`/`HasTargets()` (Java and C++ respectiv
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Check if the latest result has any targets.
boolean hasTargets = result.hasTargets();
.. code-block:: C++
.. code-block:: c++
// Check if the latest result has any targets.
bool hasTargets = result.HasTargets();
.. code-block:: Python
.. code-block:: python
# Check if the latest result has any targets.
hasTargets = result.hasTargets()
@@ -99,17 +99,17 @@ You can get a list of tracked targets using the `getTargets()`/`GetTargets()` (J
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Get a list of currently tracked targets.
List<PhotonTrackedTarget> targets = result.getTargets();
.. code-block:: C++
.. code-block:: c++
// Get a list of currently tracked targets.
wpi::ArrayRef<photonlib::PhotonTrackedTarget> targets = result.GetTargets();
.. code-block:: Python
.. code-block:: python
# Get a list of currently tracked targets.
targets = result.getTargets()
@@ -121,18 +121,18 @@ You can get the {ref}`best target <docs/reflectiveAndShape/contour-filtering:Con
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Get the current best target.
PhotonTrackedTarget target = result.getBestTarget();
.. code-block:: C++
.. code-block:: c++
// Get the current best target.
photonlib::PhotonTrackedTarget target = result.GetBestTarget();
.. code-block:: Python
.. code-block:: python
# Coming Soon!
@@ -149,7 +149,7 @@ You can get the {ref}`best target <docs/reflectiveAndShape/contour-filtering:Con
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Get information from target.
double yaw = target.getYaw();
@@ -159,7 +159,7 @@ You can get the {ref}`best target <docs/reflectiveAndShape/contour-filtering:Con
Transform2d pose = target.getCameraToTarget();
List<TargetCorner> corners = target.getCorners();
.. code-block:: C++
.. code-block:: c++
// Get information from target.
double yaw = target.GetYaw();
@@ -169,7 +169,7 @@ You can get the {ref}`best target <docs/reflectiveAndShape/contour-filtering:Con
frc::Transform2d pose = target.GetCameraToTarget();
wpi::SmallVector<std::pair<double, double>, 4> corners = target.GetCorners();
.. code-block:: Python
.. code-block:: python
# Get information from target.
yaw = target.getYaw()
@@ -193,7 +193,7 @@ All of the data above (**except skew**) is available when using AprilTags.
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Get information from target.
int targetID = target.getFiducialId();
@@ -201,7 +201,7 @@ All of the data above (**except skew**) is available when using AprilTags.
Transform3d bestCameraToTarget = target.getBestCameraToTarget();
Transform3d alternateCameraToTarget = target.getAlternateCameraToTarget();
.. code-block:: C++
.. code-block:: c++
// Get information from target.
int targetID = target.GetFiducialId();
@@ -209,7 +209,7 @@ All of the data above (**except skew**) is available when using AprilTags.
frc::Transform3d bestCameraToTarget = target.getBestCameraToTarget();
frc::Transform3d alternateCameraToTarget = target.getAlternateCameraToTarget();
.. code-block:: Python
.. code-block:: python
# Get information from target.
targetID = target.getFiducialId()
@@ -227,7 +227,7 @@ Images are stored within the PhotonVision configuration directory. Running the "
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Capture pre-process camera stream image
camera.takeInputSnapshot();
@@ -235,7 +235,7 @@ Images are stored within the PhotonVision configuration directory. Running the "
// Capture post-process camera stream image
camera.takeOutputSnapshot();
.. code-block:: C++
.. code-block:: c++
// Capture pre-process camera stream image
camera.TakeInputSnapshot();
@@ -243,7 +243,7 @@ Images are stored within the PhotonVision configuration directory. Running the "
// Capture post-process camera stream image
camera.TakeOutputSnapshot();
.. code-block:: Python
.. code-block:: python
# Capture pre-process camera stream image
camera.takeInputSnapshot()

36
docs/source/docs/programming/photonlib/using-target-data.md
View File

@@ -8,17 +8,17 @@ A `PhotonUtils` class with helpful common calculations is included within `Photo
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Calculate robot's field relative pose
if (aprilTagFieldLayout.getTagPose(target.getFiducialId()).isPresent()) {
Pose3d robotPose = PhotonUtils.estimateFieldToRobotAprilTag(target.getBestCameraToTarget(), aprilTagFieldLayout.getTagPose(target.getFiducialId()).get(), cameraToRobot);
}
.. code-block:: C++
.. code-block:: c++
//TODO
.. code-block:: Python
.. code-block:: python
# Coming Soon!
```
@@ -29,19 +29,19 @@ You can get your robot's `Pose2D` on the field using various camera data, target
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Calculate robot's field relative pose
Pose2D robotPose = PhotonUtils.estimateFieldToRobot(
kCameraHeight, kTargetHeight, kCameraPitch, kTargetPitch, Rotation2d.fromDegrees(-target.getYaw()), gyro.getRotation2d(), targetPose, cameraToRobot);
.. code-block:: C++
.. code-block:: c++
// Calculate robot's field relative pose
frc::Pose2D robotPose = photonlib::EstimateFieldToRobot(
kCameraHeight, kTargetHeight, kCameraPitch, kTargetPitch, frc::Rotation2d(units::degree_t(-target.GetYaw())), frc::Rotation2d(units::degree_t(gyro.GetRotation2d)), targetPose, cameraToRobot);
.. code-block:: Python
.. code-block:: python
# Coming Soon!
@@ -54,15 +54,15 @@ If your camera is at a fixed height on your robot and the height of the target i
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// TODO
.. code-block:: C++
.. code-block:: c++
// TODO
.. code-block:: Python
.. code-block:: python
# Coming Soon!
@@ -78,15 +78,15 @@ The C++ version of PhotonLib uses the Units library. For more information, see [
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
double distanceToTarget = PhotonUtils.getDistanceToPose(robotPose, targetPose);
.. code-block:: C++
.. code-block:: c++
//TODO
.. code-block:: Python
.. code-block:: python
# Coming Soon!
```
@@ -97,19 +97,19 @@ You can get a [translation](https://docs.wpilib.org/en/latest/docs/software/adva
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Calculate a translation from the camera to the target.
Translation2d translation = PhotonUtils.estimateCameraToTargetTranslation(
distanceMeters, Rotation2d.fromDegrees(-target.getYaw()));
.. code-block:: C++
.. code-block:: c++
// Calculate a translation from the camera to the target.
frc::Translation2d translation = photonlib::PhotonUtils::EstimateCameraToTargetTranslation(
distance, frc::Rotation2d(units::degree_t(-target.GetYaw())));
.. code-block:: Python
.. code-block:: python
# Coming Soon!
@@ -125,14 +125,14 @@ We are negating the yaw from the camera from CV (computer vision) conventions to
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
Rotation2d targetYaw = PhotonUtils.getYawToPose(robotPose, targetPose);
.. code-block:: C++
.. code-block:: c++
//TODO
.. code-block:: Python
.. code-block:: python
# Coming Soon!
```

10
docs/source/docs/quick-start/networking.md
View File

@@ -75,15 +75,15 @@ If you would like to access your Ethernet-connected vision device from a compute
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
PortForwarder.add(5800, "photonvision.local", 5800);
.. code-block:: C++
.. code-block:: c++
wpi::PortForwarder::GetInstance().Add(5800, "photonvision.local", 5800);
.. code-block:: Python
.. code-block:: python
# Coming Soon!
```
@@ -99,3 +99,7 @@ The camera streams start at 1181 with two ports for each camera (ex. 1181 and 11
:::{warning}
If your camera stream isn't sent to the same port as it's originally found on, its stream will not be visible in the UI.
:::
## SSH Access
For advanced users, SSH access is available for coprocessors running PhotonVision. This allows you to perform advanced configurations and troubleshooting. The default credentials are: `photon:vision` for all devices using an image of `v2026.0.3` or later. The legacy credentials of `pi:raspberry` will still work, but it's recommended to switch to the new credentials as the old ones will be deprecated in a future release.

57
docs/source/docs/quick-start/quick-install.md
View File

@@ -1,22 +1,30 @@
# Quick Install
# Quick Installation Guide
## Install the latest image of photonvision for your coprocessor
- For the following supported coprocessors
- {ref}`Raspberry Pi 3,4,5 <docs/quick-start/quick-install:Raspberry Pi and Orange Pi Installation>`
- {ref}`Orange Pi 5, 5B, 5 Pro <docs/quick-start/quick-install:Raspberry Pi and Orange Pi Installation>`
- {ref}`Limelight 2, 2+, 3, 3G, 4 <docs/quick-start/quick-install:LimeLight Installation>`
- {ref}`Rubik Pi 3 <docs/quick-start/quick-install:Rubik Pi 3 Installation>`
- For the supported coprocessors
- RPI 3,4,5
- Orange Pi 5
- Limelight
For installing on non-supported devices {ref}`see. <docs/advanced-installation/sw_install/index:Software Installation>`
For installing on non-supported devices {ref}`see here. <docs/advanced-installation/sw_install/index:Software Installation>`
[Download the latest preconfigured image of photonvision for your coprocessor](https://github.com/PhotonVision/photonvision/releases/latest)
| Coprocessor | Image filename | Jar |
| -------------------- | ---------------------------------------------------- | ------------------------------------- |
| OrangePi 5 | photonvision-{version}-linuxarm64_orangepi5.img.xz | photonvision-{version}-linuxarm64.jar |
| Raspberry Pi 3, 4, 5 | photonvision-{version}-linuxarm64_RaspberryPi.img.xz | photonvision-{version}-linuxarm64.jar |
| Limelight 2 | photonvision-{version}-linuxarm64_limelight2.img.xz | photonvision-{version}-linuxarm64.jar |
| Limelight 3 | photonvision-{version}-linuxarm64_limelight3.img.xz | photonvision-{version}-linuxarm64.jar |
| Coprocessor | Image filename | Jar |
| -------------------- | -------------------------------------------------------- | ------------------------------------- |
| Raspberry Pi 3, 4, 5 | photonvision-{version}-linuxarm64_RaspberryPi.img.xz | photonvision-{version}-linuxarm64.jar |
| OrangePi 5 | photonvision-{version}-linuxarm64_orangepi5.img.xz | photonvision-{version}-linuxarm64.jar |
| OrangePi 5B | photonvision-{version}-linuxarm64_orangepi5b.img.xz | photonvision-{version}-linuxarm64.jar |
| OrangePi 5 Pro | photonvision-{version}-linuxarm64_orangepi5pro.img.xz | photonvision-{version}-linuxarm64.jar |
| Limelight 2 | photonvision-{version}-linuxarm64_limelight2.img.xz | photonvision-{version}-linuxarm64.jar |
| Limelight 3 | photonvision-{version}-linuxarm64_limelight3.img.xz | photonvision-{version}-linuxarm64.jar |
| Limelight 3G | photonvision-{version}-linuxarm64_limelight3G.img.xz | photonvision-{version}-linuxarm64.jar |
| Limelight 4 | photonvision-{version}-linuxarm64_limelight4.img.xz | photonvision-{version}-linuxarm64.jar |
| Rubik Pi 3 | photonvision-{version}-linuxarm64_rubikpi3.tar.xz | photonvision-{version}-linuxarm64.jar |
Unless otherwise noted in release notes or if updating from the prior years version, to update PhotonVision after the initial installation, use the offline update option in the settings page with the downloaded jar file from the latest release.
## Raspberry Pi and Orange Pi Installation
Use the [Raspberry Pi Imager](https://www.raspberrypi.com/software/) to flash the image onto the coprocessors microSD card. Select the downloaded `.img.xz` file, select your microSD card, and flash.
@@ -24,10 +32,25 @@ Use the [Raspberry Pi Imager](https://www.raspberrypi.com/software/) to flash th
Balena Etcher can also be used, but historically has had issues such as bootlooping (the system will repeatedly boot and restart) when imaging your device. Use at your own risk.
:::
Limelights have a different installation processes. Simply connect the limelight to your computer using the proper usb cable. Select the compute module. If it doesn’t show up after 30s try using another USB port, initialization may take a while. If prompted, install the recommended missing drivers. Select the image, and flash.
## Limelight Installation
Unless otherwise noted in release notes or if updating from the prior years version, to update PhotonVision after the initial installation, use the offline update option in the settings page with the downloaded jar file from the latest release.
In order to flash your Limelight you should follow the instructions on the Limelight documentation for the relevant version. Make sure to replace the Limelight OS image with the relevant PhotonVision image.
| Limelight Version | Limelight Documentation | PhotonVision Image | |
| ----------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | --- |
| 2 | [Updating Limelight 2 OS](https://docs.limelightvision.io/docs/docs-limelight/getting-started/limelight-2#4-updating-limelightos) | photonvision-{version}-linuxarm64_limelight2.img.xz | |
| 3 | [Updating Limelight 3 OS](https://docs.limelightvision.io/docs/docs-limelight/getting-started/limelight-3#4-updating-limelightos) | photonvision-{version}-linuxarm64_limelight3.img.xz | |
| 3G | [Updating Limelight 3G OS](https://docs.limelightvision.io/docs/docs-limelight/getting-started/limelight-3g#4-updating-limelightos) | photonvision-{version}-linuxarm64_limelight3g.img.xz | |
| 4 | [Updating Limelight 4 OS](https://docs.limelightvision.io/docs/docs-limelight/getting-started/limelight-4#4-updating-limelightos) | photonvision-{version}-linuxarm64_limelight4.img.xz | |
:::{note}
Limelight 2, 2+, and 3 will need a [custom hardware config file](https://github.com/PhotonVision/photonvision/tree/main/docs/source/docs/advanced-installation/sw_install/files) for lighting to work. Currently only limelight 2 and 2+ files are available.
Limelight models will need a [custom hardware config file](https://github.com/PhotonVision/photonvision/tree/main/docs/source/docs/advanced-installation/sw_install/files) for LEDs or other hardware features to work.
:::
## Rubik Pi 3 Installation
:::{warning}
The Qualcomm Launcher caches files. If you flash multiple times, you may need to clear the cache by navigating to your temp directory, and deleting the `qualcomm-launcher` folder.
:::
To flash the Rubik Pi 3 coprocessor, it's necessary to use the [Qualcomm Launcher](https://softwarecenter.qualcomm.com/catalog/item/Qualcomm_Launcher). Upload a custom image by selecting the *Custom* option in the launcher. Choose the downloaded PhotonVision `.tar.xz` file and follow the prompts to complete the installation. It is recommended to skip the *Configure Login* process, as PhotonVision will handle the necessary settings.

24
docs/source/docs/simulation/simulation-java.md
View File

@@ -54,7 +54,7 @@ A `VisionSystemSim` represents the simulated world for one or more cameras, and
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// A vision system sim labelled as "main" in NetworkTables
VisionSystemSim visionSim = new VisionSystemSim("main");
@@ -67,7 +67,7 @@ Vision targets require a `TargetModel`, which describes the shape of the target.
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// A 0.5 x 0.25 meter rectangular target
TargetModel targetModel = new TargetModel(0.5, 0.25);
@@ -78,7 +78,7 @@ These `TargetModel` are paired with a target pose to create a `VisionTargetSim`.
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// The pose of where the target is on the field.
// Its rotation determines where "forward" or the target x-axis points.
@@ -100,7 +100,7 @@ For convenience, an `AprilTagFieldLayout` can also be added to automatically cre
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// The layout of AprilTags which we want to add to the vision system
AprilTagFieldLayout tagLayout = AprilTagFieldLayout.loadFromResource(AprilTagFields.kDefaultField.m_resourceFile);
@@ -121,7 +121,7 @@ Before adding a simulated camera, we need to define its properties. This is done
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// The simulated camera properties
SimCameraProperties cameraProp = new SimCameraProperties();
@@ -132,7 +132,7 @@ By default, this will create a 960 x 720 resolution camera with a 90 degree diag
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// A 640 x 480 camera with a 100 degree diagonal FOV.
cameraProp.setCalibration(640, 480, Rotation2d.fromDegrees(100));
@@ -150,7 +150,7 @@ These properties are used in a `PhotonCameraSim`, which handles generating captu
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// The PhotonCamera used in the real robot code.
PhotonCamera camera = new PhotonCamera("cameraName");
@@ -164,7 +164,7 @@ The `PhotonCameraSim` can now be added to the `VisionSystemSim`. We have to defi
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Our camera is mounted 0.1 meters forward and 0.5 meters up from the robot pose,
// (Robot pose is considered the center of rotation at the floor level, or Z = 0)
@@ -186,7 +186,7 @@ If the camera is mounted on a mobile mechanism (like a turret) this transform ca
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// The turret the camera is mounted on is rotated 5 degrees
Rotation3d turretRotation = new Rotation3d(0, 0, Math.toRadians(5));
@@ -203,7 +203,7 @@ To update the `VisionSystemSim`, we simply have to pass in the simulated robot p
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Update with the simulated drivetrain pose. This should be called every loop in simulation.
visionSim.update(robotPoseMeters);
@@ -218,7 +218,7 @@ Each `VisionSystemSim` has its own built-in `Field2d` for displaying object pose
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Get the built-in Field2d used by this VisionSystemSim
visionSim.getDebugField();
@@ -233,7 +233,7 @@ A `PhotonCameraSim` can also draw and publish generated camera frames to a MJPEG
```{eval-rst}
.. tab-set-code::
.. code-block:: Java
.. code-block:: java
// Enable the raw and processed streams. These are enabled by default.
cameraSim.enableRawStream(true);

6
docs/source/docs/troubleshooting/unix-commands.md
View File

@@ -122,11 +122,13 @@ systemctl status photonvision
View the PhotonVision logs:
```
journalctl -u photonvision
journalctl --output cat -u photonvision
```
View the PhotonVision logs in real-time:
```
journalctl -u photonvision -f
journalctl --output cat -u photonvision -f
```
`--output cat` is used to prevent journalctl from printing its own timestamps, because we log our own timestamps.

1
docs/source/index.md
View File

@@ -127,6 +127,7 @@ docs/troubleshooting/index
docs/additional-resources/best-practices
docs/additional-resources/config
docs/additional-resources/nt-api
docs/benchmarks/index
docs/contributing/index
```