Team2890/PhotonVision
4
0
Fork 0
mirror of https://github.com/PhotonVision/photonvision synced 2026-06-19 00:41:41 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add support for object detection on Rubik Pi 3 (#2005)

Browse Source
This commit is contained in:
Sam Freund
2025-08-09 10:02:55 -05:00
committed by GitHub
parent e23df8c9a4
commit 9277960018
26 changed files with 545 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.styleguide
View File

@@ -19,6 +19,7 @@ modifiableFileExclude {
\.webp$
\.ico$
\.rknn$
\.tflite$
\.mp4$
\.ttf$
\.woff2$

1
build.gradle
View File

@@ -40,6 +40,7 @@ ext {
javalinVersion = "5.6.2"
libcameraDriverVersion = "v2025.0.3"
rknnVersion = "dev-v2025.0.0-1-g33b6263"
rubikVersion = "v2025.1.0"
frcYear = "2025"
mrcalVersion = "v2025.0.0";

1
docs/.styleguide
View File

@@ -11,6 +11,7 @@ modifiableFileExclude {
\.webp$
\.ico$
\.rknn$
\.tflite$
\.svg$
\.woff2$
gradlew

27
docs/source/docs/objectDetection/about-object-detection.md
View File

@@ -2,7 +2,7 @@
## How does it work?
PhotonVision supports object detection using neural network accelerator hardware built into Orange Pi 5/5+ coprocessors. Please note that the Orange Pi 5/5+ are the only coprocessors that are currently supported. The Neural Processing Unit, or NPU, is [used by PhotonVision](https://github.com/PhotonVision/rknn_jni/tree/main) to massively accelerate certain math operations like those needed for running ML-based object detection.
PhotonVision supports object detection using neural network accelerator hardware, commonly known as an NPU. The two coprocessors currently supported are the {ref}`Orange Pi 5 <docs/objectDetection/opi:Orange Pi 5 (and variants) Object Detection>` and the {ref}`Rubik Pi 3 <docs/objectDetection/rubik:Rubik Pi 3 Object Detection>`.
PhotonVision currently ships with a model trained on the [COCO dataset](https://cocodataset.org/) by [Ultralytics](https://github.com/ultralytics/ultralytics) (this model is licensed under [AGPLv3](https://www.gnu.org/licenses/agpl-3.0.en.html)). This model is meant to be used for testing and other miscellaneous purposes. It is not meant to be used in competition. For the 2025 post-season, PhotonVision also ships with a pretrained ALGAE model. A model to detect coral is available in the PhotonVision discord, but will not be distributed with PhotonVision.
@@ -18,7 +18,7 @@ This model output means that while its fairly easy to say that "this rectangle p
## Tuning and Filtering
Compared to other pipelines, object detection exposes very few tuning handles. The Confidence slider changes the minimum confidence that the model needs to have in a given detection to consider it valid, as a number between 0 and 1 (with 0 meaning completely uncertain and 1 meaning maximally certain). The Non-Maximum Suppresion (NMS) Threshold slider is used to filter out overlapping detections. Lower values mean more detections are allowed through, but may result in false positives. It's generally recommended that teams leave this set at the default, unless they find they're unable to get usable results with solely the Confidence slider.
Compared to other pipelines, object detection exposes very few tuning handles. The Confidence slider changes the minimum confidence that the model needs to have in a given detection to consider it valid, as a number between 0 and 1 (with 0 meaning completely uncertain and 1 meaning maximally certain). The Non-Maximum Suppresion (NMS) Threshold slider is used to filter out overlapping detections. Higher values mean more detections are allowed through, but may result in false positives. It's generally recommended that teams leave this set at the default, unless they find they're unable to get usable results with solely the Confidence slider.
```{raw} html
<video width="85%" controls>
@@ -33,22 +33,19 @@ The same area, aspect ratio, and target orientation/sort parameters from {ref}`r
Photonvision will letterbox your camera frame to 640x640. This means that if you select a resolution that is larger than 640 it will be scaled down to fit inside a 640x640 frame with black bars if needed. Smaller frames will be scaled up with black bars if needed.
## Training Custom Models
It is recommended that you select a resolution that results in the smaller dimension being just greater than, or equal to, 640. Anything above this will not see any increased performance.
:::{warning}
Power users only. This requires some setup, such as obtaining your own dataset and installing various tools. It's additionally advised to have a general knowledge of ML before attempting to train your own model. Additionally, this is not officially supported by Photonvision, and any problems that may arise are not attributable to Photonvision.
:::
## Custom Models
Before beginning, it is necessary to install the [rknn-toolkit2](https://github.com/airockchip/rknn-toolkit2). Then, install the relevant [Ultralytics repository](https://github.com/airockchip?tab=repositories&q=yolo&type=&language=&sort=) from this list. After training your model, export it to `rknn`. This will give you an `onnx` file, formatted for conversion. Copy this file to the relevant folder in [rknn_model_zoo](https://github.com/airockchip/rknn_model_zoo), and use the conversion script located there to convert it. If necessary, modify the script to provide the path to your training database for quantization.
For information regarding converting custom models and supported models for each platform, refer to the page detailing information about your specific coprocessor.
## Managing Custom Models
- {ref}`Orange Pi 5 <docs/objectDetection/opi:Orange Pi 5 (and variants) Object Detection>`
- {ref}`Rubik Pi 3 <docs/objectDetection/rubik:Rubik Pi 3 Object Detection>`
:::{warning}
PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv5, YOLOv8, and YOLOv11 models trained and converted to `.rknn` format for RK3588 CPUs! Other models require different post-processing code and will NOT work. The model conversion process is also highly particular. Proceed with care.
:::
### Training Custom Models
:::{warning}
Non-quantized models are not supported! If you have the option, make sure quantization is enabled when exporting to .rknn format. This will represent the weights and activations of the model as 8-bit integers, instead of 32-bit floats which PhotonVision doesn't support. Quantized models are also much faster for a negligible loss in accuracy.
:::
PhotonVision does not offer any support for training custom models, only conversion. For information on which models are supported for a given coprocessor, use the links above.
Custom models can now be managed from the Object Detection tab in settings. You can upload a custom model by clicking the "Upload Model" button, selecting your `.rknn` file, and filling out the property fields. Models can also be exported, both individually and in bulk. Models exported in bulk can be imported using the `import bulk` button. Models exported individually must be re-imported as an individual model, and all the relevant metadata is stored in the filename of the model.
### Managing Custom Models
Custom models can now be managed from the Object Detection tab in settings. You can upload a custom model by clicking the "Upload Model" button, selecting your model file, and filling out the property fields. Models can also be exported, both individually and in bulk. Models exported in bulk can be imported using the `import bulk` button. Models exported individually must be re-imported as an individual model, and all the relevant metadata is stored in the filename of the model.

4
docs/source/docs/objectDetection/index.md
View File

@@ -1,8 +1,8 @@
# Object Detection
```{toctree}
:maxdepth: 0
:titlesonly: true
about-object-detection
opi
rubik
```

19
docs/source/docs/objectDetection/opi.md Normal file
View File

@@ -0,0 +1,19 @@
# Orange Pi 5 (and variants) Object Detection
## How it works
PhotonVision runs object detection on the Orange Pi 5 by use of the RKNN model architecture, and [this JNI code](https://github.com/PhotonVision/rknn_jni).
## Supported models
PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv5, YOLOv8, and YOLOv11 models trained and converted to `.rknn` format for RK3588 SOCs! Other models require different post-processing code and will NOT work.
## Converting Custom Models
:::{warning}
Only quantized models are supported, so take care when exporting to select the option for quantization.
:::
PhotonVision now ships with a [Python Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/rknn-convert-tool/rknn_conversion.ipynb) that you can use in [Google Colab](https://colab.research.google.com) or in a local environment. In Google Colab, you can simply paste the PhotonVision GitHub URL into the "GitHub" tab and select the `rknn_conversion.ipynb` notebook without needing to manually download anything.
Please ensure that the model you are attempting to convert is among the {ref}`supported models <docs/objectDetection/opi:Supported Models>` and using the PyTorch format.

25
docs/source/docs/objectDetection/rubik.md Normal file
View File

@@ -0,0 +1,25 @@
# Rubik Pi 3 Object Detection
## How it works
PhotonVision runs object detection on the Rubik Pi 3 by use of [TensorflowLite](https://github.com/tensorflow/tensorflow), and [this JNI code](https://github.com/PhotonVision/rubik_jni).
## Supported models
PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv8 and YOLOv11 models trained and converted to `.tflite` format for QCS6490 SOCs! Other models require different post-processing code and will NOT work.
## Converting Custom Models
:::{warning}
Only quantized models are supported, so take care when exporting to select the option for quantization.
:::
PhotonVision now ships with a [Python Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/rubik_conversion.ipynb) that you can use in [Google Colab](https://colab.research.google.com) or in a local environment. In Google Colab, you can simply paste the PhotonVision GitHub URL into the "GitHub" tab and select the `rubik_conversion.ipynb` notebook without needing to manually download anything.
Please ensure that the model you are attempting to convert is among the {ref}`supported models <docs/objectDetection/rubik:Supported Models>` and using the PyTorch format.
## Benchmarking
Before you can perform benchmarking, it's necessary to install `tensorflow-lite-qcom-apps` with apt.
By SSHing into your Rubik Pi and running this command, replacing `PATH/TO/MODEL` with the path to your model, `benchmark_model --graph=src/test/resources/yolov8nCoco.tflite --external_delegate_path=/usr/lib/libQnnTFLiteDelegate.so --external_delegate_options=backend_type:htp --external_delegate_options=htp_use_conv_hmx:1 --external_delegate_options=htp_performance_mode:2` you can determine how long it takes for inference to be performed with your model.

2
photon-client/src/components/dashboard/tabs/ObjectDetectionTab.vue
View File

@@ -90,7 +90,7 @@ const selectedModel = computed({
class="pt-2"
:slider-cols="interactiveCols"
label="NMS Threshold"
tooltip="The Non-Maximum Suppression threshold used to filter out overlapping detections. Lower values mean more detections are allowed through, but may result in false positives."
tooltip="The Non-Maximum Suppression threshold used to filter out overlapping detections. Higher values mean more detections are allowed through, but may result in false positives."
:min="0"
:max="1"
:step="0.01"

37
photon-client/src/components/settings/ObjectDetectionCard.vue
View File

@@ -8,7 +8,6 @@ import pvInput from "@/components/common/pv-input.vue";
import { useTheme } from "vuetify";
const theme = useTheme();
const showImportDialog = ref(false);
const showInfo = ref({ show: false, model: {} as ObjectDetectionModelProperties });
const confirmDeleteDialog = ref({ show: false, model: {} as ObjectDetectionModelProperties });
@@ -303,11 +302,33 @@ const handleBulkImport = () => {
<v-card color="surface" dark>
<v-card-title class="pb-0">Import New Object Detection Model</v-card-title>
<v-card-text>
Upload a new object detection model to this device that can be used in a pipeline. Note that ONLY
640x640 YOLOv5, YOLOv8, and YOLOv11 models trained and converted to `.rknn` format for RK3588 CPUs are
currently supported!
<span v-if="useSettingsStore().general.supportedBackends?.includes('RKNN')"
>Upload a new object detection model to this device that can be used in a pipeline. Note that ONLY
640x640 YOLOv5, YOLOv8, and YOLOv11 models trained and converted to `.rknn` format for RK3588 SOCs are
currently supporter!</span
>
<span v-else-if="useSettingsStore().general.supportedBackends?.includes('RUBIK')"
>Upload a new object detection model to this device that can be used in a pipeline. Note that ONLY
640x640 YOLOv8 and YOLOv11 models trained and converted to `.tflite` format for QCS6490 compatible
backends are currently supported!
</span>
<span v-else>
If you're seeing this, something broke; please file a ticket and tell us the details of your
situation.</span
>
<div class="pa-5 pb-0">
<v-file-input v-model="importModelFile" variant="underlined" label="Model File" accept=".rknn" />
<v-file-input
v-model="importModelFile"
variant="underlined"
label="Model File"
:accept="
useSettingsStore().general.supportedBackends?.includes('RKNN')
? '.rknn'
: useSettingsStore().general.supportedBackends?.includes('RUBIK')
? '.tflite'
: ''
"
/>
<v-text-field
v-model="importLabels"
label="Labels"
@@ -321,7 +342,11 @@ const handleBulkImport = () => {
v-model="importVersion"
variant="underlined"
label="Model Version"
:items="['YOLOv5', 'YOLOv8', 'YOLO11']"
:items="
useSettingsStore().general.supportedBackends?.includes('RKNN')
? ['YOLOv5', 'YOLOv8', 'YOLO11']
: ['YOLOv8', 'YOLO11']
"
/>
<v-btn
color="buttonActive"

2
photon-client/src/types/SettingTypes.ts
View File

@@ -20,7 +20,7 @@ export interface ObjectDetectionModelProperties {
labels: string[];
resolutionWidth: number;
resolutionHeight: number;
family: "RKNN";
family: "RKNN" | "RUBIK";
version: "YOLOV5" | "YOLOV8" | "YOLOV11";
}

6
photon-core/build.gradle
View File

@@ -36,6 +36,12 @@ dependencies {
implementation("org.photonvision:rknn_jni-java:$rknnVersion") {
transitive = false
}
implementation("org.photonvision:rubik_jni-jni:$rubikVersion:linuxarm64") {
transitive = false
}
implementation("org.photonvision:rubik_jni-java:$rubikVersion") {
transitive = false
}
implementation("org.photonvision:photon-libcamera-gl-driver-jni:$libcameraDriverVersion:linuxarm64") {
transitive = false
}

92
photon-core/src/main/java/org/photonvision/common/configuration/NeuralNetworkModelManager.java
View File

@@ -39,6 +39,7 @@ import org.photonvision.common.logging.LogGroup;
import org.photonvision.common.logging.Logger;
import org.photonvision.vision.objects.Model;
import org.photonvision.vision.objects.RknnModel;
import org.photonvision.vision.objects.RubikModel;
/**
* Manages the loading of neural network models.
@@ -54,6 +55,8 @@ public class NeuralNetworkModelManager {
/** Singleton instance of the NeuralNetworkModelManager */
private static NeuralNetworkModelManager INSTANCE;
private final List<Family> supportedBackends = new ArrayList<>();
/**
* This function stores the properties of the shipped object detection models. It is stored as a
* function so that it can be dynamic, to adjust for the models directory.
@@ -165,6 +168,26 @@ public class NeuralNetworkModelManager {
Family.RKNN,
Version.YOLOV8));
nnProps.addModelProperties(
new ModelProperties(
Path.of(modelsDirectory.getAbsolutePath(), "algae-coral-yolov8s.tflite"),
"Algae Coral v8s",
new LinkedList<String>(List.of("Algae", "Coral")),
640,
640,
Family.RUBIK,
Version.YOLOV8));
nnProps.addModelProperties(
new ModelProperties(
Path.of(modelsDirectory.getAbsolutePath(), "yolov8nCOCO.tflite"),
"COCO",
cocoLabels,
640,
640,
Family.RUBIK,
Version.YOLOV8));
return nnProps;
}
@@ -174,13 +197,17 @@ public class NeuralNetworkModelManager {
* @return The NeuralNetworkModelManager instance
*/
private NeuralNetworkModelManager() {
ArrayList<Family> backends = new ArrayList<>();
if (Platform.isRK3588()) {
backends.add(Family.RKNN);
switch (Platform.getCurrentPlatform()) {
case LINUX_QCS6490 -> supportedBackends.add(Family.RUBIK);
case LINUX_RK3588_64 -> supportedBackends.add(Family.RKNN);
default -> {
logger.warn(
"No supported neural network backends found for this platform: "
+ Platform.getCurrentPlatform());
// No supported backends, so we won't load any models
return;
}
}
supportedBackends = backends;
}
/**
@@ -199,7 +226,8 @@ public class NeuralNetworkModelManager {
private static final Logger logger = new Logger(NeuralNetworkModelManager.class, LogGroup.Config);
public enum Family {
RKNN
RKNN,
RUBIK
}
public enum Version {
@@ -208,8 +236,6 @@ public class NeuralNetworkModelManager {
YOLOV11
}
private final List<Family> supportedBackends;
/**
* Retrieves the list of supported backends.
*
@@ -264,14 +290,19 @@ public class NeuralNetworkModelManager {
}
// Do checking later on, when we create the model object
private void loadModel(ModelProperties properties) {
private void loadModel(Path path) {
if (models == null) {
models = new HashMap<>();
}
ModelProperties properties =
ConfigManager.getInstance().getConfig().neuralNetworkPropertyManager().getModel(path);
if (properties == null) {
logger.error(
"Model properties are null, this could mean the models config was unable to be found in the database");
"Model properties are null. This could mean the config for model "
+ path
+ " was unable to be found in the database.");
return;
}
@@ -292,6 +323,9 @@ public class NeuralNetworkModelManager {
case RKNN -> {
models.get(properties.family()).add(new RknnModel(properties));
}
case RUBIK -> {
models.get(properties.family()).add(new RubikModel(properties));
}
}
logger.info(
"Loaded model "
@@ -324,13 +358,7 @@ public class NeuralNetworkModelManager {
try {
Files.walk(modelsDirectory.toPath())
.filter(Files::isRegularFile)
.forEach(
path ->
loadModel(
ConfigManager.getInstance()
.getConfig()
.neuralNetworkPropertyManager()
.getModel(path)));
.forEach(path -> loadModel(path));
} catch (IOException e) {
logger.error("Failed to discover models at " + modelsDirectory.getAbsolutePath(), e);
}
@@ -357,6 +385,23 @@ public class NeuralNetworkModelManager {
public void extractModels() {
File modelsDirectory = ConfigManager.getInstance().getModelsDirectory();
// Filter shippedProprties by supportedBackends
NeuralNetworkPropertyManager supportedProperties = new NeuralNetworkPropertyManager();
for (ModelProperties model : getShippedProperties(modelsDirectory).getModels()) {
if (supportedBackends.contains(model.family())) {
supportedProperties.addModelProperties(model);
} else {
logger.warn(
"Skipping model " + model.nickname() + " as it is not supported on this platform.");
}
}
// Used for checking if the model to be extracted is supported for this architecture
ArrayList<String> supportedModelFileNames = new ArrayList<String>();
for (ModelProperties model : supportedProperties.getModels()) {
supportedModelFileNames.add(model.modelPath().getFileName().toString());
}
if (!modelsDirectory.exists() && !modelsDirectory.mkdirs()) {
throw new RuntimeException("Failed to create directory: " + modelsDirectory);
}
@@ -376,7 +421,11 @@ public class NeuralNetworkModelManager {
Path outputPath =
modelsDirectory.toPath().resolve(entry.getName().substring(resource.length() + 1));
if (Files.exists(outputPath)) {
// Check if the file already exists or if it is a supported model file
if ((Files.exists(outputPath))
|| !(entry.getName().endsWith("txt")
|| supportedModelFileNames.contains(
entry.getName().substring(entry.getName().lastIndexOf('/') + 1)))) {
logger.info("Skipping extraction of DNN resource: " + entry.getName());
continue;
}
@@ -394,11 +443,12 @@ public class NeuralNetworkModelManager {
logger.error("Error extracting models", e);
}
// Combine with existing properties
ConfigManager.getInstance()
.getConfig()
.setNeuralNetworkProperties(
getShippedProperties(modelsDirectory)
.sum(ConfigManager.getInstance().getConfig().neuralNetworkPropertyManager()));
supportedProperties.sum(
ConfigManager.getInstance().getConfig().neuralNetworkPropertyManager()));
}
public boolean clearModels() {

3
photon-core/src/main/java/org/photonvision/common/configuration/NeuralNetworkPropertyManager.java
View File

@@ -28,8 +28,7 @@ import org.photonvision.common.configuration.NeuralNetworkModelManager.Version;
public class NeuralNetworkPropertyManager {
/*
* The properties of the model. This is used to determine which model to load.
* The only family
* currently supported is RKNN.
* The only families currently supported are RKNN and Rubik (custom .tflite)
*/
public record ModelProperties(
@JsonProperty("modelPath") Path modelPath,

56
photon-core/src/main/java/org/photonvision/jni/RubikDetectorJNI.java Normal file
View File

@@ -0,0 +1,56 @@
/*
* Copyright (C) Photon Vision.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package org.photonvision.jni;
import java.io.IOException;
import java.util.List;
import org.photonvision.common.util.TestUtils;
public class RubikDetectorJNI extends PhotonJNICommon {
private boolean isLoaded;
private static RubikDetectorJNI instance = null;
private RubikDetectorJNI() {
isLoaded = false;
}
public static RubikDetectorJNI getInstance() {
if (instance == null) instance = new RubikDetectorJNI();
return instance;
}
public static synchronized void forceLoad() throws IOException {
TestUtils.loadLibraries();
forceLoad(
getInstance(),
RubikDetectorJNI.class,
List.of("tensorflowlite", "tensorflowlite_c", "external_delegate", "rubik_jni"));
}
@Override
public boolean isLoaded() {
return isLoaded;
}
@Override
public void setLoaded(boolean state) {
isLoaded = state;
}
}

2
photon-core/src/main/java/org/photonvision/vision/objects/Model.java
View File

@@ -30,4 +30,6 @@ public interface Model {
public Family getFamily();
public ModelProperties getProperties();
public String toString();
}

4
photon-core/src/main/java/org/photonvision/vision/objects/NullModel.java
View File

@@ -81,4 +81,8 @@ public class NullModel implements Model, ObjectDetector {
public List<NeuralNetworkPipeResult> detect(Mat in, double nmsThresh, double boxThresh) {
return List.of();
}
public String toString() {
return "NullModel";
}
}

5
photon-core/src/main/java/org/photonvision/vision/objects/RknnModel.java
View File

@@ -22,7 +22,6 @@ import org.opencv.core.Size;
import org.photonvision.common.configuration.NeuralNetworkModelManager.Family;
import org.photonvision.common.configuration.NeuralNetworkModelManager.Version;
import org.photonvision.common.configuration.NeuralNetworkPropertyManager.ModelProperties;
import org.photonvision.jni.RknnObjectDetector;
public class RknnModel implements Model {
public final File modelFile;
@@ -82,4 +81,8 @@ public class RknnModel implements Model {
return new RknnObjectDetector(
this, new Size(properties.resolutionWidth(), properties.resolutionHeight()));
}
public String toString() {
return "RknnModel{" + "modelFile=" + modelFile + ", properties=" + properties + '}';
}
}

6
photon-core/src/main/java/org/photonvision/jni/RknnObjectDetector.java → photon-core/src/main/java/org/photonvision/vision/objects/RknnObjectDetector.java
View File

@@ -15,7 +15,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package org.photonvision.jni;
package org.photonvision.vision.objects;
import java.awt.Color;
import java.lang.ref.Cleaner;
@@ -26,10 +26,8 @@ import org.opencv.core.Size;
import org.photonvision.common.logging.LogGroup;
import org.photonvision.common.logging.Logger;
import org.photonvision.common.util.ColorHelper;
import org.photonvision.jni.RknnDetectorJNI;
import org.photonvision.rknn.RknnJNI;
import org.photonvision.vision.objects.Letterbox;
import org.photonvision.vision.objects.ObjectDetector;
import org.photonvision.vision.objects.RknnModel;
import org.photonvision.vision.pipe.impl.NeuralNetworkPipeResult;
/** Manages an object detector using the rknn backend. */

86
photon-core/src/main/java/org/photonvision/vision/objects/RubikModel.java Normal file
View File

@@ -0,0 +1,86 @@
/*
* Copyright (C) Photon Vision.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package org.photonvision.vision.objects;
import java.io.File;
import org.opencv.core.Size;
import org.photonvision.common.configuration.NeuralNetworkModelManager.Family;
import org.photonvision.common.configuration.NeuralNetworkModelManager.Version;
import org.photonvision.common.configuration.NeuralNetworkPropertyManager.ModelProperties;
public class RubikModel implements Model {
public final File modelFile;
public final ModelProperties properties;
/**
* Rubik model constructor.
*
* @param properties The properties of the model.
* @throws IllegalArgumentException
*/
public RubikModel(ModelProperties properties) throws IllegalArgumentException {
modelFile = new File(properties.modelPath().toString());
if (!modelFile.exists()) {
throw new IllegalArgumentException("Model file does not exist: " + modelFile);
}
if (properties.labels() == null || properties.labels().isEmpty()) {
throw new IllegalArgumentException("Labels must be provided");
}
if (properties.resolutionWidth() <= 0 || properties.resolutionHeight() <= 0) {
throw new IllegalArgumentException("Resolution must be greater than 0");
}
if (properties.family() != Family.RUBIK) {
throw new IllegalArgumentException("Model family must be RUBIK");
}
if (properties.version() != Version.YOLOV8 && properties.version() != Version.YOLOV11) {
throw new IllegalArgumentException("Model version must be YOLOV8 or YOLOV11");
}
this.properties = properties;
}
/** Return the unique identifier for the model. In this case, it's the model's path. */
public String getUID() {
return properties.modelPath().toString();
}
public String getNickname() {
return properties.nickname();
}
public Family getFamily() {
return properties.family();
}
public ModelProperties getProperties() {
return properties;
}
public ObjectDetector load() {
return new RubikObjectDetector(
this, new Size(this.properties.resolutionWidth(), this.properties.resolutionHeight()));
}
public String toString() {
return "RubikModel{" + "modelFile=" + modelFile + ", properties=" + properties + '}';
}
}

162
photon-core/src/main/java/org/photonvision/vision/objects/RubikObjectDetector.java Normal file
View File

@@ -0,0 +1,162 @@
/*
* Copyright (C) Photon Vision.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package org.photonvision.vision.objects;
import java.awt.Color;
import java.lang.ref.Cleaner;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import org.opencv.core.Mat;
import org.opencv.core.Size;
import org.photonvision.common.logging.LogGroup;
import org.photonvision.common.logging.Logger;
import org.photonvision.common.util.ColorHelper;
import org.photonvision.jni.RubikDetectorJNI;
import org.photonvision.rubik.RubikJNI;
import org.photonvision.vision.pipe.impl.NeuralNetworkPipeResult;
/** Manages an object detector using the rubik backend. */
public class RubikObjectDetector implements ObjectDetector {
private static final Logger logger = new Logger(RubikDetectorJNI.class, LogGroup.General);
/** Cleaner instance to release the detector when it goes out of scope */
private final Cleaner cleaner = Cleaner.create();
/** Atomic boolean to ensure that the native object can only be released once. */
private AtomicBoolean released = new AtomicBoolean(false);
/** Pointer to the native object */
private final long ptr;
private final RubikModel model;
private final Size inputSize;
/** Returns the model in use by this detector. */
@Override
public RubikModel getModel() {
return model;
}
/**
* Creates a new rubikObjectDetector from the given model.
*
* @param model The model to create the detector from.
* @param inputSize The required image dimensions for the model. Images will be {@link
* Letterbox}ed to this shape.
*/
public RubikObjectDetector(RubikModel model, Size inputSize) {
this.model = model;
this.inputSize = inputSize;
// Create the detector
try {
ptr = RubikJNI.create(model.modelFile.getPath().toString());
} catch (Exception e) {
logger.error("Failed to create detector from path " + model.modelFile.getPath(), e);
throw new RuntimeException(
"Failed to create detector from path " + model.modelFile.getPath(), e);
}
if (!isValid()) {
logger.error(
"Failed to create detector from path "
+ model.modelFile.getPath()
+ ". Please ensure the model is valid and compatible with the Rubik backend.");
throw new RuntimeException(
"Failed to create detector from path " + model.modelFile.getPath());
}
logger.debug("Created detector for model " + model.modelFile.getName());
// Register the cleaner to release the detector when it goes out of scope
cleaner.register(this, this::release);
}
/**
* Returns the classes that the detector can detect
*
* @return The classes
*/
@Override
public List<String> getClasses() {
return model.properties.labels();
}
/**
* Detects objects in the given input image using the rubikDetector.
*
* @param in The input image to perform object detection on.
* @param nmsThresh The threshold value for non-maximum suppression.
* @param boxThresh The threshold value for bounding box detection.
* @return A list of NeuralNetworkPipeResult objects representing the detected objects. Returns an
* empty list if the detector is not initialized or if no objects are detected.
*/
@Override
public List<NeuralNetworkPipeResult> detect(Mat in, double nmsThresh, double boxThresh) {
if (!isValid()) {
logger.error(
"Detector is not initialized, and so it can't be released! Model: "
+ model.modelFile.getName());
return null;
}
// Resize the frame to the input size of the model
Mat letterboxed = new Mat();
Letterbox scale =
Letterbox.letterbox(in, letterboxed, this.inputSize, ColorHelper.colorToScalar(Color.GRAY));
if (!letterboxed.size().equals(this.inputSize)) {
letterboxed.release();
throw new RuntimeException("Letterboxed frame is not the right size!");
}
// Detect objects in the letterboxed frame
var results = RubikJNI.detect(ptr, letterboxed.getNativeObjAddr(), boxThresh, nmsThresh);
letterboxed.release();
if (results == null) {
return List.of();
}
return scale.resizeDetections(
List.of(results).stream()
.map(it -> new NeuralNetworkPipeResult(it.rect, it.class_id, it.conf))
.toList());
}
/** Thread-safe method to release the detector. */
@Override
public void release() {
// Checks if the atomic is 'false', and if so, sets it to 'true'
if (released.compareAndSet(false, true)) {
if (!isValid()) {
logger.error(
"Detector is not initialized, and so it can't be released! Model: "
+ model.modelFile.getName());
return;
}
RubikJNI.destroy(ptr);
logger.debug("Released detector for model " + model.modelFile.getName());
}
}
private boolean isValid() {
return ptr != 0;
}
}

10
photon-core/src/main/java/org/photonvision/vision/pipeline/ObjectDetectionPipeline.java
View File

@@ -107,19 +107,15 @@ public class ObjectDetectionPipeline
protected CVPipelineResult process(Frame frame, ObjectDetectionPipelineSettings settings) {
long sumPipeNanosElapsed = 0;
// ***************** change based on backend ***********************
CVPipeResult<List<NeuralNetworkPipeResult>> rknnResult =
CVPipeResult<List<NeuralNetworkPipeResult>> neuralNetworkResult =
objectDetectorPipe.run(frame.colorImage);
sumPipeNanosElapsed += rknnResult.nanosElapsed;
sumPipeNanosElapsed += neuralNetworkResult.nanosElapsed;
var names = objectDetectorPipe.getClassNames();
frame.colorImage.getMat().copyTo(frame.processedImage.getMat());
// ***************** change based on backend ***********************
var filterContoursResult = filterContoursPipe.run(rknnResult.output);
var filterContoursResult = filterContoursPipe.run(neuralNetworkResult.output);
sumPipeNanosElapsed += filterContoursResult.nanosElapsed;
CVPipeResult<List<PotentialTarget>> sortContoursResult =

20
photon-server/src/main/java/org/photonvision/Main.java
View File

@@ -40,6 +40,7 @@ import org.photonvision.common.networking.NetworkManager;
import org.photonvision.common.util.TestUtils;
import org.photonvision.jni.PhotonTargetingJniLoader;
import org.photonvision.jni.RknnDetectorJNI;
import org.photonvision.jni.RubikDetectorJNI;
import org.photonvision.mrcal.MrCalJNILoader;
import org.photonvision.raspi.LibCameraJNILoader;
import org.photonvision.server.Server;
@@ -231,12 +232,31 @@ public class Main {
try {
if (Platform.isRK3588()) {
RknnDetectorJNI.forceLoad();
if (RknnDetectorJNI.getInstance().isLoaded()) {
logger.info("RknnDetectorJNI loaded successfully.");
} else {
logger.error("Failed to load RknnDetectorJNI!");
}
} else {
logger.error("Platform does not support RKNN based machine learning!");
}
} catch (IOException e) {
logger.error("Failed to load rknn-JNI!", e);
}
try {
if (Platform.isQCS6490()) {
RubikDetectorJNI.forceLoad();
if (RubikDetectorJNI.getInstance().isLoaded()) {
logger.info("RubikDetectorJNI loaded successfully.");
} else {
logger.error("Failed to load RubikDetectorJNI!");
}
} else {
logger.error("Platform does not support Rubik based machine learning!");
}
} catch (IOException e) {
logger.error("Failed to load rubik-JNI!", e);
}
try {
MrCalJNILoader.forceLoad();
} catch (IOException e) {

39
photon-server/src/main/java/org/photonvision/server/RequestHandler.java
View File

@@ -605,13 +605,40 @@ public class RequestHandler {
return;
}
String modelFileExtension = "";
NeuralNetworkModelManager.Family family;
switch (Platform.getCurrentPlatform()) {
case LINUX_QCS6490:
modelFileExtension = "tflite";
family = NeuralNetworkModelManager.Family.RUBIK;
break;
case LINUX_RK3588_64:
modelFileExtension = "rknn";
family = NeuralNetworkModelManager.Family.RKNN;
break;
default:
ctx.status(400);
ctx.result("The current platform does not support object detection models");
logger.error("The current platform does not support object detection models");
return;
}
// If adding additional platforms, check platform matches
if (!modelFile.extension().contains("rknn")) {
if (!modelFile.extension().contains(modelFileExtension)) {
ctx.status(400);
ctx.result(
"The uploaded file was not of type 'rknn'. The uploaded file should be a .rknn file.");
"The uploaded file was not of type '"
+ modelFileExtension
+ "'. The uploaded file should be a ."
+ modelFileExtension
+ " file.");
logger.error(
"The uploaded file was not of type 'rknn'. The uploaded file should be a .rknn file.");
"The uploaded file was not of type '"
+ modelFileExtension
+ "'. The uploaded file should be a ."
+ modelFileExtension
+ " file.");
return;
}
@@ -638,13 +665,11 @@ public class RequestHandler {
.addModelProperties(
new ModelProperties(
modelPath,
modelFile.filename().replaceAll(".rknn", ""),
modelFile.filename().replaceAll("." + modelFileExtension, ""),
labels,
width,
height,
NeuralNetworkModelManager.Family.RKNN, // This can be determined by platform if
// additional platforms are
// supported
family,
version));
logger.debug(

BIN
photon-server/src/main/resources/models/algae-coral-yolov8s.tflite Normal file
View File

Binary file not shown.

BIN
photon-server/src/main/resources/models/yolov8nCOCO.tflite Normal file
View File

Binary file not shown.

5
photon-targeting/src/main/java/org/photonvision/common/hardware/Platform.java
View File

@@ -55,7 +55,7 @@ public enum Platform {
"Linux AARCH 64-bit with QCS6490",
Platform::getLinuxDeviceTreeModel,
"linuxarm64",
true,
false,
OSType.LINUX,
true), // QCS6490 SBCs
LINUX_AARCH64(
@@ -173,7 +173,6 @@ public enum Platform {
String.format("Unknown Platform. OS: %s, Architecture: %s", OS_NAME, OS_ARCH);
private static final String UnknownDeviceModelString = "Unknown";
// TODO: add rubik, but waiting for more info on architecture
public static Platform getCurrentPlatform() {
String OS_NAME;
if (Platform.OS_NAME != null) {
@@ -272,7 +271,7 @@ public enum Platform {
}
private static boolean isRubik() {
return fileHasText("/proc/device-tree/model", "Rubik");
return fileHasText("/proc/device-tree/model", "RUBIK");
}
static String getLinuxDeviceTreeModel() {