Team2890/PhotonVision
4
0
Fork 0
mirror of https://github.com/PhotonVision/photonvision synced 2026-06-24 01:31:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into py-docs

Browse Source
This commit is contained in:
Sam Freund
2025-04-13 14:50:54 -05:00
committed by GitHub
parent a5b1cc0ded cbdea5d0f1
commit a5bc63878d
8 changed files with 151 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
docs/source/docs/pipelines/about-pipelines.md
View File

@@ -10,37 +10,57 @@ A vision pipeline represents a series of steps that are used to acquire an image
## Types of Pipelines
### Reflective
### AprilTag / AruCo
This is the most common pipeline type and it is based on detecting targets with retroreflective tape. In the contours tab of this pipeline type, you can filter the area, width/height ratio, fullness, degree of speckle rejection.
This pipeline type is based on detecting AprilTag fiducial markers. More information about AprilTags can be found in the [WPILib documentation](https://docs.wpilib.org/en/stable/docs/software/vision-processing/apriltag/apriltag-intro.html). This pipeline provides easy to use 3D pose information which allows localization.
:::{note}
In order to get 3D Pose data about AprilTags, you are required to {ref}`calibrate your camera<docs/calibration/calibration:Calibrating Your Camera>`.
:::
### Object Detection
This pipeline type is based on detecting objects using a neural network. The object detection pipeline uses a pre-trained model to detect objects in the camera stream.
:::{note}
This pipeline type is only supported on the Orange Pi 5/5+ coprocessors due to its Neural Processing Unit used by PhotonVision to support running ML-based object detection.
:::
### Colored Shape
This pipeline type is based on detecting different shapes like circles, triangles, quadrilaterals, or a polygon. An example usage would be detecting yellow PowerCells from the 2020 FRC game. You can read more about the specific settings available in the contours page.
### AprilTag / AruCo
### Reflective
This pipeline type is based on detecting AprilTag fiducial markers. More information about AprilTags can be found in the WPILib documentation. While being more performance intensive than the reflective and colored shape pipeline, it has the benefit of providing easy to use 3D pose information which allows localization.
This pipeline type is based on detecting targets with reflective tape. In the contours tab of this pipeline type, you can filter the area, width/height ratio, fullness, degree of speckle rejection.
:::{note}
In order to get 3D Pose data about AprilTags, you are required to {ref}`calibrate your camera<docs/calibration/calibration:Calibrating Your Camera>`.
This pipeline type is not used anymore due to FRC's removal of retro-reflective tape from the game. It is still available as a pipeline for legacy purposes.
:::
## Note About Multiple Cameras and Pipelines
When using more than one camera, it is important to keep in mind that all cameras run one pipeline each, all publish to NT, and all send both streams. This will have a noticeable affect on performance and we recommend users limit themselves to 1-2 cameras per coprocessor.
## Pipeline Steps
## Pipeline Configuration
Reflective and Colored Shape Pipelines have 4 steps (represented as 4 tabs):
Each pipeline has a set of tabs that are used to configure the pipeline. All pipelines follow a similar structure with an Input and Output tab, as well as a set of tabs that are specific to the pipeline type.
1. Input: This tab allows the raw camera image to be modified before it gets processed. Here, you can set exposure, brightness, gain, orientation, and resolution.
2. Threshold (Only Reflective and Colored Shape): This tabs allows you to filter our specific colors/pixels in your camera stream through HSV tuning. The end goal here is having a black and white image that will only have your target lit up.
3. Contours: After thresholding, contiguous white pixels are grouped together, and described by a curve that outlines the group. This curve is called a "contour" which represent various targets on your screen. Regardless of type, you can filter how the targets are grouped, their intersection, and how the targets are sorted. Other available filters will change based on different pipeline types.
4. Output: Now that you have filtered all of your contours, this allows you to manipulate the detected target via orientation, the offset point, and offset.
- Input: This tab allows the raw camera image to be modified before it gets processed. Here, you can set exposure, brightness, gain, orientation, and resolution.
AprilTag / AruCo Pipelines have 3 steps:
- Output: This allows you to manipulate the detected target via the target offset point (for calculating pitch/yaw) and robot (crosshair) offset. In addition, it allows users to send additional (up to 5) outputs through PhotonLib.
1. Input: This is the same as the above.
2. AprilTag: This step include AprilTag specific tuning parameters, such as decimate, blur, threads, pose iterations, and more.
3. Output: This is the same as the above.
Pipielines also have additional tabs that are specific to the pipeline type. Listed below are the tabs for each pipeline type.
### AprilTag / AruCo Pipelines
- AprilTag: This tab includes AprilTag specific tuning parameters, such as decimate, blur, threads, pose iterations, and more.
### Object Detection Pipelines
- Object Detection: This tab allows you to filter results from the neural network, such as confidence, area, and width/height ratio. The end goal of this tab is to filter out any false positives.
### Reflective and Colored Shape Pipelines
- Threshold: This tab allows you to filter out specific colors/pixels in your camera stream through HSV tuning. The end goal here is having a black and white image that will only have your target lit up.
- Contours: After thresholding, contiguous white pixels are grouped together, and described by a curve that outlines the group. This curve is called a "contour" which represent various targets on your screen. Regardless of type, you can filter how the targets are grouped, their intersection, and how the targets are sorted. Other available filters will change based on different pipeline types.

15
photon-core/src/main/java/org/photonvision/vision/calibration/CameraCalibrationCoefficients.java
View File

@@ -116,6 +116,8 @@ public class CameraCalibrationCoefficients implements Releasable {
double p1 = getDistCoeffsMat().get(0, 2)[0];
double p2 = getDistCoeffsMat().get(0, 3)[0];
Size rotatedImageSize = null;
// A bunch of horrifying opaque rotation black magic. See image-rotation.md for more details.
switch (rotation) {
case DEG_0:
@@ -136,6 +138,9 @@ public class CameraCalibrationCoefficients implements Releasable {
// P2
rotatedDistCoeffs.put(0, 3, -p1);
// The rotated image size is the same as the unrotated image size, but the width and height
// are flipped
rotatedImageSize = new Size(unrotatedImageSize.height, unrotatedImageSize.width);
break;
case DEG_180_CCW:
// CX
@@ -147,6 +152,9 @@ public class CameraCalibrationCoefficients implements Releasable {
rotatedDistCoeffs.put(0, 2, -p1);
// P2
rotatedDistCoeffs.put(0, 3, -p2);
// The rotated image size is the same as the unrotated image size
rotatedImageSize = unrotatedImageSize;
break;
case DEG_90_CCW:
// FX
@@ -164,6 +172,9 @@ public class CameraCalibrationCoefficients implements Releasable {
// P2
rotatedDistCoeffs.put(0, 3, p1);
// The rotated image size is the same as the unrotated image size, but the width and height
// are flipped
rotatedImageSize = new Size(unrotatedImageSize.height, unrotatedImageSize.width);
break;
}
@@ -174,8 +185,6 @@ public class CameraCalibrationCoefficients implements Releasable {
rotatedIntrinsics.release();
rotatedDistCoeffs.release();
var rotatedImageSize = new Size(unrotatedImageSize.height, unrotatedImageSize.width);
return new CameraCalibrationCoefficients(
rotatedImageSize,
newIntrinsics,
@@ -189,7 +198,7 @@ public class CameraCalibrationCoefficients implements Releasable {
@JsonIgnore
public Mat getCameraIntrinsicsMat() {
return cameraIntrinsics.getAsMat();
return cameraIntrinsics.getAsMatOfDouble();
}
@JsonIgnore

16
photon-core/src/main/java/org/photonvision/vision/calibration/JsonMatOfDouble.java
View File

@@ -58,18 +58,25 @@ public class JsonMatOfDouble implements Releasable {
}
@JsonIgnore
public static double[] getDataFromMat(Mat mat) {
double[] data = new double[(int) (mat.total() * mat.elemSize())];
private static double[] getDataFromMat(Mat mat) {
double[] data = new double[(int) mat.total()];
mat.get(0, 0, data);
return data;
}
/**
* Returns a JsonMatOfDouble by copying the data from a Mat. The Mat type must be {@link
* CvType#CV_64FC1}.
*
* @param mat The Mat.
* @return The JsonMatOfDouble
*/
public static JsonMatOfDouble fromMat(Mat mat) {
return new JsonMatOfDouble(mat.rows(), mat.cols(), getDataFromMat(mat));
}
@JsonIgnore
public Mat getAsMat() {
private Mat getAsMat() {
if (this.type != CvType.CV_64FC1) return null;
if (wrappedMat == null) {
@@ -108,9 +115,6 @@ public class JsonMatOfDouble implements Releasable {
@Override
public void release() {
if (wrappedMat != null) {
wrappedMat.release();
}
if (wrappedMatOfDouble != null) {
wrappedMatOfDouble.release();
}

4
photon-core/src/main/java/org/photonvision/vision/pipe/impl/Calibrate3dPipe.java
View File

@@ -303,7 +303,7 @@ public class Calibrate3dPipe
Calib3d.solvePnP(
o.objectPoints,
o.imagePoints,
cameraMatrixMat.getAsMat(),
cameraMatrixMat.getAsMatOfDouble(),
distortionCoefficientsMat.getAsMatOfDouble(),
rvec,
tvec);
@@ -314,7 +314,7 @@ public class Calibrate3dPipe
List<BoardObservation> observations =
createObservations(
in,
cameraMatrixMat.getAsMat(),
cameraMatrixMat.getAsMatOfDouble(),
distortionCoefficientsMat.getAsMatOfDouble(),
rvecs,
tvecs,

5
photon-core/src/test/java/org/photonvision/vision/pipeline/Calibrate3dPipeTest.java
View File

@@ -304,7 +304,10 @@ public class Calibrate3dPipeTest {
Mat raw = Imgcodecs.imread(file.getAbsolutePath());
Mat undistorted = new Mat(new Size(imgRes.width * 2, imgRes.height * 2), raw.type());
Calib3d.undistort(
raw, undistorted, cal.cameraIntrinsics.getAsMat(), cal.distCoeffs.getAsMat());
raw,
undistorted,
cal.cameraIntrinsics.getAsMatOfDouble(),
cal.distCoeffs.getAsMatOfDouble());
TestUtils.showImage(undistorted, "undistorted " + file.getName(), 1);
raw.release();
undistorted.release();

88
photon-core/src/test/java/org/photonvision/vision/pipeline/CalibrationRotationPipeTest.java
View File

@@ -17,6 +17,7 @@
package org.photonvision.vision.pipeline;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import java.io.IOException;
@@ -173,6 +174,93 @@ public class CalibrationRotationPipeTest {
}
}
@Test
public void testRotateCoefficients180multiple() {
ImageRotationMode rot = ImageRotationMode.DEG_180_CCW;
// GIVEN A camera calibration
var res = new Size(1270, 720);
double fx = 900;
double fy = 951;
double cx = 500;
double cy = 321;
double[] intrinsics = {fx, 0, cx, 0, fy, cy, 0, 0, 1};
double[] distCoeffs = {
0.25,
-1.5,
0.0017808248356550637,
.00004,
2.179764689221826,
-0.034952777924711353,
0.09625562194891251,
-0.1860797479660746
};
CameraCalibrationCoefficients coeffs =
new CameraCalibrationCoefficients(
res,
new JsonMatOfDouble(3, 3, intrinsics),
new JsonMatOfDouble(1, 8, distCoeffs),
new double[] {},
List.of(),
new Size(),
1,
CameraLensModel.LENSMODEL_OPENCV);
// WHEN the camera calibration is rotated 180 degrees
var coeffs2 = coeffs.rotateCoefficients(rot);
// THEN the optical center should be rotated 180 degrees
double[] rotatedCamMat = {fx, 0, res.width - cx, 0, fy, res.height - cy, 0, 0, 1};
assertArrayEquals(rotatedCamMat, coeffs2.cameraIntrinsics.data);
// AND the image size should be the same
assertEquals(res, coeffs2.unrotatedImageSize);
// WHEN the camera calibration is rotated 180 degrees
var coeffs3 = coeffs2.rotateCoefficients(rot);
// THEN the camera matrix will be the same as the original
assertArrayEquals(intrinsics, coeffs3.cameraIntrinsics.data);
// AND the image size should be the same
assertEquals(res, coeffs2.unrotatedImageSize);
}
@CartesianTest
public void testCalibrationDataIsValidWithRotation(@Enum ImageRotationMode rot) {
double[] intrinsics = {
900, 0, 500,
0, 951, 321,
0, 0, 1
};
double[] distCoeffs = {
0.25,
-1.5,
0.0017808248356550637,
.00004,
2.179764689221826,
-0.034952777924711353,
0.09625562194891251,
-0.1860797479660746
};
// GIVEN A camera calibration
CameraCalibrationCoefficients coeffs =
new CameraCalibrationCoefficients(
new Size(1270, 720),
new JsonMatOfDouble(3, 3, intrinsics),
new JsonMatOfDouble(1, 8, distCoeffs),
new double[] {},
List.of(),
new Size(),
1,
CameraLensModel.LENSMODEL_OPENCV);
// WHEN A camera calibration is rotated 4 times
for (int i = 0; i < 4; i++) {
coeffs = coeffs.rotateCoefficients(rot);
}
// THEN, it should be like it was never rotated at all
assertArrayEquals(intrinsics, coeffs.cameraIntrinsics.data);
assertArrayEquals(distCoeffs, coeffs.distCoeffs.data);
}
@Test
public void testApriltagRotated() {
// matt's lifecam

4
photon-core/src/test/java/org/photonvision/vision/pipeline/CirclePNPTest.java
View File

@@ -66,16 +66,12 @@ public class CirclePNPTest {
assertNotNull(cameraCalibration);
assertEquals(3, cameraCalibration.cameraIntrinsics.rows);
assertEquals(3, cameraCalibration.cameraIntrinsics.cols);
assertEquals(3, cameraCalibration.cameraIntrinsics.getAsMat().rows());
assertEquals(3, cameraCalibration.cameraIntrinsics.getAsMat().cols());
assertEquals(3, cameraCalibration.cameraIntrinsics.getAsMatOfDouble().rows());
assertEquals(3, cameraCalibration.cameraIntrinsics.getAsMatOfDouble().cols());
assertEquals(3, cameraCalibration.getCameraIntrinsicsMat().rows());
assertEquals(3, cameraCalibration.getCameraIntrinsicsMat().cols());
assertEquals(1, cameraCalibration.distCoeffs.rows);
assertEquals(5, cameraCalibration.distCoeffs.cols);
assertEquals(1, cameraCalibration.distCoeffs.getAsMat().rows());
assertEquals(5, cameraCalibration.distCoeffs.getAsMat().cols());
assertEquals(1, cameraCalibration.distCoeffs.getAsMatOfDouble().rows());
assertEquals(5, cameraCalibration.distCoeffs.getAsMatOfDouble().cols());
assertEquals(1, cameraCalibration.getDistCoeffsMat().rows());

4
photon-core/src/test/java/org/photonvision/vision/pipeline/SolvePNPTest.java
View File

@@ -70,16 +70,12 @@ public class SolvePNPTest {
assertNotNull(cameraCalibration);
assertEquals(3, cameraCalibration.cameraIntrinsics.rows);
assertEquals(3, cameraCalibration.cameraIntrinsics.cols);
assertEquals(3, cameraCalibration.cameraIntrinsics.getAsMat().rows());
assertEquals(3, cameraCalibration.cameraIntrinsics.getAsMat().cols());
assertEquals(3, cameraCalibration.cameraIntrinsics.getAsMatOfDouble().rows());
assertEquals(3, cameraCalibration.cameraIntrinsics.getAsMatOfDouble().cols());
assertEquals(3, cameraCalibration.getCameraIntrinsicsMat().rows());
assertEquals(3, cameraCalibration.getCameraIntrinsicsMat().cols());
assertEquals(1, cameraCalibration.distCoeffs.rows);
assertEquals(5, cameraCalibration.distCoeffs.cols);
assertEquals(1, cameraCalibration.distCoeffs.getAsMat().rows());
assertEquals(5, cameraCalibration.distCoeffs.getAsMat().cols());
assertEquals(1, cameraCalibration.distCoeffs.getAsMatOfDouble().rows());
assertEquals(5, cameraCalibration.distCoeffs.getAsMatOfDouble().cols());
assertEquals(1, cameraCalibration.getDistCoeffsMat().rows());