mirror of
https://github.com/PhotonVision/photonvision
synced 2026-06-19 00:41:41 +00:00
311 lines
13 KiB
Plaintext
311 lines
13 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bb5367ce",
|
||
"metadata": {},
|
||
"source": [
|
||
"# RKNN Conversion Guide\n",
|
||
"\n",
|
||
"----------------------------\n",
|
||
"\n",
|
||
"### Before you start\n",
|
||
"\n",
|
||
"If you are not using Google Colab, it is recommended to create a separate [Python virtual environment](https://docs.python.org/3/library/venv.html) before you run the scripts or the Python notebook from this project. This ensures that packages installed for the conversion process do not conflict with other packages you may already have setup."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5f42d0a144caceb6",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Preinstallation\n",
|
||
"\n",
|
||
"This notebook requires the use of external Python scripts. Please run the installation script below to import these external scripts if you do not have them already."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7903189e",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import subprocess\n",
|
||
"\n",
|
||
"# Define scripts with URLs and inferred filenames\n",
|
||
"# DO NOT modify the filenames\n",
|
||
"scripts = [\n",
|
||
" {\n",
|
||
" \"url\": \"https://raw.githubusercontent.com/PhotonVision/photonvision/ba1c0db7e19db090ca04a8375255b00db2e0babd/scripts/rknn-convert-tool/create_onnx.py\",\n",
|
||
" \"filename\": \"create_onnx.py\" # CREATE_ONNX_SCRIPT\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"url\": \"https://raw.githubusercontent.com/PhotonVision/photonvision/ba1c0db7e19db090ca04a8375255b00db2e0babd/scripts/rknn-convert-tool/create_rknn.py\",\n",
|
||
" \"filename\": \"create_rknn.py\" # CREATE_RKNN_SCRIPT\n",
|
||
" }\n",
|
||
"]\n",
|
||
"\n",
|
||
"# Download each script\n",
|
||
"for script in scripts:\n",
|
||
" try:\n",
|
||
" subprocess.run([\"wget\", script[\"url\"], \"-O\", script[\"filename\"]]).check_returncode()\n",
|
||
" print(f\"Successfully downloaded: {script['filename']}\")\n",
|
||
" except subprocess.CalledProcessError as e:\n",
|
||
" print(f\"Failed to download script from URL: {script['url']}\")\n",
|
||
" print(e)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d68be4aba4d3022b",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### *Numpy Fix* - Important for Google Colab Users\n",
|
||
"\n",
|
||
"Google Colab comes with an incompatible version of Numpy installed. To fix this, please run the following cells below and **restart your session** when prompted."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "de0310a3e4401233",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%pip uninstall numpy -y\n",
|
||
"%pip install \"numpy>=1.23.0,<2.0.0\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d498ed79",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Step 1: Convert to ONNX\n",
|
||
"\n",
|
||
"To convert to ONNX, simply run the `create_onnx.py` script, providing the path to your model weights and specifying the model version, as shown below."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "0659e15f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# where version is either yolov5, yolov8, or yolov11, and model_path is the path to your weights file (.pt)\n",
|
||
"%run create_onnx.py --version yolov8 --model_path myyolov8model.pt"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "86ff07e6",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Step 2: Download RKNN API\n",
|
||
"\n",
|
||
"You can either use `pip` below to automatically detect and install the correct RKNN API Python library for you, or install it manually.\n",
|
||
"\n",
|
||
"#### Automatic installation\n",
|
||
"\n",
|
||
"Please run `pip` below. If it does not work, refer to the instructions for manual installation.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7ec11f96",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%pip install rknn-toolkit2"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "8b57fe4d",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### Manual Installation (If Automatic Installation Fails)\n",
|
||
"Visit the [RKNN Toolkit 2](https://github.com/airockchip/rknn-toolkit2) Github repository, then click on rknn-toolkit2, followed by packages.\n",
|
||
"If you are running an x86_64 CPU (e.g., most Intel and AMD processors), select that option; otherwise, choose arm64 for ARM-based computers (e.g., M-series Macs or Snapdragon processors). If you're unsure which CPU you're using, check your system settings for processor architecture information.\n",
|
||
"\n",
|
||
"Once you've selected the correct CPU architecture, you'll see multiple packages. The file names will look something like:\n",
|
||
"`rknn_toolkit2-2.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`.\n",
|
||
"The numbers after cp correspond to your Python version. For example, if you're using Python 3.10, look for a package with cp310 in the name. For Python 3.8, look for cp38; for Python 3.7, cp37, and so on.\n",
|
||
"\n",
|
||
"Once you've found the correct package, click the \"Raw\" button to download the .whl file. Then, run the following command in your terminal, replacing rknn_toolkit2.whl with the actual path to the file you downloaded:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7414b120",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%pip install rknn_toolkit2.whl"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c1db5ef0",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Step 3: Convert to RKNN\n",
|
||
"\n",
|
||
"Please review the notes about quantization before running the RKNN conversion."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5e56b2f64bf6e85f",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Quantization\n",
|
||
"\n",
|
||
"When performing quantization, it is critical to provide representative images of the objects or scenes you are trying to detect. These images are used to calibrate the model’s internal activations and greatly influence the final performance.\n",
|
||
"\n",
|
||
"It is recommended to use 300–500 representative images that reflect the real-world input your model will encounter. As the old saying goes, it’s quality over quantity — having a diverse, relevant set matters more than simply having many images.\n",
|
||
"\n",
|
||
"Quantization will cause some loss in model accuracy. However, if your calibration images are chosen wisely, this accuracy drop should be minimal and acceptable. If the sampled images are too uniform or unrelated, your quantized model's performance may worsen significantly.\n",
|
||
"\n",
|
||
"The script will automatically sample representative images randomly from the provided dataset. While this usually works well, please verify that the dataset contains diverse and relevant examples of your target objects. As a reminder, the images used to quantize the model are stored in the text file specified by `--img_dataset_txt`.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "93e0d0622df170e",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Optional: Download a dataset from Roboflow for quantization\n",
|
||
"\n",
|
||
"If you do not already have a dataset or set of images containing the objects you want to detect, follow the steps below to download one from Roboflow Universe.\n",
|
||
"\n",
|
||
"#### **Step 1: Search for a Dataset**\n",
|
||
"\n",
|
||
"Go to [Roboflow Universe](https://universe.roboflow.com) and use the search bar to locate a dataset relevant to what you want to detect.\n",
|
||
"**Note:** The dataset must include the classes or object types you intend to detect.\n",
|
||
"\n",
|
||
"#### **Step 2: Access the Dataset Tab**\n",
|
||
"\n",
|
||
"After selecting a suitable project, navigate to the **Dataset** tab. Click the **\"Download Dataset\"** button. A prompt will appear with several options, including:\n",
|
||
"\n",
|
||
"- Train a model with this dataset\n",
|
||
"- Train from a portion of this dataset\n",
|
||
"- Download dataset\n",
|
||
"\n",
|
||
"Select **Download dataset**.\n",
|
||
"\n",
|
||
"#### **Step 3: Choose Format and View Download Code**\n",
|
||
"\n",
|
||
"- Under **Image and Annotation Format**, choose the version of YOLO you are using:\n",
|
||
" - For **YOLOv5**, choose `YOLOv5 PyTorch`\n",
|
||
" - For **YOLOv8**, choose `YOLOv8`\n",
|
||
" - For **YOLOv11**, choose `YOLOv11`\n",
|
||
"- If multiple annotation formats are listed for your model, always select the one ending in **\"PyTorch\"**.\n",
|
||
"\n",
|
||
"Then, under **Download Options**, click **\"Show Download Code\"** and continue.\n",
|
||
"\n",
|
||
"In the resulting screen, you will see three tabs:\n",
|
||
"- **Jupyter**\n",
|
||
"- **Terminal**\n",
|
||
"- **Raw URL**\n",
|
||
"\n",
|
||
"Select the **Terminal** tab and copy the provided command.\n",
|
||
"\n",
|
||
"#### **Step 4: Paste and Run**\n",
|
||
"\n",
|
||
"Paste the copied command into the notebook cell below and run it. This will download and extract the dataset into your environment, making it ready for use in the quantization process.\n",
|
||
"\n",
|
||
"Make sure to prefix the command with \"`!`\" so it executes properly in this Jupyter Notebook environment."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "8bf75c9dcb328c84",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"!curl -L \"https://universe.roboflow.com/ds/FaF3HbDmF7?key=iMoJR25O9H\" > roboflow.zip; unzip roboflow.zip; rm roboflow.zip"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "72bad9cac670f1ab",
|
||
"metadata": {},
|
||
"source": [
|
||
"### RKNN Conversion Script\n",
|
||
"To get started, run the `create_rknn.py` script below, replacing the arguments with your own values. Refer to the table below for detailed information on each argument’s purpose and usage. The `--model_path` argument should point to your exported ONNX model from Step 1, and `--img_dir` must reference a valid directory containing either a dataset or a set of images to be used for quantization.\n",
|
||
"\n",
|
||
"#### Overview of the `create_rknn.py` script\n",
|
||
"\n",
|
||
"This script converts a YOLO ONNX model to RKNN format using a set of calibration images. It's designed to work with either:\n",
|
||
"\n",
|
||
"- A flat directory of images (e.g. `train/images`), **or**\n",
|
||
"- A dataset directory containing a `data.yaml` file that defines `train`, `val`, and/or `test` folders.\n",
|
||
"\n",
|
||
"##### Arguments\n",
|
||
"\n",
|
||
"| Argument | Type | Description |\n",
|
||
"|----------|------|-----------------------------------------------------------------------------------------------------------------|\n",
|
||
"| `--img_dir` (`-d`) | `str` (required) | Path to your image directory. This can either be a folder of images **or** a dataset folder with a `data.yaml`. |\n",
|
||
"| `--model_path` (`-m`) | `str` (required) | Path to your YOLO ONNX model, created in Step 1. |\n",
|
||
"| `--num_imgs` (`-ni`) | `int` (default: `300`) | Number of images to use for quantization calibration. |\n",
|
||
"| `--disable_quantize` (`-dq`) | `bool` (default: `False`) | Set to `True` to skip quantization entirely. Not recommended for performance, and should not be used for deployment on PhotonVision, which requires quantization. |\n",
|
||
"| `--rknn_output` (`-o`) | `str` (default: `out.rknn`) | File path where the final RKNN model should be saved. |\n",
|
||
"| `--img_dataset_txt` (`-ds`) | `str` (default: `imgs.txt`) | File path to store the list of images used during quantization. |\n",
|
||
"| `--verbose` (`-vb`) | `bool` (default: `False`) | Enable detailed logging from the RKNN API during conversion. |\n",
|
||
"\n",
|
||
"\n",
|
||
"##### *Notes*\n",
|
||
"\n",
|
||
"1. This script is designed for use with [PhotonVision](https://photonvision.org), and by default sets the target platform for RKNN conversion to `RK3588`, a chipset commonly found in many variants of the Orange Pi 5 series (e.g., Orange Pi 5, 5 Pro, 5 Plus, 5 Max, etc.). You may modify the `DEFAULT_PLATFORM` value in the `create_rknn.py` script to match your specific hardware or deployment requirements if necessary.\n",
|
||
"\n",
|
||
"2. If you followed the Roboflow dataset download instructions from the previous section, the dataset will have been extracted to your **current working directory**. In that case, you can simply set `--img_dir` to \"`.`\" to reference the current directory."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "b09656dd",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%run create_rknn.py --img_dir ./datasets --model_path weights.onnx"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5b3a6806",
|
||
"metadata": {},
|
||
"source": [
|
||
"And that’s it! Your RKNN model file is now ready for deployment on an Orange Pi."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": ".venv",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.11.13"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|