Add support for vkapriltag

[yojobama]

Is your feature request related to a problem? Please describe.

PhotonVision currently relies on CPU-based libapriltag for AprilTag detection. On ARM-based embedded coprocessors without CUDA support (such as the Orange Pi 5 / RK3588 Mali GPU), CPU-bound detection heavily limits frame rates and consumes CPU headroom needed for other vision pipelines. Existing GPU detectors are almost exclusively CUDA-dependent or inefficient on non-NVIDIA hardware.

Describe the solution you'd like

Add an optional Vulkan-accelerated backend (vkapriltag) for AprilTag detection alongside libapriltag. The settings UI would ideally support:

• A toggle to select between CPU (libapriltag) and Vulkan GPU (vkapriltag).
• A device selector dropdown to choose the active Vulkan compute device when multiple GPUs/devices are present.
• Graceful fallback to CPU libapriltag if Vulkan 1.1+ initialization fails.

Describe alternatives you've considered

• cuAprilTags / CUDA detectors: Incompatible with ARM Mali GPUs on non-NVIDIA single-board computers like the Orange Pi 5.
• Standard CPU detection: High CPU usage and limited FPS scaling at higher camera resolutions.

To solve this, I created vkapriltag, a custom Vulkan 1.1+ compute detector. On my personal Orange Pi 5 plus, offloading detection to the GPU educed CPU load and improved latency to ~16.1ms at 1920*1080.

Additional context

• Repository: https://github.com/yojobama/vkapriltag
• Requirements: Vulkan 1.1+ driver support.
• Build Integration: CMake-compatible C/C++ library designed for straightforward native linking.
• Contribution: I can assist with C++ native integration, benchmarking, and testing on RK3588 hardware.

i.e: it requires vulkan drivers to be installed, that means adding the appropriate drivers to every image file (like libmali) if they're not already installed.

a test run on my system (ryzen 5600x, rx 9060 xt 16gb, 64gb ram)

[mcm001]

Hi. Prior art here includes @Eeshwar-Krishnan 's halide-apriltag repository. They were last I heard collecting a validation dataset, I'd love to apply that to this detector as well.

As a general reminder, our AI usage policy allows their usage (and this project looks powerful! It might be a difficult conversation if we can't maintain it without LLM help), but issues/pull requests/conversations in comments should always be authored entirely by a human :)

[yojobama]

Hi, I tested the detector on all the apriltag images in the halide-apritlag repo (in all three branches), most of the results are valid as far as I can tell (they do somewhat diverge from libapriltag (sub 1 pixel), but all the detected tags exist in the image and it detects all the tags libapriltag does) (only one image is inconsistant when it comes to which tags are detected, it's a 4k image (testimages/bingus.jpg) and I'm working on diagnosing it).

Regarding the part of maintaining the project without LLM help: Even though most of the project is "vibe-coded" I am familiar with the codebase and able to debug, test, and maintain it without LLM help.

and regarding the integration of the project into photonvision, I guess the next step will be to create a different repo for the JNI bindings (like what you did with RKNN) (I guess I'll open the repo under my account and hand it off to you when it's ready?) and for me to fork the main repo and add support for vkapriltag before submitting a pull-request to integrate it?

[gerth2]

Hey Yoav, really awesome project. PhotonVision does prioritize making bleeding-edge technology accessible to the masses, and having libraries like these works toward that goal.

We do also have a goal of long term viability and stability for users, and as others mentioned here this is pretty new/obscure technology to most of the core development team. We're pumped that you're interested and proactively looking at integration, and would want to encourage that to continue. A few caveats as you go:

1. As a team, we don't have much in the way of AI credits, so the ability to consume and process large PR's is limited
2. We have to consider long-term maintenance, and may ask for architectural choices to keep the new content "optional" behind a new pipeline type or toggle, and to minimize disturbance to existing flow.
3. Depending on speed and how many other developers latch on to it, it may stay in "beta" for a year or two, with a risk of being removed if we can't keep it up to date.
4. Repeatable "same as or better than" testing may be needed against libapriltag - ideally as a unit test in this repo?

Appreciate the time and work though - Open some draft PR's whenever you can, that's the best way for folks to at least have a chance of getting you early feedback.