tcxAzureKinectBodyTracking

Skeleton/bodytracking for TrussC, on top of tcxAzureKinect. Wraps the Azure Kinect Body Tracking SDK (k4abt) — a free, local DNN (no cloud, no usage fees) — and turns its output into TrussC types (Body, BodyJoint, 32 Joints).

It shares the single k4a_capture that the AzureKinect backend already grabs each frame (via AzureKinect::getNativeCapture()), so one device feeds both the point cloud and the skeleton — no second device open.

Status: working — hardware-verified. Builds and runs against the Azure Kinect Body Tracking SDK 1.1.2; example-basic tracks a skeleton overlaid on the live point cloud from a real device.

Tested on Windows 11 with an Orbbec Femto Bolt (via the Orbbec K4A Wrapper), in CPU and DirectML inference modes. Due to the limits of the test machine (NVIDIA Quadro P1000), CUDA / TensorRT modes are not yet validated here — they will be tested on another machine. See TODO.md.

Platforms: Windows / Linux only. k4abt has no macOS support (like k4a).

Requirements

• Azure Kinect Body Tracking SDK (k4abt) 1.1.2 installed.
  • Windows: the MSI from Microsoft installs to C:\Program Files\Azure Kinect Body Tracking SDK. This addon's CMakeLists.txt locates the headers + k4abt.lib there automatically (override with -DK4ABT_SDK_ROOT="…/sdk"), and copies the runtime DLLs + DNN model next to the app exe.
  • GPU inference: DirectML (default here) needs no CUDA — any DX12 GPU works. CUDA / TensorRT are optional faster paths (BodyTracker::Mode).
• The tcxAzureKinect and tcxDepthCamera addons (declared as dependencies), built with the getNativeCapture() / getNativeCalibration() handles.

Using an Orbbec Femto Bolt

k4abt drives a Femto Bolt unchanged — the code here is device-agnostic. Per Orbbec's "Access AKDK Application Software with Femto Bolt" guide:

1. Install the Body Tracking SDK 1.1.2 (above).
2. Register UVC metadata once (admin): obsensor_metadata_win10.ps1 from the Orbbec K4A Wrapper (already done if you ran the tcxAzureKinect point-cloud example).
3. Replace the k4a runtime with the Orbbec K4A Wrapper's — k4a.dll, OrbbecSDK.dll, k4arecord.dll, depthengine_2_0.dll + extensions\ — next to the exe, so it binds the Femto instead of a real Azure Kinect.
4. Verify with the SDK's simple_3d_viewer.exe (admin) before running the example.

So the app's bin/ ends up with both: k4abt runtime (k4abt.dll, onnxruntime*.dll, directml.dll, dnn_model_*.onnx) + the Wrapper's k4a DLLs.

Usage

#include <tcxAzureKinect.h>
#include <tcxAzureKinectBodyTracking.h>
using namespace tcx;

AzureKinect cam;
BodyTracker tracker;

cam.setThreaded(false);            // body tracking shares the per-frame capture
cam.enableDepth();
cam.setup();
tracker.setup(cam);                // builds the k4abt tracker from cam calibration

// per frame:
cam.update();
if (cam.isFrameNew() && tracker.update(cam)) {
    for (const Body& b : tracker.bodies()) {
        Vec3 head = b.joint(Joint::Head).position;     // meters, depth-cam space
        for (const auto& bone : boneList()) {
            const BodyJoint& a = b.joint(bone.first);
            const BodyJoint& c = b.joint(bone.second);
            // draw a -> c ...
        }
    }
}

Joint positions are in meters, depth-camera space — the same convention as tcxAzureKinect's point cloud (toMesh() / frame.world), so skeletons overlay the cloud directly. See example-basic/ for a point cloud + skeleton viewer.

Notes

• Threading: getNativeCapture() is swapped on the grab thread when the camera runs setThreaded(true). Run the camera non-threaded when feeding the tracker (the example does), or drive cam.update() and tracker.update() on one thread.
• Confidence: joints below JointConfidence::Low (i.e. None) are unobserved — skip them when drawing. The SDK currently never reports High.
• One device, one capture: the tracker does not open the device; it reuses the capture tcxAzureKinect already grabbed, so depth/cloud and skeleton are the same frame.

License

MIT. See LICENSES.md. Depends on the Azure Kinect Body Tracking SDK (MIT, Microsoft) which must be installed separately.