feat(deviceio): generate passthrough trackers from a TOML manifest #853
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| function(isaac_teleop_generate_trackers) | ||
| set(_TRACKER_MANIFEST "${CMAKE_SOURCE_DIR}/src/core/deviceio_trackers/trackers.toml") | ||
| set(_TRACKER_DEFAULTS "${CMAKE_SOURCE_DIR}/src/core/deviceio_trackers/defaults.toml") | ||
| set(_GENERATOR "${CMAKE_SOURCE_DIR}/src/core/codegen/generate_trackers.py") | ||
| set(_CODEGEN_TEMPLATES_DIR "${CMAKE_SOURCE_DIR}/src/core/codegen/templates") | ||
| file(GLOB_RECURSE _CODEGEN_IN_TEMPLATES CONFIGURE_DEPENDS | ||
| "${_CODEGEN_TEMPLATES_DIR}/*.template") | ||
| set(_OUT_DIR "${CMAKE_BINARY_DIR}/generated/trackers") | ||
| set(_CMAKE_OUT "${_OUT_DIR}/generated_sources.cmake") | ||
|
|
||
| # Pass --prune-stale so renamed/removed trackers cannot leave orphan headers that | ||
| # still satisfy stale #includes on the generated include path. | ||
| execute_process( | ||
| COMMAND "${Python3_EXECUTABLE}" "${_GENERATOR}" | ||
| --manifest "${_TRACKER_MANIFEST}" | ||
| --defaults "${_TRACKER_DEFAULTS}" | ||
| --out-dir "${_OUT_DIR}" | ||
| --emit-cmake "${_CMAKE_OUT}" | ||
| --prune-stale | ||
| RESULT_VARIABLE _tracker_gen_rc | ||
| OUTPUT_VARIABLE _tracker_gen_out | ||
| ERROR_VARIABLE _tracker_gen_err) | ||
| if(NOT _tracker_gen_rc EQUAL 0) | ||
| message(FATAL_ERROR "tracker codegen failed:\n${_tracker_gen_out}\n${_tracker_gen_err}") | ||
| endif() | ||
|
|
||
| include("${_CMAKE_OUT}") | ||
| set(GENERATED_TRACKER_DIR "${GENERATED_TRACKER_DIR}" PARENT_SCOPE) | ||
| set(GENERATED_TRACKER_FACADE_SOURCES "${GENERATED_TRACKER_FACADE_SOURCES}" PARENT_SCOPE) | ||
| set(GENERATED_TRACKER_LIVE_SOURCES "${GENERATED_TRACKER_LIVE_SOURCES}" PARENT_SCOPE) | ||
| set(GENERATED_TRACKER_REPLAY_SOURCES "${GENERATED_TRACKER_REPLAY_SOURCES}" PARENT_SCOPE) | ||
| set(GENERATED_TRACKER_DEVICEIO_BASE_INC_DIR "${GENERATED_TRACKER_DEVICEIO_BASE_INC_DIR}" PARENT_SCOPE) | ||
| set(GENERATED_TRACKER_DEVICEIO_TRACKERS_INC_DIR "${GENERATED_TRACKER_DEVICEIO_TRACKERS_INC_DIR}" PARENT_SCOPE) | ||
| set(GENERATED_TRACKER_LIVE_IMPL_INC_DIR "${GENERATED_TRACKER_LIVE_IMPL_INC_DIR}" PARENT_SCOPE) | ||
| set(GENERATED_TRACKER_REPLAY_IMPL_INC_DIR "${GENERATED_TRACKER_REPLAY_IMPL_INC_DIR}" PARENT_SCOPE) | ||
| set(GENERATED_TRACKER_INC_DIR "${GENERATED_TRACKER_INC_DIR}" PARENT_SCOPE) | ||
| set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS | ||
| "${_TRACKER_MANIFEST}" "${_TRACKER_DEFAULTS}" "${_GENERATOR}" | ||
| "${CMAKE_SOURCE_DIR}/src/core/codegen/manifest.py" | ||
| "${CMAKE_SOURCE_DIR}/src/core/codegen/templates.py" | ||
| "${CMAKE_SOURCE_DIR}/src/core/codegen/in_renderer.py" | ||
| ${_CODEGEN_IN_TEMPLATES}) | ||
| endfunction() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,165 @@ | ||
| .. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| .. SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| Generated Tracker Code | ||
| ====================== | ||
|
|
||
| Trackers that only move a FlatBuffer between a plugin and the host — *passthrough* trackers — | ||
| are not hand-written. They are declared in a TOML manifest and their C++ and Python glue is | ||
| generated when you configure the build. Adding one is a ``.fbs`` schema plus roughly ten lines | ||
| of TOML; see :ref:`Passthrough trackers (manifest) <passthrough-tracker-manifest>` for the | ||
| authoring workflow. | ||
|
|
||
| This page records what is generated today, what is deliberately still hand-written, and what | ||
| would have to change to generate the rest. | ||
|
|
||
| How it works | ||
| ------------ | ||
|
|
||
| Three inputs drive the generator: | ||
|
|
||
| - :code-file:`src/core/deviceio_trackers/trackers.toml` — one ``[[tracker]]`` entry per tracker, | ||
| carrying only the values that differ from the defaults. | ||
| - :code-file:`src/core/deviceio_trackers/defaults.toml` — the defaults, expressed with | ||
| ``%placeholder%`` substitutions (``%name%``, ``%name_CamelCase%``, and so on). | ||
| - :code-dir:`src/core/codegen` — ``manifest.py`` resolves the placeholders, ``templates.py`` | ||
| derives type and file names, ``templates/`` holds the C++ bodies as ``*.hpp.template`` / | ||
| ``*.cpp.template`` (``@KEY@`` substitution via ``in_renderer.py``), and ``generate_trackers.py`` | ||
| renders the output. | ||
|
|
||
| :code-file:`cmake/GenerateTrackers.cmake` runs the generator through ``execute_process`` at | ||
| **configure** time, not build time, because CMake needs the resulting source list before it can | ||
| define targets. Output lands in ``${CMAKE_BINARY_DIR}/generated/trackers/`` and is **not** checked | ||
| into git, exactly like the ``flatc`` output. Sources are wired through an explicit | ||
| ``generated_sources.cmake`` list (not by globbing that directory). Two consequences worth knowing: | ||
|
|
||
| - Grepping ``src/`` for a generated class such as ``Se3Tracker`` finds the manifest entry and the | ||
| generator, not a ``.cpp``. Searching, debugging, and clangd all need a configured build tree. | ||
| - The manifests, every ``.py`` under ``src/core/codegen``, and every template under | ||
| ``templates/`` are registered as ``CMAKE_CONFIGURE_DEPENDS``, so editing any of them re-runs | ||
| configure on the next build. | ||
|
|
||
| The generator rewrites a file only when its content changed. Configure also passes | ||
| ``--prune-stale`` so that after renaming or removing a tracker, orphan headers cannot remain on | ||
| the generated include path and keep satisfying a stale ``#include``. | ||
|
|
||
| What each entry produces | ||
| ------------------------ | ||
|
|
||
| Per manifest entry, in ``${CMAKE_BINARY_DIR}/generated/trackers/``: | ||
|
|
||
| .. code-block:: text | ||
| :class: code-100col | ||
|
|
||
| deviceio_base/<name>_tracker_base.hpp # I<Name>TrackerImpl interface | ||
| deviceio_trackers/inc/deviceio_trackers/<name>_tracker.hpp | ||
| deviceio_trackers/<name>_tracker.cpp # the ITracker facade | ||
| live_trackers/live_<name>_tracker_impl.{hpp,cpp} # wraps SchemaTracker / SchemaPusher | ||
| replay_trackers/replay_<name>_tracker_impl.{hpp,cpp} | ||
|
|
||
| The shared registration points stay hand-written and ``#include`` a generated ``.inc`` fragment, | ||
| so hand-written trackers keep their own rows. The fragments under ``generated/trackers/inc/`` cover | ||
| the live and replay factories (includes, forward declarations, try-create thunks, dispatch rows, | ||
| factory declarations and definitions), the MCAP recording traits in | ||
| :code-file:`src/core/mcap/cpp/inc/mcap/recording_traits.hpp`, the pybind blocks in | ||
| :code-file:`src/core/deviceio_trackers/python/tracker_bindings.cpp`, and a | ||
| ``_generated_tracker_exports.py`` that :code-file:`src/python/isaacteleop/deviceio_trackers/__init__.py` | ||
| star-imports. Because that last one is spliced into ``__all__``, a new manifest entry needs no | ||
| Python edit at all. | ||
|
|
||
| ``.pyi`` stubs need no work either: :code-file:`src/core/python/generate_stubs.py` derives them from | ||
| the built module. | ||
|
|
||
| Generated today | ||
| --------------- | ||
|
|
||
| .. list-table:: | ||
| :header-rows: 1 | ||
| :widths: 26 26 48 | ||
|
|
||
| * - Manifest entry | ||
| - Class | ||
| - Notes | ||
| * - ``joint_state`` | ||
| - ``JointStateTracker`` | ||
| - One generic joint-space device (leader arm, exoskeleton, ...) | ||
| * - ``se3_tracker`` | ||
| - ``Se3Tracker`` | ||
| - Overrides ``class``; the default would give ``Se3TrackerTracker`` | ||
| * - ``oglo_tactile`` | ||
| - ``OgloTactileTracker`` | ||
| - MCAP channels are ``oglo``/``oglo_tracked``, so ``channel`` is overridden | ||
| * - ``generic_3axis_pedal`` | ||
| - ``Generic3AxisPedalTracker`` | ||
| - Schema lives in ``pedals.fbs``, so ``schema`` and ``channel`` are overridden | ||
| * - ``haptic_command`` | ||
| - ``HapticCommandPushTracker`` | ||
| - ``direction = "out"``: a typed producer wrapping ``SchemaPusher`` | ||
|
|
||
| Still hand-written | ||
| ------------------ | ||
|
|
||
| .. list-table:: | ||
| :header-rows: 1 | ||
| :widths: 34 66 | ||
|
|
||
| * - Tracker | ||
| - Why it is not generated | ||
| * - ``HeadTracker``, ``HandTracker``, ``ControllerTracker`` | ||
| - Real ``xrLocate*`` / hand-tracking calls, not FlatBuffer passthrough | ||
| * - ``FullBodyTracker`` (PICO vendor) | ||
| - Native ``XR_BD_body_tracking`` | ||
| * - ``FullBodyTracker`` (Noitom vendor) | ||
| - Genuinely a passthrough, but a **vendor** of an existing facade — see below | ||
| * - ``MessageChannelTracker`` | ||
| - ``XR_NV_opaque_data_channel`` with its own connection state machine | ||
| * - ``HapticCommandReaderTracker`` | ||
| - Cross-process consumer: ``read_all_samples`` on one collection, buckets by | ||
| ``HapticCommand.endpoint`` (left/right). Paired with generated ``HapticCommandPushTracker``. | ||
| * - ``FrameMetadataTrackerOak`` | ||
| - Holds N ``SchemaTracker`` s, one collection per stream — see below | ||
| * - ``TensorPushTracker`` | ||
| - Deliberately kept as the untyped ``bytes`` escape hatch | ||
|
|
||
| A quick way to tell the two groups apart: only passthrough impls mention ``SchemaTracker`` or | ||
| ``SchemaPusher``. Of the hand-written live impls that use those helpers, each is listed above. | ||
|
|
||
| Future work | ||
| ----------- | ||
|
|
||
| **Multi-endpoint reader (rejected generated shape).** We briefly generated | ||
| ``HapticCommandReaderTracker`` with a ``multi_endpoint`` template (bucket samples by | ||
| ``HapticCommand.endpoint`` on one push-tensor collection). That shape was removed: it made the | ||
| codegen tree branchy for a single device, and a generic multi-endpoint reader does not fit | ||
| ``in/`` or ``out``. The hand-written reader remains the supported approach; a future redesign | ||
| (split tensors/collections per endpoint, or a dedicated generator) is optional follow-up. | ||
|
|
||
| **Multi-stream shape (OAK).** ``FrameMetadataTrackerOak`` owns one ``SchemaTracker`` per camera | ||
| stream, with collection ids derived as ``{prefix}/{StreamName}``. It needs a template variant that | ||
| emits an N-collection reader plus per-stream accessors. This was sequenced last during the initial | ||
| migration precisely so it could not distort the common template, and it remains the closest | ||
| candidate. | ||
|
|
||
| **Vendored shape (Noitom).** ``LiveFullBodyTrackerNoitomImpl`` reads | ||
| ``SchemaTracker<FullBodyPoseRecord, FullBodyPose>``, so it is passthrough by mechanism, but it is a | ||
| second *vendor* of the hand-written ``FullBodyTracker`` rather than its own tracker type. Generating | ||
| it would need a shape that emits **only** a live impl and a vendor-keyed dispatch row — no facade, | ||
| base interface, recording traits, replay half, or pybind block, since it shares all of those with the | ||
| PICO vendor. Its ``collection_id`` and ``max_flatbuffer_size`` also arrive through | ||
| ``TrackerVendor::params`` at runtime instead of being fixed at generation time. | ||
|
|
||
| **Schema pybind bindings.** The field-by-field ``src/core/schema/python/*_bindings.h`` files are | ||
|
Collaborator
There was a problem hiding this comment. @aristarkhovNV, I think you can follow up here for using flatbuffer objs rather than the manually coded bindings as the next step. |
||
| still hand-written. They are derivable from the ``.bfbs`` reflection data ``flatc`` already emits | ||
| (``--bfbs-gen-embed --reflect-names --reflect-types``), so this is a viable follow-up, but it is a | ||
| separate generator with its own risks. | ||
|
|
||
| **Manifest ``python_accessor`` (pybind method name).** Generated ``in`` trackers use | ||
| ``fragments/pybind_in.template``, which binds whatever string ``python_accessor`` holds (default | ||
| ``get_data``; ``oglo_tactile`` and ``generic_3axis_pedal`` override with device-specific names). | ||
| ``direction = "out"`` uses ``push`` via defaults. Future work: remove the manifest key and | ||
| hard-code ``get_data`` on readers (or maybe ``pull``, paired with ``push`` on producers). | ||
| Maybe rename the C++ facade to match if we pick ``pull``, so Python and C++ stay aligned without | ||
| per-tracker overrides. | ||
|
|
||
| If a prospective shape makes the templates hard to follow, leaving that tracker hand-written is the | ||
| correct outcome — the generator exists to remove mechanical duplication, not to model every device. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.