diff --git a/docs/source/device/add_device.rst b/docs/source/device/add_device.rst index 5ad1dd792..fb5a11668 100644 --- a/docs/source/device/add_device.rst +++ b/docs/source/device/add_device.rst @@ -65,12 +65,9 @@ Reference schema: :code-file:`src/core/schema/fbs/pedals.fbs` - **Output table** — The primary payload type (e.g. ``Generic3AxisPedalOutput``) with the device fields. This is what the plugin serializes and pushes. -- **Tracked wrapper** — A table that wraps the output in an optional ``data`` field - (e.g. ``Generic3AxisPedalOutputTracked``). Used by the in-memory tracker API so that - ``data`` can be null when no sample is available. - **Record wrapper** — A table that wraps the output plus ``DeviceDataTimestamp`` (e.g. ``Generic3AxisPedalOutputRecord``). This is the root type written to MCAP channels - by the recorder; trackers serialize into this type in ``serialize_all()``. + by the recorder. - **root_type** — Set to the Record type (e.g. ``root_type Generic3AxisPedalOutputRecord;``). Include ``timestamp.fbs`` for ``DeviceDataTimestamp``; include other shared types (e.g. @@ -126,32 +123,35 @@ tensor samples from OpenXR. Implement a concrete tracker class (e.g. - **Factory registration** — Register your tracker in the live factory dispatch table (see ``LiveDeviceIOFactory``). The factory constructs an ``ITrackerImpl`` that holds a ``SchemaTracker``, builds a ``SchemaTrackerConfig`` from the tracker's stored - configuration, and implements ``update(XrTime)`` and - ``serialize_all(channel_index, callback)``. + configuration, and implements ``update(int64_t monotonic_time_ns)``. In the **Impl**: -- **update()** — Call ``m_schema_reader.read_all_samples(pending_records)``. If the - collection is not present, clear the tracked state (e.g. set ``m_tracked.data = nullptr``). - Otherwise, deserialize the latest sample (or all samples) into your tracked type and - keep the last one for ``get_data()``. -- **serialize_all()** — For each sample in the pending batch, deserialize, build the - Record FlatBuffer (output table + ``DeviceDataTimestamp``), and invoke the callback with - ``(log_time_ns, buffer_ptr, size)``. The buffer is only valid during the callback. If the - device disappeared and there are no samples, you may emit one record with null data and - the update-tick timestamp so the MCAP stream marks absence. +- **Construction** — Build the ``SchemaTrackerConfig`` from the tracker's configuration and + hand it to the ``SchemaTracker``, along with the MCAP channels (or ``nullptr`` when + recording is disabled) and the sub-channel indices to write. +- **update()** — Call ``m_schema_reader.update(m_tracked)``, where ``m_tracked`` is the + published ``Serialized`` handle. That one call reads the pending + samples, writes each of them to MCAP when channels are attached, and publishes the final + one. A tick with no new samples leaves the last-known handle in place; an absent + collection empties it. +- **get_data()** — Return the published handle. See + :ref:`Reading a payload ` for what a consumer may assume about it. + +Recording needs no per-tracker serialization code: ``SchemaTracker`` writes through +``McapTrackerChannels``, which wraps the payload in the Record type with its +``DeviceDataTimestamp``. Reference implementation — split across facade and live backend: - **Tracker facade** — :code-file:`src/core/deviceio_trackers/cpp/generic_3axis_pedal_tracker.cpp` (class ``Generic3AxisPedalTracker``): holds collection configuration, implements ``ITracker``, and exposes ``get_data(session)`` returning - ``Generic3AxisPedalOutputTrackedT`` by dispatching to the session’s + ``Serialized`` by dispatching to the session’s ``IGeneric3AxisPedalTrackerImpl`` (see :code-file:`src/core/deviceio_base/cpp/inc/deviceio_base/generic_3axis_pedal_tracker_base.hpp`). - **Live backend** — :code-file:`src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.cpp` - (``LiveGeneric3AxisPedalTrackerImpl``): composes ``SchemaTracker``, implements ``update()`` and - ``serialize_all()``, and uses ``SchemaTracker::read_all_samples()`` with - ``std::vector`` for the pending batch. See + (``LiveGeneric3AxisPedalTrackerImpl``): composes ``SchemaTracker``, owns the MCAP channels, and + implements ``update()`` as a single ``SchemaTracker::update()`` call. See :code-file:`src/core/live_trackers/cpp/inc/live_trackers/schema_tracker.hpp` for ``SchemaTracker`` and ``SampleResult`` (buffer + timestamp metadata). @@ -166,8 +166,8 @@ the collection and prints samples. Pattern (see :code-file:`examples/schemaio/pe 2. Get required extensions with ``DeviceIOSession::get_required_extensions(trackers)`` and create an ``OpenXRSession``. 3. Create a ``DeviceIOSession`` with ``DeviceIOSession::run(trackers, oxr_session->get_handles())``. -4. Loop: call ``session->update()``, then read ``tracker->get_data(*session)``. If - ``tracked.data`` is non-null, use the latest sample; otherwise sleep briefly and repeat. +4. Loop: call ``session->update()``, then read ``tracker->get_data(*session)``. If the + returned handle is non-empty, use the latest sample; otherwise sleep briefly and repeat. Use the same ``collection_id`` (and optionally ``tensor_identifier``) as the plugin. See :ref:`Schema IO example: build and run ` above for building and running @@ -214,12 +214,13 @@ Both exit after 100 samples, or press Ctrl+C to exit early. the configured identifier, provides ``push_buffer()`` for raw serialized data. Use composition to create typed wrappers (e.g. ``Generic3AxisPedalPusher`` in :code-file:`examples/schemaio/pedal_pusher.cpp`). - **SchemaTracker** (``live_trackers``) — Helper for reading FlatBuffer schema data via - OpenXR tensor collections: discovers collections by identifier, exposes ``read_all_samples()`` into - ``SampleResult`` values. Live tracker implementations (e.g. ``LiveGeneric3AxisPedalTrackerImpl``) - compose a ``SchemaTracker`` and implement ``ITrackerImpl::update()`` / ``serialize_all()``. + OpenXR tensor collections: discovers collections by identifier, reads pending ``SampleResult`` + values, records them when MCAP channels are attached, and publishes the final one as a + ``Serialized<...>``. Live tracker implementations (e.g. ``LiveGeneric3AxisPedalTrackerImpl``) + compose a ``SchemaTracker`` and implement ``ITrackerImpl::update()`` on top of it. - **Generic3AxisPedalTracker** (tracker facade in ``deviceio_trackers``) — Concrete ``ITracker`` for ``Generic3AxisPedalOutput``: holds configuration and - ``get_data(session)`` returning ``Generic3AxisPedalOutputTrackedT`` via the session’s + ``get_data(session)`` returning ``Serialized`` via the session’s ``IGeneric3AxisPedalTrackerImpl``. - **DeviceIOSession** — Session manager: collects required OpenXR extensions from registered trackers, creates tracker implementations with session handles, and calls ``update()`` on all diff --git a/docs/source/device/oglo.rst b/docs/source/device/oglo.rst index 022aac992..ef303818a 100644 --- a/docs/source/device/oglo.rst +++ b/docs/source/device/oglo.rst @@ -28,7 +28,7 @@ Components ---------- - **Schema** — :code-file:`src/core/schema/fbs/oglo_tactile.fbs` - (``OgloGloveSample`` / ``OgloGloveSampleTracked`` / ``OgloGloveSampleRecord``). + (``OgloGloveSample`` / ``OgloGloveSampleRecord``). - **Plugin** — :code-dir:`src/plugins/oglo_tactile` (BLE read → parse → OpenXR push). - **Tracker** — ``OgloTactileTracker`` (:code-file:`src/core/deviceio_trackers/cpp/inc/deviceio_trackers/oglo_tactile_tracker.hpp`) diff --git a/docs/source/device/trackers.rst b/docs/source/device/trackers.rst index 3334ff0f0..eee424df6 100644 --- a/docs/source/device/trackers.rst +++ b/docs/source/device/trackers.rst @@ -44,23 +44,17 @@ Data Schema Convention ---------------------- Every tracker's data is defined by a FlatBuffers schema under -:code-dir:`src/core/schema/fbs`. Each schema follows a three-tier convention: +:code-dir:`src/core/schema/fbs`. Each schema follows a two-tier convention: .. code-block:: idl - // 1. Inner data table -- the actual payload + // 1. Payload table -- the actual data, and what trackers hand to consumers. table Xxx { field_a: SomeType (id: 0); field_b: AnotherType (id: 1); } - // 2. Tracked wrapper -- used by the in-memory tracker API. - // data is null when the tracked entity is inactive. - table XxxTracked { - data: Xxx (id: 0); - } - - // 3. Record wrapper -- used as the MCAP recording root type. + // 2. Record wrapper -- used as the MCAP recording root type. // Adds a DeviceDataTimestamp alongside the payload. table XxxRecord { data: Xxx (id: 0); @@ -69,18 +63,39 @@ Every tracker's data is defined by a FlatBuffers schema under root_type XxxRecord; -- **Inner data table** (e.g. ``HeadPose``, ``HandPose``, ``ControllerSnapshot``) -- - contains the device-specific fields. All fields are present when the parent - wrapper's ``data`` pointer is non-null. - -- **Tracked wrapper** (e.g. ``HeadPoseTracked``) -- wraps the inner data in an - optional ``data`` field. The in-memory ``get_*()`` accessors return a reference - to this wrapper. When ``data`` is ``nullptr`` (C++) or ``None`` (Python), the - device is inactive or no sample has arrived yet. +- **Payload table** (e.g. ``HeadPose``, ``HandPose``, ``ControllerSnapshot``) -- + contains the device-specific fields. All fields are present whenever the table + itself is present. -- **Record wrapper** (e.g. ``HeadPoseRecord``) -- wraps the inner data plus a +- **Record wrapper** (e.g. ``HeadPoseRecord``) -- wraps the payload plus a ``DeviceDataTimestamp``. This is the ``root_type`` written to MCAP channels by - the recorder via ``serialize_all()``. + the recorder. + +Reading a payload +~~~~~~~~~~~~~~~~~ + +The ``get_*()`` accessors hand out the payload table itself as an owning handle +over the encoded bytes -- ``Serialized`` in C++ +(:code-file:`src/core/schema/cpp/inc/schema/serialized.hpp`), a read-only view +class (``HeadPose``) in Python. Reads go straight into the buffer, so there is no +unpack step and joint arrays come back as zero-copy NumPy views. + +An **empty handle is the absent payload**: the device is inactive, no sample has +arrived yet, or replay hit a gap. Test it with ``if (handle)`` in C++; in Python +the accessor returns ``None``. + +Each ``session.update()`` publishes a *new* buffer rather than refilling the +previous one, so a handle read this frame keeps its values after the next update. + +To build a payload from Python, pass every field to its constructor -- these +views are immutable, so there are no setters. + +.. note:: + + ``MessageChannelMessagesTracked`` wraps its payload in a table, because that + payload is a **list** and something has to hold the vector. ``get_messages()`` + always returns a non-empty handle; an absent ``data`` vector means no messages + arrived this frame. Shared Types ~~~~~~~~~~~~ diff --git a/examples/lerobot/record.py b/examples/lerobot/record.py index 7523e6d3f..d6f4d9a1a 100644 --- a/examples/lerobot/record.py +++ b/examples/lerobot/record.py @@ -120,32 +120,28 @@ def main(): session.update() # Get hand data - left_tracked: schema.HandPoseTrackedT = ( - hand_tracker.get_left_hand(session) - ) - right_tracked: schema.HandPoseTrackedT = ( - hand_tracker.get_right_hand(session) + left_tracked: schema.HandPose = hand_tracker.get_left_hand( + session ) - head_tracked: schema.HeadPoseTrackedT = head_tracker.get_head( + right_tracked: schema.HandPose = hand_tracker.get_right_hand( session ) + head_tracked: schema.HeadPose = head_tracker.get_head(session) # Extract positions and orientations (with defaults for invalid data) left_pos = np.zeros(3, dtype=np.float32) right_pos = np.zeros(3, dtype=np.float32) - if left_tracked.data is not None and left_tracked.data.joints: - wrist = left_tracked.data.joints.poses(deviceio.JOINT_WRIST) + if left_tracked and left_tracked.joints: + wrist = left_tracked.joints.poses(deviceio.JOINT_WRIST) if wrist.is_valid: pos = wrist.pose.position left_pos = np.array( [pos.x, pos.y, pos.z], dtype=np.float32 ) - if right_tracked.data is not None and right_tracked.data.joints: - wrist = right_tracked.data.joints.poses( - deviceio.JOINT_WRIST - ) + if right_tracked and right_tracked.joints: + wrist = right_tracked.joints.poses(deviceio.JOINT_WRIST) if wrist.is_valid: pos = wrist.pose.position right_pos = np.array( @@ -153,8 +149,8 @@ def main(): ) head_pos = np.zeros(3, dtype=np.float32) - if head_tracked.data is not None and head_tracked.data.is_valid: - pos = head_tracked.data.pose.position + if head_tracked and head_tracked.is_valid: + pos = head_tracked.pose.position head_pos = np.array([pos.x, pos.y, pos.z], dtype=np.float32) # STEP 3: Record frame to dataset diff --git a/examples/mcap_record_replay/cpp/record_full_body.cpp b/examples/mcap_record_replay/cpp/record_full_body.cpp index b09c44038..ab818efa5 100644 --- a/examples/mcap_record_replay/cpp/record_full_body.cpp +++ b/examples/mcap_record_replay/cpp/record_full_body.cpp @@ -26,6 +26,7 @@ #include #include #include +#include #include #include @@ -70,12 +71,12 @@ std::string resolve_output_path(const std::string& arg) return arg; } -uint32_t count_valid_joints(const core::FullBodyPoseT& data) +uint32_t count_valid_joints(const core::FullBodyPose& data) { uint32_t valid_count = 0; for (uint32_t i = 0; i < core::FullBodyTracker::JOINT_COUNT; ++i) { - if ((*data.joints->joints())[i]->is_valid()) + if ((*data.joints()->joints())[i]->is_valid()) { ++valid_count; } @@ -123,12 +124,11 @@ try if (frame_count % 60 == 0) { - const auto& tracked = tracker->get_body_pose(*session); + const auto* body = tracker->get_body_pose(*session).get(); std::cout << "[record] t=" << std::fixed << std::setprecision(2) << elapsed_s << "s frame=" << frame_count; - if (tracked.data) + if (body != nullptr) { - std::cout << " joints=" << count_valid_joints(*tracked.data) << "/" - << core::FullBodyTracker::JOINT_COUNT; + std::cout << " joints=" << count_valid_joints(*body) << "/" << core::FullBodyTracker::JOINT_COUNT; } else { diff --git a/examples/oglo_tactile/oglo_teleop_record.py b/examples/oglo_tactile/oglo_teleop_record.py index d948e7083..ba8efb705 100755 --- a/examples/oglo_tactile/oglo_teleop_record.py +++ b/examples/oglo_tactile/oglo_teleop_record.py @@ -140,9 +140,9 @@ def _make_overlay_layer( def _taxels(tracked) -> np.ndarray | None: - if tracked is None or tracked.data is None: + if not tracked: return None - t = tracked.data.taxels + t = tracked.taxels if not t or len(t) < NUM_TAXELS: return None return np.asarray(t, dtype=np.float32)[:NUM_TAXELS] diff --git a/examples/oxr/cpp/oxr_session_sharing.cpp b/examples/oxr/cpp/oxr_session_sharing.cpp index 91d586378..02c77d346 100644 --- a/examples/oxr/cpp/oxr_session_sharing.cpp +++ b/examples/oxr/cpp/oxr_session_sharing.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -85,12 +86,13 @@ try if (i % 3 == 0) { + const bool head_valid = head_tracked && head_tracked->is_valid(); std::cout << "Frame " << i << ": " - << "Hands=" << (left_tracked.data ? "ACTIVE" : "INACTIVE") << " | " - << "Head=" << ((head_tracked.data && head_tracked.data->is_valid) ? "VALID" : "INVALID"); - if (head_tracked.data && head_tracked.data->is_valid && head_tracked.data->pose) + << "Hands=" << (left_tracked ? "ACTIVE" : "INACTIVE") << " | " + << "Head=" << (head_valid ? "VALID" : "INVALID"); + if (head_valid && head_tracked->pose() != nullptr) { - const auto& pos = head_tracked.data->pose->position(); + const auto& pos = head_tracked->pose()->position(); std::cout << " [" << pos.x() << ", " << pos.y() << ", " << pos.z() << "]"; } std::cout << std::endl; diff --git a/examples/oxr/cpp/oxr_simple_api_demo.cpp b/examples/oxr/cpp/oxr_simple_api_demo.cpp index 15d0176bb..265b11261 100644 --- a/examples/oxr/cpp/oxr_simple_api_demo.cpp +++ b/examples/oxr/cpp/oxr_simple_api_demo.cpp @@ -1,10 +1,11 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #include #include #include #include +#include #include #include @@ -84,14 +85,13 @@ try const auto& head_tracked = head_tracker->get_head(*session); std::cout << "Frame " << i << ":" << std::endl; - std::cout << " Left hand: " << (left_tracked.data ? "ACTIVE" : "INACTIVE") << std::endl; - std::cout << " Right hand: " << (right_tracked.data ? "ACTIVE" : "INACTIVE") << std::endl; - std::cout << " Head pose: " << ((head_tracked.data && head_tracked.data->is_valid) ? "VALID" : "INVALID") - << std::endl; + std::cout << " Left hand: " << (left_tracked ? "ACTIVE" : "INACTIVE") << std::endl; + std::cout << " Right hand: " << (right_tracked ? "ACTIVE" : "INACTIVE") << std::endl; + std::cout << " Head pose: " << ((head_tracked && head_tracked->is_valid()) ? "VALID" : "INVALID") << std::endl; - if (head_tracked.data && head_tracked.data->is_valid) + if (head_tracked && head_tracked->is_valid()) { - const auto& pos = head_tracked.data->pose->position(); + const auto& pos = head_tracked->pose()->position(); std::cout << " Position: [" << pos.x() << ", " << pos.y() << ", " << pos.z() << "]" << std::endl; } std::cout << std::endl; diff --git a/examples/oxr/python/modular_example.py b/examples/oxr/python/modular_example.py index 525a164ed..6298a26ef 100755 --- a/examples/oxr/python/modular_example.py +++ b/examples/oxr/python/modular_example.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 """ @@ -75,15 +75,15 @@ def main(): print(f"[{elapsed:4.1f}s] Frame {frame_count}") # Get hand data - left_tracked: schema.HandPoseTrackedT = ( - hand_tracker.get_left_hand(session) + left_tracked: schema.HandPose = hand_tracker.get_left_hand( + session ) - right_tracked: schema.HandPoseTrackedT = ( - hand_tracker.get_right_hand(session) + right_tracked: schema.HandPose = hand_tracker.get_right_hand( + session ) - if left_tracked.data is not None: - pos = left_tracked.data.joints.poses( + if left_tracked: + pos = left_tracked.joints.poses( deviceio.JOINT_WRIST ).pose.position print( @@ -92,8 +92,8 @@ def main(): else: print(" Left hand: inactive") - if right_tracked.data is not None: - pos = right_tracked.data.joints.poses( + if right_tracked: + pos = right_tracked.joints.poses( deviceio.JOINT_WRIST ).pose.position print( @@ -103,11 +103,9 @@ def main(): print(" Right hand: inactive") # Get head data - head_tracked: schema.HeadPoseTrackedT = head_tracker.get_head( - session - ) - if head_tracked.data is not None: - pos = head_tracked.data.pose.position + head_tracked: schema.HeadPose = head_tracker.get_head(session) + if head_tracked: + pos = head_tracked.pose.position print( f" Head pos: [{pos.x:6.3f}, {pos.y:6.3f}, {pos.z:6.3f}]" ) diff --git a/examples/oxr/python/test_controller_tracker.py b/examples/oxr/python/test_controller_tracker.py index 4115b6e30..8415615da 100644 --- a/examples/oxr/python/test_controller_tracker.py +++ b/examples/oxr/python/test_controller_tracker.py @@ -111,8 +111,8 @@ def assert_trackers_consistent(label, ta, tb): print(f" [{elapsed:5.2f}s] Frame {frame_count:4d}") - left_data = left_tracked.data - if left_data is not None: + left_data = left_tracked + if left_data: li = left_data.inputs print( f" L: Trig={li.trigger_value:.2f} Sq={li.squeeze_value:.2f}" @@ -122,8 +122,8 @@ def assert_trackers_consistent(label, ta, tb): else: print(" L: INACTIVE") - right_data = right_tracked.data - if right_data is not None: + right_data = right_tracked + if right_data: ri = right_data.inputs print( f" R: Trig={ri.trigger_value:.2f} Sq={ri.squeeze_value:.2f}" @@ -146,12 +146,12 @@ def assert_trackers_consistent(label, ta, tb): def print_controller_summary(hand_name, tracked): print(f" {hand_name} Controller:") - if tracked.data is not None: - pos = tracked.data.grip_pose.pose.position + if tracked: + pos = tracked.grip_pose.pose.position print(f" Grip position: [{pos.x:+.3f}, {pos.y:+.3f}, {pos.z:+.3f}]") - pos = tracked.data.aim_pose.pose.position + pos = tracked.aim_pose.pose.position print(f" Aim position: [{pos.x:+.3f}, {pos.y:+.3f}, {pos.z:+.3f}]") - inputs = tracked.data.inputs + inputs = tracked.inputs print(f" Trigger: {inputs.trigger_value:.2f}") print(f" Squeeze: {inputs.squeeze_value:.2f}") print( diff --git a/examples/oxr/python/test_extensions.py b/examples/oxr/python/test_extensions.py index 869cbe183..2cbf0ff41 100755 --- a/examples/oxr/python/test_extensions.py +++ b/examples/oxr/python/test_extensions.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 """ @@ -91,13 +91,13 @@ left_tracked = hand.get_left_hand(session) head_tracked = head.get_head(session) print(" ✅ Update successful") - if left_tracked.data is not None: - pos = left_tracked.data.joints.poses(deviceio.JOINT_WRIST).pose.position + if left_tracked: + pos = left_tracked.joints.poses(deviceio.JOINT_WRIST).pose.position print(f" Left wrist: [{pos.x:.3f}, {pos.y:.3f}, {pos.z:.3f}]") else: print(" Left hand: inactive") - if head_tracked.data is not None: - pos = head_tracked.data.pose.position + if head_tracked: + pos = head_tracked.pose.position print(f" Head pos: [{pos.x:.3f}, {pos.y:.3f}, {pos.z:.3f}]") else: print(" Head: inactive") diff --git a/examples/oxr/python/test_full_body_tracker.py b/examples/oxr/python/test_full_body_tracker.py index 18fda3241..130f78148 100644 --- a/examples/oxr/python/test_full_body_tracker.py +++ b/examples/oxr/python/test_full_body_tracker.py @@ -63,15 +63,13 @@ # Test 6: Check initial body tracking state print("[Test 6] Checking body tracking state...") body_tracked = body_tracker.get_body_pose(session) - print( - f" Body tracking active: {'YES' if body_tracked.data is not None else 'NO'}" - ) + print(f" Body tracking active: {'YES' if body_tracked else 'NO'}") - if body_tracked.data is not None: + if body_tracked: valid_count = sum( 1 for i in range(schema.BodyJoint.NUM_JOINTS) - if body_tracked.data.joints.joints(i).is_valid + if body_tracked.joints.joints(i).is_valid ) print(f" Valid joints: {valid_count}/{schema.BodyJoint.NUM_JOINTS}") print() @@ -94,11 +92,11 @@ elapsed = current_time - start_time body_tracked = body_tracker.get_body_pose(session) - if body_tracked.data is not None: - pelvis_pos = body_tracked.data.joints.joints( + if body_tracked: + pelvis_pos = body_tracked.joints.joints( int(schema.BodyJoint.PELVIS) ).pose.position - head_pos = body_tracked.data.joints.joints( + head_pos = body_tracked.joints.joints( int(schema.BodyJoint.HEAD) ).pose.position print( @@ -121,15 +119,13 @@ print("[Test 8] Final body pose state...") body_tracked = body_tracker.get_body_pose(session) - print( - f" Body tracking active: {'YES' if body_tracked.data is not None else 'NO'}" - ) + print(f" Body tracking active: {'YES' if body_tracked else 'NO'}") - if body_tracked.data is not None: + if body_tracked: print() print(" Joint positions:") for i in range(schema.BodyJoint.NUM_JOINTS): - joint = body_tracked.data.joints.joints(i) + joint = body_tracked.joints.joints(i) name = schema.BodyJoint(i).name pos = joint.pose.position rot = joint.pose.orientation diff --git a/examples/oxr/python/test_modular.py b/examples/oxr/python/test_modular.py index c1f1cd5e0..3dff5b3f2 100755 --- a/examples/oxr/python/test_modular.py +++ b/examples/oxr/python/test_modular.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 """ @@ -53,17 +53,13 @@ # Test 5: Check hand data print("[Test 5] Checking hand tracking data...") - left_tracked: schema.HandPoseTrackedT = hand_tracker.get_left_hand(session) - right_tracked: schema.HandPoseTrackedT = hand_tracker.get_right_hand(session) - print( - f" Left hand: {'ACTIVE' if left_tracked.data is not None else 'INACTIVE'}" - ) - print( - f" Right hand: {'ACTIVE' if right_tracked.data is not None else 'INACTIVE'}" - ) - - if left_tracked.data is not None: - pos = left_tracked.data.joints.poses(deviceio.JOINT_WRIST).pose.position + left_tracked: schema.HandPose = hand_tracker.get_left_hand(session) + right_tracked: schema.HandPose = hand_tracker.get_right_hand(session) + print(f" Left hand: {'ACTIVE' if left_tracked else 'INACTIVE'}") + print(f" Right hand: {'ACTIVE' if right_tracked else 'INACTIVE'}") + + if left_tracked: + pos = left_tracked.joints.poses(deviceio.JOINT_WRIST).pose.position print(f" Left wrist position: [{pos.x:.3f}, {pos.y:.3f}, {pos.z:.3f}]") else: print(" Left hand: inactive") @@ -71,10 +67,10 @@ # Test 6: Check head data print("[Test 6] Checking head tracking data...") - head_tracked: schema.HeadPoseTrackedT = head_tracker.get_head(session) - if head_tracked.data is not None: - pos = head_tracked.data.pose.position - ori = head_tracked.data.pose.orientation + head_tracked: schema.HeadPose = head_tracker.get_head(session) + if head_tracked: + pos = head_tracked.pose.position + ori = head_tracked.pose.orientation print(f" Head position: [{pos.x:.3f}, {pos.y:.3f}, {pos.z:.3f}]") print( f" Head orientation: [{ori.x:.3f}, {ori.y:.3f}, {ori.z:.3f}, {ori.w:.3f}]" @@ -96,15 +92,13 @@ left_tracked = hand_tracker.get_left_hand(session) head_tracked = head_tracker.get_head(session) print(f" [{elapsed:4.1f}s] Frame {frame_count:3d}:") - if left_tracked.data is not None: - pos = left_tracked.data.joints.poses( - deviceio.JOINT_WRIST - ).pose.position + if left_tracked: + pos = left_tracked.joints.poses(deviceio.JOINT_WRIST).pose.position print(f" Left wrist: [{pos.x:6.3f}, {pos.y:6.3f}, {pos.z:6.3f}]") else: print(" Left hand: inactive") - if head_tracked.data is not None: - pos = head_tracked.data.pose.position + if head_tracked: + pos = head_tracked.pose.position print(f" Head pos: [{pos.x:6.3f}, {pos.y:6.3f}, {pos.z:6.3f}]") else: print(" Head: inactive") diff --git a/examples/oxr/python/test_oak_camera.py b/examples/oxr/python/test_oak_camera.py index 6647e4092..abb898247 100755 --- a/examples/oxr/python/test_oak_camera.py +++ b/examples/oxr/python/test_oak_camera.py @@ -90,12 +90,9 @@ def _run_schema_pusher( elapsed = time.time() - start_time for idx, name in enumerate(stream_names): tracked = tracker.get_stream_data(session, idx) - if ( - tracked.data is not None - and tracked.data.sequence_number != last_seq.get(name, -1) - ): + if tracked and tracked.sequence_number != last_seq.get(name, -1): metadata_samples[name] = metadata_samples.get(name, 0) + 1 - last_seq[name] = tracked.data.sequence_number + last_seq[name] = tracked.sequence_number if int(elapsed) > last_print_time: last_print_time = int(elapsed) diff --git a/examples/oxr/python/test_session_sharing.py b/examples/oxr/python/test_session_sharing.py index 2eba6c65c..81b1c0558 100755 --- a/examples/oxr/python/test_session_sharing.py +++ b/examples/oxr/python/test_session_sharing.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 """ @@ -95,15 +95,13 @@ head_tracked = head_tracker.get_head(session2) print(f"[{elapsed:4.1f}s] Frame {frame_count:3d}:") - if left_tracked.data is not None: - pos = left_tracked.data.joints.poses( - deviceio.JOINT_WRIST - ).pose.position + if left_tracked: + pos = left_tracked.joints.poses(deviceio.JOINT_WRIST).pose.position print(f" Left wrist: [{pos.x:6.3f}, {pos.y:6.3f}, {pos.z:6.3f}]") else: print(" Left hand: inactive") - if head_tracked.data is not None: - pos = head_tracked.data.pose.position + if head_tracked: + pos = head_tracked.pose.position print(f" Head pos: [{pos.x:6.3f}, {pos.y:6.3f}, {pos.z:6.3f}]") else: print(" Head: inactive") diff --git a/examples/oxr/python/test_synthetic_hands.py b/examples/oxr/python/test_synthetic_hands.py index 5b626d9d0..4688b8abb 100644 --- a/examples/oxr/python/test_synthetic_hands.py +++ b/examples/oxr/python/test_synthetic_hands.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 """ @@ -103,15 +103,15 @@ def run_test(): right_tracked = hand_tracker.get_right_hand(deviceio_session) print(f"Frame {frame_count}:") - if left_tracked.data is not None: - pos = left_tracked.data.joints.poses( + if left_tracked: + pos = left_tracked.joints.poses( deviceio.JOINT_WRIST ).pose.position print(f" Left wrist: [{pos.x:.3f}, {pos.y:.3f}, {pos.z:.3f}]") else: print(" Left hand: inactive") - if right_tracked.data is not None: - pos = right_tracked.data.joints.poses( + if right_tracked: + pos = right_tracked.joints.poses( deviceio.JOINT_WRIST ).pose.position print(f" Right wrist: [{pos.x:.3f}, {pos.y:.3f}, {pos.z:.3f}]") diff --git a/examples/schemaio/frame_metadata_printer.cpp b/examples/schemaio/frame_metadata_printer.cpp index b7ce5ac2e..8c7f3da81 100644 --- a/examples/schemaio/frame_metadata_printer.cpp +++ b/examples/schemaio/frame_metadata_printer.cpp @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -95,10 +96,9 @@ try std::vector> last_sequences(stream_count); for (size_t i = 0; i < stream_count; ++i) { - const auto& tracked = tracker->get_stream_data(*session, i); - if (tracked.data) + if (const auto* metadata = tracker->get_stream_data(*session, i).get()) { - last_sequences[i] = tracked.data->sequence_number; + last_sequences[i] = metadata->sequence_number(); } } @@ -119,10 +119,9 @@ try // data is already present so we don't reprint an existing sample. for (size_t i = old_count; i < stream_count; ++i) { - const auto& tracked = tracker->get_stream_data(*session, i); - if (tracked.data) + if (const auto* metadata = tracker->get_stream_data(*session, i).get()) { - last_sequences[i] = tracked.data->sequence_number; + last_sequences[i] = metadata->sequence_number(); } } } @@ -130,15 +129,15 @@ try // Print one line per stream that has a new sample. for (size_t i = 0; i < stream_count; ++i) { - const auto& tracked = tracker->get_stream_data(*session, i); - if (!tracked.data || - (last_sequences[i].has_value() && tracked.data->sequence_number == last_sequences[i].value())) + const auto* metadata = tracker->get_stream_data(*session, i).get(); + if (metadata == nullptr || + (last_sequences[i].has_value() && metadata->sequence_number() == last_sequences[i].value())) { continue; } - last_sequences[i] = tracked.data->sequence_number; - std::cout << "Sample " << ++received_count << ": " << core::EnumNameStreamType(tracked.data->stream) - << " seq=" << tracked.data->sequence_number << std::endl; + last_sequences[i] = metadata->sequence_number(); + std::cout << "Sample " << ++received_count << ": " << core::EnumNameStreamType(metadata->stream()) + << " seq=" << metadata->sequence_number() << std::endl; } auto now = std::chrono::steady_clock::now(); diff --git a/examples/schemaio/full_body_printer.cpp b/examples/schemaio/full_body_printer.cpp index 42f4a5db5..bab1d9fb6 100644 --- a/examples/schemaio/full_body_printer.cpp +++ b/examples/schemaio/full_body_printer.cpp @@ -22,6 +22,7 @@ #include #include #include +#include #include #include @@ -51,9 +52,9 @@ void print_joint(const char* label, const core::BodyJointPose& joint) } } -void print_body_pose(const core::FullBodyPoseT& data, size_t sample_count) +void print_body_pose(const core::FullBodyPose& data, size_t sample_count) { - const auto& joints = *data.joints->joints(); + const auto& joints = *data.joints()->joints(); uint32_t valid_count = 0; for (uint32_t i = 0; i < core::FullBodyTracker::JOINT_COUNT; ++i) @@ -115,9 +116,9 @@ try // Print current data if available. tracked.data is null only in limp mode (body tracking // unsupported); a supported-but-untracked body still delivers data with valid=0/24 joints. const auto& tracked = tracker->get_body_pose(*session); - if (tracked.data) + if (const auto* body = tracked.get()) { - print_body_pose(*tracked.data, received_count++); + print_body_pose(*body, received_count++); } else if (tick_count % 30 == 0) { diff --git a/examples/schemaio/pedal_printer.cpp b/examples/schemaio/pedal_printer.cpp index e00740d0a..265b647a0 100644 --- a/examples/schemaio/pedal_printer.cpp +++ b/examples/schemaio/pedal_printer.cpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 /*! @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -27,12 +28,12 @@ using namespace schemaio_example; -void print_pedal_data(const core::Generic3AxisPedalOutputT& data, size_t sample_count) +void print_pedal_data(const core::Generic3AxisPedalOutput& data, size_t sample_count) { std::cout << "Sample " << sample_count; - std::cout << std::fixed << std::setprecision(3) << " [left=" << data.left_pedal << ", right=" << data.right_pedal - << ", rudder=" << data.rudder << "]"; + std::cout << std::fixed << std::setprecision(3) << " [left=" << data.left_pedal() + << ", right=" << data.right_pedal() << ", rudder=" << data.rudder() << "]"; std::cout << std::endl; } @@ -73,9 +74,9 @@ try // Print current data if available const auto& tracked = tracker->get_data(*session); - if (tracked.data) + if (const auto* pedals = tracked.get()) { - print_pedal_data(*tracked.data, received_count++); + print_pedal_data(*pedals, received_count++); } else { diff --git a/examples/schemaio/se3_printer.cpp b/examples/schemaio/se3_printer.cpp index 825aa97ba..8ff70b223 100644 --- a/examples/schemaio/se3_printer.cpp +++ b/examples/schemaio/se3_printer.cpp @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -30,11 +31,11 @@ using namespace schemaio_example; -void print_se3_data(const core::Se3TrackerPoseT& data, size_t sample_count) +void print_se3_data(const core::Se3TrackerPose& data, size_t sample_count) { std::cout << "Sample " << sample_count; - if (!data.is_valid) + if (!data.is_valid()) { // Pose contents are unspecified while tracking is lost (see se3_tracker.fbs) — // gate on is_valid, never on pose values. @@ -42,8 +43,8 @@ void print_se3_data(const core::Se3TrackerPoseT& data, size_t sample_count) return; } - const auto& position = data.pose->position(); - const auto& orientation = data.pose->orientation(); + const auto& position = data.pose()->position(); + const auto& orientation = data.pose()->orientation(); std::cout << std::fixed << std::setprecision(3) << " pos=[" << position.x() << ", " << position.y() << ", " << position.z() << "] quat(xyzw)=[" << orientation.x() << ", " << orientation.y() << ", " << orientation.z() << ", " << orientation.w() << "]"; @@ -93,9 +94,9 @@ try // last-known sample between pushes, so without the fixed-rate sleep below this // loop would spin and reprint stale data as fast as the CPU allows. const auto& tracked = tracker->get_data(*session); - if (tracked.data) + if (const auto* pose = tracked.get()) { - print_se3_data(*tracked.data, received_count++); + print_se3_data(*pose, received_count++); } // Tick at ~30 Hz. Each update drains all pending samples, so the printer keeps up diff --git a/examples/teleop_session_manager/python/message_channel_example.py b/examples/teleop_session_manager/python/message_channel_example.py index e5c607089..0c2c7b658 100755 --- a/examples/teleop_session_manager/python/message_channel_example.py +++ b/examples/teleop_session_manager/python/message_channel_example.py @@ -21,7 +21,7 @@ message_channel_config, ) from isaacteleop.retargeting_engine.interface import TensorGroup -from isaacteleop.schema import MessageChannelMessages, MessageChannelMessagesTrackedT +from isaacteleop.schema import MessageChannelMessages, MessageChannelMessagesTracked from isaacteleop.teleop_session_manager import TeleopSession, TeleopSessionConfig @@ -50,7 +50,7 @@ def _parse_uuid_bytes(uuid_text: str) -> bytes: def _enqueue_outbound_message(sink, payload: bytes) -> None: """Push one outbound message through MessageChannelSink.""" tg = TensorGroup(sink.input_spec()["messages_tracked"]) - tg[0] = MessageChannelMessagesTrackedT([MessageChannelMessages(payload)]) + tg[0] = MessageChannelMessagesTracked([MessageChannelMessages(payload)]) sink.compute({"messages_tracked": tg}, {}) diff --git a/src/core/deviceio_base/AGENTS.md b/src/core/deviceio_base/AGENTS.md index e3a92a338..e02df33b9 100644 --- a/src/core/deviceio_base/AGENTS.md +++ b/src/core/deviceio_base/AGENTS.md @@ -11,6 +11,9 @@ SPDX-License-Identifier: Apache-2.0 - **`ITrackerImpl::update`** takes **`int64_t monotonic_time_ns`** (system monotonic clock, same domain as `core::os_monotonic_now_ns()`). - **Do not** use `XrTime`, ``, or OpenXR link targets in this library. Keep the tracker abstraction runtime-agnostic. +- **Query accessors return `Serialized`, never a generated `-T` and never a wrapper table.** The object-API types are an implementation detail of whoever assembles the payload; they must not appear in any `ITrackerImpl` or `ITracker` signature. See ``. +- **Keep `Serialized` schema-agnostic.** It owns a buffer and re-points within it; it knows nothing about any field. Anything that assumes a wrapper's `data` field belongs in `` — `payload(handle)` — so a translation unit's includes say which it depends on. +- **An empty handle is the absent payload** — device inactive, no sample yet, replay gap. Consumers test one condition (`if (handle)`). Do **not** reintroduce a wrapper table to carry optionality: that was what the `Tracked` tables did before the handle became nullable, and it made every read a two-step null check. The exception is the message channel, whose payload is a **list**: a batch needs a table to hold the vector, and "nothing this frame" is an empty batch rather than an absent one. ## CMake diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/controller_tracker_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/controller_tracker_base.hpp index 717ab4c41..405625816 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/controller_tracker_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/controller_tracker_base.hpp @@ -5,17 +5,19 @@ #include "tracker.hpp" +#include + namespace core { -struct ControllerSnapshotTrackedT; +struct ControllerSnapshot; // Abstract base interface for controller tracker implementations. class IControllerTrackerImpl : public ITrackerImpl { public: - virtual const ControllerSnapshotTrackedT& get_left_controller() const = 0; - virtual const ControllerSnapshotTrackedT& get_right_controller() const = 0; + virtual const Serialized& get_left_controller() const = 0; + virtual const Serialized& get_right_controller() const = 0; /// Apply one frame of haptic vibration to the left / right controller. /// diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/frame_metadata_tracker_oak_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/frame_metadata_tracker_oak_base.hpp index cebeb7c13..665ccddc9 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/frame_metadata_tracker_oak_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/frame_metadata_tracker_oak_base.hpp @@ -5,18 +5,20 @@ #include "tracker.hpp" +#include + #include namespace core { -struct FrameMetadataOakTrackedT; +struct FrameMetadataOak; // Abstract base interface for FrameMetadataTrackerOak implementations. class IFrameMetadataTrackerOakImpl : public ITrackerImpl { public: - virtual const FrameMetadataOakTrackedT& get_stream_data(size_t stream_index) const = 0; + virtual const Serialized& get_stream_data(size_t stream_index) const = 0; }; } // namespace core diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/full_body_tracker_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/full_body_tracker_base.hpp index c7648538a..f1ed03f84 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/full_body_tracker_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/full_body_tracker_base.hpp @@ -5,18 +5,20 @@ #include "tracker.hpp" +#include + namespace core { -struct FullBodyPoseTrackedT; +struct FullBodyPose; // Abstract base interface for full body tracker implementations. // Vendor-agnostic: every live/replay backend (native XR, pushed tensor, ...) -// implements this and produces the same FullBodyPoseTrackedT payload. +// implements this and produces the same Serialized payload. class IFullBodyTrackerImpl : public ITrackerImpl { public: - virtual const FullBodyPoseTrackedT& get_body_pose() const = 0; + virtual const Serialized& get_body_pose() const = 0; }; } // namespace core diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/generic_3axis_pedal_tracker_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/generic_3axis_pedal_tracker_base.hpp index 9b9724a05..3326b2512 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/generic_3axis_pedal_tracker_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/generic_3axis_pedal_tracker_base.hpp @@ -1,20 +1,22 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #pragma once #include "tracker.hpp" +#include + namespace core { -struct Generic3AxisPedalOutputTrackedT; +struct Generic3AxisPedalOutput; // Abstract base interface for Generic3AxisPedalTracker implementations. class IGeneric3AxisPedalTrackerImpl : public ITrackerImpl { public: - virtual const Generic3AxisPedalOutputTrackedT& get_data() const = 0; + virtual const Serialized& get_data() const = 0; }; } // namespace core diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/hand_tracker_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/hand_tracker_base.hpp index e0ee121d1..b0c07d687 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/hand_tracker_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/hand_tracker_base.hpp @@ -1,21 +1,23 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #pragma once #include "tracker.hpp" +#include + namespace core { -struct HandPoseTrackedT; +struct HandPose; // Abstract base interface for hand tracker implementations. class IHandTrackerImpl : public ITrackerImpl { public: - virtual const HandPoseTrackedT& get_left_hand() const = 0; - virtual const HandPoseTrackedT& get_right_hand() const = 0; + virtual const Serialized& get_left_hand() const = 0; + virtual const Serialized& get_right_hand() const = 0; }; } // namespace core diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/haptic_command_reader_tracker_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/haptic_command_reader_tracker_base.hpp index eecd1c72e..e714bad21 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/haptic_command_reader_tracker_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/haptic_command_reader_tracker_base.hpp @@ -5,12 +5,14 @@ #include "tracker.hpp" +#include + #include namespace core { -struct HapticCommandTrackedT; +struct HapticCommand; // Abstract base interface for HapticCommandReaderTracker implementations. class IHapticCommandReaderTrackerImpl : public ITrackerImpl @@ -19,13 +21,13 @@ class IHapticCommandReaderTrackerImpl : public ITrackerImpl // Latest command across all endpoints. Correct for a single-endpoint device; // for a multi-endpoint device it returns whichever endpoint was pushed last, // so prefer the endpoint overload there. Kept for backward compatibility. - virtual const HapticCommandTrackedT& get_data() const = 0; + virtual const Serialized& get_data() const = 0; - // Latest command for `endpoint` ("left"/"right"/...); `data` is null until a - // sample for that endpoint arrives. Endpoints are tracked independently so + // Latest command for `endpoint` ("left"/"right"/...); the handle is empty until + // a sample for that endpoint arrives. Endpoints are tracked independently so // commands pushed for different endpoints on one collection do not clobber // each other. - virtual const HapticCommandTrackedT& get_data(std::string_view endpoint) const = 0; + virtual const Serialized& get_data(std::string_view endpoint) const = 0; }; } // namespace core diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/head_tracker_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/head_tracker_base.hpp index 514c1afc3..ff7a89fae 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/head_tracker_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/head_tracker_base.hpp @@ -1,20 +1,22 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #pragma once #include "tracker.hpp" +#include + namespace core { -struct HeadPoseTrackedT; +struct HeadPose; // Abstract base interface for head tracker implementations. class IHeadTrackerImpl : public ITrackerImpl { public: - virtual const HeadPoseTrackedT& get_head() const = 0; + virtual const Serialized& get_head() const = 0; }; } // namespace core diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/joint_state_tracker_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/joint_state_tracker_base.hpp index b9567e09f..cc36a017f 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/joint_state_tracker_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/joint_state_tracker_base.hpp @@ -5,10 +5,12 @@ #include "tracker.hpp" +#include + namespace core { -struct JointStateOutputTrackedT; +struct JointStateOutput; // Abstract base interface for JointStateTracker implementations. // @@ -17,7 +19,7 @@ struct JointStateOutputTrackedT; class IJointStateTrackerImpl : public ITrackerImpl { public: - virtual const JointStateOutputTrackedT& get_data() const = 0; + virtual const Serialized& get_data() const = 0; }; } // namespace core diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/message_channel_tracker_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/message_channel_tracker_base.hpp index d0ad376d0..054a6773f 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/message_channel_tracker_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/message_channel_tracker_base.hpp @@ -5,14 +5,15 @@ #include "tracker.hpp" +#include + #include #include namespace core { -struct MessageChannelMessagesT; -struct MessageChannelMessagesTrackedT; +struct MessageChannelMessagesTracked; enum class MessageChannelStatus : int32_t { @@ -27,7 +28,7 @@ class IMessageChannelTrackerImpl : public ITrackerImpl { public: virtual MessageChannelStatus get_status() const = 0; - virtual const MessageChannelMessagesTrackedT& get_messages() const = 0; + virtual const Serialized& get_messages() const = 0; virtual void send_message(const std::vector& payload) const = 0; }; diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/oglo_tactile_tracker_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/oglo_tactile_tracker_base.hpp index 846010742..3676e4cf9 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/oglo_tactile_tracker_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/oglo_tactile_tracker_base.hpp @@ -5,16 +5,18 @@ #include "tracker.hpp" +#include + namespace core { -struct OgloGloveSampleTrackedT; +struct OgloGloveSample; // Abstract base interface for OgloTactileTracker implementations. class IOgloTactileTrackerImpl : public ITrackerImpl { public: - virtual const OgloGloveSampleTrackedT& get_data() const = 0; + virtual const Serialized& get_data() const = 0; }; } // namespace core diff --git a/src/core/deviceio_base/cpp/inc/deviceio_base/se3_tracker_base.hpp b/src/core/deviceio_base/cpp/inc/deviceio_base/se3_tracker_base.hpp index 97054b067..4c62e3970 100644 --- a/src/core/deviceio_base/cpp/inc/deviceio_base/se3_tracker_base.hpp +++ b/src/core/deviceio_base/cpp/inc/deviceio_base/se3_tracker_base.hpp @@ -5,10 +5,12 @@ #include "tracker.hpp" +#include + namespace core { -struct Se3TrackerPoseTrackedT; +struct Se3TrackerPose; // Abstract base interface for Se3Tracker implementations. // @@ -18,7 +20,7 @@ struct Se3TrackerPoseTrackedT; class ISe3TrackerImpl : public ITrackerImpl { public: - virtual const Se3TrackerPoseTrackedT& get_data() const = 0; + virtual const Serialized& get_data() const = 0; }; } // namespace core diff --git a/src/core/deviceio_trackers/AGENTS.md b/src/core/deviceio_trackers/AGENTS.md index 64649df18..16dda2ae5 100644 --- a/src/core/deviceio_trackers/AGENTS.md +++ b/src/core/deviceio_trackers/AGENTS.md @@ -12,6 +12,13 @@ SPDX-License-Identifier: Apache-2.0 - **`deviceio_trackers`** must **not** link **`OpenXR::headers`**, **`oxr::oxr_utils`**, or vendor extension targets, and must **not** `#include` OpenXR headers. Public API stays schema + **`deviceio_base`** only. - This includes **`tracker_bindings.cpp`**: do not add `#include ` or any `XR_NV_*` extension headers here, even when the bound tracker wraps an OpenXR concept. The UUID is `std::array` at the `deviceio_trackers` boundary—no OpenXR types leak through. +## Python never sees FlatBuffers `-T` types + +- The schema bindings expose **one Python class per FlatBuffer table**, backed by `Serialized`: reads go through the generated accessors into the buffer. Adding a `py::class_` over a `-T` re-introduces a mutable parallel type and is not allowed. +- **To build one from Python, add a constructor**, not setters. The constructor assembles a `-T` as a C++ local and `pack()`s it, so the encoder is always the generated `Pack` and the `-T` stays invisible. Tests construct their inputs this way rather than mutating fields. +- `Record` wrapper bindings are generated by `bind_record()` in `schema_serialized.h`; a new schema only needs to describe its own payload table. Payload constructors always encode, so a payload view reaching Python is never empty — absence arrives as `None`, mapped by `to_python()` in `tracker_bindings.cpp`. +- **Structs are unaffected** — flatc emits one struct type for both APIs, so `Pose` / `HandJoints` bindings and the zero-copy NumPy views keep working unchanged. + ## Related docs - Base interface: [`../deviceio_base/AGENTS.md`](../deviceio_base/AGENTS.md) diff --git a/src/core/deviceio_trackers/cpp/controller_tracker.cpp b/src/core/deviceio_trackers/cpp/controller_tracker.cpp index 4875a6ce3..d7cb8f7bd 100644 --- a/src/core/deviceio_trackers/cpp/controller_tracker.cpp +++ b/src/core/deviceio_trackers/cpp/controller_tracker.cpp @@ -10,12 +10,12 @@ namespace core // ControllerTracker Public Interface // ============================================================================ -const ControllerSnapshotTrackedT& ControllerTracker::get_left_controller(const ITrackerSession& session) const +const Serialized& ControllerTracker::get_left_controller(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_left_controller(); } -const ControllerSnapshotTrackedT& ControllerTracker::get_right_controller(const ITrackerSession& session) const +const Serialized& ControllerTracker::get_right_controller(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_right_controller(); } diff --git a/src/core/deviceio_trackers/cpp/frame_metadata_tracker_oak.cpp b/src/core/deviceio_trackers/cpp/frame_metadata_tracker_oak.cpp index 5c7485448..91374777f 100644 --- a/src/core/deviceio_trackers/cpp/frame_metadata_tracker_oak.cpp +++ b/src/core/deviceio_trackers/cpp/frame_metadata_tracker_oak.cpp @@ -35,8 +35,8 @@ FrameMetadataTrackerOak::FrameMetadataTrackerOak(const std::string& collection_p } } -const FrameMetadataOakTrackedT& FrameMetadataTrackerOak::get_stream_data(const ITrackerSession& session, - size_t stream_index) const +const Serialized& FrameMetadataTrackerOak::get_stream_data(const ITrackerSession& session, + size_t stream_index) const { return static_cast(session.get_tracker_impl(*this)).get_stream_data(stream_index); } diff --git a/src/core/deviceio_trackers/cpp/full_body_tracker.cpp b/src/core/deviceio_trackers/cpp/full_body_tracker.cpp index 30c59108f..a31050593 100644 --- a/src/core/deviceio_trackers/cpp/full_body_tracker.cpp +++ b/src/core/deviceio_trackers/cpp/full_body_tracker.cpp @@ -6,7 +6,7 @@ namespace core { -const FullBodyPoseTrackedT& FullBodyTracker::get_body_pose(const ITrackerSession& session) const +const Serialized& FullBodyTracker::get_body_pose(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_body_pose(); } diff --git a/src/core/deviceio_trackers/cpp/generic_3axis_pedal_tracker.cpp b/src/core/deviceio_trackers/cpp/generic_3axis_pedal_tracker.cpp index 6fc13dc5b..399df1cf4 100644 --- a/src/core/deviceio_trackers/cpp/generic_3axis_pedal_tracker.cpp +++ b/src/core/deviceio_trackers/cpp/generic_3axis_pedal_tracker.cpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #include "inc/deviceio_trackers/generic_3axis_pedal_tracker.hpp" @@ -15,7 +15,7 @@ Generic3AxisPedalTracker::Generic3AxisPedalTracker(const std::string& collection { } -const Generic3AxisPedalOutputTrackedT& Generic3AxisPedalTracker::get_data(const ITrackerSession& session) const +const Serialized& Generic3AxisPedalTracker::get_data(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_data(); } diff --git a/src/core/deviceio_trackers/cpp/hand_tracker.cpp b/src/core/deviceio_trackers/cpp/hand_tracker.cpp index f5c8d6e8d..2ba66cbca 100644 --- a/src/core/deviceio_trackers/cpp/hand_tracker.cpp +++ b/src/core/deviceio_trackers/cpp/hand_tracker.cpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #include "inc/deviceio_trackers/hand_tracker.hpp" @@ -10,12 +10,12 @@ namespace core // HandTracker // ============================================================================ -const HandPoseTrackedT& HandTracker::get_left_hand(const ITrackerSession& session) const +const Serialized& HandTracker::get_left_hand(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_left_hand(); } -const HandPoseTrackedT& HandTracker::get_right_hand(const ITrackerSession& session) const +const Serialized& HandTracker::get_right_hand(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_right_hand(); } diff --git a/src/core/deviceio_trackers/cpp/haptic_command_reader_tracker.cpp b/src/core/deviceio_trackers/cpp/haptic_command_reader_tracker.cpp index 55215b202..11a0271fc 100644 --- a/src/core/deviceio_trackers/cpp/haptic_command_reader_tracker.cpp +++ b/src/core/deviceio_trackers/cpp/haptic_command_reader_tracker.cpp @@ -21,13 +21,13 @@ HapticCommandReaderTracker::HapticCommandReaderTracker(const std::string& collec } } -const HapticCommandTrackedT& HapticCommandReaderTracker::get_data(const ITrackerSession& session) const +const Serialized& HapticCommandReaderTracker::get_data(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_data(); } -const HapticCommandTrackedT& HapticCommandReaderTracker::get_data(const ITrackerSession& session, - std::string_view endpoint) const +const Serialized& HapticCommandReaderTracker::get_data(const ITrackerSession& session, + std::string_view endpoint) const { return static_cast(session.get_tracker_impl(*this)).get_data(endpoint); } diff --git a/src/core/deviceio_trackers/cpp/head_tracker.cpp b/src/core/deviceio_trackers/cpp/head_tracker.cpp index 8485963a8..c91b9f274 100644 --- a/src/core/deviceio_trackers/cpp/head_tracker.cpp +++ b/src/core/deviceio_trackers/cpp/head_tracker.cpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #include "inc/deviceio_trackers/head_tracker.hpp" @@ -10,7 +10,7 @@ namespace core // HeadTracker // ============================================================================ -const HeadPoseTrackedT& HeadTracker::get_head(const ITrackerSession& session) const +const Serialized& HeadTracker::get_head(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_head(); } diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/controller_tracker.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/controller_tracker.hpp index 4c30d70e4..e516bfc3f 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/controller_tracker.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/controller_tracker.hpp @@ -21,10 +21,10 @@ class ControllerTracker : public ITracker } // Query methods: - // - tracked.data is null when the controller is inactive. - // - when tracked.data is non-null, nested fields in ControllerSnapshotT are safe to read. - const ControllerSnapshotTrackedT& get_left_controller(const ITrackerSession& session) const; - const ControllerSnapshotTrackedT& get_right_controller(const ITrackerSession& session) const; + // - the handle is empty when the controller is inactive. + // - when it is non-empty, nested fields in ControllerSnapshot are safe to read. + const Serialized& get_left_controller(const ITrackerSession& session) const; + const Serialized& get_right_controller(const ITrackerSession& session) const; /// Drive the left/right controller's haptic actuator for one frame. /// diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/frame_metadata_tracker_oak.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/frame_metadata_tracker_oak.hpp index 105c81cbb..b04f02e95 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/frame_metadata_tracker_oak.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/frame_metadata_tracker_oak.hpp @@ -26,8 +26,8 @@ namespace core * // ... create session with tracker ... * session->update(); * const auto& color = tracker->get_stream_data(*session, 0); - * if (color.data) - * std::cout << EnumNameStreamType(color.data->stream) << " seq=" << color.data->sequence_number << std::endl; + * if (color) + * std::cout << EnumNameStreamType(color->stream()) << " seq=" << color->sequence_number() << std::endl; * @endcode */ class FrameMetadataTrackerOak : public ITracker @@ -56,12 +56,11 @@ class FrameMetadataTrackerOak : public ITracker * @brief Get per-stream frame metadata. * @param session Active ITrackerSession. * @param stream_index Index into the streams vector passed at construction. - * @return Reference to the FrameMetadataOakTrackedT for that stream. - * The inner @c data pointer is null until the first frame arrives. - * When @c data is non-null, nested fields in FrameMetadataOakT are - * safe to read. + * @return Reference to the Serialized for that stream. + * The handle is empty until the first frame arrives. When non-empty, + * nested fields in FrameMetadataOak are safe to read. */ - const FrameMetadataOakTrackedT& get_stream_data(const ITrackerSession& session, size_t stream_index) const; + const Serialized& get_stream_data(const ITrackerSession& session, size_t stream_index) const; //! Number of streams this tracker is configured for. size_t get_stream_count() const diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/full_body_tracker.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/full_body_tracker.hpp index f1d06e746..cd625b3b1 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/full_body_tracker.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/full_body_tracker.hpp @@ -32,9 +32,9 @@ class FullBodyTracker : public ITracker } // Query method: - // - tracked.data is null when the body tracker is inactive. - // - when tracked.data is non-null, nested fields in FullBodyPoseT are safe to read. - const FullBodyPoseTrackedT& get_body_pose(const ITrackerSession& session) const; + // - the handle is empty when the body tracker is inactive. + // - when it is non-empty, nested fields in FullBodyPose are safe to read. + const Serialized& get_body_pose(const ITrackerSession& session) const; private: static constexpr const char* TRACKER_NAME = "FullBodyTracker"; diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/generic_3axis_pedal_tracker.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/generic_3axis_pedal_tracker.hpp index 30462457a..c2bf38f82 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/generic_3axis_pedal_tracker.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/generic_3axis_pedal_tracker.hpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #pragma once @@ -13,17 +13,17 @@ namespace core { /*! - * @brief Facade for three-axis pedal state exposed as ``Generic3AxisPedalOutputTrackedT``. + * @brief Facade for three-axis pedal state exposed as ``Serialized``. * * Semantic contract: ``left_pedal``, ``right_pedal``, and ``rudder`` are scalar floats matching the * ``Generic3AxisPedalOutput`` schema (axis semantics are left/right/rudder as named). Units, range, * and calibration (e.g. normalized vs raw device values) are defined by the data producer unless * documented elsewhere. After each ``ITrackerSession::update()`` that includes this tracker, ``get_data(session)`` * reflects the implementation’s tracked snapshot. The live backend (``LiveGeneric3AxisPedalTrackerImpl::update``) - * may retain the **last-known** sample when a tick has **no new** samples (e.g. ``m_pending_records`` empty after - * ``read_all_samples``) while the tensor collection remains available — in that case ``data`` stays non-null but - * may be **stale** relative to the latest device state. Separately, **absent** data (``data`` null) means no sample - * has been unpacked yet or the collection/source is unavailable and the implementation has cleared state. + * may retain the **last-known** sample when a tick has **no new** samples while the tensor collection remains + * available — in that case the handle stays non-empty but may be **stale** relative to the latest device state. + * Separately, an **empty** handle means no sample has arrived yet or the collection/source is unavailable and + * the implementation has cleared state. * Implementations may obtain these values through different * backends; transport-specific setup (buffers, extensions, discovery) is documented with the live * ``ITrackerImpl`` and session factory. @@ -59,13 +59,13 @@ class Generic3AxisPedalTracker : public ITracker /*! * @brief Pedal snapshot from the session’s implementation. * - * ``tracked.data`` is null when there is no valid last-known sample (source never provided data or - * implementation cleared state when the collection is gone). When non-null, values may still be **unchanged** - * from the previous ``update()`` if that tick produced no new samples (see ``LiveGeneric3AxisPedalTrackerImpl`` - * and ``m_pending_records``). When ``tracked.data`` is non-null, nested fields in - * ``Generic3AxisPedalOutputT`` are safe to read. + * The handle is empty when there is no valid last-known sample (source never provided data or + * implementation cleared state when the collection is gone). When non-empty, values may still be + * **unchanged** from the previous ``update()`` if that tick produced no new samples (see + * ``LiveGeneric3AxisPedalTrackerImpl`` and ``m_pending_records``); nested fields in + * ``Generic3AxisPedalOutput`` are safe to read. */ - const Generic3AxisPedalOutputTrackedT& get_data(const ITrackerSession& session) const; + const Serialized& get_data(const ITrackerSession& session) const; const std::string& collection_id() const { diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/hand_tracker.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/hand_tracker.hpp index f68d76308..cd23da57b 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/hand_tracker.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/hand_tracker.hpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #pragma once @@ -19,10 +19,10 @@ class HandTracker : public ITracker } // Query methods: - // - tracked.data is null when the hand is inactive. - // - when tracked.data is non-null, nested fields in HandPoseT are safe to read. - const HandPoseTrackedT& get_left_hand(const ITrackerSession& session) const; - const HandPoseTrackedT& get_right_hand(const ITrackerSession& session) const; + // - the handle is empty when the hand is inactive. + // - when it is non-empty, nested fields in HandPose are safe to read. + const Serialized& get_left_hand(const ITrackerSession& session) const; + const Serialized& get_right_hand(const ITrackerSession& session) const; private: static constexpr const char* TRACKER_NAME = "HandTracker"; diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/haptic_command_reader_tracker.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/haptic_command_reader_tracker.hpp index db92dc756..09ea1a61a 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/haptic_command_reader_tracker.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/haptic_command_reader_tracker.hpp @@ -36,11 +36,11 @@ class HapticCommandReaderTracker : public ITracker // Latest command across all endpoints (backward compatible). Correct for a // single-endpoint device; a multi-endpoint device should use the endpoint // overload, as this returns whichever endpoint was pushed last. - const HapticCommandTrackedT& get_data(const ITrackerSession& session) const; + const Serialized& get_data(const ITrackerSession& session) const; - // Latest command for `endpoint`; `tracked.data` is null until a sample for - // that endpoint arrives, or after the producer collection disappears. - const HapticCommandTrackedT& get_data(const ITrackerSession& session, std::string_view endpoint) const; + // Latest command for `endpoint`; the handle is empty until a sample for that + // endpoint arrives, or after the producer collection disappears. + const Serialized& get_data(const ITrackerSession& session, std::string_view endpoint) const; const std::string& collection_id() const { diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/head_tracker.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/head_tracker.hpp index 9688dbb18..1b65ebdae 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/head_tracker.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/head_tracker.hpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #pragma once @@ -19,9 +19,9 @@ class HeadTracker : public ITracker } // Query method: - // - tracked.data is null when no head sample is available for the frame. - // - when tracked.data is non-null, nested fields in HeadPoseT are safe to read. - const HeadPoseTrackedT& get_head(const ITrackerSession& session) const; + // - the handle is empty when no head sample is available for the frame. + // - when it is non-empty, nested fields in HeadPose are safe to read. + const Serialized& get_head(const ITrackerSession& session) const; private: static constexpr const char* TRACKER_NAME = "HeadTracker"; diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/joint_state_tracker.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/joint_state_tracker.hpp index 97a53b406..79ddb006d 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/joint_state_tracker.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/joint_state_tracker.hpp @@ -13,7 +13,7 @@ namespace core { /*! - * @brief Facade for a generic joint-space device exposed as ``JointStateOutputTrackedT``. + * @brief Facade for a generic joint-space device exposed as ``Serialized``. * * Generic across joint-space input devices (leader arms, exoskeletons, gloves, ...): the payload * is a list of named joints (``JointStateOutput.joints``, keyed by ``JointState.name``) plus an @@ -24,8 +24,8 @@ namespace core * After each ``ITrackerSession::update()`` that includes this tracker, ``get_data(session)`` * reflects the implementation's tracked snapshot. As with other ``SchemaTracker``-backed trackers, * the live backend may retain the last-known sample when a tick has no new samples while the - * collection remains available (``data`` stays non-null but may be stale); ``data`` is null only - * when no sample has arrived yet or the collection is unavailable. + * collection remains available (the handle stays non-empty but may be stale); the handle is empty + * only when no sample has arrived yet or the collection is unavailable. * * Usage: * @code @@ -59,10 +59,10 @@ class JointStateTracker : public ITracker /*! * @brief Joint-state snapshot from the session's implementation. * - * ``tracked.data`` is null when no valid sample exists. When non-null, the nested - * ``JointStateOutputT`` (joints, device_id, optional ee_pose) is safe to read. + * The handle is empty when no valid sample exists. When non-empty, the nested + * ``JointStateOutput`` fields (joints, device_id, optional ee_pose) are safe to read. */ - const JointStateOutputTrackedT& get_data(const ITrackerSession& session) const; + const Serialized& get_data(const ITrackerSession& session) const; const std::string& collection_id() const { diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/message_channel_tracker.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/message_channel_tracker.hpp index 74a152af2..e7b676b8b 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/message_channel_tracker.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/message_channel_tracker.hpp @@ -32,7 +32,7 @@ class MessageChannelTracker : public ITracker } MessageChannelStatus get_status(const ITrackerSession& session) const; - const MessageChannelMessagesTrackedT& get_messages(const ITrackerSession& session) const; + const Serialized& get_messages(const ITrackerSession& session) const; void send_message(const ITrackerSession& session, const std::vector& payload) const; const std::array& channel_uuid() const diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/oglo_tactile_tracker.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/oglo_tactile_tracker.hpp index f1c9a9944..85a9ccba2 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/oglo_tactile_tracker.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/oglo_tactile_tracker.hpp @@ -13,13 +13,13 @@ namespace core { /*! - * @brief Facade for one OGLO tactile glove exposed as ``OgloGloveSampleTrackedT``. + * @brief Facade for one OGLO tactile glove exposed as ``Serialized``. * * Reads a tensor collection pushed by the ``oglo_tactile`` plugin * (``--collection-prefix``). One tracker per hand: construct with the matching * ``collection_id`` (e.g. ``"oglo/left"`` / ``"oglo/right"``). After each * ``ITrackerSession::update()`` that includes this tracker, ``get_data(session)`` - * reflects the latest decoded sample; ``data`` is null until the first sample + * reflects the latest decoded sample; the handle is empty until the first sample * arrives or when the collection is unavailable. * * Usage: @@ -28,7 +28,7 @@ namespace core * // ... register with a session, then each tick: ... * session->update(); * const auto& tracked = glove->get_data(*session); - * if (tracked.data) { auto& taxels = tracked.data->taxels; ... } + * if (tracked) { const auto* taxels = tracked->taxels(); ... } * @endcode */ class OgloTactileTracker : public ITracker @@ -54,10 +54,10 @@ class OgloTactileTracker : public ITracker /*! * @brief Glove snapshot from the session's implementation. - * @c tracked.data is null when no valid sample is available; when non-null, - * @c data->taxels (80 values) and the IMU fields are safe to read. + * The handle is empty when no valid sample is available; when non-empty, + * @c taxels() (80 values) and the IMU fields are safe to read. */ - const OgloGloveSampleTrackedT& get_data(const ITrackerSession& session) const; + const Serialized& get_data(const ITrackerSession& session) const; const std::string& collection_id() const { diff --git a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/se3_tracker.hpp b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/se3_tracker.hpp index f2b701aa5..6952311be 100644 --- a/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/se3_tracker.hpp +++ b/src/core/deviceio_trackers/cpp/inc/deviceio_trackers/se3_tracker.hpp @@ -14,7 +14,7 @@ namespace core { /*! - * @brief Facade for a generic SE3 (6-DoF pose) tracker device exposed as ``Se3TrackerPoseTrackedT``. + * @brief Facade for a generic SE3 (6-DoF pose) tracker device exposed as ``Serialized``. * * Generic across rigid-body pose sources (tracker pucks, mocap rigid bodies, logical trackers * derived from other devices, ...): the payload is a single pose plus a validity flag. The @@ -25,9 +25,9 @@ namespace core * After each ``ITrackerSession::update()`` that includes this tracker, ``get_data(session)`` * reflects the implementation's tracked snapshot. As with other ``SchemaTracker``-backed trackers, * the live backend may retain the last-known sample when a tick has no new samples while the - * collection remains available (``data`` stays non-null but may be stale); ``data`` is null only - * when no sample has arrived yet or the collection is unavailable. Independently, - * ``data->is_valid == false`` means the producer is streaming but tracking is lost — the pose + * collection remains available (the handle stays non-empty but may be stale); the handle is empty + * only when no sample has arrived yet or the collection is unavailable. Independently, + * ``is_valid() == false`` means the producer is streaming but tracking is lost — the pose * contents are then unspecified. * * Note: ``collection_id`` (stream instance), ``TENSOR_IDENTIFIER`` (tensor name within the @@ -69,11 +69,11 @@ class Se3Tracker : public ITracker /*! * @brief SE3 tracker snapshot from the session's implementation. * - * ``tracked.data`` is null when no sample has arrived yet or the collection is unavailable. - * When non-null, gate on ``data->is_valid`` before consuming ``data->pose`` — the pose is + * The handle is empty when no sample has arrived yet or the collection is unavailable. + * When non-empty, gate on ``is_valid()`` before consuming ``pose()`` — the pose is * unspecified while tracking is lost. */ - const Se3TrackerPoseTrackedT& get_data(const ITrackerSession& session) const; + const Serialized& get_data(const ITrackerSession& session) const; const std::string& collection_id() const { diff --git a/src/core/deviceio_trackers/cpp/joint_state_tracker.cpp b/src/core/deviceio_trackers/cpp/joint_state_tracker.cpp index 5766ffef5..06464b332 100644 --- a/src/core/deviceio_trackers/cpp/joint_state_tracker.cpp +++ b/src/core/deviceio_trackers/cpp/joint_state_tracker.cpp @@ -15,7 +15,7 @@ JointStateTracker::JointStateTracker(const std::string& collection_id, size_t ma { } -const JointStateOutputTrackedT& JointStateTracker::get_data(const ITrackerSession& session) const +const Serialized& JointStateTracker::get_data(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_data(); } diff --git a/src/core/deviceio_trackers/cpp/message_channel_tracker.cpp b/src/core/deviceio_trackers/cpp/message_channel_tracker.cpp index 3774c06b2..ebd492bdd 100644 --- a/src/core/deviceio_trackers/cpp/message_channel_tracker.cpp +++ b/src/core/deviceio_trackers/cpp/message_channel_tracker.cpp @@ -24,7 +24,7 @@ MessageChannelStatus MessageChannelTracker::get_status(const ITrackerSession& se return static_cast(session.get_tracker_impl(*this)).get_status(); } -const MessageChannelMessagesTrackedT& MessageChannelTracker::get_messages(const ITrackerSession& session) const +const Serialized& MessageChannelTracker::get_messages(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_messages(); } diff --git a/src/core/deviceio_trackers/cpp/oglo_tactile_tracker.cpp b/src/core/deviceio_trackers/cpp/oglo_tactile_tracker.cpp index 851911d15..6b02858b0 100644 --- a/src/core/deviceio_trackers/cpp/oglo_tactile_tracker.cpp +++ b/src/core/deviceio_trackers/cpp/oglo_tactile_tracker.cpp @@ -15,7 +15,7 @@ OgloTactileTracker::OgloTactileTracker(const std::string& collection_id, size_t { } -const OgloGloveSampleTrackedT& OgloTactileTracker::get_data(const ITrackerSession& session) const +const Serialized& OgloTactileTracker::get_data(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_data(); } diff --git a/src/core/deviceio_trackers/cpp/se3_tracker.cpp b/src/core/deviceio_trackers/cpp/se3_tracker.cpp index 1f09ebeab..f20bd63e5 100644 --- a/src/core/deviceio_trackers/cpp/se3_tracker.cpp +++ b/src/core/deviceio_trackers/cpp/se3_tracker.cpp @@ -15,7 +15,7 @@ Se3Tracker::Se3Tracker(const std::string& collection_id, size_t max_flatbuffer_s { } -const Se3TrackerPoseTrackedT& Se3Tracker::get_data(const ITrackerSession& session) const +const Serialized& Se3Tracker::get_data(const ITrackerSession& session) const { return static_cast(session.get_tracker_impl(*this)).get_data(); } diff --git a/src/core/deviceio_trackers/python/tracker_bindings.cpp b/src/core/deviceio_trackers/python/tracker_bindings.cpp index a36b99213..aac9f85b0 100644 --- a/src/core/deviceio_trackers/python/tracker_bindings.cpp +++ b/src/core/deviceio_trackers/python/tracker_bindings.cpp @@ -27,41 +27,28 @@ namespace py = pybind11; namespace { -// Hand a tracker's tracked snapshot to Python without cloning its payload. +// Hand an encoded snapshot to Python, mapping "no payload" onto None. // -// Every Tracked wrapper holds its payload behind a shared_ptr, but flatc gives each generated -// -T a deep-copying copy constructor, so returning one by value clones the whole payload on -// every call: for a hand that is two allocations and 936 bytes of joints, twice per frame. -// Copying the shared_ptr instead costs one small wrapper allocation and a refcount bump, which -// is also what the .data property on these wrappers has always done. -// -// The payload is therefore live tracker storage. Most impls refill it in place on the next -// session.update(), so the returned object is a view valid until that call, not a snapshot -- -// see the note repeated on each accessor below. -template -std::shared_ptr share_tracked(const TrackedT& tracked) +// C++ spells absence as an empty handle, but exposing that to Python would make an +// inactive device answer field reads with defaults -- a disconnected pedal would read +// 0.0, indistinguishable from a pedal at rest. None makes the same mistake an +// AttributeError instead, and is what the caller already tests for. +template +py::object to_python(const core::Serialized& handle) { - // Only `data` is carried over, so a Tracked wrapper that grows a second field (the parallel - // Record wrappers already carry a timestamp) would be silently dropped here with nothing to - // flag it. NativeTable is empty, so a one-field wrapper is exactly as wide as its member. - static_assert(sizeof(TrackedT) == sizeof(decltype(TrackedT::data)), - "Tracked wrapper has fields beyond .data; share_tracked() would drop them"); - auto shared = std::make_shared(); - shared->data = tracked.data; - return shared; + return handle ? py::cast(handle) : py::none(); } } // namespace -// Appended to the docstring of every accessor that returns a Tracked wrapper. A macro so it -// concatenates with the surrounding literal at compile time; pybind11 takes a const char*. -#define TRACKED_LIFETIME_DOC \ - "\n\nThe returned wrapper shares the tracker's storage rather than copying it: its contents are " \ - "refilled by the next session.update(). Read what you need before that call, or copy it." +// Handing a snapshot to Python copies a shared_ptr to an immutable buffer, so there is no +// payload clone on the read path and no aliasing of live tracker storage: what a caller +// reads this frame keeps its values after the next session.update(), which publishes a new +// buffer rather than refilling this one. PYBIND11_MODULE(_deviceio_trackers, m) { - // Load schema pybind converters (TrackedT / schema types) before exposing tracker accessors. + // Load schema pybind converters (the encoded table views) before exposing tracker accessors. py::module_::import("isaacteleop.schema._schema"); m.doc() = "Isaac Teleop DeviceIO - Tracker classes"; @@ -75,37 +62,34 @@ PYBIND11_MODULE(_deviceio_trackers, m) .def( "get_left_hand", [](const core::HandTracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_left_hand(session)); }, - // No lifetime note on either hand: like full body, the impls install freshly allocated - // joint storage each frame rather than refilling it, so the result is a snapshot of the - // frame it was read on. Call again to see newer hand data. - py::arg("session"), "Get the left hand tracked state (data is None if inactive)") + { return to_python(self.get_left_hand(session)); }, + py::arg("session"), "Get the left hand tracked state (None if inactive)") .def( "get_right_hand", [](const core::HandTracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_right_hand(session)); }, - py::arg("session"), "Get the right hand tracked state (data is None if inactive)"); + { return to_python(self.get_right_hand(session)); }, + py::arg("session"), "Get the right hand tracked state (None if inactive)"); py::class_>(m, "HeadTracker") .def(py::init<>()) .def( "get_head", [](const core::HeadTracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_head(session)); }, - py::arg("session"), "Get the head tracked state (data is None if inactive)" TRACKED_LIFETIME_DOC); + { return to_python(self.get_head(session)); }, + py::arg("session"), "Get the head tracked state (None if inactive)"); py::class_>(m, "ControllerTracker") .def(py::init<>()) .def( "get_left_controller", [](const core::ControllerTracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_left_controller(session)); }, - py::arg("session"), "Get the left controller tracked state (data is None if inactive)" TRACKED_LIFETIME_DOC) + { return to_python(self.get_left_controller(session)); }, + py::arg("session"), "Get the left controller tracked state (None if inactive)") .def( "get_right_controller", [](const core::ControllerTracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_right_controller(session)); }, - py::arg("session"), "Get the right controller tracked state (data is None if inactive)" TRACKED_LIFETIME_DOC) + { return to_python(self.get_right_controller(session)); }, + py::arg("session"), "Get the right controller tracked state (None if inactive)") .def( "apply_left_haptic_feedback", [](const core::ControllerTracker& self, const core::ITrackerSession& session, float amplitude, @@ -151,9 +135,7 @@ PYBIND11_MODULE(_deviceio_trackers, m) .def( "get_messages", [](const core::MessageChannelTracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_messages(session)); }, - // No lifetime note here: this payload is a vector, so share_tracked() copies the list - // itself, and every drained message is freshly allocated. The result is a snapshot. + { return to_python(self.get_messages(session)); }, py::arg("session"), "Get all messages drained during the last update (possibly empty)") .def( "get_status", @@ -163,7 +145,12 @@ PYBIND11_MODULE(_deviceio_trackers, m) .def( "send_message", [](const core::MessageChannelTracker& self, const core::ITrackerSession& session, - const core::MessageChannelMessagesT& message) { self.send_message(session, message.payload); }, + const core::Serialized& message) + { + const auto* payload = message ? message->payload() : nullptr; + self.send_message(session, payload != nullptr ? std::vector(payload->begin(), payload->end()) : + std::vector{}); + }, py::arg("session"), py::arg("message"), "Send a MessageChannelMessages payload over the message channel"); py::class_>( @@ -175,10 +162,9 @@ PYBIND11_MODULE(_deviceio_trackers, m) .def( "get_stream_data", [](const core::FrameMetadataTrackerOak& self, const core::ITrackerSession& session, size_t stream_index) - { return share_tracked(self.get_stream_data(session, stream_index)); }, + { return to_python(self.get_stream_data(session, stream_index)); }, py::arg("session"), py::arg("stream_index"), - "Get FrameMetadataOakTrackedT for a specific stream by index; .data is None until first frame " - "arrives" TRACKED_LIFETIME_DOC) + "Get the frame metadata for a specific stream by index; None until the first frame arrives") .def_property_readonly("stream_count", &core::FrameMetadataTrackerOak::get_stream_count, "Number of streams this tracker is configured for"); @@ -190,9 +176,8 @@ PYBIND11_MODULE(_deviceio_trackers, m) .def( "get_pedal_data", [](const core::Generic3AxisPedalTracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_data(session)); }, - py::arg("session"), - "Get the current foot pedal tracked state (data is None when no data available)" TRACKED_LIFETIME_DOC); + { return to_python(self.get_data(session)); }, + py::arg("session"), "Get the current foot pedal tracked state (None when no data available)"); py::class_>( m, "OgloTactileTracker") @@ -203,9 +188,8 @@ PYBIND11_MODULE(_deviceio_trackers, m) .def( "get_glove_data", [](const core::OgloTactileTracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_data(session)); }, - py::arg("session"), - "Get the current tactile glove tracked state (data is None when no data available)" TRACKED_LIFETIME_DOC); + { return to_python(self.get_data(session)); }, + py::arg("session"), "Get the current tactile glove tracked state (None when no data available)"); py::class_> tensor_push_tracker( m, "TensorPushTracker"); @@ -234,9 +218,8 @@ PYBIND11_MODULE(_deviceio_trackers, m) .def( "get_data", [](const core::JointStateTracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_data(session)); }, - py::arg("session"), - "Get the current joint-state tracked state (data is None when no data available)" TRACKED_LIFETIME_DOC); + { return to_python(self.get_data(session)); }, + py::arg("session"), "Get the current joint-state tracked state (None when no data available)"); py::class_>(m, "Se3Tracker") .def(py::init(), py::arg("collection_id"), @@ -246,10 +229,10 @@ PYBIND11_MODULE(_deviceio_trackers, m) .def( "get_data", [](const core::Se3Tracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_data(session)); }, + { return to_python(self.get_data(session)); }, py::arg("session"), - "Get the current SE3 tracked state (data is None when no data available; gate on " - "data.is_valid before consuming the pose)" TRACKED_LIFETIME_DOC); + "Get the current SE3 tracked state (None when no data available; gate on " + "is_valid before consuming the pose)"); py::class_>(m, "FullBodyTracker") .def(py::init<>(), @@ -258,11 +241,8 @@ PYBIND11_MODULE(_deviceio_trackers, m) .def( "get_body_pose", [](const core::FullBodyTracker& self, const core::ITrackerSession& session) - { return share_tracked(self.get_body_pose(session)); }, - // No lifetime note here: the full-body impls install freshly allocated joint storage - // each frame rather than refilling it, so the result is a snapshot of the frame it was - // read on. Call again to see newer body data. - py::arg("session"), "Get full body pose tracked state (data is None if inactive)"); + { return to_python(self.get_body_pose(session)); }, + py::arg("session"), "Get full body pose tracked state (None if inactive)"); m.attr("NUM_JOINTS") = static_cast(core::HandJoint_NUM_JOINTS); m.attr("JOINT_PALM") = static_cast(core::HandJoint_PALM); diff --git a/src/core/live_trackers/AGENTS.md b/src/core/live_trackers/AGENTS.md index 8d882310d..e0d78f70c 100644 --- a/src/core/live_trackers/AGENTS.md +++ b/src/core/live_trackers/AGENTS.md @@ -41,6 +41,13 @@ When adding MCAP support to a new tracker impl, all of the following are require 7. **Always build** (`cmake --build -- -j$(nproc)`) before treating work as done. Pre-commit alone does not catch compile errors or clang-format violations enforced at build time. 8. Read `AGENTS.md` before starting. Not after CI breaks. +## Publishing tracker output + +- An impl may keep a `-T` as **assembly scratch** (name it `native_`), but what it publishes is a `Serialized` encoded once per `update()`. Getters return the published handle; the scratch never escapes. +- **Encode on every exit path of `update()`**, including early returns and the throwing ones (limp mode, locate failure) — otherwise a consumer keeps reading last frame's snapshot after the device drops out. +- Encode into a **new** buffer each frame rather than over the previous one. Consumers hold snapshots, so a caller that read last frame must keep seeing last frame's values; this is what removed the old "valid until the next `session.update()`" caveat. +- `SchemaTracker` does this for tensor-sourced trackers, and does it without encoding at all: the wire already carries the payload table, so it **adopts the sample's buffer**. Do not reintroduce an unpack on that path — the only reason it materialises a native is MCAP recording, which is why that unpack is gated on `mcap_channels_`. + ## Related docs - Session update loop: [`../deviceio_session/AGENTS.md`](../deviceio_session/AGENTS.md) diff --git a/src/core/live_trackers/cpp/inc/live_trackers/schema_tracker.hpp b/src/core/live_trackers/cpp/inc/live_trackers/schema_tracker.hpp index 9dc8cfb1a..8fcff930c 100644 --- a/src/core/live_trackers/cpp/inc/live_trackers/schema_tracker.hpp +++ b/src/core/live_trackers/cpp/inc/live_trackers/schema_tracker.hpp @@ -7,6 +7,7 @@ #include #include +#include #include #include @@ -21,7 +22,8 @@ namespace core * read from the tensor can be automatically written to an MCAP channel. * * @tparam RecordT FlatBuffer record wrapper (e.g. Generic3AxisPedalOutputRecord). - * @tparam DataTableT FlatBuffer data table (e.g. Generic3AxisPedalOutput). + * @tparam DataTableT FlatBuffer data table (e.g. Generic3AxisPedalOutput). This is both + * the wire type and the type published to consumers. */ template class SchemaTracker : public SchemaTrackerBase @@ -55,19 +57,20 @@ class SchemaTracker : public SchemaTrackerBase /** * @brief Read all pending samples; write each to MCAP if channels are set. * - * Each sample is unpacked, repacked into a Record with its timestamp, - * and written to the MCAP channel. The last sample's unpacked data is - * returned via out_latest (if non-null and samples were read). + * The wire already carries `DataTableT`, which is exactly what consumers read, so + * the final sample is published by taking ownership of its buffer -- no unpack and + * no re-encode. Recording is the only reason to materialise a native, so samples are + * unpacked solely when MCAP channels are attached. * - * @param out_latest If non-null and samples were read, receives the unpacked - * data from the last sample. Cleared when the tensor collection - * is absent. + * @param out Receives the final sample of this tick when any were read; left + * untouched when the collection is present but produced nothing (the + * last-known sample is retained); emptied when the collection is absent. * @throws std::runtime_error On critical OpenXR/tensor API failures propagated * from SchemaTrackerBase. * @note Missing collection, temporary collection loss, and "no new sample" * are treated as common non-fatal conditions and do not throw. */ - void update(std::shared_ptr& out_latest) + void update(Serialized& out) { samples_.clear(); bool present = read_all_samples(samples_); @@ -76,39 +79,44 @@ class SchemaTracker : public SchemaTrackerBase { if (!present) { - out_latest.reset(); + out.reset(); } return; } - DeviceDataTimestamp last_timestamp{}; - for (const auto& sample : samples_) + if (mcap_channels_) { - auto fb = flatbuffers::GetRoot(sample.buffer.data()); - if (!fb) + DeviceDataTimestamp last_timestamp{}; + for (const auto& sample : samples_) { - continue; - } + auto fb = flatbuffers::GetRoot(sample.buffer.data()); + if (!fb) + { + continue; + } - if (!out_latest) - { - out_latest = std::make_shared(); + if (!recording_scratch_) + { + recording_scratch_ = std::make_shared(); + } + fb->UnPackTo(recording_scratch_.get()); + last_timestamp = sample.timestamp; + + // write() serializes synchronously and does not retain the shared_ptr, + // so reusing the scratch across loop iterations is safe. + mcap_channels_->write(mcap_channel_index_, sample.timestamp, recording_scratch_); } - fb->UnPackTo(out_latest.get()); - last_timestamp = sample.timestamp; - // write() serializes synchronously and does not retain the shared_ptr, - // so reusing out_latest across loop iterations is safe. - if (mcap_channels_) + if (mcap_channel_tracked_index_ && recording_scratch_) { - mcap_channels_->write(mcap_channel_index_, sample.timestamp, out_latest); + mcap_channels_->write(*mcap_channel_tracked_index_, last_timestamp, recording_scratch_); } } - if (mcap_channel_tracked_index_ && mcap_channels_ && out_latest) - { - mcap_channels_->write(*mcap_channel_tracked_index_, last_timestamp, out_latest); - } + // Adopt the final sample's bytes rather than copying them: the wire type is the + // published type. Each tick owns its own buffer, so a consumer still holding last + // tick's handle keeps last tick's values. + out = Serialized::adopt(std::move(samples_.back().buffer)); } private: @@ -116,6 +124,9 @@ class SchemaTracker : public SchemaTrackerBase size_t mcap_channel_index_; std::optional mcap_channel_tracked_index_; std::vector samples_; + // Unpack target for MCAP only -- McapTrackerChannels::write takes a native. Stays + // null while recording is disabled, which is what keeps the read path unpack-free. + std::shared_ptr recording_scratch_; }; } // namespace core diff --git a/src/core/live_trackers/cpp/live_controller_tracker_impl.cpp b/src/core/live_trackers/cpp/live_controller_tracker_impl.cpp index 3aba49490..cd4d380a2 100644 --- a/src/core/live_trackers/cpp/live_controller_tracker_impl.cpp +++ b/src/core/live_trackers/cpp/live_controller_tracker_impl.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -357,18 +358,20 @@ void LiveControllerTrackerImpl::update(int64_t monotonic_time_ns) { // Policy: action sync failure is a critical tracker/runtime error. // Ensure callers do not observe stale controller data after sync failure. - left_tracked_.data.reset(); - right_tracked_.data.reset(); + left_native_.reset(); + right_native_.reset(); + left_tracked_.reset(); + right_tracked_.reset(); throw std::runtime_error("[ControllerTracker] xrSyncActions2NV failed: " + std::to_string(result)); } auto update_controller = [&](XrPath hand_path, const XrSpacePtr& grip_space, const XrSpacePtr& aim_space, - ControllerSnapshotTrackedT& tracked) + std::shared_ptr& tracked) { if (!get_pose_action_active(session_, core_funcs_, grip_pose_action_, hand_path)) { // Policy: controller not active is a common runtime condition. - tracked.data.reset(); + tracked.reset(); return; } @@ -379,7 +382,7 @@ void LiveControllerTrackerImpl::update(int64_t monotonic_time_ns) result = core_funcs_.xrLocateSpace(grip_space.get(), base_space_, xr_time, &grip_location); if (XR_FAILED(result)) { - tracked.data.reset(); + tracked.reset(); throw std::runtime_error("[ControllerTracker] xrLocateSpace(grip) failed: " + std::to_string(result)); } if (XR_SUCCEEDED(result)) @@ -400,7 +403,7 @@ void LiveControllerTrackerImpl::update(int64_t monotonic_time_ns) result = core_funcs_.xrLocateSpace(aim_space.get(), base_space_, xr_time, &aim_location); if (XR_FAILED(result)) { - tracked.data.reset(); + tracked.reset(); throw std::runtime_error("[ControllerTracker] xrLocateSpace(aim) failed: " + std::to_string(result)); } if (XR_SUCCEEDED(result)) @@ -430,32 +433,35 @@ void LiveControllerTrackerImpl::update(int64_t monotonic_time_ns) ControllerInputState inputs(primary_click, secondary_click, thumbstick_click, menu_click, thumbstick_x, thumbstick_y, squeeze_value, trigger_value); - if (!tracked.data) + if (!tracked) { - tracked.data = std::make_shared(); + tracked = std::make_shared(); } - tracked.data->grip_pose = std::make_shared(grip_pose); - tracked.data->aim_pose = std::make_shared(aim_pose); - tracked.data->inputs = std::make_shared(inputs); + tracked->grip_pose = std::make_shared(grip_pose); + tracked->aim_pose = std::make_shared(aim_pose); + tracked->inputs = std::make_shared(inputs); }; - update_controller(left_hand_path_, left_grip_space_, left_aim_space_, left_tracked_); - update_controller(right_hand_path_, right_grip_space_, right_aim_space_, right_tracked_); + update_controller(left_hand_path_, left_grip_space_, left_aim_space_, left_native_); + update_controller(right_hand_path_, right_grip_space_, right_aim_space_, right_native_); + + left_tracked_ = pack_optional(left_native_); + right_tracked_ = pack_optional(right_native_); if (mcap_channels_) { DeviceDataTimestamp timestamp(last_update_time_, last_update_time_, xr_time); - mcap_channels_->write(0, timestamp, left_tracked_.data); - mcap_channels_->write(1, timestamp, right_tracked_.data); + mcap_channels_->write(0, timestamp, left_native_); + mcap_channels_->write(1, timestamp, right_native_); } } -const ControllerSnapshotTrackedT& LiveControllerTrackerImpl::get_left_controller() const +const Serialized& LiveControllerTrackerImpl::get_left_controller() const { return left_tracked_; } -const ControllerSnapshotTrackedT& LiveControllerTrackerImpl::get_right_controller() const +const Serialized& LiveControllerTrackerImpl::get_right_controller() const { return right_tracked_; } diff --git a/src/core/live_trackers/cpp/live_controller_tracker_impl.hpp b/src/core/live_trackers/cpp/live_controller_tracker_impl.hpp index c61200bf3..0f7fad6a2 100644 --- a/src/core/live_trackers/cpp/live_controller_tracker_impl.hpp +++ b/src/core/live_trackers/cpp/live_controller_tracker_impl.hpp @@ -42,8 +42,8 @@ class LiveControllerTrackerImpl : public IControllerTrackerImpl LiveControllerTrackerImpl& operator=(LiveControllerTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const ControllerSnapshotTrackedT& get_left_controller() const override; - const ControllerSnapshotTrackedT& get_right_controller() const override; + const Serialized& get_left_controller() const override; + const Serialized& get_right_controller() const override; void apply_left_haptic_feedback(float amplitude, float frequency_hz, float duration_s) const override; void apply_right_haptic_feedback(float amplitude, float frequency_hz, float duration_s) const override; @@ -90,8 +90,12 @@ class LiveControllerTrackerImpl : public IControllerTrackerImpl XrSpacePtr left_aim_space_; XrSpacePtr right_aim_space_; - ControllerSnapshotTrackedT left_tracked_; - ControllerSnapshotTrackedT right_tracked_; + // Assembly scratch for the OpenXR query, and the encoded snapshots published from it + // each frame. Only the latter leave this class. + std::shared_ptr left_native_; + std::shared_ptr right_native_; + Serialized left_tracked_; + Serialized right_tracked_; int64_t last_update_time_ = 0; // Once-per-side log gates for OpenXR haptic call failures. Indexed by diff --git a/src/core/live_trackers/cpp/live_frame_metadata_tracker_oak_impl.cpp b/src/core/live_trackers/cpp/live_frame_metadata_tracker_oak_impl.cpp index fa28deb80..c996e7a00 100644 --- a/src/core/live_trackers/cpp/live_frame_metadata_tracker_oak_impl.cpp +++ b/src/core/live_trackers/cpp/live_frame_metadata_tracker_oak_impl.cpp @@ -65,11 +65,11 @@ void LiveFrameMetadataTrackerOakImpl::update(int64_t /*monotonic_time_ns*/) // Missing stream collection/no fresh sample are treated as common non-fatal cases. for (auto& stream : m_streams) { - stream.reader->update(stream.tracked.data); + stream.reader->update(stream.tracked); } } -const FrameMetadataOakTrackedT& LiveFrameMetadataTrackerOakImpl::get_stream_data(size_t stream_index) const +const Serialized& LiveFrameMetadataTrackerOakImpl::get_stream_data(size_t stream_index) const { if (stream_index >= m_streams.size()) { diff --git a/src/core/live_trackers/cpp/live_frame_metadata_tracker_oak_impl.hpp b/src/core/live_trackers/cpp/live_frame_metadata_tracker_oak_impl.hpp index f839eca7b..9820a1730 100644 --- a/src/core/live_trackers/cpp/live_frame_metadata_tracker_oak_impl.hpp +++ b/src/core/live_trackers/cpp/live_frame_metadata_tracker_oak_impl.hpp @@ -42,13 +42,13 @@ class LiveFrameMetadataTrackerOakImpl : public IFrameMetadataTrackerOakImpl LiveFrameMetadataTrackerOakImpl& operator=(LiveFrameMetadataTrackerOakImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const FrameMetadataOakTrackedT& get_stream_data(size_t stream_index) const override; + const Serialized& get_stream_data(size_t stream_index) const override; private: struct StreamState { std::unique_ptr reader; - FrameMetadataOakTrackedT tracked; + Serialized tracked; }; std::unique_ptr mcap_channels_; diff --git a/src/core/live_trackers/cpp/live_full_body_tracker_noitom_impl.cpp b/src/core/live_trackers/cpp/live_full_body_tracker_noitom_impl.cpp index 2b0a636eb..1fe1a9e3e 100644 --- a/src/core/live_trackers/cpp/live_full_body_tracker_noitom_impl.cpp +++ b/src/core/live_trackers/cpp/live_full_body_tracker_noitom_impl.cpp @@ -83,10 +83,10 @@ LiveFullBodyTrackerNoitomImpl::LiveFullBodyTrackerNoitomImpl(const OpenXRSession void LiveFullBodyTrackerNoitomImpl::update(int64_t /*monotonic_time_ns*/) { - schema_reader_.update(tracked_.data); + schema_reader_.update(tracked_); } -const FullBodyPoseTrackedT& LiveFullBodyTrackerNoitomImpl::get_body_pose() const +const Serialized& LiveFullBodyTrackerNoitomImpl::get_body_pose() const { return tracked_; } diff --git a/src/core/live_trackers/cpp/live_full_body_tracker_noitom_impl.hpp b/src/core/live_trackers/cpp/live_full_body_tracker_noitom_impl.hpp index 0743d0292..6b892feb7 100644 --- a/src/core/live_trackers/cpp/live_full_body_tracker_noitom_impl.hpp +++ b/src/core/live_trackers/cpp/live_full_body_tracker_noitom_impl.hpp @@ -48,12 +48,12 @@ class LiveFullBodyTrackerNoitomImpl : public IFullBodyTrackerImpl LiveFullBodyTrackerNoitomImpl& operator=(LiveFullBodyTrackerNoitomImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const FullBodyPoseTrackedT& get_body_pose() const override; + const Serialized& get_body_pose() const override; private: std::unique_ptr mcap_channels_; FullBodyNoitomSchemaTracker schema_reader_; - FullBodyPoseTrackedT tracked_; + Serialized tracked_; }; } // namespace core diff --git a/src/core/live_trackers/cpp/live_full_body_tracker_pico_impl.cpp b/src/core/live_trackers/cpp/live_full_body_tracker_pico_impl.cpp index 59e4af5be..e346a9a7b 100644 --- a/src/core/live_trackers/cpp/live_full_body_tracker_pico_impl.cpp +++ b/src/core/live_trackers/cpp/live_full_body_tracker_pico_impl.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -105,7 +106,8 @@ void LiveFullBodyTrackerPicoImpl::update(int64_t monotonic_time_ns) if (body_tracker_ == XR_NULL_HANDLE) { // Policy: limp mode (feature unsupported/unavailable) is non-fatal. - tracked_.data.reset(); + native_.reset(); + tracked_.reset(); return; } @@ -126,13 +128,11 @@ void LiveFullBodyTrackerPicoImpl::update(int64_t monotonic_time_ns) XrResult result = pfn_locate_body_joints_(body_tracker_, &locate_info, &locations); if (XR_FAILED(result)) { - tracked_.data.reset(); + native_.reset(); + tracked_.reset(); throw std::runtime_error("[FullBodyTracker] xrLocateBodyJointsBD failed: " + std::to_string(result)); } - // Publish freshly allocated joint storage each frame instead of refilling the previous - // frame's. The query API hands the pose out by reference and callers may still hold an - // earlier frame's joints, so an in-place refill would change data already handed out. auto data = std::make_shared(); data->all_joint_poses_tracked = locations.allJointPosesTracked; data->joints = std::make_shared(); @@ -153,16 +153,17 @@ void LiveFullBodyTrackerPicoImpl::update(int64_t monotonic_time_ns) data->joints->mutable_joints()->Mutate(i, joint_pose); } - tracked_.data = std::move(data); + native_ = std::move(data); + tracked_ = pack(*native_); if (mcap_channels_) { DeviceDataTimestamp timestamp(last_update_time_, last_update_time_, xr_time); - mcap_channels_->write(0, timestamp, tracked_.data); + mcap_channels_->write(0, timestamp, native_); } } -const FullBodyPoseTrackedT& LiveFullBodyTrackerPicoImpl::get_body_pose() const +const Serialized& LiveFullBodyTrackerPicoImpl::get_body_pose() const { return tracked_; } diff --git a/src/core/live_trackers/cpp/live_full_body_tracker_pico_impl.hpp b/src/core/live_trackers/cpp/live_full_body_tracker_pico_impl.hpp index 7c11b5d72..9c44da861 100644 --- a/src/core/live_trackers/cpp/live_full_body_tracker_pico_impl.hpp +++ b/src/core/live_trackers/cpp/live_full_body_tracker_pico_impl.hpp @@ -45,13 +45,16 @@ class LiveFullBodyTrackerPicoImpl : public IFullBodyTrackerImpl LiveFullBodyTrackerPicoImpl& operator=(LiveFullBodyTrackerPicoImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const FullBodyPoseTrackedT& get_body_pose() const override; + const Serialized& get_body_pose() const override; private: XrTimeConverter time_converter_; XrSpace base_space_; XrBodyTrackerBD body_tracker_; - FullBodyPoseTrackedT tracked_; + // Assembly scratch for the OpenXR query, and the encoded snapshot published from it + // each frame. Only the latter leaves this class. + std::shared_ptr native_; + Serialized tracked_; int64_t last_update_time_ = 0; PFN_xrCreateBodyTrackerBD pfn_create_body_tracker_; diff --git a/src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.cpp b/src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.cpp index 3a4791856..47a231b36 100644 --- a/src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.cpp +++ b/src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.cpp @@ -52,10 +52,10 @@ void LiveGeneric3AxisPedalTrackerImpl::update(int64_t /*monotonic_time_ns*/) { // Policy: SchemaTracker throws on critical OpenXR/tensor API failures. // Missing collection/no new data are treated as common non-fatal cases. - m_schema_reader.update(m_tracked.data); + m_schema_reader.update(m_tracked); } -const Generic3AxisPedalOutputTrackedT& LiveGeneric3AxisPedalTrackerImpl::get_data() const +const Serialized& LiveGeneric3AxisPedalTrackerImpl::get_data() const { return m_tracked; } diff --git a/src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.hpp b/src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.hpp index f2d403591..586dc25b3 100644 --- a/src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.hpp +++ b/src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.hpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #pragma once @@ -40,12 +40,12 @@ class LiveGeneric3AxisPedalTrackerImpl : public IGeneric3AxisPedalTrackerImpl LiveGeneric3AxisPedalTrackerImpl& operator=(LiveGeneric3AxisPedalTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const Generic3AxisPedalOutputTrackedT& get_data() const override; + const Serialized& get_data() const override; private: std::unique_ptr mcap_channels_; PedalSchemaTracker m_schema_reader; - Generic3AxisPedalOutputTrackedT m_tracked; + Serialized m_tracked; }; } // namespace core diff --git a/src/core/live_trackers/cpp/live_hand_tracker_impl.cpp b/src/core/live_trackers/cpp/live_hand_tracker_impl.cpp index df6d1de6e..3171de474 100644 --- a/src/core/live_trackers/cpp/live_hand_tracker_impl.cpp +++ b/src/core/live_trackers/cpp/live_hand_tracker_impl.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -176,23 +177,26 @@ void LiveHandTrackerImpl::update(int64_t monotonic_time_ns) { last_update_time_ = monotonic_time_ns; const XrTime xr_time = time_converter_.convert_monotonic_ns_to_xrtime(monotonic_time_ns); - update_hand(left_hand_trackers_, xr_time, left_tracked_); - update_hand(right_hand_trackers_, xr_time, right_tracked_); + update_hand(left_hand_trackers_, xr_time, left_native_); + update_hand(right_hand_trackers_, xr_time, right_native_); + + left_tracked_ = pack_optional(left_native_); + right_tracked_ = pack_optional(right_native_); if (mcap_channels_) { DeviceDataTimestamp timestamp(last_update_time_, last_update_time_, xr_time); - mcap_channels_->write(0, timestamp, left_tracked_.data); - mcap_channels_->write(1, timestamp, right_tracked_.data); + mcap_channels_->write(0, timestamp, left_native_); + mcap_channels_->write(1, timestamp, right_native_); } } -const HandPoseTrackedT& LiveHandTrackerImpl::get_left_hand() const +const Serialized& LiveHandTrackerImpl::get_left_hand() const { return left_tracked_; } -const HandPoseTrackedT& LiveHandTrackerImpl::get_right_hand() const +const Serialized& LiveHandTrackerImpl::get_right_hand() const { return right_tracked_; } @@ -360,11 +364,13 @@ void LiveHandTrackerImpl::destroy_xdev_list() } } -void LiveHandTrackerImpl::update_hand(const std::vector& trackers, XrTime time, HandPoseTrackedT& tracked) +void LiveHandTrackerImpl::update_hand(const std::vector& trackers, + XrTime time, + std::shared_ptr& tracked) { for (XrHandTrackerEXT tracker : trackers) { - HandPoseTrackedT candidate; + std::shared_ptr candidate; if (try_update_hand(tracker, time, candidate)) { tracked = std::move(candidate); @@ -372,14 +378,14 @@ void LiveHandTrackerImpl::update_hand(const std::vector& track } } - tracked.data.reset(); + tracked.reset(); } -bool LiveHandTrackerImpl::try_update_hand(XrHandTrackerEXT tracker, XrTime time, HandPoseTrackedT& tracked) +bool LiveHandTrackerImpl::try_update_hand(XrHandTrackerEXT tracker, XrTime time, std::shared_ptr& tracked) { if (tracker == XR_NULL_HANDLE) { - tracked.data.reset(); + tracked.reset(); return false; } @@ -397,20 +403,19 @@ bool LiveHandTrackerImpl::try_update_hand(XrHandTrackerEXT tracker, XrTime time, XrResult result = pfn_locate_hand_joints_(tracker, &locate_info, &locations); if (XR_FAILED(result)) { - tracked.data.reset(); + tracked.reset(); return false; } if (!locations.isActive) { // Policy: inactive hand is a common runtime condition; non-fatal. - tracked.data.reset(); + tracked.reset(); return false; } - // Publish freshly allocated joint storage each frame instead of refilling the previous - // frame's. The query API hands the pose out by reference and callers may still hold an - // earlier frame's joints, so an in-place refill would change data already handed out. + // Scratch storage for this candidate: update_hand() only commits it once the locate + // call succeeds, and update() encodes the winner into the published buffer. auto data = std::make_shared(); data->joints = std::make_shared(); @@ -430,7 +435,7 @@ bool LiveHandTrackerImpl::try_update_hand(XrHandTrackerEXT tracker, XrTime time, data->joints->mutable_poses()->Mutate(i, joint_pose); } - tracked.data = std::move(data); + tracked = std::move(data); return true; } diff --git a/src/core/live_trackers/cpp/live_hand_tracker_impl.hpp b/src/core/live_trackers/cpp/live_hand_tracker_impl.hpp index e247119f3..9bf894b4c 100644 --- a/src/core/live_trackers/cpp/live_hand_tracker_impl.hpp +++ b/src/core/live_trackers/cpp/live_hand_tracker_impl.hpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #pragma once @@ -37,8 +37,8 @@ class LiveHandTrackerImpl : public IHandTrackerImpl LiveHandTrackerImpl& operator=(LiveHandTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const HandPoseTrackedT& get_left_hand() const override; - const HandPoseTrackedT& get_right_hand() const override; + const Serialized& get_left_hand() const override; + const Serialized& get_right_hand() const override; private: void initialize_xdev_hand_trackers(const OpenXRSessionHandles& handles); @@ -49,8 +49,8 @@ class LiveHandTrackerImpl : public IHandTrackerImpl bool try_create_default_hand_tracker(XrSession session, XrHandEXT hand, std::vector& trackers); void destroy_hand_trackers(std::vector& trackers); void destroy_xdev_list(); - void update_hand(const std::vector& trackers, XrTime time, HandPoseTrackedT& tracked); - bool try_update_hand(XrHandTrackerEXT tracker, XrTime time, HandPoseTrackedT& tracked); + void update_hand(const std::vector& trackers, XrTime time, std::shared_ptr& tracked); + bool try_update_hand(XrHandTrackerEXT tracker, XrTime time, std::shared_ptr& tracked); XrTimeConverter time_converter_; XrSpace base_space_; @@ -59,8 +59,12 @@ class LiveHandTrackerImpl : public IHandTrackerImpl std::vector right_hand_trackers_; XrXDevListMNDX xdev_list_; - HandPoseTrackedT left_tracked_; - HandPoseTrackedT right_tracked_; + // Assembly scratch for the OpenXR query, and the encoded snapshots published from + // it each frame. Only the latter leave this class. + std::shared_ptr left_native_; + std::shared_ptr right_native_; + Serialized left_tracked_; + Serialized right_tracked_; int64_t last_update_time_ = 0; PFN_xrCreateHandTrackerEXT pfn_create_hand_tracker_; diff --git a/src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.cpp b/src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.cpp index 74acf011b..678d397ef 100644 --- a/src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.cpp +++ b/src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.cpp @@ -4,6 +4,7 @@ #include "live_haptic_command_reader_tracker_impl.hpp" #include +#include #include #include @@ -61,34 +62,32 @@ void LiveHapticCommandReaderTrackerImpl::update(int64_t /*monotonic_time_ns*/) // collection interleaves all endpoints, so bucket by HapticCommand.endpoint // instead of collapsing to one latest sample (which would drop every endpoint // but the last one pushed each frame). - for (const auto& sample : samples_) + for (auto& sample : samples_) { const auto* fb = flatbuffers::GetRoot(sample.buffer.data()); if (fb == nullptr) { continue; } - const std::string endpoint = fb->endpoint() != nullptr ? fb->endpoint()->str() : std::string{}; - HapticCommandTrackedT& tracked = tracked_by_endpoint_[endpoint]; - if (!tracked.data) - { - tracked.data = std::make_unique(); - } - fb->UnPackTo(tracked.data.get()); - latest_endpoint_ = endpoint; + std::string endpoint = fb->endpoint() != nullptr ? fb->endpoint()->str() : std::string{}; + // The wire already carries HapticCommand, so adopt the sample's bytes instead of + // unpacking and re-encoding them. `fb` dangles past this point; read the endpoint + // out first. + tracked_by_endpoint_[endpoint] = Serialized::adopt(std::move(sample.buffer)); + latest_endpoint_ = std::move(endpoint); } } -const HapticCommandTrackedT& LiveHapticCommandReaderTrackerImpl::get_data() const +const Serialized& LiveHapticCommandReaderTrackerImpl::get_data() const { // Backward-compatible latest-across-all-endpoints view: the endpoint of the // most recently drained sample. return get_data(latest_endpoint_); } -const HapticCommandTrackedT& LiveHapticCommandReaderTrackerImpl::get_data(std::string_view endpoint) const +const Serialized& LiveHapticCommandReaderTrackerImpl::get_data(std::string_view endpoint) const { - static const HapticCommandTrackedT kEmpty{}; + static const Serialized kEmpty{}; const auto it = tracked_by_endpoint_.find(endpoint); return it != tracked_by_endpoint_.end() ? it->second : kEmpty; } diff --git a/src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.hpp b/src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.hpp index 26c2b6981..089510b9a 100644 --- a/src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.hpp +++ b/src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.hpp @@ -40,8 +40,8 @@ class LiveHapticCommandReaderTrackerImpl : public IHapticCommandReaderTrackerImp LiveHapticCommandReaderTrackerImpl& operator=(LiveHapticCommandReaderTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const HapticCommandTrackedT& get_data() const override; - const HapticCommandTrackedT& get_data(std::string_view endpoint) const override; + const Serialized& get_data() const override; + const Serialized& get_data(std::string_view endpoint) const override; private: HapticCommandSchemaTracker schema_reader_; @@ -49,7 +49,7 @@ class LiveHapticCommandReaderTrackerImpl : public IHapticCommandReaderTrackerImp // endpoint's command (each tagged by HapticCommand.endpoint); bucketing by // that tag keeps concurrent endpoints (e.g. left/right) from overwriting one // another the way a single latest-sample slot would. - std::map> tracked_by_endpoint_; + std::map, std::less<>> tracked_by_endpoint_; // Endpoint of the most recently drained sample, backing the no-arg get_data(). std::string latest_endpoint_; std::vector samples_; diff --git a/src/core/live_trackers/cpp/live_head_tracker_impl.cpp b/src/core/live_trackers/cpp/live_head_tracker_impl.cpp index 0119e86fe..33d7ecb7b 100644 --- a/src/core/live_trackers/cpp/live_head_tracker_impl.cpp +++ b/src/core/live_trackers/cpp/live_head_tracker_impl.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -37,7 +38,7 @@ LiveHeadTrackerImpl::LiveHeadTrackerImpl(const OpenXRSessionHandles& handles, { .type = XR_TYPE_REFERENCE_SPACE_CREATE_INFO, .referenceSpaceType = XR_REFERENCE_SPACE_TYPE_VIEW, .poseInReferenceSpace = { .orientation = { 0, 0, 0, 1 } } })), - tracked_{}, + native_{}, mcap_channels_(std::move(mcap_channels)) { } @@ -53,41 +54,44 @@ void LiveHeadTrackerImpl::update(int64_t monotonic_time_ns) if (XR_FAILED(result)) { - tracked_.data.reset(); + native_.reset(); + tracked_.reset(); throw std::runtime_error("[HeadTracker] xrLocateSpace failed: " + std::to_string(result)); } bool position_valid = (location.locationFlags & XR_SPACE_LOCATION_POSITION_VALID_BIT) != 0; bool orientation_valid = (location.locationFlags & XR_SPACE_LOCATION_ORIENTATION_VALID_BIT) != 0; - if (!tracked_.data) + if (!native_) { - tracked_.data = std::make_shared(); + native_ = std::make_shared(); } - tracked_.data->is_valid = position_valid && orientation_valid; + native_->is_valid = position_valid && orientation_valid; - if (tracked_.data->is_valid) + if (native_->is_valid) { Point position(location.pose.position.x, location.pose.position.y, location.pose.position.z); Quaternion orientation(location.pose.orientation.x, location.pose.orientation.y, location.pose.orientation.z, location.pose.orientation.w); - tracked_.data->pose = std::make_shared(position, orientation); + native_->pose = std::make_shared(position, orientation); } else { // Keep pose populated whenever data is present; validity is indicated by is_valid. - tracked_.data->pose = std::make_shared(); + native_->pose = std::make_shared(); } + tracked_ = pack(*native_); + if (mcap_channels_) { DeviceDataTimestamp timestamp(last_update_time_, last_update_time_, xr_time); - mcap_channels_->write(0, timestamp, tracked_.data); + mcap_channels_->write(0, timestamp, native_); } } -const HeadPoseTrackedT& LiveHeadTrackerImpl::get_head() const +const Serialized& LiveHeadTrackerImpl::get_head() const { return tracked_; } diff --git a/src/core/live_trackers/cpp/live_head_tracker_impl.hpp b/src/core/live_trackers/cpp/live_head_tracker_impl.hpp index a212b1875..ed8cc7f01 100644 --- a/src/core/live_trackers/cpp/live_head_tracker_impl.hpp +++ b/src/core/live_trackers/cpp/live_head_tracker_impl.hpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #pragma once @@ -38,14 +38,17 @@ class LiveHeadTrackerImpl : public IHeadTrackerImpl LiveHeadTrackerImpl& operator=(LiveHeadTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const HeadPoseTrackedT& get_head() const override; + const Serialized& get_head() const override; private: const OpenXRCoreFunctions core_funcs_; XrTimeConverter time_converter_; XrSpace base_space_; XrSpacePtr view_space_; - HeadPoseTrackedT tracked_; + // Assembly scratch for the OpenXR query, and the encoded snapshot published from it + // each frame. Only the latter leaves this class. + std::shared_ptr native_; + Serialized tracked_; int64_t last_update_time_ = 0; std::unique_ptr mcap_channels_; }; diff --git a/src/core/live_trackers/cpp/live_joint_state_tracker_impl.cpp b/src/core/live_trackers/cpp/live_joint_state_tracker_impl.cpp index 6aa679810..a13b7a181 100644 --- a/src/core/live_trackers/cpp/live_joint_state_tracker_impl.cpp +++ b/src/core/live_trackers/cpp/live_joint_state_tracker_impl.cpp @@ -53,10 +53,10 @@ void LiveJointStateTrackerImpl::update(int64_t /*monotonic_time_ns*/) { // Policy: SchemaTracker throws on critical OpenXR/tensor API failures. // Missing collection/no new data are treated as common non-fatal cases. - m_schema_reader.update(m_tracked.data); + m_schema_reader.update(m_tracked); } -const JointStateOutputTrackedT& LiveJointStateTrackerImpl::get_data() const +const Serialized& LiveJointStateTrackerImpl::get_data() const { return m_tracked; } diff --git a/src/core/live_trackers/cpp/live_joint_state_tracker_impl.hpp b/src/core/live_trackers/cpp/live_joint_state_tracker_impl.hpp index b39a7e5ae..e33f86a2b 100644 --- a/src/core/live_trackers/cpp/live_joint_state_tracker_impl.hpp +++ b/src/core/live_trackers/cpp/live_joint_state_tracker_impl.hpp @@ -41,12 +41,12 @@ class LiveJointStateTrackerImpl : public IJointStateTrackerImpl LiveJointStateTrackerImpl& operator=(LiveJointStateTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const JointStateOutputTrackedT& get_data() const override; + const Serialized& get_data() const override; private: std::unique_ptr mcap_channels_; JointStateSchemaTracker m_schema_reader; - JointStateOutputTrackedT m_tracked; + Serialized m_tracked; }; } // namespace core diff --git a/src/core/live_trackers/cpp/live_message_channel_tracker_impl.cpp b/src/core/live_trackers/cpp/live_message_channel_tracker_impl.cpp index df01926c0..54d81fff9 100644 --- a/src/core/live_trackers/cpp/live_message_channel_tracker_impl.cpp +++ b/src/core/live_trackers/cpp/live_message_channel_tracker_impl.cpp @@ -60,7 +60,7 @@ void LiveMessageChannelTrackerImpl::update(int64_t monotonic_time_ns) last_update_time_ = monotonic_time_ns; const XrTime xr_time = time_converter_.convert_monotonic_ns_to_xrtime(monotonic_time_ns); - messages_.data.clear(); + native_.data.clear(); const MessageChannelStatus status = query_status(); if (status == MessageChannelStatus::DISCONNECTED) @@ -76,6 +76,11 @@ void LiveMessageChannelTrackerImpl::update(int64_t monotonic_time_ns) // are drained but the sentinel write below still advances the // replay frame clock. + // Always encode, including for an empty drain: unlike the single-payload trackers, + // `data` here is a list, and "no messages this frame" is an empty batch rather than + // an absent one. + messages_ = pack(native_); + if (mcap_channels_) { // The message channel is the replay impl's own frame clock: @@ -87,13 +92,13 @@ void LiveMessageChannelTrackerImpl::update(int64_t monotonic_time_ns) // the replay from the per-frame trackers (head / hand / ...) // by the duration of the gap. DeviceDataTimestamp timestamp(last_update_time_, last_update_time_, xr_time); - if (messages_.data.empty()) + if (native_.data.empty()) { mcap_channels_->write(0, timestamp, nullptr); } else { - for (const auto& msg : messages_.data) + for (const auto& msg : native_.data) { mcap_channels_->write(0, timestamp, msg); } @@ -165,7 +170,7 @@ void LiveMessageChannelTrackerImpl::drain_messages() auto message = std::make_shared(); message->payload.assign(receive_buffer_.begin(), receive_buffer_.begin() + read_count); - messages_.data.push_back(message); + native_.data.push_back(message); } } @@ -174,7 +179,7 @@ MessageChannelStatus LiveMessageChannelTrackerImpl::get_status() const return query_status(); } -const MessageChannelMessagesTrackedT& LiveMessageChannelTrackerImpl::get_messages() const +const Serialized& LiveMessageChannelTrackerImpl::get_messages() const { return messages_; } diff --git a/src/core/live_trackers/cpp/live_message_channel_tracker_impl.hpp b/src/core/live_trackers/cpp/live_message_channel_tracker_impl.hpp index a9bc5259f..e7927fc8c 100644 --- a/src/core/live_trackers/cpp/live_message_channel_tracker_impl.hpp +++ b/src/core/live_trackers/cpp/live_message_channel_tracker_impl.hpp @@ -44,7 +44,7 @@ class LiveMessageChannelTrackerImpl : public IMessageChannelTrackerImpl void update(int64_t monotonic_time_ns) override; MessageChannelStatus get_status() const override; - const MessageChannelMessagesTrackedT& get_messages() const override; + const Serialized& get_messages() const override; void send_message(const std::vector& payload) const override; private: @@ -73,7 +73,10 @@ class LiveMessageChannelTrackerImpl : public IMessageChannelTrackerImpl XrTimeConverter time_converter_; int64_t last_update_time_ = 0; - MessageChannelMessagesTrackedT messages_; + // Assembly scratch for the drained batch, and the encoded snapshot published + // from it each frame. Only the latter leaves this class. + MessageChannelMessagesTrackedT native_; + Serialized messages_; std::vector receive_buffer_; std::unique_ptr mcap_channels_; }; diff --git a/src/core/live_trackers/cpp/live_oglo_tactile_tracker_impl.cpp b/src/core/live_trackers/cpp/live_oglo_tactile_tracker_impl.cpp index eb7fa2def..076a0bc60 100644 --- a/src/core/live_trackers/cpp/live_oglo_tactile_tracker_impl.cpp +++ b/src/core/live_trackers/cpp/live_oglo_tactile_tracker_impl.cpp @@ -52,10 +52,10 @@ void LiveOgloTactileTrackerImpl::update(int64_t /*monotonic_time_ns*/) { // SchemaTracker throws on critical OpenXR/tensor failures; missing collection // and "no new data" are non-fatal. - m_schema_reader.update(m_tracked.data); + m_schema_reader.update(m_tracked); } -const OgloGloveSampleTrackedT& LiveOgloTactileTrackerImpl::get_data() const +const Serialized& LiveOgloTactileTrackerImpl::get_data() const { return m_tracked; } diff --git a/src/core/live_trackers/cpp/live_oglo_tactile_tracker_impl.hpp b/src/core/live_trackers/cpp/live_oglo_tactile_tracker_impl.hpp index 57e3362c9..492f77e38 100644 --- a/src/core/live_trackers/cpp/live_oglo_tactile_tracker_impl.hpp +++ b/src/core/live_trackers/cpp/live_oglo_tactile_tracker_impl.hpp @@ -40,12 +40,12 @@ class LiveOgloTactileTrackerImpl : public IOgloTactileTrackerImpl LiveOgloTactileTrackerImpl& operator=(LiveOgloTactileTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const OgloGloveSampleTrackedT& get_data() const override; + const Serialized& get_data() const override; private: std::unique_ptr mcap_channels_; OgloSchemaTracker m_schema_reader; - OgloGloveSampleTrackedT m_tracked; + Serialized m_tracked; }; } // namespace core diff --git a/src/core/live_trackers/cpp/live_se3_tracker_impl.cpp b/src/core/live_trackers/cpp/live_se3_tracker_impl.cpp index fd211743a..d81411086 100644 --- a/src/core/live_trackers/cpp/live_se3_tracker_impl.cpp +++ b/src/core/live_trackers/cpp/live_se3_tracker_impl.cpp @@ -56,10 +56,10 @@ void LiveSe3TrackerImpl::update(int64_t /*monotonic_time_ns*/) { // Policy: SchemaTracker throws on critical OpenXR/tensor API failures. // Missing collection/no new data are treated as common non-fatal cases. - m_schema_reader.update(m_tracked.data); + m_schema_reader.update(m_tracked); } -const Se3TrackerPoseTrackedT& LiveSe3TrackerImpl::get_data() const +const Serialized& LiveSe3TrackerImpl::get_data() const { return m_tracked; } diff --git a/src/core/live_trackers/cpp/live_se3_tracker_impl.hpp b/src/core/live_trackers/cpp/live_se3_tracker_impl.hpp index 2286d0e5a..deee945a9 100644 --- a/src/core/live_trackers/cpp/live_se3_tracker_impl.hpp +++ b/src/core/live_trackers/cpp/live_se3_tracker_impl.hpp @@ -41,12 +41,12 @@ class LiveSe3TrackerImpl : public ISe3TrackerImpl LiveSe3TrackerImpl& operator=(LiveSe3TrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const Se3TrackerPoseTrackedT& get_data() const override; + const Serialized& get_data() const override; private: std::unique_ptr mcap_channels_; Se3TrackerSchemaTracker m_schema_reader; - Se3TrackerPoseTrackedT m_tracked; + Serialized m_tracked; }; } // namespace core diff --git a/src/core/mcap/cpp/inc/mcap/tracker_channels.hpp b/src/core/mcap/cpp/inc/mcap/tracker_channels.hpp index 49e10b590..80f332748 100644 --- a/src/core/mcap/cpp/inc/mcap/tracker_channels.hpp +++ b/src/core/mcap/cpp/inc/mcap/tracker_channels.hpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -165,12 +166,21 @@ class McapTrackerViewers } /** - * @brief Read and deserialize the next record. + * @brief Read the next record as an encoded handle. * @param channel_index Index into the sub_channels list passed at construction. - * @return The deserialized Record (data member is null when the tracker - * was inactive), or std::nullopt when no more messages remain. + * @return A handle owning the recorded bytes, empty when no more messages remain. + * + * The two levels of absence are the handle and its payload, not a wrapper around + * either: an empty handle means the stream had nothing left, while a non-empty handle + * whose `payload()` is null means a record was read for a tracker that was inactive. + * A record that was read always yields a non-empty handle, so there is nothing for an + * optional to say that the handle does not. + * + * The recorded root is a Record whose `data` is byte-for-byte the payload table + * consumers read, so a caller narrows to it rather than unpacking: `record.narrow(...)` + * shares this buffer instead of allocating a second one. */ - std::optional read(size_t channel_index) + Serialized read_serialized(size_t channel_index) { if (channel_index >= channels_.size()) { @@ -191,7 +201,7 @@ class McapTrackerViewers { const auto& msg_view = *(tracker_view_->it); size_t idx = find_channel_idx(msg_view.channel->topic); - NativeRecordT record = deserialize(msg_view.message, idx); + Serialized record = adopt_message(msg_view.message, idx); ++(tracker_view_->it); @@ -204,14 +214,38 @@ class McapTrackerViewers channels_[idx].buffer.push_back(std::move(record)); } - return std::nullopt; + return Serialized(); + } + + /** + * @brief Read the next record as an object-API value. + * @param channel_index Index into the sub_channels list passed at construction. + * @return The deserialized Record (data member is null when the tracker + * was inactive), or std::nullopt when no more messages remain. + * + * For callers that compose a new table out of what they read and so need owning, + * mutable members; a `-T` is not nullable, so this one does need the optional. + * Prefer read_serialized() when the recorded payload is what gets published: this + * unpacks it. + */ + std::optional read(size_t channel_index) + { + const Serialized record = read_serialized(channel_index); + if (!record) + { + return std::nullopt; + } + + NativeRecordT native; + record->UnPackTo(&native); + return native; } private: struct ChannelBuffer { std::string topic; - std::deque buffer; + std::deque> buffer; }; struct TrackerView @@ -236,7 +270,10 @@ class McapTrackerViewers throw std::runtime_error("McapTrackerViewers: unexpected topic '" + topic + "'"); } - NativeRecordT deserialize(const mcap::Message& msg, size_t channel_index) const + // Verifies the recorded bytes and takes ownership of a copy of them. The copy is + // needed either way: the iterator owns the message storage and reuses it on the next + // advance, so the bytes cannot outlive this call by reference. + Serialized adopt_message(const mcap::Message& msg, size_t channel_index) const { flatbuffers::Verifier verifier(reinterpret_cast(msg.data), msg.dataSize); if (!verifier.VerifyBuffer()) @@ -245,10 +282,8 @@ class McapTrackerViewers std::to_string(channel_index) + " at sequence " + std::to_string(msg.sequence)); } - auto* fb_record = flatbuffers::GetRoot(msg.data); - NativeRecordT record; - fb_record->UnPackTo(&record); - return record; + const auto* bytes = reinterpret_cast(msg.data); + return Serialized::adopt(std::vector(bytes, bytes + msg.dataSize)); } std::unique_ptr reader_; diff --git a/src/core/oxr_utils/cpp/inc/oxr_utils/pose_conversions.hpp b/src/core/oxr_utils/cpp/inc/oxr_utils/pose_conversions.hpp index 3710336d2..c3eddbc4f 100644 --- a/src/core/oxr_utils/cpp/inc/oxr_utils/pose_conversions.hpp +++ b/src/core/oxr_utils/cpp/inc/oxr_utils/pose_conversions.hpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #pragma once @@ -31,18 +31,18 @@ inline XrPosef to_xr_posef(const core::ControllerPose& controller_pose, bool& ou return to_xr_posef(controller_pose.pose()); } -// Convert core::ControllerSnapshotT to get aim pose as XrPosef -inline XrPosef get_aim_pose(const core::ControllerSnapshotT& snapshot, bool& out_valid) +// Convert core::ControllerSnapshot to get aim pose as XrPosef +inline XrPosef get_aim_pose(const core::ControllerSnapshot& snapshot, bool& out_valid) { - out_valid = snapshot.aim_pose->is_valid(); - return to_xr_posef(snapshot.aim_pose->pose()); + out_valid = snapshot.aim_pose()->is_valid(); + return to_xr_posef(snapshot.aim_pose()->pose()); } -// Convert core::ControllerSnapshotT to get grip pose as XrPosef -inline XrPosef get_grip_pose(const core::ControllerSnapshotT& snapshot, bool& out_valid) +// Convert core::ControllerSnapshot to get grip pose as XrPosef +inline XrPosef get_grip_pose(const core::ControllerSnapshot& snapshot, bool& out_valid) { - out_valid = snapshot.grip_pose->is_valid(); - return to_xr_posef(snapshot.grip_pose->pose()); + out_valid = snapshot.grip_pose()->is_valid(); + return to_xr_posef(snapshot.grip_pose()->pose()); } } // namespace oxr_utils diff --git a/src/core/replay_deviceio_session_tests/cpp/test_replay_session.cpp b/src/core/replay_deviceio_session_tests/cpp/test_replay_session.cpp index c43237d5e..c5da24289 100644 --- a/src/core/replay_deviceio_session_tests/cpp/test_replay_session.cpp +++ b/src/core/replay_deviceio_session_tests/cpp/test_replay_session.cpp @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -139,9 +140,17 @@ std::vector to_string_vec(auto traits_channels) return std::vector(traits_channels.begin(), traits_channels.end()); } -std::string payload_string(const std::shared_ptr& msg) +// FlatBuffers omits an empty vector rather than encoding a zero-length one, so an +// absent `data` field is how a drained-nothing frame arrives. +size_t message_count(const core::Serialized& msgs) { - return std::string(msg->payload.begin(), msg->payload.end()); + const auto* messages = payload(msgs); + return messages != nullptr ? messages->size() : 0; +} + +std::string payload_string(const core::MessageChannelMessages* msg) +{ + return std::string(msg->payload()->begin(), msg->payload()->end()); } std::array make_test_uuid() @@ -186,15 +195,15 @@ TEST_CASE("ReplaySession: head tracker round-trip with multiple frames", "[repla { session->update(); const auto& head = head_tracker.get_head(*session); - REQUIRE(head.data); + REQUIRE(head); float v = static_cast(i + 1); - CHECK(head.data->pose->position().x() == v); - CHECK(head.data->pose->position().y() == v * 10.0f); - CHECK(head.data->pose->position().z() == v * 100.0f); + CHECK(head->pose()->position().x() == v); + CHECK(head->pose()->position().y() == v * 10.0f); + CHECK(head->pose()->position().z() == v * 100.0f); } session->update(); - CHECK_FALSE(head_tracker.get_head(*session).data); + CHECK_FALSE(head_tracker.get_head(*session)); } // ============================================================================= @@ -240,19 +249,19 @@ TEST_CASE("ReplaySession: se3 tracker round-trip and null at EOF", "[replay][ses { session->update(); const auto& tracked = tracker.get_data(*session); - REQUIRE(tracked.data); - CHECK(tracked.data->is_valid); + REQUIRE(tracked); + CHECK(tracked->is_valid()); float v = static_cast(i + 1); - CHECK(tracked.data->pose->position().x() == v); - CHECK(tracked.data->pose->position().y() == v * 10.0f); - CHECK(tracked.data->pose->position().z() == v * 100.0f); + CHECK(tracked->pose()->position().x() == v); + CHECK(tracked->pose()->position().y() == v * 10.0f); + CHECK(tracked->pose()->position().z() == v * 100.0f); } // Replay nulls data at gap/EOF — intentionally different from the live impl's // stale-sample retention on sample-less ticks. See the "Record/replay fidelity" // paragraph in docs/se3-tracker-design.md before "fixing" this toward sample-and-hold. session->update(); - CHECK_FALSE(tracker.get_data(*session).data); + CHECK_FALSE(tracker.get_data(*session)); } // ============================================================================= @@ -293,13 +302,13 @@ TEST_CASE("ReplaySession: hand tracker round-trip with left and right", "[replay session->update(); const auto& left = hand_tracker.get_left_hand(*session); const auto& right = hand_tracker.get_right_hand(*session); - CHECK(left.data != nullptr); - CHECK(right.data != nullptr); + CHECK(left); + CHECK(right); } session->update(); - CHECK_FALSE(hand_tracker.get_left_hand(*session).data); - CHECK_FALSE(hand_tracker.get_right_hand(*session).data); + CHECK_FALSE(hand_tracker.get_left_hand(*session)); + CHECK_FALSE(hand_tracker.get_right_hand(*session)); } // ============================================================================= @@ -354,19 +363,19 @@ TEST_CASE("ReplaySession: head and hand trackers in one session", "[replay][sess float v = static_cast(i + 1); const auto& head = head_tracker.get_head(*session); - REQUIRE(head.data); - CHECK(head.data->pose->position().x() == v); - CHECK(head.data->pose->position().y() == v * 2.0f); - CHECK(head.data->pose->position().z() == v * 3.0f); + REQUIRE(head); + CHECK(head->pose()->position().x() == v); + CHECK(head->pose()->position().y() == v * 2.0f); + CHECK(head->pose()->position().z() == v * 3.0f); - CHECK(hand_tracker.get_left_hand(*session).data != nullptr); - CHECK(hand_tracker.get_right_hand(*session).data != nullptr); + CHECK(hand_tracker.get_left_hand(*session)); + CHECK(hand_tracker.get_right_hand(*session)); } session->update(); - CHECK_FALSE(head_tracker.get_head(*session).data); - CHECK_FALSE(hand_tracker.get_left_hand(*session).data); - CHECK_FALSE(hand_tracker.get_right_hand(*session).data); + CHECK_FALSE(head_tracker.get_head(*session)); + CHECK_FALSE(hand_tracker.get_left_hand(*session)); + CHECK_FALSE(hand_tracker.get_right_hand(*session)); } // ============================================================================= @@ -417,15 +426,15 @@ TEST_CASE("ReplaySession: message channel drains records on their recorded frame session->update(); { const auto& msgs = ctrl_tracker.get_messages(*session); - REQUIRE(msgs.data.size() == 3); - CHECK(payload_string(msgs.data[0]) == "start"); - CHECK(payload_string(msgs.data[1]) == "stop"); - CHECK(payload_string(msgs.data[2]) == "reset"); + REQUIRE(message_count(msgs) == 3); + CHECK(payload_string(payload(msgs)->Get(0)) == "start"); + CHECK(payload_string(payload(msgs)->Get(1)) == "stop"); + CHECK(payload_string(payload(msgs)->Get(2)) == "reset"); } // EOF: subsequent updates produce empty batches (no double-emission). session->update(); - CHECK(ctrl_tracker.get_messages(*session).data.empty()); + CHECK(message_count(ctrl_tracker.get_messages(*session)) == 0); } TEST_CASE("ReplaySession: message channel fans recorded events across update ticks", "[replay][session][message_channel]") @@ -462,26 +471,26 @@ TEST_CASE("ReplaySession: message channel fans recorded events across update tic session->update(); { const auto& msgs = ctrl_tracker.get_messages(*session); - REQUIRE(msgs.data.size() == 1); - CHECK(payload_string(msgs.data[0]) == "start"); + REQUIRE(message_count(msgs) == 1); + CHECK(payload_string(payload(msgs)->Get(0)) == "start"); } session->update(); { const auto& msgs = ctrl_tracker.get_messages(*session); - REQUIRE(msgs.data.size() == 1); - CHECK(payload_string(msgs.data[0]) == "stop"); + REQUIRE(message_count(msgs) == 1); + CHECK(payload_string(payload(msgs)->Get(0)) == "stop"); } session->update(); { const auto& msgs = ctrl_tracker.get_messages(*session); - REQUIRE(msgs.data.size() == 1); - CHECK(payload_string(msgs.data[0]) == "reset"); + REQUIRE(message_count(msgs) == 1); + CHECK(payload_string(payload(msgs)->Get(0)) == "reset"); } session->update(); - CHECK(ctrl_tracker.get_messages(*session).data.empty()); + CHECK(message_count(ctrl_tracker.get_messages(*session)) == 0); } TEST_CASE("ReplaySession: message channel emits at recorded frame regardless of replay-loop speed", @@ -547,17 +556,17 @@ TEST_CASE("ReplaySession: message channel emits at recorded frame regardless of const auto& msgs = ctrl_tracker.get_messages(*session); if (frame == kStartFrame) { - REQUIRE(msgs.data.size() == 1); - CHECK(payload_string(msgs.data[0]) == "start"); + REQUIRE(message_count(msgs) == 1); + CHECK(payload_string(payload(msgs)->Get(0)) == "start"); } else if (frame == kStopFrame) { - REQUIRE(msgs.data.size() == 1); - CHECK(payload_string(msgs.data[0]) == "stop"); + REQUIRE(message_count(msgs) == 1); + CHECK(payload_string(payload(msgs)->Get(0)) == "stop"); } else { - CHECK(msgs.data.empty()); + CHECK(message_count(msgs) == 0); } } } @@ -600,21 +609,21 @@ TEST_CASE("ReplaySession: message channel drains payloads alongside sentinels in REQUIRE(session != nullptr); session->update(); - CHECK(ctrl_tracker.get_messages(*session).data.empty()); + CHECK(message_count(ctrl_tracker.get_messages(*session)) == 0); session->update(); { const auto& msgs = ctrl_tracker.get_messages(*session); - REQUIRE(msgs.data.size() == 2); - CHECK(payload_string(msgs.data[0]) == "hello"); - CHECK(payload_string(msgs.data[1]) == "world"); + REQUIRE(message_count(msgs) == 2); + CHECK(payload_string(payload(msgs)->Get(0)) == "hello"); + CHECK(payload_string(payload(msgs)->Get(1)) == "world"); } session->update(); - CHECK(ctrl_tracker.get_messages(*session).data.empty()); + CHECK(message_count(ctrl_tracker.get_messages(*session)) == 0); session->update(); - CHECK(ctrl_tracker.get_messages(*session).data.empty()); + CHECK(message_count(ctrl_tracker.get_messages(*session)) == 0); } // ============================================================================= diff --git a/src/core/replay_trackers/cpp/replay_controller_tracker_impl.cpp b/src/core/replay_trackers/cpp/replay_controller_tracker_impl.cpp index eb209c604..2a96d6ec5 100644 --- a/src/core/replay_trackers/cpp/replay_controller_tracker_impl.cpp +++ b/src/core/replay_trackers/cpp/replay_controller_tracker_impl.cpp @@ -5,7 +5,9 @@ #include #include +#include #include +#include #include #include @@ -28,38 +30,38 @@ ReplayControllerTrackerImpl::ReplayControllerTrackerImpl(std::unique_ptr& ReplayControllerTrackerImpl::get_left_controller() const { return left_tracked_; } -const ControllerSnapshotTrackedT& ReplayControllerTrackerImpl::get_right_controller() const +const Serialized& ReplayControllerTrackerImpl::get_right_controller() const { return right_tracked_; } void ReplayControllerTrackerImpl::update(int64_t /*monotonic_time_ns*/) { - auto left_record = mcap_viewers_->read(0); - auto right_record = mcap_viewers_->read(1); + auto left_record = mcap_viewers_->read_serialized(0); + auto right_record = mcap_viewers_->read_serialized(1); if (left_record) { - left_tracked_.data = std::move(left_record->data); + left_tracked_ = left_record.narrow(payload(left_record)); } else { std::cerr << "ReplayControllerTrackerImpl: left controller data not found" << std::endl; - left_tracked_.data.reset(); + left_tracked_.reset(); } if (right_record) { - right_tracked_.data = std::move(right_record->data); + right_tracked_ = right_record.narrow(payload(right_record)); } else { std::cerr << "ReplayControllerTrackerImpl: right controller data not found" << std::endl; - right_tracked_.data.reset(); + right_tracked_.reset(); } } diff --git a/src/core/replay_trackers/cpp/replay_controller_tracker_impl.hpp b/src/core/replay_trackers/cpp/replay_controller_tracker_impl.hpp index db8b9446d..bba630d69 100644 --- a/src/core/replay_trackers/cpp/replay_controller_tracker_impl.hpp +++ b/src/core/replay_trackers/cpp/replay_controller_tracker_impl.hpp @@ -27,8 +27,8 @@ class ReplayControllerTrackerImpl : public IControllerTrackerImpl ReplayControllerTrackerImpl& operator=(ReplayControllerTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const ControllerSnapshotTrackedT& get_left_controller() const override; - const ControllerSnapshotTrackedT& get_right_controller() const override; + const Serialized& get_left_controller() const override; + const Serialized& get_right_controller() const override; // Replay sessions do not drive hardware — haptic feedback is a no-op here. void apply_left_haptic_feedback(float /*amplitude*/, float /*frequency_hz*/, float /*duration_s*/) const override { @@ -38,8 +38,8 @@ class ReplayControllerTrackerImpl : public IControllerTrackerImpl } private: - ControllerSnapshotTrackedT left_tracked_; - ControllerSnapshotTrackedT right_tracked_; + Serialized left_tracked_; + Serialized right_tracked_; std::unique_ptr mcap_viewers_; }; diff --git a/src/core/replay_trackers/cpp/replay_full_body_tracker_impl.cpp b/src/core/replay_trackers/cpp/replay_full_body_tracker_impl.cpp index 5b3f7f86a..390cf4f0d 100644 --- a/src/core/replay_trackers/cpp/replay_full_body_tracker_impl.cpp +++ b/src/core/replay_trackers/cpp/replay_full_body_tracker_impl.cpp @@ -5,7 +5,9 @@ #include #include +#include #include +#include #include #include @@ -27,22 +29,22 @@ ReplayFullBodyTrackerImpl::ReplayFullBodyTrackerImpl(std::unique_ptr& ReplayFullBodyTrackerImpl::get_body_pose() const { return tracked_; } void ReplayFullBodyTrackerImpl::update(int64_t /*monotonic_time_ns*/) { - auto record = mcap_viewers_->read(0); + auto record = mcap_viewers_->read_serialized(0); if (record) { - tracked_.data = std::move(record->data); + tracked_ = record.narrow(payload(record)); } else { std::cerr << "ReplayFullBodyTrackerImpl: body data not found" << std::endl; - tracked_.data.reset(); + tracked_.reset(); } } diff --git a/src/core/replay_trackers/cpp/replay_full_body_tracker_impl.hpp b/src/core/replay_trackers/cpp/replay_full_body_tracker_impl.hpp index 38cdc8251..3c43936e1 100644 --- a/src/core/replay_trackers/cpp/replay_full_body_tracker_impl.hpp +++ b/src/core/replay_trackers/cpp/replay_full_body_tracker_impl.hpp @@ -29,10 +29,10 @@ class ReplayFullBodyTrackerImpl : public IFullBodyTrackerImpl ReplayFullBodyTrackerImpl& operator=(ReplayFullBodyTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const FullBodyPoseTrackedT& get_body_pose() const override; + const Serialized& get_body_pose() const override; private: - FullBodyPoseTrackedT tracked_; + Serialized tracked_; std::unique_ptr mcap_viewers_; }; diff --git a/src/core/replay_trackers/cpp/replay_generic_3axis_pedal_tracker_impl.cpp b/src/core/replay_trackers/cpp/replay_generic_3axis_pedal_tracker_impl.cpp index 9a6c99d9b..ca2165c88 100644 --- a/src/core/replay_trackers/cpp/replay_generic_3axis_pedal_tracker_impl.cpp +++ b/src/core/replay_trackers/cpp/replay_generic_3axis_pedal_tracker_impl.cpp @@ -5,7 +5,9 @@ #include #include +#include #include +#include #include #include @@ -28,22 +30,22 @@ ReplayGeneric3AxisPedalTrackerImpl::ReplayGeneric3AxisPedalTrackerImpl(std::uniq { } -const Generic3AxisPedalOutputTrackedT& ReplayGeneric3AxisPedalTrackerImpl::get_data() const +const Serialized& ReplayGeneric3AxisPedalTrackerImpl::get_data() const { return tracked_; } void ReplayGeneric3AxisPedalTrackerImpl::update(int64_t /*monotonic_time_ns*/) { - auto record = mcap_viewers_->read(0); + auto record = mcap_viewers_->read_serialized(0); if (record) { - tracked_.data = std::move(record->data); + tracked_ = record.narrow(payload(record)); } else { std::cerr << "ReplayGeneric3AxisPedalTrackerImpl: pedal data not found" << std::endl; - tracked_.data.reset(); + tracked_.reset(); } } diff --git a/src/core/replay_trackers/cpp/replay_generic_3axis_pedal_tracker_impl.hpp b/src/core/replay_trackers/cpp/replay_generic_3axis_pedal_tracker_impl.hpp index 8974cc234..0d7420e5e 100644 --- a/src/core/replay_trackers/cpp/replay_generic_3axis_pedal_tracker_impl.hpp +++ b/src/core/replay_trackers/cpp/replay_generic_3axis_pedal_tracker_impl.hpp @@ -27,10 +27,10 @@ class ReplayGeneric3AxisPedalTrackerImpl : public IGeneric3AxisPedalTrackerImpl ReplayGeneric3AxisPedalTrackerImpl& operator=(ReplayGeneric3AxisPedalTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const Generic3AxisPedalOutputTrackedT& get_data() const override; + const Serialized& get_data() const override; private: - Generic3AxisPedalOutputTrackedT tracked_; + Serialized tracked_; std::unique_ptr mcap_viewers_; }; diff --git a/src/core/replay_trackers/cpp/replay_hand_tracker_impl.cpp b/src/core/replay_trackers/cpp/replay_hand_tracker_impl.cpp index b66fef790..0642b6de9 100644 --- a/src/core/replay_trackers/cpp/replay_hand_tracker_impl.cpp +++ b/src/core/replay_trackers/cpp/replay_hand_tracker_impl.cpp @@ -5,7 +5,9 @@ #include #include +#include #include +#include #include #include @@ -27,38 +29,38 @@ ReplayHandTrackerImpl::ReplayHandTrackerImpl(std::unique_ptr r { } -const HandPoseTrackedT& ReplayHandTrackerImpl::get_left_hand() const +const Serialized& ReplayHandTrackerImpl::get_left_hand() const { return left_tracked_; } -const HandPoseTrackedT& ReplayHandTrackerImpl::get_right_hand() const +const Serialized& ReplayHandTrackerImpl::get_right_hand() const { return right_tracked_; } void ReplayHandTrackerImpl::update(int64_t /*monotonic_time_ns*/) { - auto left_record = mcap_viewers_->read(0); - auto right_record = mcap_viewers_->read(1); + auto left_record = mcap_viewers_->read_serialized(0); + auto right_record = mcap_viewers_->read_serialized(1); if (left_record) { - left_tracked_.data = std::move(left_record->data); + left_tracked_ = left_record.narrow(payload(left_record)); } else { std::cerr << "ReplayHandTrackerImpl: left hand data not found" << std::endl; - left_tracked_.data.reset(); + left_tracked_.reset(); } if (right_record) { - right_tracked_.data = std::move(right_record->data); + right_tracked_ = right_record.narrow(payload(right_record)); } else { std::cerr << "ReplayHandTrackerImpl: right hand data not found" << std::endl; - right_tracked_.data.reset(); + right_tracked_.reset(); } } diff --git a/src/core/replay_trackers/cpp/replay_hand_tracker_impl.hpp b/src/core/replay_trackers/cpp/replay_hand_tracker_impl.hpp index 55bb7dc8a..7a05b8c2e 100644 --- a/src/core/replay_trackers/cpp/replay_hand_tracker_impl.hpp +++ b/src/core/replay_trackers/cpp/replay_hand_tracker_impl.hpp @@ -27,12 +27,12 @@ class ReplayHandTrackerImpl : public IHandTrackerImpl ReplayHandTrackerImpl& operator=(ReplayHandTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const HandPoseTrackedT& get_left_hand() const override; - const HandPoseTrackedT& get_right_hand() const override; + const Serialized& get_left_hand() const override; + const Serialized& get_right_hand() const override; private: - HandPoseTrackedT left_tracked_; - HandPoseTrackedT right_tracked_; + Serialized left_tracked_; + Serialized right_tracked_; std::unique_ptr mcap_viewers_; }; diff --git a/src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.cpp b/src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.cpp index 028514026..923ae963b 100644 --- a/src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.cpp +++ b/src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.cpp @@ -10,12 +10,12 @@ void ReplayHapticCommandReaderTrackerImpl::update(int64_t /*monotonic_time_ns*/) { } -const HapticCommandTrackedT& ReplayHapticCommandReaderTrackerImpl::get_data() const +const Serialized& ReplayHapticCommandReaderTrackerImpl::get_data() const { return tracked_; } -const HapticCommandTrackedT& ReplayHapticCommandReaderTrackerImpl::get_data(std::string_view /*endpoint*/) const +const Serialized& ReplayHapticCommandReaderTrackerImpl::get_data(std::string_view /*endpoint*/) const { return tracked_; } diff --git a/src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.hpp b/src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.hpp index 5cafe671b..c84916f52 100644 --- a/src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.hpp +++ b/src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.hpp @@ -20,11 +20,11 @@ class ReplayHapticCommandReaderTrackerImpl : public IHapticCommandReaderTrackerI ReplayHapticCommandReaderTrackerImpl() = default; void update(int64_t monotonic_time_ns) override; - const HapticCommandTrackedT& get_data() const override; - const HapticCommandTrackedT& get_data(std::string_view endpoint) const override; + const Serialized& get_data() const override; + const Serialized& get_data(std::string_view endpoint) const override; private: - HapticCommandTrackedT tracked_; + Serialized tracked_; }; } // namespace core diff --git a/src/core/replay_trackers/cpp/replay_head_tracker_impl.cpp b/src/core/replay_trackers/cpp/replay_head_tracker_impl.cpp index af5eb3d87..0c7d208e1 100644 --- a/src/core/replay_trackers/cpp/replay_head_tracker_impl.cpp +++ b/src/core/replay_trackers/cpp/replay_head_tracker_impl.cpp @@ -5,7 +5,9 @@ #include #include +#include #include +#include #include #include @@ -27,22 +29,22 @@ ReplayHeadTrackerImpl::ReplayHeadTrackerImpl(std::unique_ptr r { } -const HeadPoseTrackedT& ReplayHeadTrackerImpl::get_head() const +const Serialized& ReplayHeadTrackerImpl::get_head() const { return tracked_; } void ReplayHeadTrackerImpl::update(int64_t /*monotonic_time_ns*/) { - auto record = mcap_viewers_->read(0); + auto record = mcap_viewers_->read_serialized(0); if (record) { - tracked_.data = std::move(record->data); + tracked_ = record.narrow(payload(record)); } else { std::cerr << "ReplayHeadTrackerImpl: head data not found" << std::endl; - tracked_.data.reset(); + tracked_.reset(); } } diff --git a/src/core/replay_trackers/cpp/replay_head_tracker_impl.hpp b/src/core/replay_trackers/cpp/replay_head_tracker_impl.hpp index b4d448d86..ad23af806 100644 --- a/src/core/replay_trackers/cpp/replay_head_tracker_impl.hpp +++ b/src/core/replay_trackers/cpp/replay_head_tracker_impl.hpp @@ -27,10 +27,10 @@ class ReplayHeadTrackerImpl : public IHeadTrackerImpl ReplayHeadTrackerImpl& operator=(ReplayHeadTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const HeadPoseTrackedT& get_head() const override; + const Serialized& get_head() const override; private: - HeadPoseTrackedT tracked_; + Serialized tracked_; std::unique_ptr mcap_viewers_; }; diff --git a/src/core/replay_trackers/cpp/replay_joint_state_tracker_impl.cpp b/src/core/replay_trackers/cpp/replay_joint_state_tracker_impl.cpp index 00567e051..7586c13a7 100644 --- a/src/core/replay_trackers/cpp/replay_joint_state_tracker_impl.cpp +++ b/src/core/replay_trackers/cpp/replay_joint_state_tracker_impl.cpp @@ -5,7 +5,9 @@ #include #include +#include #include +#include #include #include @@ -29,17 +31,17 @@ ReplayJointStateTrackerImpl::ReplayJointStateTrackerImpl(std::unique_ptr& ReplayJointStateTrackerImpl::get_data() const { return tracked_; } void ReplayJointStateTrackerImpl::update(int64_t /*monotonic_time_ns*/) { - auto record = mcap_viewers_->read(0); + auto record = mcap_viewers_->read_serialized(0); if (record) { - tracked_.data = std::move(record->data); + tracked_ = record.narrow(payload(record)); warned_no_data_ = false; } else @@ -50,7 +52,7 @@ void ReplayJointStateTrackerImpl::update(int64_t /*monotonic_time_ns*/) std::cerr << "ReplayJointStateTrackerImpl: joint state data not found" << std::endl; warned_no_data_ = true; } - tracked_.data.reset(); + tracked_.reset(); } } diff --git a/src/core/replay_trackers/cpp/replay_joint_state_tracker_impl.hpp b/src/core/replay_trackers/cpp/replay_joint_state_tracker_impl.hpp index 63dd9e174..4230f9ecc 100644 --- a/src/core/replay_trackers/cpp/replay_joint_state_tracker_impl.hpp +++ b/src/core/replay_trackers/cpp/replay_joint_state_tracker_impl.hpp @@ -27,10 +27,10 @@ class ReplayJointStateTrackerImpl : public IJointStateTrackerImpl ReplayJointStateTrackerImpl& operator=(ReplayJointStateTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const JointStateOutputTrackedT& get_data() const override; + const Serialized& get_data() const override; private: - JointStateOutputTrackedT tracked_; + Serialized tracked_; std::unique_ptr mcap_viewers_; // Warn only on the first frame of a no-data gap (EOF / sparse stream) to avoid per-frame spam. bool warned_no_data_ = false; diff --git a/src/core/replay_trackers/cpp/replay_message_channel_tracker_impl.cpp b/src/core/replay_trackers/cpp/replay_message_channel_tracker_impl.cpp index 34373c80f..bdd047fa3 100644 --- a/src/core/replay_trackers/cpp/replay_message_channel_tracker_impl.cpp +++ b/src/core/replay_trackers/cpp/replay_message_channel_tracker_impl.cpp @@ -45,29 +45,32 @@ void ReplayMessageChannelTrackerImpl::update(int64_t /*monotonic_time_ns*/) // sharing the first pending record's timestamp. See the class // docstring for the invariant this relies on (the live recorder // writes ≥1 record per session.update()). - messages_.data.clear(); + native_.data.clear(); if (!pending_record_) { pending_record_ = mcap_viewers_->read(0); } - if (!pending_record_) - { - return; - } - const int64_t frame_ns = record_monotonic_ns(*pending_record_); - while (pending_record_ && record_monotonic_ns(*pending_record_) == frame_ns) + if (pending_record_) { - // Sentinel records carry no data and only exist to mark a - // frame boundary; skip them but still advance the iterator so - // the next update reads the following frame. - if (pending_record_->data) + const int64_t frame_ns = record_monotonic_ns(*pending_record_); + while (pending_record_ && record_monotonic_ns(*pending_record_) == frame_ns) { - messages_.data.push_back(std::move(pending_record_->data)); + // Sentinel records carry no data and only exist to mark a + // frame boundary; skip them but still advance the iterator so + // the next update reads the following frame. + if (pending_record_->data) + { + native_.data.push_back(std::move(pending_record_->data)); + } + pending_record_ = mcap_viewers_->read(0); } - pending_record_ = mcap_viewers_->read(0); } + + // Always encode, including for an empty batch: `data` is a list here, so "no + // messages this frame" is an empty batch rather than an absent one. + messages_ = pack(native_); } MessageChannelStatus ReplayMessageChannelTrackerImpl::get_status() const @@ -78,7 +81,7 @@ MessageChannelStatus ReplayMessageChannelTrackerImpl::get_status() const return MessageChannelStatus::CONNECTED; } -const MessageChannelMessagesTrackedT& ReplayMessageChannelTrackerImpl::get_messages() const +const Serialized& ReplayMessageChannelTrackerImpl::get_messages() const { return messages_; } diff --git a/src/core/replay_trackers/cpp/replay_message_channel_tracker_impl.hpp b/src/core/replay_trackers/cpp/replay_message_channel_tracker_impl.hpp index dcc2984bb..9dc3443fd 100644 --- a/src/core/replay_trackers/cpp/replay_message_channel_tracker_impl.hpp +++ b/src/core/replay_trackers/cpp/replay_message_channel_tracker_impl.hpp @@ -57,13 +57,16 @@ class ReplayMessageChannelTrackerImpl : public IMessageChannelTrackerImpl void update(int64_t monotonic_time_ns) override; MessageChannelStatus get_status() const override; - const MessageChannelMessagesTrackedT& get_messages() const override; + const Serialized& get_messages() const override; void send_message(const std::vector& payload) const override; private: static int64_t record_monotonic_ns(const MessageChannelMessagesRecordT& record); - MessageChannelMessagesTrackedT messages_; + // Assembly scratch for the frame's batch, and the encoded snapshot published + // from it. Only the latter leaves this class. + MessageChannelMessagesTrackedT native_; + Serialized messages_; std::unique_ptr mcap_viewers_; // Holds the first record of the next frame, peeked but not yet // consumed. McapTrackerViewers::read() advances the underlying diff --git a/src/core/replay_trackers/cpp/replay_oglo_tactile_tracker_impl.cpp b/src/core/replay_trackers/cpp/replay_oglo_tactile_tracker_impl.cpp index 4310ba771..9edcb67fe 100644 --- a/src/core/replay_trackers/cpp/replay_oglo_tactile_tracker_impl.cpp +++ b/src/core/replay_trackers/cpp/replay_oglo_tactile_tracker_impl.cpp @@ -5,7 +5,9 @@ #include #include +#include #include +#include #include #include @@ -28,22 +30,22 @@ ReplayOgloTactileTrackerImpl::ReplayOgloTactileTrackerImpl(std::unique_ptr& ReplayOgloTactileTrackerImpl::get_data() const { return tracked_; } void ReplayOgloTactileTrackerImpl::update(int64_t /*monotonic_time_ns*/) { - auto record = mcap_viewers_->read(0); + auto record = mcap_viewers_->read_serialized(0); if (record) { - tracked_.data = std::move(record->data); + tracked_ = record.narrow(payload(record)); } else { std::cerr << "ReplayOgloTactileTrackerImpl: glove data not found" << std::endl; - tracked_.data.reset(); + tracked_.reset(); } } diff --git a/src/core/replay_trackers/cpp/replay_oglo_tactile_tracker_impl.hpp b/src/core/replay_trackers/cpp/replay_oglo_tactile_tracker_impl.hpp index 11a84a1b2..f76a265ed 100644 --- a/src/core/replay_trackers/cpp/replay_oglo_tactile_tracker_impl.hpp +++ b/src/core/replay_trackers/cpp/replay_oglo_tactile_tracker_impl.hpp @@ -27,10 +27,10 @@ class ReplayOgloTactileTrackerImpl : public IOgloTactileTrackerImpl ReplayOgloTactileTrackerImpl& operator=(ReplayOgloTactileTrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const OgloGloveSampleTrackedT& get_data() const override; + const Serialized& get_data() const override; private: - OgloGloveSampleTrackedT tracked_; + Serialized tracked_; std::unique_ptr mcap_viewers_; }; diff --git a/src/core/replay_trackers/cpp/replay_se3_tracker_impl.cpp b/src/core/replay_trackers/cpp/replay_se3_tracker_impl.cpp index a88faa707..d755fcda4 100644 --- a/src/core/replay_trackers/cpp/replay_se3_tracker_impl.cpp +++ b/src/core/replay_trackers/cpp/replay_se3_tracker_impl.cpp @@ -5,7 +5,9 @@ #include #include +#include #include +#include #include #include @@ -29,17 +31,17 @@ ReplaySe3TrackerImpl::ReplaySe3TrackerImpl(std::unique_ptr rea { } -const Se3TrackerPoseTrackedT& ReplaySe3TrackerImpl::get_data() const +const Serialized& ReplaySe3TrackerImpl::get_data() const { return tracked_; } void ReplaySe3TrackerImpl::update(int64_t /*monotonic_time_ns*/) { - auto record = mcap_viewers_->read(0); + auto record = mcap_viewers_->read_serialized(0); if (record) { - tracked_.data = std::move(record->data); + tracked_ = record.narrow(payload(record)); warned_no_data_ = false; } else @@ -50,7 +52,7 @@ void ReplaySe3TrackerImpl::update(int64_t /*monotonic_time_ns*/) std::cerr << no_data_message_ << std::endl; warned_no_data_ = true; } - tracked_.data.reset(); + tracked_.reset(); } } diff --git a/src/core/replay_trackers/cpp/replay_se3_tracker_impl.hpp b/src/core/replay_trackers/cpp/replay_se3_tracker_impl.hpp index a90a54bd5..1553af59f 100644 --- a/src/core/replay_trackers/cpp/replay_se3_tracker_impl.hpp +++ b/src/core/replay_trackers/cpp/replay_se3_tracker_impl.hpp @@ -28,10 +28,10 @@ class ReplaySe3TrackerImpl : public ISe3TrackerImpl ReplaySe3TrackerImpl& operator=(ReplaySe3TrackerImpl&&) = delete; void update(int64_t monotonic_time_ns) override; - const Se3TrackerPoseTrackedT& get_data() const override; + const Serialized& get_data() const override; private: - Se3TrackerPoseTrackedT tracked_; + Serialized tracked_; std::unique_ptr mcap_viewers_; // Pre-baked warn-once message including the base_name, so multi-tracker replays are // distinguishable in the log. diff --git a/src/core/retargeting_engine_tests/python/test_joint_state_source.py b/src/core/retargeting_engine_tests/python/test_joint_state_source.py index 69d15f819..aa7ac0085 100644 --- a/src/core/retargeting_engine_tests/python/test_joint_state_source.py +++ b/src/core/retargeting_engine_tests/python/test_joint_state_source.py @@ -13,7 +13,7 @@ from isaacteleop.retargeting_engine.deviceio_source_nodes import JointStateSource from isaacteleop.retargeting_engine.interface.base_retargeter import _make_output_group from isaacteleop.retargeting_engine.interface.tensor_group import TensorGroup -from isaacteleop.schema import JointState, JointStateOutput, JointStateOutputTrackedT +from isaacteleop.schema import JointState, JointStateOutput SO101_JOINTS = [ "shoulder_pan", @@ -41,10 +41,10 @@ def _outputs(source): def _make_output(joint_values: dict) -> JointStateOutput: - out = JointStateOutput() - out.device_id = "so101_leader" - out.joints = [JointState(name, pos) for name, pos in joint_values.items()] - return out + return JointStateOutput( + joints=[JointState(name, pos) for name, pos in joint_values.items()], + device_id="so101_leader", + ) class TestJointStateSource: @@ -75,7 +75,7 @@ def test_active_conversion(self): values = {n: round(0.1 * (i + 1), 3) for i, n in enumerate(SO101_JOINTS)} inputs = _make_inputs( src, - {"deviceio_joint_state": [JointStateOutputTrackedT(_make_output(values))]}, + {"deviceio_joint_state": [_make_output(values)]}, ) outputs = _outputs(src) src.compute(inputs, outputs) @@ -92,9 +92,7 @@ def test_name_order_independent(self): ) # Schema joints intentionally in reverse order. out = _make_output({"c": 3.0, "a": 1.0, "b": 2.0}) - inputs = _make_inputs( - src, {"deviceio_joint_state": [JointStateOutputTrackedT(out)]} - ) + inputs = _make_inputs(src, {"deviceio_joint_state": [out]}) outputs = _outputs(src) src.compute(inputs, outputs) group = outputs[JointStateSource.JOINTS] @@ -107,9 +105,7 @@ def test_missing_joint_defaults_zero(self): name="leader", collection_id="so101_leader", joint_names=["a", "missing"] ) out = _make_output({"a": 1.5}) - inputs = _make_inputs( - src, {"deviceio_joint_state": [JointStateOutputTrackedT(out)]} - ) + inputs = _make_inputs(src, {"deviceio_joint_state": [out]}) outputs = _outputs(src) src.compute(inputs, outputs) group = outputs[JointStateSource.JOINTS] @@ -120,10 +116,8 @@ def test_inactive_sets_none(self): src = JointStateSource( name="leader", collection_id="so101_leader", joint_names=SO101_JOINTS ) - # TrackedT with no data -> device inactive. - inputs = _make_inputs( - src, {"deviceio_joint_state": [JointStateOutputTrackedT()]} - ) + # Tracked wrapper with no data -> device inactive. + inputs = _make_inputs(src, {"deviceio_joint_state": [None]}) outputs = _outputs(src) src.compute(inputs, outputs) assert outputs[JointStateSource.JOINTS].is_none diff --git a/src/core/retargeting_engine_tests/python/test_message_channel_nodes.py b/src/core/retargeting_engine_tests/python/test_message_channel_nodes.py index e0863cf23..5268222ba 100644 --- a/src/core/retargeting_engine_tests/python/test_message_channel_nodes.py +++ b/src/core/retargeting_engine_tests/python/test_message_channel_nodes.py @@ -5,7 +5,7 @@ from isaacteleop.schema import ( MessageChannelMessages, - MessageChannelMessagesTrackedT, + MessageChannelMessagesTracked, ) from isaacteleop.retargeting_engine.deviceio_source_nodes import ( MessageChannelConnectionStatus, @@ -21,7 +21,7 @@ class DummyTracker: def __init__(self): self.sent_payloads = [] - self._drained = MessageChannelMessagesTrackedT() + self._drained = MessageChannelMessagesTracked() self.connected = True def send_message(self, session, payload): @@ -51,7 +51,7 @@ def test_message_channel_source_active_message(): tracker = DummyTracker() source = MessageChannelSource("msg_source", tracker, deque()) message = MessageChannelMessages(b"hello") - tracker._drained = MessageChannelMessagesTrackedT([message]) + tracker._drained = MessageChannelMessagesTracked([message]) inputs = source.poll_tracker(deviceio_session=object()) outputs = {k: _make_output_group(v) for k, v in source.output_spec().items()} @@ -66,7 +66,7 @@ def test_message_channel_source_active_message(): def test_message_channel_source_inactive_message(): tracker = DummyTracker() source = MessageChannelSource("msg_source", tracker, deque()) - tracker._drained = MessageChannelMessagesTrackedT() + tracker._drained = MessageChannelMessagesTracked() inputs = source.poll_tracker(deviceio_session=object()) outputs = {k: _make_output_group(v) for k, v in source.output_spec().items()} @@ -83,7 +83,7 @@ def test_message_channel_sink_enqueues_message(): m1 = MessageChannelMessages(b"echo") m2 = MessageChannelMessages(b"pong") - batch = MessageChannelMessagesTrackedT([m1, m2]) + batch = MessageChannelMessagesTracked([m1, m2]) inputs = _make_inputs(sink, {"messages_tracked": [batch]}) outputs = {k: _make_output_group(v) for k, v in sink.output_spec().items()} sink.compute(inputs, outputs) @@ -102,7 +102,7 @@ def test_message_channel_source_returns_all_drained_messages(): source = MessageChannelSource("msg_source_list", tracker, deque()) m1 = MessageChannelMessages(b"x") m2 = MessageChannelMessages(b"y") - tracker._drained = MessageChannelMessagesTrackedT([m1, m2]) + tracker._drained = MessageChannelMessagesTracked([m1, m2]) inputs = source.poll_tracker(deviceio_session=object()) outputs = {k: _make_output_group(v) for k, v in source.output_spec().items()} @@ -122,12 +122,8 @@ def test_message_channel_source_keeps_outbound_queue_while_disconnected(): outbound_queue = deque() source = MessageChannelSource("msg_source_disconnected", tracker, outbound_queue) - outbound_queue.append( - MessageChannelMessagesTrackedT([MessageChannelMessages(b"a")]) - ) - outbound_queue.append( - MessageChannelMessagesTrackedT([MessageChannelMessages(b"b")]) - ) + outbound_queue.append(MessageChannelMessagesTracked([MessageChannelMessages(b"a")])) + outbound_queue.append(MessageChannelMessagesTracked([MessageChannelMessages(b"b")])) inputs = source.poll_tracker(deviceio_session=object()) outputs = {k: _make_output_group(v) for k, v in source.output_spec().items()} @@ -156,9 +152,9 @@ def test_message_channel_sink_bounded_queue_drops_oldest(): m2 = MessageChannelMessages(b"2") m3 = MessageChannelMessages(b"3") - b1 = MessageChannelMessagesTrackedT([m1]) - b2 = MessageChannelMessagesTrackedT([m2]) - b3 = MessageChannelMessagesTrackedT([m3]) + b1 = MessageChannelMessagesTracked([m1]) + b2 = MessageChannelMessagesTracked([m2]) + b3 = MessageChannelMessagesTracked([m3]) outputs = {k: _make_output_group(v) for k, v in sink.output_spec().items()} sink.compute(_make_inputs(sink, {"messages_tracked": [b1]}), outputs) sink.compute(_make_inputs(sink, {"messages_tracked": [b2]}), outputs) diff --git a/src/core/retargeting_engine_tests/python/test_sources.py b/src/core/retargeting_engine_tests/python/test_sources.py index a3e1eaa96..539a36162 100644 --- a/src/core/retargeting_engine_tests/python/test_sources.py +++ b/src/core/retargeting_engine_tests/python/test_sources.py @@ -6,7 +6,7 @@ Tests the stateless converters that transform raw DeviceIO flatbuffer data into standard retargeting engine tensor formats, using real schema types -(TrackedT wrappers and table T types) constructed via Python bindings. +(Tracked wrappers and their payload tables) constructed via Python bindings. """ import pytest @@ -32,9 +32,7 @@ ControllerPose, ControllerInputState, ControllerSnapshot, - ControllerSnapshotTrackedT, - HeadPoseT, - HeadPoseTrackedT, + HeadPose, ) @@ -133,7 +131,7 @@ def test_controllers_source_compute(self): """Test that ControllersSource converts DeviceIO data correctly.""" source = ControllersSource(name="controllers") - # Create raw DeviceIO flatbuffer inputs wrapped in TrackedT types + # Create raw DeviceIO flatbuffer inputs wrapped in Tracked types left_snapshot = create_controller_snapshot( grip_pos=(0.1, 0.2, 0.3), aim_pos=(0.4, 0.5, 0.6), trigger_val=0.5 ) @@ -141,14 +139,12 @@ def test_controllers_source_compute(self): grip_pos=(0.4, 0.5, 0.6), aim_pos=(0.7, 0.8, 0.9), trigger_val=0.8 ) - # Prepare input dict with TrackedT wrappers (active controllers) + # Prepare input dict with Tracked wrappers (active controllers) inputs = _make_inputs( source, { - "deviceio_controller_left": [ControllerSnapshotTrackedT(left_snapshot)], - "deviceio_controller_right": [ - ControllerSnapshotTrackedT(right_snapshot) - ], + "deviceio_controller_left": [left_snapshot], + "deviceio_controller_right": [right_snapshot], }, ) @@ -246,12 +242,12 @@ def test_head_source_compute_active(self): """Test that HeadSource converts active tracked data correctly.""" source = HeadSource(name="head") - head_data = HeadPoseT( + head_data = HeadPose( Pose(Point(1.0, 2.0, 3.0), Quaternion(0.0, 0.0, 0.0, 1.0)), True, ) - inputs = _make_inputs(source, {"deviceio_head": [HeadPoseTrackedT(head_data)]}) + inputs = _make_inputs(source, {"deviceio_head": [head_data]}) outputs = { name: _make_output_group(gt) for name, gt in source.output_spec().items() } @@ -268,10 +264,10 @@ def test_head_source_compute_active(self): assert head[HeadInputIndex.IS_VALID] is True def test_head_source_compute_inactive(self): - """Test that inactive head (TrackedT.data is None) produces absent output.""" + """Test that inactive head (Tracked.data is None) produces absent output.""" source = HeadSource(name="head") - inputs = _make_inputs(source, {"deviceio_head": [HeadPoseTrackedT()]}) + inputs = _make_inputs(source, {"deviceio_head": [None]}) outputs = { name: _make_output_group(gt) for name, gt in source.output_spec().items() } @@ -297,7 +293,7 @@ def test_output_spec_is_optional(self): assert output_spec["controller_right"].is_optional def test_active_controller_produces_data(self): - """Active controllers (TrackedT.data is not None) produce non-absent OptionalTensorGroups.""" + """Active controllers (Tracked.data is not None) produce non-absent OptionalTensorGroups.""" source = ControllersSource(name="controllers") left_snapshot = create_controller_snapshot( @@ -310,10 +306,8 @@ def test_active_controller_produces_data(self): inputs = _make_inputs( source, { - "deviceio_controller_left": [ControllerSnapshotTrackedT(left_snapshot)], - "deviceio_controller_right": [ - ControllerSnapshotTrackedT(right_snapshot) - ], + "deviceio_controller_left": [left_snapshot], + "deviceio_controller_right": [right_snapshot], }, ) outputs = { @@ -331,7 +325,7 @@ def test_active_controller_produces_data(self): ] == pytest.approx(0.9) def test_inactive_controller_sets_none(self): - """Inactive controllers (TrackedT.data is None) produce absent OptionalTensorGroups.""" + """Inactive controllers (Tracked.data is None) produce absent OptionalTensorGroups.""" source = ControllersSource(name="controllers") right_snapshot = create_controller_snapshot( @@ -341,10 +335,8 @@ def test_inactive_controller_sets_none(self): inputs = _make_inputs( source, { - "deviceio_controller_left": [ControllerSnapshotTrackedT()], - "deviceio_controller_right": [ - ControllerSnapshotTrackedT(right_snapshot) - ], + "deviceio_controller_left": [None], + "deviceio_controller_right": [right_snapshot], }, ) outputs = { @@ -362,8 +354,8 @@ def test_absent_controller_raises_on_access(self): inputs = _make_inputs( source, { - "deviceio_controller_left": [ControllerSnapshotTrackedT()], - "deviceio_controller_right": [ControllerSnapshotTrackedT()], + "deviceio_controller_left": [None], + "deviceio_controller_right": [None], }, ) outputs = { @@ -431,15 +423,15 @@ def test_output_groups_are_optional_tensor_groups(self): assert isinstance(outputs["head"], OptionalTensorGroup) def test_active_head_produces_data(self): - """Active head (TrackedT.data is not None) produces non-absent OptionalTensorGroup.""" + """Active head (Tracked.data is not None) produces non-absent OptionalTensorGroup.""" source = HeadSource(name="head") - head_data = HeadPoseT( + head_data = HeadPose( Pose(Point(0.5, 1.5, 0.0), Quaternion(0.0, 0.707, 0.0, 0.707)), True, ) - inputs = _make_inputs(source, {"deviceio_head": [HeadPoseTrackedT(head_data)]}) + inputs = _make_inputs(source, {"deviceio_head": [head_data]}) outputs = { name: _make_output_group(gt) for name, gt in source.output_spec().items() } @@ -451,10 +443,10 @@ def test_active_head_produces_data(self): ) def test_inactive_head_sets_none(self): - """Inactive head (TrackedT.data is None) produces absent OptionalTensorGroup.""" + """Inactive head (Tracked.data is None) produces absent OptionalTensorGroup.""" source = HeadSource(name="head") - inputs = _make_inputs(source, {"deviceio_head": [HeadPoseTrackedT()]}) + inputs = _make_inputs(source, {"deviceio_head": [None]}) outputs = { name: _make_output_group(gt) for name, gt in source.output_spec().items() } @@ -466,7 +458,7 @@ def test_absent_head_raises_on_access(self): """Accessing fields of an absent head output raises ValueError.""" source = HeadSource(name="head") - inputs = _make_inputs(source, {"deviceio_head": [HeadPoseTrackedT()]}) + inputs = _make_inputs(source, {"deviceio_head": [None]}) outputs = { name: _make_output_group(gt) for name, gt in source.output_spec().items() } diff --git a/src/core/schema/cpp/inc/schema/full_body_compat.hpp b/src/core/schema/cpp/inc/schema/full_body_compat.hpp index 549426055..0f85f6ddb 100644 --- a/src/core/schema/cpp/inc/schema/full_body_compat.hpp +++ b/src/core/schema/cpp/inc/schema/full_body_compat.hpp @@ -21,8 +21,6 @@ namespace core // ---- Table / struct / record type aliases ------------------------------------- using FullBodyPosePico [[deprecated("renamed to core::FullBodyPose")]] = FullBodyPose; using FullBodyPosePicoT [[deprecated("renamed to core::FullBodyPoseT")]] = FullBodyPoseT; -using FullBodyPosePicoTracked [[deprecated("renamed to core::FullBodyPoseTracked")]] = FullBodyPoseTracked; -using FullBodyPosePicoTrackedT [[deprecated("renamed to core::FullBodyPoseTrackedT")]] = FullBodyPoseTrackedT; using FullBodyPosePicoRecord [[deprecated("renamed to core::FullBodyPoseRecord")]] = FullBodyPoseRecord; using FullBodyPosePicoRecordT [[deprecated("renamed to core::FullBodyPoseRecordT")]] = FullBodyPoseRecordT; using BodyJointsPico [[deprecated("renamed to core::BodyJoints")]] = BodyJoints; diff --git a/src/core/schema/cpp/inc/schema/serialized.hpp b/src/core/schema/cpp/inc/schema/serialized.hpp new file mode 100644 index 000000000..2f63baf22 --- /dev/null +++ b/src/core/schema/cpp/inc/schema/serialized.hpp @@ -0,0 +1,173 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Owning handle to any FlatBuffer table, used in place of the generated object-API +// (`-T`) types. +// +// A `-T` type is a tree of std::vector / std::string / std::unique_ptr members that +// only exists after an UnPack. Handing one out means a tracker either copies it per +// read or lends out storage it will refill next frame. `Serialized` instead points +// straight at the encoded bytes: readers address the buffer, so there is no unpack +// step, no per-field allocation, and no `-T` in any public signature. +// +// Two properties the rest of the tracker stack leans on: +// +// - Copying is a refcount bump, and nothing rewrites the bytes once encoded, so a copy +// taken this frame stays valid after the tracker moves on. Consumers hold snapshots, +// not views into live tracker storage. +// +// Immutability is a contract, not an enforcement: the Python bindings hand out +// writable NumPy views over the joint arrays, because NumPy cannot export a +// read-only array over DLPack before 2.1 (see schema_array_views.h). Writing through +// one changes what every holder of that buffer sees, so callers that intend to +// modify must copy first. +// - `ptr_` need not be the buffer root. Narrowing to a nested table shares the +// parent's owner and just re-points, so one allocation backs a whole tree of views. +// +// An empty handle (`get() == nullptr`, contextually false) means "no table here". +// +// Nullable on purpose, and for a format-level reason rather than a domain one: a +// FlatBuffers table field is optional, so the generated accessor already returns null +// when it is unset. A handle is that pointer plus its owner, so a non-nullable handle +// could represent less than the pointer it wraps -- `narrow()` would have to hand back +// an optional, which relocates the null rather than removing it, and costs the +// by-reference returns and default-constructibility the tracker impls rely on. +// +// This type is deliberately schema-agnostic: it knows how to own and re-point a buffer +// and nothing about any field. Helpers for this repo's Tracked/Record wrapper shape +// live in . + +#pragma once + +#include + +#include +#include +#include +#include + +namespace core +{ + +template +class Serialized +{ +public: + //! No table: no buffer, `get()` is null. See the note on nullability above. + Serialized() = default; + + /*! + * @brief Wraps `ptr` and keeps `owner` alive for as long as this handle (or any + * copy, or any handle narrowed from it) exists. + * + * `owner` is type-erased because the bytes can be backed by a builder's + * `DetachedBuffer`, a `std::vector` read off the wire, or the owner of a + * parent handle this one was narrowed from. Prefer `adopt()` / `narrow()` over + * calling this directly. + */ + Serialized(std::shared_ptr owner, const T* ptr) : owner_(std::move(owner)), ptr_(ptr) + { + } + + /*! + * @brief Takes ownership of a finished builder's buffer, rooted at `T`. + * + * The builder must have had `Finish()` called on an offset of type `T`; it is + * reset by the `Release()` and can be reused for the next frame. + */ + static Serialized adopt(flatbuffers::FlatBufferBuilder& fbb) + { + auto owner = std::make_shared(fbb.Release()); + return Serialized(owner, flatbuffers::GetRoot(owner->data())); + } + + /*! + * @brief Takes ownership of encoded bytes rooted at `T`, as read off the wire. + * + * The counterpart to the builder overload for the other owner kind named above: a + * buffer that arrived already encoded, so there is nothing to build and nothing to + * copy. `bytes` must hold a complete buffer whose root table is `T`. + */ + static Serialized adopt(std::vector&& bytes) + { + auto owner = std::make_shared>(std::move(bytes)); + return Serialized(owner, flatbuffers::GetRoot(owner->data())); + } + + //! Narrows to a table nested inside this buffer, sharing the owner. Null `ptr` + //! yields an empty handle, so `narrow(parent->child())` maps an unset nested-table + //! field onto an absent handle without a branch at the call site. + template + Serialized narrow(const U* ptr) const + { + return ptr != nullptr ? Serialized(owner_, ptr) : Serialized(); + } + + //! Encoded table, or null when this handle points at nothing. + const T* get() const noexcept + { + return ptr_; + } + + //! Precondition: the handle is non-empty. Test with `operator bool` (or reach the + //! field through a null-safe accessor) before dereferencing. + const T* operator->() const noexcept + { + assert(ptr_ != nullptr && "dereferenced an empty Serialized handle"); + return ptr_; + } + + //! Same precondition as `operator->`. + const T& operator*() const noexcept + { + assert(ptr_ != nullptr && "dereferenced an empty Serialized handle"); + return *ptr_; + } + + explicit operator bool() const noexcept + { + return ptr_ != nullptr; + } + + //! Drops the table and releases this handle's claim on the buffer. Spells "the payload + //! went away" without naming the table type, which the assignment form has to repeat. + void reset() noexcept + { + owner_.reset(); + ptr_ = nullptr; + } + +private: + std::shared_ptr owner_; + const T* ptr_ = nullptr; +}; + +/*! + * @brief Encodes a native (`-T`) value into a standalone `Serialized`. + * + * The bridge for producers that still assemble a `-T` — a tracker impl filling one + * from an OpenXR query, or a Python binding constructor taking loose arguments. The + * `-T` stays a local of the caller; only the encoded buffer escapes. + */ +template +Serialized pack(const typename T::NativeTableType& native) +{ + flatbuffers::FlatBufferBuilder fbb; + fbb.Finish(T::Pack(fbb, &native)); + return Serialized::adopt(fbb); +} + +/*! + * @brief Encodes `native` if it is present, otherwise yields an empty handle. + * + * The shape a producer of optional data needs: a device that went inactive, a sample + * that never arrived, a replay gap. Absence stays one state rather than becoming a + * present-but-empty buffer. + */ +template +Serialized pack_optional(const std::shared_ptr& native) +{ + return native ? pack(*native) : Serialized(); +} + +} // namespace core diff --git a/src/core/schema/cpp/inc/schema/tracked.hpp b/src/core/schema/cpp/inc/schema/tracked.hpp new file mode 100644 index 000000000..b41316ab3 --- /dev/null +++ b/src/core/schema/cpp/inc/schema/tracked.hpp @@ -0,0 +1,37 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Helper for the wrapper tables in `fbs/` that front a payload behind a `data` field: +// the `Record` family that MCAP writes, and `MessageChannelMessagesTracked`. +// +// Wrappers used to front the tracker query API too, expressing "no data" with a null +// `data`. `Serialized` says that with an empty handle, so those are gone and trackers +// hand out their payload table directly. The message-channel batch survives because its +// `data` is a *list*: a drained batch needs a table to hold the vector, and "no messages +// this frame" is an empty batch rather than an absent one. +// +// Including this header is the signal that a translation unit depends on that shape. + +#pragma once + +#include + +namespace core +{ + +/*! + * @brief The `data` field of a wrapper, or null when there is none. + * + * Collapses "is the handle non-empty" and "is its `data` set" into one test. + * + * @note FlatBuffers omits an empty vector rather than encoding a zero-length one, so a + * null return means an empty batch, not missing data. Callers treat the two the + * same; do not read it as an error. + */ +template +auto payload(const Serialized& wrapper) +{ + return wrapper ? wrapper->data() : nullptr; +} + +} // namespace core diff --git a/src/core/schema/fbs/controller.fbs b/src/core/schema/fbs/controller.fbs index 7ab9cfe4d..b59c7336a 100644 --- a/src/core/schema/fbs/controller.fbs +++ b/src/core/schema/fbs/controller.fbs @@ -31,7 +31,7 @@ struct ControllerPose { } // Snapshot data for a single controller. -// All fields are always present when the parent Tracked/Record wrapper's data is non-null. +// All fields are always present whenever this table itself is present. table ControllerSnapshot { // Grip pose - represents the physical controller position/orientation grip_pose: ControllerPose (id: 0); @@ -43,11 +43,6 @@ table ControllerSnapshot { inputs: ControllerInputState (id: 2); } -// Tracked wrapper for the in-memory tracker API (data is null when controller is inactive). -table ControllerSnapshotTracked { - data: ControllerSnapshot (id: 0); -} - // MCAP recording wrapper for ControllerSnapshot. // // Record types are the root types written to MCAP channels by the McapRecorder. diff --git a/src/core/schema/fbs/full_body.fbs b/src/core/schema/fbs/full_body.fbs index a8abcc5be..2ae0ab090 100644 --- a/src/core/schema/fbs/full_body.fbs +++ b/src/core/schema/fbs/full_body.fbs @@ -54,7 +54,7 @@ struct BodyJoints { } // Full body pose data (XR_BD_body_tracking joint layout, vendor-neutral). -// All fields are always present when the parent Tracked/Record wrapper's data is non-null. +// All fields are always present whenever this table itself is present. table FullBodyPose { // Vector of BodyJointPose. // For XR_BD_body_tracking, this is 24 joints. @@ -62,15 +62,10 @@ table FullBodyPose { // allJointPosesTracked from the OpenXR extension — quality flag only. // When false, individual joint is_valid flags should be consulted. - // Does not affect whether the data pointer is null; see the Tracked wrapper. + // Does not affect whether this table is present at all. all_joint_poses_tracked: bool (id: 1); } -// Tracked wrapper for the in-memory tracker API (data is null when body tracking is inactive). -table FullBodyPoseTracked { - data: FullBodyPose (id: 0); -} - // MCAP recording wrapper for FullBodyPose. // // Record types are the root types written to MCAP channels by the McapRecorder. diff --git a/src/core/schema/fbs/hand.fbs b/src/core/schema/fbs/hand.fbs index a0ce49a26..18908155e 100644 --- a/src/core/schema/fbs/hand.fbs +++ b/src/core/schema/fbs/hand.fbs @@ -63,18 +63,13 @@ struct HandJoints { } // Hand pose data. -// All fields are always present when the parent Tracked/Record wrapper's data is non-null. +// All fields are always present whenever this table itself is present. table HandPose { // Fixed-size hand joint poses in HandJoint / OpenXR order (struct HandJoints, not a variable-length vector). // The HandJoints.poses array length must stay in sync with HandJoint::NUM_JOINTS. joints: HandJoints (id: 0); } -// Tracked wrapper for the in-memory tracker API (data is null when hand is inactive). -table HandPoseTracked { - data: HandPose (id: 0); -} - // MCAP recording wrapper for HandPose. // // Record types are the root types written to MCAP channels by the McapRecorder. diff --git a/src/core/schema/fbs/haptic_command.fbs b/src/core/schema/fbs/haptic_command.fbs index 5aee4ecdf..b499107ad 100644 --- a/src/core/schema/fbs/haptic_command.fbs +++ b/src/core/schema/fbs/haptic_command.fbs @@ -17,12 +17,6 @@ table HapticCommand { values: [float] (id: 1); } -// Tracked wrapper for the in-memory reader API; data is null until the -// first sample arrives or after the producer collection disappears. -table HapticCommandTracked { - data: HapticCommand (id: 0); -} - // MCAP recording wrapper required by the SchemaTracker // template. Recording is disabled here -- the live reader passes // mcap_channels=nullptr. diff --git a/src/core/schema/fbs/head.fbs b/src/core/schema/fbs/head.fbs index e7894c355..a5c6a3c67 100644 --- a/src/core/schema/fbs/head.fbs +++ b/src/core/schema/fbs/head.fbs @@ -7,7 +7,7 @@ include "timestamp.fbs"; namespace core; // Head pose data. -// All fields are always present when the parent Tracked/Record wrapper's data is non-null. +// All fields are always present whenever this table itself is present. table HeadPose { // The concrete pose data pose: Pose (id: 0); @@ -16,11 +16,6 @@ table HeadPose { is_valid: bool (id: 1); } -// Tracked wrapper for the in-memory tracker API (data is null when head is inactive). -table HeadPoseTracked { - data: HeadPose (id: 0); -} - // MCAP recording wrapper for HeadPose. // // Record types are the root types written to MCAP channels by the McapRecorder. diff --git a/src/core/schema/fbs/joint_state.fbs b/src/core/schema/fbs/joint_state.fbs index 54570ac66..3768e84b9 100644 --- a/src/core/schema/fbs/joint_state.fbs +++ b/src/core/schema/fbs/joint_state.fbs @@ -30,8 +30,8 @@ table JointState { } // Per-frame state of a generic joint-space input device (leader arm, exoskeleton, glove, or -// any joint-encoder source), as name:value joint records. All fields are present when the -// parent Tracked/Record wrapper's data is non-null. +// any joint-encoder source), as name:value joint records. +// All fields are always present whenever this table itself is present. table JointStateOutput { // One entry per actuated DOF, keyed by JointState.name. The reference JointStateSource maps // these into the configured joint order BY NAME (so wire order does not matter). `name` is a @@ -55,11 +55,6 @@ table JointStateOutput { ee_pose_valid: bool (id: 5); } -// Tracked wrapper for the in-memory tracker API (data is null when the device is inactive). -table JointStateOutputTracked { - data: JointStateOutput (id: 0); -} - // MCAP recording wrapper for JointStateOutput. // // Record types are the root types written to MCAP channels by the McapRecorder. diff --git a/src/core/schema/fbs/message_channel.fbs b/src/core/schema/fbs/message_channel.fbs index 28f039d5e..c3e7ec758 100644 --- a/src/core/schema/fbs/message_channel.fbs +++ b/src/core/schema/fbs/message_channel.fbs @@ -10,7 +10,12 @@ table MessageChannelMessages { payload: [ubyte] (id: 0); } -// Tracked wrapper for drained batches (data is null when no messages were drained). +// A batch of messages drained during one update. +// +// The one table that still fronts a tracker's payload, because that payload is a *list*: +// something has to hold the vector. Trackers publish this unconditionally, so the handle +// is always non-empty; FlatBuffers omits an empty vector rather than encoding a +// zero-length one, so a null `data` means "no messages this frame", not missing data. table MessageChannelMessagesTracked { data: [MessageChannelMessages] (id: 0); } diff --git a/src/core/schema/fbs/oak.fbs b/src/core/schema/fbs/oak.fbs index d46ee1408..7dfd9d1bd 100644 --- a/src/core/schema/fbs/oak.fbs +++ b/src/core/schema/fbs/oak.fbs @@ -9,7 +9,7 @@ namespace core; enum StreamType : byte { Color = 0, MonoLeft = 1, MonoRight = 2 } // Per-frame metadata pushed by the OAK camera plugin (one per stream). -// All fields are always present when the parent Tracked/Record wrapper's data is non-null. +// All fields are always present whenever this table itself is present. // Each stream is recorded as a separate MCAP channel using this as the root type. table FrameMetadataOak { // Which camera sensor this metadata belongs to. @@ -19,11 +19,6 @@ table FrameMetadataOak { sequence_number: uint64 (id: 1); } -// Tracked wrapper for the in-memory tracker API (data is null when no metadata available). -table FrameMetadataOakTracked { - data: FrameMetadataOak (id: 0); -} - // MCAP recording wrapper for FrameMetadataOak. // // Record types are the root types written to MCAP channels by the McapRecorder. diff --git a/src/core/schema/fbs/oglo_tactile.fbs b/src/core/schema/fbs/oglo_tactile.fbs index ed23af52b..44ef38992 100644 --- a/src/core/schema/fbs/oglo_tactile.fbs +++ b/src/core/schema/fbs/oglo_tactile.fbs @@ -39,12 +39,6 @@ table OgloGloveSample { gyro_z: int16 (id: 8); } -// Tracked wrapper for the in-memory tracker API (data is null when no glove -// sample is available, e.g. the glove is disconnected). -table OgloGloveSampleTracked { - data: OgloGloveSample (id: 0); -} - // MCAP recording wrapper. Record types are the root types written to MCAP // channels; trackers serialize into Record types in serialize_all(). table OgloGloveSampleRecord { diff --git a/src/core/schema/fbs/pedals.fbs b/src/core/schema/fbs/pedals.fbs index 8f6a231c9..720495e60 100644 --- a/src/core/schema/fbs/pedals.fbs +++ b/src/core/schema/fbs/pedals.fbs @@ -7,7 +7,7 @@ include "timestamp.fbs"; namespace core; // Output from a generic 3-axis foot pedal device. -// All fields are always present when the parent Tracked/Record wrapper's data is non-null. +// All fields are always present whenever this table itself is present. // // Validated devices: // - https://www.logitechg.com/en-us/shop/p/flight-simulator-rudder-pedals @@ -19,11 +19,6 @@ table Generic3AxisPedalOutput { rudder: float (id: 2); } -// Tracked wrapper for the in-memory tracker API (data is null when no pedal data available). -table Generic3AxisPedalOutputTracked { - data: Generic3AxisPedalOutput (id: 0); -} - // MCAP recording wrapper for Generic3AxisPedalOutput. // // Record types are the root types written to MCAP channels by the McapRecorder. diff --git a/src/core/schema/fbs/se3_tracker.fbs b/src/core/schema/fbs/se3_tracker.fbs index 228f24cac..2aceacb95 100644 --- a/src/core/schema/fbs/se3_tracker.fbs +++ b/src/core/schema/fbs/se3_tracker.fbs @@ -17,13 +17,13 @@ namespace core; // XR-sourced producers MUST express the pose in the OpenXR session base reference // space (the XrSpace in OpenXRSessionHandles::space — the same space the head and // controller channels are located in). -// - Two-level validity: the parent Tracked wrapper's data is null when no sample has -// arrived yet or the source collection is unavailable; is_valid is false when the +// - Two-level validity: this table is absent when no sample has arrived yet or the +// source collection is unavailable; is_valid is false when the // producer is streaming but tracking is lost. When is_valid is false the pose // contents are UNSPECIFIED (producers may send an identity filler) — consumers must // gate on is_valid and freeze, never act on the pose value. // -// All fields are always present when the parent Tracked/Record wrapper's data is non-null. +// All fields are always present whenever this table itself is present. table Se3TrackerPose { // The concrete pose data pose: Pose (id: 0); @@ -32,11 +32,6 @@ table Se3TrackerPose { is_valid: bool (id: 1); } -// Tracked wrapper for the in-memory tracker API (data is null when no sample is available). -table Se3TrackerPoseTracked { - data: Se3TrackerPose (id: 0); -} - // MCAP recording wrapper for Se3TrackerPose. // // Record types are the root types written to MCAP channels by the McapRecorder. diff --git a/src/core/schema/python/CMakeLists.txt b/src/core/schema/python/CMakeLists.txt index 36512edff..5f01ee311 100644 --- a/src/core/schema/python/CMakeLists.txt +++ b/src/core/schema/python/CMakeLists.txt @@ -14,6 +14,7 @@ pybind11_add_module(schema_py pedals_bindings.h pose_bindings.h schema_array_views.h + schema_serialized.h se3_tracker_bindings.h schema_module.cpp ) diff --git a/src/core/schema/python/controller_bindings.h b/src/core/schema/python/controller_bindings.h index 627922fb3..520c0ccea 100644 --- a/src/core/schema/python/controller_bindings.h +++ b/src/core/schema/python/controller_bindings.h @@ -3,16 +3,20 @@ // Python bindings for the Controller FlatBuffer schema. // ControllerInputState, ControllerPose are structs. -// ControllerSnapshot is a table (exposed via ControllerSnapshotT native type). +// ControllerSnapshot is a table, exposed as an encoded view. #pragma once +#include "pose_bindings.h" +#include "schema_serialized.h" + #include #include #include #include #include +#include namespace py = pybind11; @@ -55,100 +59,51 @@ inline void bind_controller(py::module& m) .def("__repr__", [](const ControllerPose& self) { - std::string pose_str = "Pose(position=Point(x=" + std::to_string(self.pose().position().x()) + - ", y=" + std::to_string(self.pose().position().y()) + - ", z=" + std::to_string(self.pose().position().z()) + - "), orientation=Quaternion(x=" + std::to_string(self.pose().orientation().x()) + - ", y=" + std::to_string(self.pose().orientation().y()) + - ", z=" + std::to_string(self.pose().orientation().z()) + - ", w=" + std::to_string(self.pose().orientation().w()) + "))"; - - return "ControllerPose(pose=" + pose_str + ", is_valid=" + (self.is_valid() ? "True" : "False") + ")"; + return "ControllerPose(pose=" + pose_repr(self.pose()) + + ", is_valid=" + (self.is_valid() ? "True" : "False") + ")"; }); - // Bind ControllerSnapshot table (via ControllerSnapshotT native type) - py::class_>(m, "ControllerSnapshot") - .def(py::init( - []() - { - auto obj = std::make_shared(); - obj->grip_pose = std::make_shared(); - obj->aim_pose = std::make_shared(); - obj->inputs = std::make_shared(); - return obj; - })) + serialized_class( + m, "ControllerSnapshot", "Encoded controller snapshot: grip and aim poses plus the input state.") .def(py::init( [](const ControllerPose& grip_pose, const ControllerPose& aim_pose, const ControllerInputState& inputs) { - auto obj = std::make_shared(); - obj->grip_pose = std::make_shared(grip_pose); - obj->aim_pose = std::make_shared(aim_pose); - obj->inputs = std::make_shared(inputs); - return obj; + ControllerSnapshotT native; + native.grip_pose = std::make_shared(grip_pose); + native.aim_pose = std::make_shared(aim_pose); + native.inputs = std::make_shared(inputs); + return pack(native); }), - py::arg("grip_pose"), py::arg("aim_pose"), py::arg("inputs")) + py::arg("grip_pose") = ControllerPose(), py::arg("aim_pose") = ControllerPose(), + py::arg("inputs") = ControllerInputState(), + "Encode a controller snapshot. Omitted poses are all-zero and not valid.") .def_property_readonly( - "grip_pose", [](const ControllerSnapshotT& self) -> const ControllerPose* { return self.grip_pose.get(); }, + "grip_pose", [](const Serialized& self) { return self ? self->grip_pose() : nullptr; }, py::return_value_policy::reference_internal) .def_property_readonly( - "aim_pose", [](const ControllerSnapshotT& self) -> const ControllerPose* { return self.aim_pose.get(); }, + "aim_pose", [](const Serialized& self) { return self ? self->aim_pose() : nullptr; }, py::return_value_policy::reference_internal) .def_property_readonly( - "inputs", [](const ControllerSnapshotT& self) -> const ControllerInputState* { return self.inputs.get(); }, + "inputs", [](const Serialized& self) { return self ? self->inputs() : nullptr; }, py::return_value_policy::reference_internal) .def("__repr__", - [](const ControllerSnapshotT& self) + [](const Serialized& self) { - std::string grip_str = - self.grip_pose ? - "ControllerPose(is_valid=" + std::string(self.grip_pose->is_valid() ? "True" : "False") + ")" : - "None"; - std::string aim_str = - self.aim_pose ? - "ControllerPose(is_valid=" + std::string(self.aim_pose->is_valid() ? "True" : "False") + ")" : - "None"; - return "ControllerSnapshot(grip_pose=" + grip_str + ", aim_pose=" + aim_str + ")"; - }); - - py::class_>(m, "ControllerSnapshotRecord") - .def(py::init<>()) - .def(py::init( - [](const ControllerSnapshotT& data, const DeviceDataTimestamp& timestamp) + if (!self) { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - obj->timestamp = std::make_shared(timestamp); - return obj; - }), - py::arg("data"), py::arg("timestamp")) - .def_property_readonly("data", - [](const ControllerSnapshotRecordT& self) -> std::shared_ptr - { return self.data; }) - .def_readonly("timestamp", &ControllerSnapshotRecordT::timestamp) - .def("__repr__", - [](const ControllerSnapshotRecordT& self) { - return "ControllerSnapshotRecord(data=" + std::string(self.data ? "ControllerSnapshot(...)" : "None") + - ")"; - }); - - py::class_>(m, "ControllerSnapshotTrackedT") - .def(py::init<>()) - .def(py::init( - [](const ControllerSnapshotT& data) + return std::string("ControllerSnapshot()"); + } + auto pose_str = [](const ControllerPose* pose) { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - return obj; - }), - py::arg("data")) - .def_property_readonly("data", - [](const ControllerSnapshotTrackedT& self) -> std::shared_ptr - { return self.data; }) - .def("__repr__", - [](const ControllerSnapshotTrackedT& self) { - return std::string("ControllerSnapshotTrackedT(data=") + - (self.data ? "ControllerSnapshot(...)" : "None") + ")"; + return pose != nullptr ? + "ControllerPose(is_valid=" + std::string(pose->is_valid() ? "True" : "False") + ")" : + std::string("None"); + }; + return "ControllerSnapshot(grip_pose=" + pose_str(self->grip_pose()) + + ", aim_pose=" + pose_str(self->aim_pose()) + ")"; }); + + bind_record(m, "ControllerSnapshotRecord", "ControllerSnapshot"); } } // namespace core diff --git a/src/core/schema/python/full_body_bindings.h b/src/core/schema/python/full_body_bindings.h index 25e30997a..2e76ed45c 100644 --- a/src/core/schema/python/full_body_bindings.h +++ b/src/core/schema/python/full_body_bindings.h @@ -6,7 +6,9 @@ #pragma once +#include "pose_bindings.h" #include "schema_array_views.h" +#include "schema_serialized.h" #include #include @@ -18,6 +20,7 @@ #include #include #include +#include #include namespace py = pybind11; @@ -79,14 +82,8 @@ inline void bind_full_body(py::module& m) .def("__repr__", [](const BodyJointPose& self) { - return "BodyJointPose(pose=Pose(position=Point(x=" + std::to_string(self.pose().position().x()) + - ", y=" + std::to_string(self.pose().position().y()) + - ", z=" + std::to_string(self.pose().position().z()) + - "), orientation=Quaternion(x=" + std::to_string(self.pose().orientation().x()) + - ", y=" + std::to_string(self.pose().orientation().y()) + - ", z=" + std::to_string(self.pose().orientation().z()) + - ", w=" + std::to_string(self.pose().orientation().w()) + - ")), is_valid=" + (self.is_valid() ? "True" : "False") + ")"; + return "BodyJointPose(pose=" + pose_repr(self.pose()) + + ", is_valid=" + (self.is_valid() ? "True" : "False") + ")"; }); // Bind BodyJoints struct (fixed-size array of 24 BodyJointPose). @@ -136,68 +133,35 @@ inline void bind_full_body(py::module& m) "caveats.") .def("__repr__", [](const BodyJoints&) { return "BodyJoints(joints=[...24 BodyJointPose entries...])"; }); - // Bind FullBodyPoseT class (FlatBuffers object API for tables). - py::class_>(m, "FullBodyPoseT") + serialized_class(m, "FullBodyPose", "Encoded full body pose: 24 joints in BodyJoint order.") .def(py::init( - []() - { - auto obj = std::make_shared(); - obj->joints = std::make_shared(); - return obj; - })) - .def(py::init( - [](const BodyJoints& joints) + [](const BodyJoints& joints, bool all_joint_poses_tracked) { - auto obj = std::make_shared(); - obj->joints = std::make_shared(joints); - return obj; + FullBodyPoseT native; + native.joints = std::make_shared(joints); + native.all_joint_poses_tracked = all_joint_poses_tracked; + return pack(native); }), - py::arg("joints")) + py::arg("joints") = BodyJoints(), py::arg("all_joint_poses_tracked") = false, + "Encode a body pose. Defaults to all-zero joints.") .def_property_readonly( - "joints", [](const FullBodyPoseT& self) -> const BodyJoints* { return self.joints.get(); }, + "joints", [](const Serialized& self) { return self ? self->joints() : nullptr; }, py::return_value_policy::reference_internal) + .def_property_readonly("all_joint_poses_tracked", [](const Serialized& self) + { return self && self->all_joint_poses_tracked(); }) .def("__repr__", - [](const FullBodyPoseT& self) + [](const Serialized& self) { - std::string joints_str = "None"; - if (self.joints) + if (!self) { - joints_str = "BodyJoints(joints=[...24 entries...])"; + return std::string("FullBodyPose()"); } - return "FullBodyPoseT(joints=" + joints_str + ")"; + const std::string joints_str = + self->joints() != nullptr ? "BodyJoints(joints=[...24 entries...])" : "None"; + return "FullBodyPose(joints=" + joints_str + ")"; }); - py::class_>(m, "FullBodyPoseRecord") - .def(py::init<>()) - .def(py::init( - [](const FullBodyPoseT& data, const DeviceDataTimestamp& timestamp) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - obj->timestamp = std::make_shared(timestamp); - return obj; - }), - py::arg("data"), py::arg("timestamp")) - .def_property_readonly( - "data", [](const FullBodyPoseRecordT& self) -> std::shared_ptr { return self.data; }) - .def_readonly("timestamp", &FullBodyPoseRecordT::timestamp) - .def("__repr__", [](const FullBodyPoseRecordT& self) - { return "FullBodyPoseRecord(data=" + std::string(self.data ? "FullBodyPoseT(...)" : "None") + ")"; }); - - py::class_>(m, "FullBodyPoseTrackedT") - .def(py::init<>()) - .def(py::init( - [](const FullBodyPoseT& data) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - return obj; - }), - py::arg("data")) - .def_property_readonly( - "data", [](const FullBodyPoseTrackedT& self) -> std::shared_ptr { return self.data; }) - .def("__repr__", [](const FullBodyPoseTrackedT& self) - { return std::string("FullBodyPoseTrackedT(data=") + (self.data ? "FullBodyPoseT(...)" : "None") + ")"; }); + bind_record(m, "FullBodyPoseRecord", "FullBodyPose"); } } // namespace core diff --git a/src/core/schema/python/hand_bindings.h b/src/core/schema/python/hand_bindings.h index 14a19d95c..7e75802f7 100644 --- a/src/core/schema/python/hand_bindings.h +++ b/src/core/schema/python/hand_bindings.h @@ -6,7 +6,9 @@ #pragma once +#include "pose_bindings.h" #include "schema_array_views.h" +#include "schema_serialized.h" #include #include @@ -84,14 +86,8 @@ inline void bind_hand(py::module& m) .def("__repr__", [](const HandJointPose& self) { - return "HandJointPose(pose=Pose(position=Point(x=" + std::to_string(self.pose().position().x()) + - ", y=" + std::to_string(self.pose().position().y()) + - ", z=" + std::to_string(self.pose().position().z()) + - "), orientation=Quaternion(x=" + std::to_string(self.pose().orientation().x()) + - ", y=" + std::to_string(self.pose().orientation().y()) + - ", z=" + std::to_string(self.pose().orientation().z()) + - ", w=" + std::to_string(self.pose().orientation().w()) + - ")), is_valid=" + (self.is_valid() ? "True" : "False") + + return "HandJointPose(pose=" + pose_repr(self.pose()) + + ", is_valid=" + (self.is_valid() ? "True" : "False") + ", radius=" + std::to_string(self.radius()) + ")"; }); @@ -158,68 +154,31 @@ inline void bind_hand(py::module& m) .def("__repr__", [](const HandJoints&) { return "HandJoints(poses=[...HandJoint.NUM_JOINTS HandJointPose entries...])"; }); - // Bind HandPoseT class (FlatBuffers object API for tables). - py::class_>(m, "HandPoseT") - .def(py::init( - []() - { - auto obj = std::make_shared(); - obj->joints = std::make_shared(); - return obj; - })) + serialized_class(m, "HandPose", "Encoded hand pose: 26 joints in HandJoint / OpenXR order.") .def(py::init( [](const HandJoints& joints) { - auto obj = std::make_shared(); - obj->joints = std::make_shared(joints); - return obj; + HandPoseT native; + native.joints = std::make_shared(joints); + return pack(native); }), - py::arg("joints")) + py::arg("joints") = HandJoints(), "Encode a hand pose. Defaults to all-zero joints.") .def_property_readonly( - "joints", [](const HandPoseT& self) -> const HandJoints* { return self.joints.get(); }, + "joints", [](const Serialized& self) { return self ? self->joints() : nullptr; }, py::return_value_policy::reference_internal) .def("__repr__", - [](const HandPoseT& self) + [](const Serialized& self) { - std::string joints_str = "None"; - if (self.joints) + if (!self) { - joints_str = "HandJoints(poses=[...26 entries...])"; + return std::string("HandPose()"); } - return "HandPoseT(joints=" + joints_str + ")"; + const std::string joints_str = + self->joints() != nullptr ? "HandJoints(poses=[...26 entries...])" : "None"; + return "HandPose(joints=" + joints_str + ")"; }); - py::class_>(m, "HandPoseRecord") - .def(py::init<>()) - .def(py::init( - [](const HandPoseT& data, const DeviceDataTimestamp& timestamp) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - obj->timestamp = std::make_shared(timestamp); - return obj; - }), - py::arg("data"), py::arg("timestamp")) - .def_property_readonly( - "data", [](const HandPoseRecordT& self) -> std::shared_ptr { return self.data; }) - .def_readonly("timestamp", &HandPoseRecordT::timestamp) - .def("__repr__", [](const HandPoseRecordT& self) - { return "HandPoseRecord(data=" + std::string(self.data ? "HandPoseT(...)" : "None") + ")"; }); - - py::class_>(m, "HandPoseTrackedT") - .def(py::init<>()) - .def(py::init( - [](const HandPoseT& data) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - return obj; - }), - py::arg("data")) - .def_property_readonly( - "data", [](const HandPoseTrackedT& self) -> std::shared_ptr { return self.data; }) - .def("__repr__", [](const HandPoseTrackedT& self) - { return std::string("HandPoseTrackedT(data=") + (self.data ? "HandPoseT(...)" : "None") + ")"; }); + bind_record(m, "HandPoseRecord", "HandPose"); } } // namespace core diff --git a/src/core/schema/python/haptic_command_bindings.h b/src/core/schema/python/haptic_command_bindings.h index 3143b4335..ecb813a2f 100644 --- a/src/core/schema/python/haptic_command_bindings.h +++ b/src/core/schema/python/haptic_command_bindings.h @@ -2,17 +2,19 @@ // SPDX-License-Identifier: Apache-2.0 // Python bindings for the vendor-neutral HapticCommand FlatBuffer schema. -// Types: HapticCommand (table) + a pack helper that serialises it to the -// bytes a TensorPushTracker pushes to a peer-process device plugin. +// Types: HapticCommand (table) and HapticCommandRecord, plus a pack helper that +// serialises a command to the bytes a TensorPushTracker pushes to a peer-process +// device plugin. #pragma once +#include "schema_serialized.h" + #include #include #include #include -#include #include #include @@ -23,28 +25,37 @@ namespace core inline void bind_haptic_command(py::module& m) { - py::class_>(m, "HapticCommand") - .def(py::init([]() { return std::make_shared(); })) + serialized_class(m, "HapticCommand", "Encoded haptic command for one named actuator endpoint.") .def(py::init( [](const std::string& endpoint, const std::vector& values) { - auto obj = std::make_shared(); - obj->endpoint = endpoint; - obj->values = values; - return obj; + HapticCommandT native; + native.endpoint = endpoint; + native.values = values; + return pack(native); }), - py::arg("endpoint"), py::arg("values")) - .def_property( - "endpoint", [](const HapticCommandT& self) { return self.endpoint; }, - [](HapticCommandT& self, const std::string& v) { self.endpoint = v; }) - .def_property( - "values", [](const HapticCommandT& self) { return self.values; }, - [](HapticCommandT& self, const std::vector& v) { self.values = v; }); - - // Producer-side encode: serialise a HapticCommand (endpoint + values) to - // the FlatBuffer bytes that TensorPushTracker.push() carries to the - // consumer. Uses the generated Pack so the wire layout always matches the - // C++ SchemaTracker reader. + py::arg("endpoint") = std::string{}, py::arg("values") = std::vector{}, "Encode a haptic command.") + .def_property_readonly("endpoint", + [](const Serialized& self) + { + const auto* endpoint = self ? self->endpoint() : nullptr; + return endpoint != nullptr ? endpoint->str() : std::string{}; + }) + .def_property_readonly("values", + [](const Serialized& self) + { + // FlatBuffers omits an empty vector rather than encoding a + // zero-length one, so an absent field is "no values". + const auto* values = self ? self->values() : nullptr; + return values != nullptr ? std::vector(values->begin(), values->end()) : + std::vector{}; + }); + + bind_record(m, "HapticCommandRecord", "HapticCommand"); + + // Producer-side encode: serialise a HapticCommand (endpoint + values) to the raw + // FlatBuffer bytes that TensorPushTracker.push() carries to the consumer. Distinct + // from the HapticCommand constructor, which yields a view rather than the wire bytes. m.def( "pack_haptic_command", [](const std::string& endpoint, const std::vector& values) -> py::bytes diff --git a/src/core/schema/python/head_bindings.h b/src/core/schema/python/head_bindings.h index de14404a1..72e89c677 100644 --- a/src/core/schema/python/head_bindings.h +++ b/src/core/schema/python/head_bindings.h @@ -1,16 +1,20 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 // Python bindings for the HeadPose FlatBuffer schema. -// HeadPoseT is a table type (mutable object-API) with pose and is_valid fields. +// HeadPose is a table (pose + is_valid), exposed as an encoded view. #pragma once +#include "pose_bindings.h" +#include "schema_serialized.h" + #include #include #include #include +#include namespace py = pybind11; @@ -19,76 +23,34 @@ namespace core inline void bind_head(py::module& m) { - // Bind HeadPoseT class (FlatBuffers object API for tables). - py::class_>(m, "HeadPoseT") - .def(py::init( - []() - { - auto obj = std::make_shared(); - obj->pose = std::make_shared(); - return obj; - })) + serialized_class(m, "HeadPose", "Encoded head pose: a pose plus its validity flag.") .def(py::init( [](const Pose& pose, bool is_valid) { - auto obj = std::make_shared(); - obj->pose = std::make_shared(pose); - obj->is_valid = is_valid; - return obj; + HeadPoseT native; + native.pose = std::make_shared(pose); + native.is_valid = is_valid; + return pack(native); }), - py::arg("pose"), py::arg("is_valid")) + py::arg("pose") = Pose(), py::arg("is_valid") = false, + "Encode a head pose. Defaults to an all-zero pose that is not valid.") .def_property_readonly( - "pose", [](const HeadPoseT& self) -> const Pose* { return self.pose.get(); }, + "pose", [](const Serialized& self) { return self ? self->pose() : nullptr; }, py::return_value_policy::reference_internal) - .def_readonly("is_valid", &HeadPoseT::is_valid) + .def_property_readonly("is_valid", [](const Serialized& self) { return self && self->is_valid(); }) .def("__repr__", - [](const HeadPoseT& self) + [](const Serialized& self) { - std::string pose_str = "None"; - if (self.pose) + if (!self) { - pose_str = "Pose(position=Point(x=" + std::to_string(self.pose->position().x()) + - ", y=" + std::to_string(self.pose->position().y()) + - ", z=" + std::to_string(self.pose->position().z()) + - "), orientation=Quaternion(x=" + std::to_string(self.pose->orientation().x()) + - ", y=" + std::to_string(self.pose->orientation().y()) + - ", z=" + std::to_string(self.pose->orientation().z()) + - ", w=" + std::to_string(self.pose->orientation().w()) + "))"; + return std::string("HeadPose()"); } - return "HeadPoseT(pose=" + pose_str + ", is_valid=" + (self.is_valid ? "True" : "False") + ")"; + const Pose* pose = self->pose(); + const std::string pose_str = pose != nullptr ? pose_repr(*pose) : "None"; + return "HeadPose(pose=" + pose_str + ", is_valid=" + (self->is_valid() ? "True" : "False") + ")"; }); - py::class_>(m, "HeadPoseRecord") - .def(py::init<>()) - .def(py::init( - [](const HeadPoseT& data, const DeviceDataTimestamp& timestamp) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - obj->timestamp = std::make_shared(timestamp); - return obj; - }), - py::arg("data"), py::arg("timestamp")) - .def_property_readonly( - "data", [](const HeadPoseRecordT& self) -> std::shared_ptr { return self.data; }) - .def_readonly("timestamp", &HeadPoseRecordT::timestamp) - .def("__repr__", [](const HeadPoseRecordT& self) - { return "HeadPoseRecord(data=" + std::string(self.data ? "HeadPoseT(...)" : "None") + ")"; }); - - py::class_>(m, "HeadPoseTrackedT") - .def(py::init<>()) - .def(py::init( - [](const HeadPoseT& data) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - return obj; - }), - py::arg("data")) - .def_property_readonly( - "data", [](const HeadPoseTrackedT& self) -> std::shared_ptr { return self.data; }) - .def("__repr__", [](const HeadPoseTrackedT& self) - { return std::string("HeadPoseTrackedT(data=") + (self.data ? "HeadPoseT(...)" : "None") + ")"; }); + bind_record(m, "HeadPoseRecord", "HeadPose"); } } // namespace core diff --git a/src/core/schema/python/joint_state_bindings.h b/src/core/schema/python/joint_state_bindings.h index 0370f9877..244465cf3 100644 --- a/src/core/schema/python/joint_state_bindings.h +++ b/src/core/schema/python/joint_state_bindings.h @@ -2,10 +2,13 @@ // SPDX-License-Identifier: Apache-2.0 // Python bindings for the JointState FlatBuffer schema. -// Types: JointState (table), JointStateOutput (table), and the Tracked / Record wrappers. +// Types: JointState (table), JointStateOutput (table) and JointStateOutputRecord, +// all exposed as encoded views. #pragma once +#include "schema_serialized.h" + #include #include #include @@ -13,6 +16,7 @@ #include #include +#include namespace py = pybind11; @@ -22,95 +26,112 @@ namespace core inline void bind_joint_state(py::module& m) { // One named DOF (name -> position [+ optional velocity/effort/valid]). - py::class_>(m, "JointState") - .def(py::init([]() { return std::make_shared(); })) + serialized_class(m, "JointState", "Encoded state of one named joint.") .def(py::init( [](const std::string& name, float position, float velocity, float effort, bool valid) { - auto obj = std::make_shared(); - obj->name = name; - obj->position = position; - obj->velocity = velocity; - obj->effort = effort; - obj->valid = valid; - return obj; + JointStateT native; + native.name = name; + native.position = position; + native.velocity = velocity; + native.effort = effort; + native.valid = valid; + return pack(native); }), py::arg("name"), py::arg("position") = 0.0f, py::arg("velocity") = 0.0f, py::arg("effort") = 0.0f, - py::arg("valid") = true) - .def_property( - "name", [](const JointStateT& self) { return self.name; }, - [](JointStateT& self, const std::string& val) { self.name = val; }) - .def_property( - "position", [](const JointStateT& self) { return self.position; }, - [](JointStateT& self, float val) { self.position = val; }) - .def_property( - "velocity", [](const JointStateT& self) { return self.velocity; }, - [](JointStateT& self, float val) { self.velocity = val; }) - .def_property( - "effort", [](const JointStateT& self) { return self.effort; }, - [](JointStateT& self, float val) { self.effort = val; }) - .def_property( - "valid", [](const JointStateT& self) { return self.valid; }, - [](JointStateT& self, bool val) { self.valid = val; }) - .def("__repr__", [](const JointStateT& self) - { return "JointState(name=" + self.name + ", position=" + std::to_string(self.position) + ")"; }); - - // Per-frame device state: a list of named joints plus identity / capability flags. - py::class_>(m, "JointStateOutput") - .def(py::init([]() { return std::make_shared(); })) - .def_property( - "joints", [](const JointStateOutputT& self) { return self.joints; }, - [](JointStateOutputT& self, std::vector> val) { self.joints = std::move(val); }) - .def_property( - "device_id", [](const JointStateOutputT& self) { return self.device_id; }, - [](JointStateOutputT& self, const std::string& val) { self.device_id = val; }) - .def_property( - "has_velocity", [](const JointStateOutputT& self) { return self.has_velocity; }, - [](JointStateOutputT& self, bool val) { self.has_velocity = val; }) - .def_property( - "has_effort", [](const JointStateOutputT& self) { return self.has_effort; }, - [](JointStateOutputT& self, bool val) { self.has_effort = val; }) - .def_property( - "ee_pose_valid", [](const JointStateOutputT& self) { return self.ee_pose_valid; }, - [](JointStateOutputT& self, bool val) { self.ee_pose_valid = val; }) + py::arg("valid") = true, "Encode one joint's state.") + .def_property_readonly("name", + [](const Serialized& self) + { + const auto* name = self ? self->name() : nullptr; + return name != nullptr ? name->str() : std::string{}; + }) + .def_property_readonly( + "position", [](const Serialized& self) { return self ? self->position() : 0.0f; }) + .def_property_readonly( + "velocity", [](const Serialized& self) { return self ? self->velocity() : 0.0f; }) + .def_property_readonly("effort", [](const Serialized& self) { return self ? self->effort() : 0.0f; }) + .def_property_readonly("valid", [](const Serialized& self) { return self && self->valid(); }) .def("__repr__", - [](const JointStateOutputT& self) { - return "JointStateOutput(device_id=" + self.device_id + - ", joints=" + std::to_string(self.joints.size()) + ")"; + [](const Serialized& self) + { + if (!self) + { + return std::string("JointState()"); + } + const auto* name = self->name(); + return "JointState(name=" + (name != nullptr ? name->str() : std::string{}) + + ", position=" + std::to_string(self->position()) + ")"; }); - py::class_>(m, "JointStateOutputRecord") - .def(py::init<>()) + // Per-frame device state: a list of named joints plus identity / capability flags. + serialized_class( + m, "JointStateOutput", "Encoded joint-space device state: named joints plus capability flags.") .def(py::init( - [](const JointStateOutputT& data, const DeviceDataTimestamp& timestamp) + [](const std::vector>& joints, const std::string& device_id, bool has_velocity, + bool has_effort, const Pose* ee_pose, bool ee_pose_valid) { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - obj->timestamp = std::make_shared(timestamp); - return obj; + JointStateOutputT native; + native.joints = to_native_vector(joints, "joints"); + native.device_id = device_id; + native.has_velocity = has_velocity; + native.has_effort = has_effort; + if (ee_pose != nullptr) + { + native.ee_pose = std::make_shared(*ee_pose); + } + native.ee_pose_valid = ee_pose_valid; + return pack(native); }), - py::arg("data"), py::arg("timestamp")) + py::arg("joints") = std::vector>{}, py::arg("device_id") = std::string{}, + py::arg("has_velocity") = false, py::arg("has_effort") = false, py::arg("ee_pose") = nullptr, + py::arg("ee_pose_valid") = false, "Encode a joint-space device state.") + .def_property_readonly("joints", + [](const Serialized& self) + { + // FlatBuffers omits an empty vector rather than encoding a zero-length one, + // so an absent field is "no joints", not missing data. + std::vector> joints; + const auto* encoded = self ? self->joints() : nullptr; + if (encoded != nullptr) + { + joints.reserve(encoded->size()); + for (const auto* joint : *encoded) + { + joints.push_back(self.narrow(joint)); + } + } + return joints; + }) + .def_property_readonly("device_id", + [](const Serialized& self) + { + const auto* device_id = self ? self->device_id() : nullptr; + return device_id != nullptr ? device_id->str() : std::string{}; + }) .def_property_readonly( - "data", [](const JointStateOutputRecordT& self) -> std::shared_ptr { return self.data; }) - .def_readonly("timestamp", &JointStateOutputRecordT::timestamp); - - py::class_>(m, "JointStateOutputTrackedT") - .def(py::init<>()) - .def(py::init( - [](const JointStateOutputT& data) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - return obj; - }), - py::arg("data")) + "has_velocity", [](const Serialized& self) { return self && self->has_velocity(); }) + .def_property_readonly( + "has_effort", [](const Serialized& self) { return self && self->has_effort(); }) .def_property_readonly( - "data", [](const JointStateOutputTrackedT& self) -> std::shared_ptr { return self.data; }) + "ee_pose", [](const Serialized& self) { return self ? self->ee_pose() : nullptr; }, + py::return_value_policy::reference_internal) + .def_property_readonly( + "ee_pose_valid", [](const Serialized& self) { return self && self->ee_pose_valid(); }) .def("__repr__", - [](const JointStateOutputTrackedT& self) { - return std::string("JointStateOutputTrackedT(data=") + (self.data ? "JointStateOutput(...)" : "None") + - ")"; + [](const Serialized& self) + { + if (!self) + { + return std::string("JointStateOutput()"); + } + const auto* device_id = self->device_id(); + const auto* joints = self->joints(); + return "JointStateOutput(device_id=" + (device_id != nullptr ? device_id->str() : std::string{}) + + ", joints=" + std::to_string(joints != nullptr ? joints->size() : 0) + ")"; }); + + bind_record(m, "JointStateOutputRecord", "JointStateOutput"); } } // namespace core diff --git a/src/core/schema/python/message_channel_bindings.h b/src/core/schema/python/message_channel_bindings.h index 4f398a7bc..c5a4f007b 100644 --- a/src/core/schema/python/message_channel_bindings.h +++ b/src/core/schema/python/message_channel_bindings.h @@ -1,14 +1,24 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +// Python bindings for the message channel FlatBuffer schema. +// Types: MessageChannelMessages (table) and its Tracked / Record wrappers, exposed as +// encoded views. This is the one surviving wrapper: its payload is a list, so a table +// is still needed to hold the vector. + #pragma once +#include "schema_serialized.h" + #include #include #include #include +#include #include +#include +#include namespace py = pybind11; @@ -17,59 +27,66 @@ namespace core inline void bind_message_channel(py::module& m) { - py::class_>(m, "MessageChannelMessages") - .def(py::init([]() { return std::make_shared(); })) + serialized_class(m, "MessageChannelMessages", "Encoded opaque message payload.") .def(py::init( [](py::bytes payload) { - auto obj = std::make_shared(); - std::string data = payload; - obj->payload.assign(data.begin(), data.end()); - return obj; + MessageChannelMessagesT native; + const std::string data = payload; + native.payload.assign(data.begin(), data.end()); + return pack(native); }), - py::arg("payload")) - .def_property( - "payload", - [](const MessageChannelMessagesT& self) - { return py::bytes(reinterpret_cast(self.payload.data()), self.payload.size()); }, - [](MessageChannelMessagesT& self, py::bytes payload) - { - std::string data = payload; - self.payload.assign(data.begin(), data.end()); - }); + py::arg("payload") = py::bytes(), "Encode a message payload.") + .def_property_readonly("payload", + [](const Serialized& self) + { + const auto* payload = self ? self->payload() : nullptr; + return payload != nullptr ? + py::bytes(reinterpret_cast(payload->data()), payload->size()) : + py::bytes(); + }); - py::class_>( - m, "MessageChannelMessagesRecord") - .def(py::init<>()) - .def(py::init( - [](const MessageChannelMessagesT& data, const DeviceDataTimestamp& timestamp) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - obj->timestamp = std::make_shared(timestamp); - return obj; - }), - py::arg("data"), py::arg("timestamp")) - .def_property_readonly("data", - [](const MessageChannelMessagesRecordT& self) -> std::shared_ptr - { return self.data; }) - .def_readonly("timestamp", &MessageChannelMessagesRecordT::timestamp); + bind_record( + m, "MessageChannelMessagesRecord", "MessageChannelMessages"); - py::class_>( - m, "MessageChannelMessagesTrackedT") - .def(py::init<>()) + // Unlike the single-payload wrappers, `data` here is the batch drained in one frame. + // It stays a list (empty, never None) so "nothing arrived" and "channel inactive" read + // the same way they always have. + serialized_class( + m, "MessageChannelMessagesTracked", "Encoded batch of messages drained during one update.") .def(py::init( - [](const std::vector>& data) + [](const std::vector>& data) { - auto obj = std::make_shared(); - obj->data = data; - return obj; + MessageChannelMessagesTrackedT native; + native.data = to_native_vector(data, "data"); + return pack(native); }), - py::arg("data")) - .def_property_readonly( - "data", - [](const MessageChannelMessagesTrackedT& self) -> std::vector> - { return self.data; }); + py::arg("data") = std::vector>{}, + "Encode a batch of messages. Omit `data` for an empty batch.") + .def_property_readonly("data", + [](const Serialized& self) + { + // FlatBuffers omits an empty vector rather than encoding a + // zero-length one, so an absent field is an empty batch. + std::vector> messages; + const auto* encoded = payload(self); + if (encoded != nullptr) + { + messages.reserve(encoded->size()); + for (const auto* message : *encoded) + { + messages.push_back(self.narrow(message)); + } + } + return messages; + }) + .def("__repr__", + [](const Serialized& self) + { + const auto* encoded = payload(self); + return "MessageChannelMessagesTracked(data=[" + + std::to_string(encoded != nullptr ? encoded->size() : 0) + " messages])"; + }); } } // namespace core diff --git a/src/core/schema/python/oak_bindings.h b/src/core/schema/python/oak_bindings.h index 215bacc80..13fc8889b 100644 --- a/src/core/schema/python/oak_bindings.h +++ b/src/core/schema/python/oak_bindings.h @@ -2,16 +2,18 @@ // SPDX-License-Identifier: Apache-2.0 // Python bindings for the OAK FlatBuffer schema. -// Types: StreamType (enum), FrameMetadataOak (table). +// Types: StreamType (enum), FrameMetadataOak (table), exposed as an encoded view. #pragma once +#include "schema_serialized.h" + #include -#include #include #include -#include +#include +#include namespace py = pybind11; @@ -25,63 +27,36 @@ inline void bind_oak(py::module& m) .value("MonoLeft", StreamType_MonoLeft) .value("MonoRight", StreamType_MonoRight); - py::class_>(m, "FrameMetadataOak") - .def(py::init([]() { return std::make_shared(); })) + serialized_class(m, "FrameMetadataOak", "Encoded per-frame OAK camera metadata.") .def(py::init( [](StreamType stream, uint64_t sequence_number) { - auto obj = std::make_shared(); - obj->stream = stream; - obj->sequence_number = sequence_number; - return obj; - }), - py::arg("stream"), py::arg("sequence_number")) - .def_property( - "stream", [](const FrameMetadataOakT& self) { return self.stream; }, - [](FrameMetadataOakT& self, StreamType val) { self.stream = val; }, - "Get or set the stream type that produced this frame") - .def_readwrite("sequence_number", &FrameMetadataOakT::sequence_number, "Get or set the per-stream sequence number") - .def("__repr__", - [](const FrameMetadataOakT& metadata) - { - return "FrameMetadataOak(stream=" + std::string(EnumNameStreamType(metadata.stream)) + - ", sequence_number=" + std::to_string(metadata.sequence_number) + ")"; - }); - - py::class_>(m, "FrameMetadataOakRecord") - .def(py::init<>()) - .def(py::init( - [](const FrameMetadataOakT& data, const DeviceDataTimestamp& timestamp) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - obj->timestamp = std::make_shared(timestamp); - return obj; + FrameMetadataOakT native; + native.stream = stream; + native.sequence_number = sequence_number; + return pack(native); }), - py::arg("data"), py::arg("timestamp")) + py::arg("stream") = StreamType_Color, py::arg("sequence_number") = 0, + "Encode frame metadata. Defaults to stream Color at sequence number 0.") .def_property_readonly( - "data", [](const FrameMetadataOakRecordT& self) -> std::shared_ptr { return self.data; }) - .def_readonly("timestamp", &FrameMetadataOakRecordT::timestamp) - .def("__repr__", [](const FrameMetadataOakRecordT& self) - { return "FrameMetadataOakRecord(data=" + std::string(self.data ? "FrameMetadataOak(...)" : "None") + ")"; }); - - py::class_>(m, "FrameMetadataOakTrackedT") - .def(py::init<>()) - .def(py::init( - [](const FrameMetadataOakT& data) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - return obj; - }), - py::arg("data")) + "stream", [](const Serialized& self) { return self ? self->stream() : StreamType_Color; }, + "The stream type that produced this frame") .def_property_readonly( - "data", [](const FrameMetadataOakTrackedT& self) -> std::shared_ptr { return self.data; }) + "sequence_number", + [](const Serialized& self) -> uint64_t { return self ? self->sequence_number() : 0; }, + "The per-stream sequence number") .def("__repr__", - [](const FrameMetadataOakTrackedT& self) { - return std::string("FrameMetadataOakTrackedT(data=") + (self.data ? "FrameMetadataOak(...)" : "None") + - ")"; + [](const Serialized& self) + { + if (!self) + { + return std::string("FrameMetadataOak()"); + } + return "FrameMetadataOak(stream=" + std::string(EnumNameStreamType(self->stream())) + + ", sequence_number=" + std::to_string(self->sequence_number()) + ")"; }); + + bind_record(m, "FrameMetadataOakRecord", "FrameMetadataOak"); } } // namespace core diff --git a/src/core/schema/python/oglo_tactile_bindings.h b/src/core/schema/python/oglo_tactile_bindings.h index 1ef75ab91..fba40c5f5 100644 --- a/src/core/schema/python/oglo_tactile_bindings.h +++ b/src/core/schema/python/oglo_tactile_bindings.h @@ -2,16 +2,20 @@ // SPDX-License-Identifier: Apache-2.0 // Python bindings for the OGLO tactile glove FlatBuffer schema. -// Types: OgloGloveSample (table), OgloGloveSampleRecord, OgloGloveSampleTrackedT. +// Types: OgloGloveSample (table) and OgloGloveSampleRecord, exposed as encoded views. #pragma once +#include "schema_serialized.h" + #include #include #include #include -#include +#include +#include +#include namespace py = pybind11; @@ -20,60 +24,68 @@ namespace core inline void bind_oglo_tactile(py::module& m) { - py::class_>(m, "OgloGloveSample") - .def(py::init([]() { return std::make_shared(); })) - .def_property( - "seq", [](const OgloGloveSampleT& s) { return s.seq; }, [](OgloGloveSampleT& s, uint32_t v) { s.seq = v; }) - .def_property( - "device_time_us", [](const OgloGloveSampleT& s) { return s.device_time_us; }, - [](OgloGloveSampleT& s, uint32_t v) { s.device_time_us = v; }) - .def_property( - "taxels", [](const OgloGloveSampleT& s) { return s.taxels; }, - [](OgloGloveSampleT& s, std::vector v) { s.taxels = std::move(v); }, + serialized_class(m, "OgloGloveSample", "Encoded tactile glove sample.") + .def(py::init( + [](uint32_t seq, uint32_t device_time_us, std::vector taxels, int16_t accel_x, + int16_t accel_y, int16_t accel_z, int16_t gyro_x, int16_t gyro_y, int16_t gyro_z) + { + OgloGloveSampleT native; + native.seq = seq; + native.device_time_us = device_time_us; + native.taxels = std::move(taxels); + native.accel_x = accel_x; + native.accel_y = accel_y; + native.accel_z = accel_z; + native.gyro_x = gyro_x; + native.gyro_y = gyro_y; + native.gyro_z = gyro_z; + return pack(native); + }), + py::arg("seq") = 0, py::arg("device_time_us") = 0, py::arg("taxels") = std::vector{}, + py::arg("accel_x") = 0, py::arg("accel_y") = 0, py::arg("accel_z") = 0, py::arg("gyro_x") = 0, + py::arg("gyro_y") = 0, py::arg("gyro_z") = 0, "Encode a tactile glove sample.") + .def_property_readonly( + "seq", [](const Serialized& self) -> uint32_t { return self ? self->seq() : 0; }) + .def_property_readonly("device_time_us", + [](const Serialized& self) -> uint32_t + { return self ? self->device_time_us() : 0; }) + .def_property_readonly( + "taxels", + [](const Serialized& self) + { + // A FlatBuffers vector field is omitted when empty, so an absent field is an + // empty reading rather than missing data. + const auto* taxels = self ? self->taxels() : nullptr; + return taxels != nullptr ? std::vector(taxels->begin(), taxels->end()) : + std::vector{}; + }, "80 raw 12-bit taxels (0..4095) in finger,row,col order") - .def_property( - "accel_x", [](const OgloGloveSampleT& s) { return s.accel_x; }, - [](OgloGloveSampleT& s, int16_t v) { s.accel_x = v; }) - .def_property( - "accel_y", [](const OgloGloveSampleT& s) { return s.accel_y; }, - [](OgloGloveSampleT& s, int16_t v) { s.accel_y = v; }) - .def_property( - "accel_z", [](const OgloGloveSampleT& s) { return s.accel_z; }, - [](OgloGloveSampleT& s, int16_t v) { s.accel_z = v; }) - .def_property( - "gyro_x", [](const OgloGloveSampleT& s) { return s.gyro_x; }, - [](OgloGloveSampleT& s, int16_t v) { s.gyro_x = v; }) - .def_property( - "gyro_y", [](const OgloGloveSampleT& s) { return s.gyro_y; }, - [](OgloGloveSampleT& s, int16_t v) { s.gyro_y = v; }) - .def_property( - "gyro_z", [](const OgloGloveSampleT& s) { return s.gyro_z; }, - [](OgloGloveSampleT& s, int16_t v) { s.gyro_z = v; }) - .def("__repr__", - [](const OgloGloveSampleT& s) - { - return "OgloGloveSample(seq=" + std::to_string(s.seq) + - ", device_time_us=" + std::to_string(s.device_time_us) + - ", taxels=" + std::to_string(s.taxels.size()) + ")"; - }); - - py::class_>(m, "OgloGloveSampleRecord") - .def(py::init<>()) .def_property_readonly( - "data", [](const OgloGloveSampleRecordT& self) -> std::shared_ptr { return self.data; }) - .def_readonly("timestamp", &OgloGloveSampleRecordT::timestamp) - .def("__repr__", [](const OgloGloveSampleRecordT& self) - { return "OgloGloveSampleRecord(data=" + std::string(self.data ? "OgloGloveSample(...)" : "None") + ")"; }); - - py::class_>(m, "OgloGloveSampleTrackedT") - .def(py::init<>()) + "accel_x", [](const Serialized& self) -> int16_t { return self ? self->accel_x() : 0; }) + .def_property_readonly( + "accel_y", [](const Serialized& self) -> int16_t { return self ? self->accel_y() : 0; }) + .def_property_readonly( + "accel_z", [](const Serialized& self) -> int16_t { return self ? self->accel_z() : 0; }) + .def_property_readonly( + "gyro_x", [](const Serialized& self) -> int16_t { return self ? self->gyro_x() : 0; }) .def_property_readonly( - "data", [](const OgloGloveSampleTrackedT& self) -> std::shared_ptr { return self.data; }) + "gyro_y", [](const Serialized& self) -> int16_t { return self ? self->gyro_y() : 0; }) + .def_property_readonly( + "gyro_z", [](const Serialized& self) -> int16_t { return self ? self->gyro_z() : 0; }) .def("__repr__", - [](const OgloGloveSampleTrackedT& self) { - return std::string("OgloGloveSampleTrackedT(data=") + (self.data ? "OgloGloveSample(...)" : "None") + - ")"; + [](const Serialized& self) + { + if (!self) + { + return std::string("OgloGloveSample()"); + } + const auto* taxels = self->taxels(); + return "OgloGloveSample(seq=" + std::to_string(self->seq()) + + ", device_time_us=" + std::to_string(self->device_time_us()) + + ", taxels=" + std::to_string(taxels != nullptr ? taxels->size() : 0) + ")"; }); + + bind_record(m, "OgloGloveSampleRecord", "OgloGloveSample"); } } // namespace core diff --git a/src/core/schema/python/pedals_bindings.h b/src/core/schema/python/pedals_bindings.h index f04186ba6..c2b59063e 100644 --- a/src/core/schema/python/pedals_bindings.h +++ b/src/core/schema/python/pedals_bindings.h @@ -2,15 +2,17 @@ // SPDX-License-Identifier: Apache-2.0 // Python bindings for the Pedals FlatBuffer schema. -// Types: Generic3AxisPedalOutput (table). +// Types: Generic3AxisPedalOutput (table), exposed as an encoded view. #pragma once +#include "schema_serialized.h" + #include #include #include -#include +#include namespace py = pybind11; @@ -19,81 +21,38 @@ namespace core inline void bind_pedals(py::module& m) { - // Bind Generic3AxisPedalOutput table using the native type (Generic3AxisPedalOutputT). - py::class_>(m, "Generic3AxisPedalOutput") - .def(py::init([]() { return std::make_shared(); })) + serialized_class(m, "Generic3AxisPedalOutput", "Encoded three-axis pedal state.") .def(py::init( [](float left_pedal, float right_pedal, float rudder) { - auto obj = std::make_shared(); - obj->left_pedal = left_pedal; - obj->right_pedal = right_pedal; - obj->rudder = rudder; - return obj; + Generic3AxisPedalOutputT native; + native.left_pedal = left_pedal; + native.right_pedal = right_pedal; + native.rudder = rudder; + return pack(native); }), - py::arg("left_pedal"), py::arg("right_pedal"), py::arg("rudder")) - .def_property( - "left_pedal", [](const Generic3AxisPedalOutputT& self) { return self.left_pedal; }, - [](Generic3AxisPedalOutputT& self, float val) { self.left_pedal = val; }) - .def_property( - "right_pedal", [](const Generic3AxisPedalOutputT& self) { return self.right_pedal; }, - [](Generic3AxisPedalOutputT& self, float val) { self.right_pedal = val; }) - .def_property( - "rudder", [](const Generic3AxisPedalOutputT& self) { return self.rudder; }, - [](Generic3AxisPedalOutputT& self, float val) { self.rudder = val; }) + py::arg("left_pedal") = 0.0f, py::arg("right_pedal") = 0.0f, py::arg("rudder") = 0.0f, + "Encode a pedal state. Omitted axes are zero.") + .def_property_readonly("left_pedal", [](const Serialized& self) + { return self ? self->left_pedal() : 0.0f; }) + .def_property_readonly("right_pedal", [](const Serialized& self) + { return self ? self->right_pedal() : 0.0f; }) + .def_property_readonly( + "rudder", [](const Serialized& self) { return self ? self->rudder() : 0.0f; }) .def("__repr__", - [](const Generic3AxisPedalOutputT& output) + [](const Serialized& self) { - std::string result = "Generic3AxisPedalOutput(left_pedal=" + std::to_string(output.left_pedal); - result += ", right_pedal=" + std::to_string(output.right_pedal); - result += ", rudder=" + std::to_string(output.rudder); - result += ")"; - return result; - }); - - py::class_>( - m, "Generic3AxisPedalOutputRecord") - .def(py::init<>()) - .def(py::init( - [](const Generic3AxisPedalOutputT& data, const DeviceDataTimestamp& timestamp) + if (!self) { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - obj->timestamp = std::make_shared(timestamp); - return obj; - }), - py::arg("data"), py::arg("timestamp")) - .def_property_readonly("data", - [](const Generic3AxisPedalOutputRecordT& self) -> std::shared_ptr - { return self.data; }) - .def_readonly("timestamp", &Generic3AxisPedalOutputRecordT::timestamp) - .def("__repr__", - [](const Generic3AxisPedalOutputRecordT& self) - { - return "Generic3AxisPedalOutputRecord(data=" + - std::string(self.data ? "Generic3AxisPedalOutput(...)" : "None") + ")"; + return std::string("Generic3AxisPedalOutput()"); + } + return "Generic3AxisPedalOutput(left_pedal=" + std::to_string(self->left_pedal()) + + ", right_pedal=" + std::to_string(self->right_pedal()) + + ", rudder=" + std::to_string(self->rudder()) + ")"; }); - py::class_>( - m, "Generic3AxisPedalOutputTrackedT") - .def(py::init<>()) - .def(py::init( - [](const Generic3AxisPedalOutputT& data) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - return obj; - }), - py::arg("data")) - .def_property_readonly("data", - [](const Generic3AxisPedalOutputTrackedT& self) -> std::shared_ptr - { return self.data; }) - .def("__repr__", - [](const Generic3AxisPedalOutputTrackedT& self) - { - return std::string("Generic3AxisPedalOutputTrackedT(data=") + - (self.data ? "Generic3AxisPedalOutput(...)" : "None") + ")"; - }); + bind_record( + m, "Generic3AxisPedalOutputRecord", "Generic3AxisPedalOutput"); } } // namespace core diff --git a/src/core/schema/python/pose_bindings.h b/src/core/schema/python/pose_bindings.h index 0df346f40..54d3adf32 100644 --- a/src/core/schema/python/pose_bindings.h +++ b/src/core/schema/python/pose_bindings.h @@ -9,11 +9,24 @@ #include #include +#include + namespace py = pybind11; namespace core { +//! The `Pose(position=..., orientation=...)` text used by `Pose.__repr__` and by every +//! table repr that nests a pose, so the five of them cannot drift apart. +inline std::string pose_repr(const Pose& p) +{ + return "Pose(position=Point(x=" + std::to_string(p.position().x()) + ", y=" + std::to_string(p.position().y()) + + ", z=" + std::to_string(p.position().z()) + + "), orientation=Quaternion(x=" + std::to_string(p.orientation().x()) + + ", y=" + std::to_string(p.orientation().y()) + ", z=" + std::to_string(p.orientation().z()) + + ", w=" + std::to_string(p.orientation().w()) + "))"; +} + inline void bind_pose(py::module& m) { // Bind Point struct (x, y, z). @@ -51,15 +64,7 @@ inline void bind_pose(py::module& m) .def(py::init(), py::arg("position"), py::arg("orientation")) .def_property_readonly("position", &Pose::position) .def_property_readonly("orientation", &Pose::orientation) - .def("__repr__", - [](const Pose& p) - { - return "Pose(position=Point(x=" + std::to_string(p.position().x()) + - ", y=" + std::to_string(p.position().y()) + ", z=" + std::to_string(p.position().z()) + - "), orientation=Quaternion(x=" + std::to_string(p.orientation().x()) + - ", y=" + std::to_string(p.orientation().y()) + ", z=" + std::to_string(p.orientation().z()) + - ", w=" + std::to_string(p.orientation().w()) + "))"; - }); + .def("__repr__", [](const Pose& p) { return pose_repr(p); }); } } // namespace core diff --git a/src/core/schema/python/schema_serialized.h b/src/core/schema/python/schema_serialized.h new file mode 100644 index 000000000..33811fbca --- /dev/null +++ b/src/core/schema/python/schema_serialized.h @@ -0,0 +1,126 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Shared scaffolding for binding FlatBuffer tables to Python as encoded views. +// +// Python sees one class per table, backed by core::Serialized
: reads go through +// the generated accessors straight into the buffer, and no object-API (`-T`) type is +// ever exposed. Structs (Pose, HandJoints, ...) are unaffected -- flatc emits one +// struct type for both APIs, so their existing bindings and the zero-copy NumPy views +// in schema_array_views.h keep working, reached through a table view that owns the +// buffer they alias. +// +// Construction from Python goes the other way: a constructor builds a `-T` as a local, +// encodes it, and returns the view. That keeps the encoder honest (it is the generated +// Pack, so the layout always matches the C++ readers) while the `-T` stays invisible. +// +// Every Record wrapper has the same shape, so bind_record() below covers them; each +// schema's binding header only has to describe its own payload table. Trackers publish +// their payload table directly -- an empty view is how absence is expressed -- so there +// is no wrapper binding here. + +#pragma once + +#include +#include +#include +#include + +#include +#include +#include +#include + +namespace py = pybind11; + +namespace core +{ + +//! Python class for a table view. Chain the table's own fields onto the result. +template +py::class_> serialized_class(py::module& m, const char* name, const char* doc) +{ + return py::class_>(m, name, doc) + // Always true in practice: an absent payload reaches Python as None, never as an + // empty view. Kept so that a view which somehow arrived empty still reports it + // rather than silently answering field reads with defaults. + .def( + "__bool__", [](const Serialized& self) { return static_cast(self); }, + "False when the payload is absent."); +} + +/*! + * @brief Converts handles into the native element vector a table's vector field takes. + * + * FlatBuffers cannot splice a finished buffer into another one, so a constructor composing + * a table around payloads the caller already built has to go back through the object API. + * A construction-time cost only -- the read path never unpacks. + * + * A vector field is the one place a null element is fatal: the generated `Pack` null-checks + * an optional table field but dereferences every vector element unconditionally. `field` + * names the vector in the message. + */ +template +std::vector> to_native_vector(const std::vector>& handles, + const char* field) +{ + std::vector> natives; + natives.reserve(handles.size()); + for (const auto& handle : handles) + { + if (!handle) + { + throw py::value_error(std::string(field) + ": entries must be non-empty"); + } + auto native = std::make_shared(); + handle->UnPackTo(native.get()); + natives.push_back(std::move(native)); + } + return natives; +} + +//! Binds a `Record` wrapper (an MCAP payload: data plus its capture timestamp). +template +void bind_record(py::module& m, const char* name, const char* data_name) +{ + serialized_class(m, name, "Encoded MCAP record: a payload plus the timestamp it was captured at.") + .def(py::init<>(), "Construct an empty record (.data and .timestamp are None).") + .def(py::init( + [](const Serialized* data, const DeviceDataTimestamp& timestamp) + { + typename RecordT::NativeTableType native; + if (data != nullptr && *data) + { + native.data = std::make_shared(); + (*data)->UnPackTo(native.data.get()); + } + native.timestamp = std::make_shared(timestamp); + return pack(native); + }), + py::arg("data").none(true), py::arg("timestamp"), + "Encode a record from a payload and its timestamp. `data` may be None: MCAP " + "carries payload-less records, such as the message channel's frame sentinel.") + + // Unlike Tracked, a Record cannot fold its no-arg form into defaults: `timestamp` is a + // struct with no meaningful default, and an empty record is a distinct thing from one + // stamped at time zero. + .def_property_readonly( + "data", + [](const Serialized& self) -> py::object + { + const DataT* data = payload(self); + return data != nullptr ? py::cast(self.narrow(data)) : py::none(); + }, + "The recorded payload, or None when absent.") + .def_property_readonly( + "timestamp", [](const Serialized& self) { return self ? self->timestamp() : nullptr; }, + py::return_value_policy::reference_internal, "Capture timestamp, or None when absent.") + .def("__repr__", + [name, data_name](const Serialized& self) + { + return std::string(name) + + "(data=" + (payload(self) != nullptr ? std::string(data_name) + "(...)" : "None") + ")"; + }); +} + +} // namespace core diff --git a/src/core/schema/python/se3_tracker_bindings.h b/src/core/schema/python/se3_tracker_bindings.h index 04a2e167d..5a192a856 100644 --- a/src/core/schema/python/se3_tracker_bindings.h +++ b/src/core/schema/python/se3_tracker_bindings.h @@ -2,15 +2,19 @@ // SPDX-License-Identifier: Apache-2.0 // Python bindings for the Se3TrackerPose FlatBuffer schema. -// Se3TrackerPoseT is a table type (mutable object-API) with pose and is_valid fields. +// Se3TrackerPose is a table (pose + is_valid), exposed as an encoded view. #pragma once +#include "pose_bindings.h" +#include "schema_serialized.h" + #include #include #include #include +#include namespace py = pybind11; @@ -19,78 +23,37 @@ namespace core inline void bind_se3_tracker(py::module& m) { - // Bind Se3TrackerPoseT class (FlatBuffers object API for tables). - py::class_>(m, "Se3TrackerPoseT") - .def(py::init( - []() - { - auto obj = std::make_shared(); - obj->pose = std::make_shared(); - return obj; - })) + serialized_class(m, "Se3TrackerPose", + "Encoded SE3 tracker pose. Gate on is_valid before consuming pose -- the pose " + "contents are unspecified while tracking is lost.") .def(py::init( [](const Pose& pose, bool is_valid) { - auto obj = std::make_shared(); - obj->pose = std::make_shared(pose); - obj->is_valid = is_valid; - return obj; + Se3TrackerPoseT native; + native.pose = std::make_shared(pose); + native.is_valid = is_valid; + return pack(native); }), - py::arg("pose"), py::arg("is_valid")) + py::arg("pose") = Pose(), py::arg("is_valid") = false, + "Encode an SE3 pose. Defaults to an all-zero pose that is not valid.") .def_property_readonly( - "pose", [](const Se3TrackerPoseT& self) -> const Pose* { return self.pose.get(); }, + "pose", [](const Serialized& self) { return self ? self->pose() : nullptr; }, py::return_value_policy::reference_internal) - .def_readonly("is_valid", &Se3TrackerPoseT::is_valid) + .def_property_readonly( + "is_valid", [](const Serialized& self) { return self && self->is_valid(); }) .def("__repr__", - [](const Se3TrackerPoseT& self) + [](const Serialized& self) { - std::string pose_str = "None"; - if (self.pose) + if (!self) { - pose_str = "Pose(position=Point(x=" + std::to_string(self.pose->position().x()) + - ", y=" + std::to_string(self.pose->position().y()) + - ", z=" + std::to_string(self.pose->position().z()) + - "), orientation=Quaternion(x=" + std::to_string(self.pose->orientation().x()) + - ", y=" + std::to_string(self.pose->orientation().y()) + - ", z=" + std::to_string(self.pose->orientation().z()) + - ", w=" + std::to_string(self.pose->orientation().w()) + "))"; + return std::string("Se3TrackerPose()"); } - return "Se3TrackerPoseT(pose=" + pose_str + ", is_valid=" + (self.is_valid ? "True" : "False") + ")"; + const Pose* pose = self->pose(); + const std::string pose_str = pose != nullptr ? pose_repr(*pose) : "None"; + return "Se3TrackerPose(pose=" + pose_str + ", is_valid=" + (self->is_valid() ? "True" : "False") + ")"; }); - py::class_>(m, "Se3TrackerPoseRecord") - .def(py::init<>()) - .def(py::init( - [](const Se3TrackerPoseT& data, const DeviceDataTimestamp& timestamp) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - obj->timestamp = std::make_shared(timestamp); - return obj; - }), - py::arg("data"), py::arg("timestamp")) - .def_property_readonly( - "data", [](const Se3TrackerPoseRecordT& self) -> std::shared_ptr { return self.data; }) - .def_readonly("timestamp", &Se3TrackerPoseRecordT::timestamp) - .def("__repr__", [](const Se3TrackerPoseRecordT& self) - { return "Se3TrackerPoseRecord(data=" + std::string(self.data ? "Se3TrackerPoseT(...)" : "None") + ")"; }); - - py::class_>(m, "Se3TrackerPoseTrackedT") - .def(py::init<>()) - .def(py::init( - [](const Se3TrackerPoseT& data) - { - auto obj = std::make_shared(); - obj->data = std::make_shared(data); - return obj; - }), - py::arg("data")) - .def_property_readonly( - "data", [](const Se3TrackerPoseTrackedT& self) -> std::shared_ptr { return self.data; }) - .def("__repr__", - [](const Se3TrackerPoseTrackedT& self) { - return std::string("Se3TrackerPoseTrackedT(data=") + (self.data ? "Se3TrackerPoseT(...)" : "None") + ")"; - }); + bind_record(m, "Se3TrackerPoseRecord", "Se3TrackerPose"); } } // namespace core diff --git a/src/core/schema_tests/cpp/CMakeLists.txt b/src/core/schema_tests/cpp/CMakeLists.txt index a60d932b6..0c0ef3a12 100644 --- a/src/core/schema_tests/cpp/CMakeLists.txt +++ b/src/core/schema_tests/cpp/CMakeLists.txt @@ -12,6 +12,7 @@ add_executable(schema_tests test_controller.cpp test_pedals.cpp test_se3_tracker.cpp + test_serialized.cpp ) target_link_libraries(schema_tests PRIVATE isaacteleop_schema diff --git a/src/core/schema_tests/cpp/test_full_body.cpp b/src/core/schema_tests/cpp/test_full_body.cpp index 33cebec4b..85385f501 100644 --- a/src/core/schema_tests/cpp/test_full_body.cpp +++ b/src/core/schema_tests/cpp/test_full_body.cpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 // Unit tests for the generated FullBodyPose FlatBuffer message. @@ -64,8 +64,6 @@ static_assert(sizeof(core::BodyJoints) == 24 * sizeof(core::BodyJointPose), // Type aliases resolve to the renamed generated types. static_assert(std::is_same_v); static_assert(std::is_same_v); -static_assert(std::is_same_v); -static_assert(std::is_same_v); static_assert(std::is_same_v); static_assert(std::is_same_v); static_assert(std::is_same_v); diff --git a/src/core/schema_tests/cpp/test_pedals.cpp b/src/core/schema_tests/cpp/test_pedals.cpp index 204ba78cf..31c615324 100644 --- a/src/core/schema_tests/cpp/test_pedals.cpp +++ b/src/core/schema_tests/cpp/test_pedals.cpp @@ -9,6 +9,7 @@ // Include generated FlatBuffer headers. #include +#include #include // ============================================================================= @@ -64,37 +65,58 @@ TEST_CASE("Generic3AxisPedalOutputT can store full output", "[pedals][native]") } // ============================================================================= -// Generic3AxisPedalOutputTrackedT Tests +// Optionality Tests +// +// A tracker expresses "no pedal data" with an empty Serialized handle rather than a +// wrapper table holding a null payload, so these cover the handle's absent state. // ============================================================================= -TEST_CASE("Generic3AxisPedalOutputTrackedT default construction has null data", "[pedals][tracked]") +TEST_CASE("Serialized pedal output is empty by default", "[pedals][tracked]") { - core::Generic3AxisPedalOutputTrackedT tracked; + core::Serialized data; - CHECK(tracked.data == nullptr); + CHECK_FALSE(static_cast(data)); + CHECK(data.get() == nullptr); } -TEST_CASE("Generic3AxisPedalOutputTrackedT with data assigned", "[pedals][tracked]") +TEST_CASE("Serialized pedal output round-trips its fields", "[pedals][tracked]") { - core::Generic3AxisPedalOutputTrackedT tracked; - tracked.data = std::make_shared(); - tracked.data->left_pedal = 0.5f; - tracked.data->right_pedal = 0.3f; - tracked.data->rudder = -0.2f; - - CHECK(tracked.data->left_pedal == Catch::Approx(0.5f)); - CHECK(tracked.data->right_pedal == Catch::Approx(0.3f)); - CHECK(tracked.data->rudder == Catch::Approx(-0.2f)); + core::Generic3AxisPedalOutputT native; + native.left_pedal = 0.5f; + native.right_pedal = 0.3f; + native.rudder = -0.2f; + + const auto data = core::pack(native); + + REQUIRE(static_cast(data)); + CHECK(data->left_pedal() == Catch::Approx(0.5f)); + CHECK(data->right_pedal() == Catch::Approx(0.3f)); + CHECK(data->rudder() == Catch::Approx(-0.2f)); +} + +TEST_CASE("Serialized pedal output can be returned to empty", "[pedals][tracked]") +{ + core::Generic3AxisPedalOutputT native; + native.left_pedal = 0.8f; + + auto data = core::pack(native); + REQUIRE(static_cast(data)); + + data = core::Serialized(); + + CHECK_FALSE(static_cast(data)); } -TEST_CASE("Generic3AxisPedalOutputTrackedT data can be reset to null", "[pedals][tracked]") +TEST_CASE("pack_optional maps a null payload onto an empty handle", "[pedals][tracked]") { - core::Generic3AxisPedalOutputTrackedT tracked; - tracked.data = std::make_shared(); - tracked.data->left_pedal = 0.8f; + const std::shared_ptr absent; + CHECK_FALSE(static_cast(core::pack_optional(absent))); - tracked.data.reset(); + auto present = std::make_shared(); + present->rudder = 0.25f; + const auto data = core::pack_optional(present); - CHECK(tracked.data == nullptr); + REQUIRE(static_cast(data)); + CHECK(data->rudder() == Catch::Approx(0.25f)); } TEST_CASE("Generic3AxisPedalOutputRecord serialization with tracked data", "[pedals][tracked][serialize]") diff --git a/src/core/schema_tests/cpp/test_se3_tracker.cpp b/src/core/schema_tests/cpp/test_se3_tracker.cpp index db41b850a..4cdbb7ffa 100644 --- a/src/core/schema_tests/cpp/test_se3_tracker.cpp +++ b/src/core/schema_tests/cpp/test_se3_tracker.cpp @@ -24,7 +24,6 @@ static_assert(core::Se3TrackerPose::VT_POSE == VT(0)); static_assert(core::Se3TrackerPose::VT_IS_VALID == VT(1)); -static_assert(core::Se3TrackerPoseTracked::VT_DATA == VT(0)); static_assert(core::Se3TrackerPoseRecord::VT_DATA == VT(0)); static_assert(core::Se3TrackerPoseRecord::VT_TIMESTAMP == VT(1)); diff --git a/src/core/schema_tests/cpp/test_serialized.cpp b/src/core/schema_tests/cpp/test_serialized.cpp new file mode 100644 index 000000000..8b47b5927 --- /dev/null +++ b/src/core/schema_tests/cpp/test_serialized.cpp @@ -0,0 +1,48 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Unit tests for narrowing a Serialized handle onto a table nested in its buffer. +// The handle's other states (empty, packed, pack_optional) are covered in test_pedals.cpp. + +#include +#include +#include +#include +#include + +#include + +TEST_CASE("Narrowing to a nested table shares the parent buffer", "[serialized]") +{ + auto joint = std::make_shared(); + joint->name = "wrist"; + joint->position = 0.75f; + + core::JointStateOutputT native; + native.joints.push_back(std::move(joint)); + + const auto output = core::pack(native); + core::Serialized nested = output.narrow(output->joints()->Get(0)); + + REQUIRE(nested); + CHECK(nested->name()->str() == "wrist"); + + // The narrowed handle keeps the buffer alive on its own: one allocation backs a whole + // tree of views, and a copy outlives the handle it came from. + core::Serialized survivor = nested; + nested.reset(); + CHECK(!nested); + CHECK(survivor->position() == 0.75f); +} + +TEST_CASE("Narrowing an absent nested table yields an empty handle", "[serialized]") +{ + core::JointStateOutputRecordT native; + native.timestamp = std::make_shared(1, 2, 3); + + const auto record = core::pack(native); + + REQUIRE(record); + REQUIRE(record->data() == nullptr); + CHECK(!record.narrow(record->data())); +} diff --git a/src/core/schema_tests/python/test_camera.py b/src/core/schema_tests/python/test_camera.py index ca8c441d0..1a8f4881b 100644 --- a/src/core/schema_tests/python/test_camera.py +++ b/src/core/schema_tests/python/test_camera.py @@ -50,28 +50,26 @@ def test_repr(self): class TestFrameMetadataOakStream: - """Tests for FrameMetadataOak stream property.""" + """Tests that the stream field round-trips through the encoding.""" - def test_set_stream_color(self): - metadata = FrameMetadataOak() - metadata.stream = StreamType.Color - assert metadata.stream == StreamType.Color + def test_stream_color(self): + assert FrameMetadataOak(stream=StreamType.Color).stream == StreamType.Color - def test_set_stream_mono_left(self): - metadata = FrameMetadataOak() - metadata.stream = StreamType.MonoLeft - assert metadata.stream == StreamType.MonoLeft + def test_stream_mono_left(self): + assert ( + FrameMetadataOak(stream=StreamType.MonoLeft).stream == StreamType.MonoLeft + ) - def test_set_stream_mono_right(self): - metadata = FrameMetadataOak() - metadata.stream = StreamType.MonoRight - assert metadata.stream == StreamType.MonoRight + def test_stream_mono_right(self): + assert ( + FrameMetadataOak(stream=StreamType.MonoRight).stream == StreamType.MonoRight + ) - def test_overwrite_stream(self): - metadata = FrameMetadataOak() - metadata.stream = StreamType.MonoLeft - metadata.stream = StreamType.MonoRight - assert metadata.stream == StreamType.MonoRight + def test_each_encoding_is_independent(self): + left = FrameMetadataOak(stream=StreamType.MonoLeft) + right = FrameMetadataOak(stream=StreamType.MonoRight) + assert left.stream == StreamType.MonoLeft + assert right.stream == StreamType.MonoRight class TestFrameMetadataOakSequenceNumber: @@ -81,14 +79,11 @@ def test_default_sequence_number(self): metadata = FrameMetadataOak() assert metadata.sequence_number == 0 - def test_set_sequence_number(self): - metadata = FrameMetadataOak() - metadata.sequence_number = 42 - assert metadata.sequence_number == 42 + def test_sequence_number(self): + assert FrameMetadataOak(sequence_number=42).sequence_number == 42 def test_large_sequence_number(self): - metadata = FrameMetadataOak() - metadata.sequence_number = 2**64 - 1 + metadata = FrameMetadataOak(sequence_number=2**64 - 1) assert metadata.sequence_number == 2**64 - 1 @@ -96,9 +91,7 @@ class TestFrameMetadataOakCombined: """Tests for FrameMetadataOak with multiple fields set.""" def test_full_metadata(self): - metadata = FrameMetadataOak() - metadata.stream = StreamType.MonoLeft - metadata.sequence_number = 99 + metadata = FrameMetadataOak(stream=StreamType.MonoLeft, sequence_number=99) assert metadata.stream == StreamType.MonoLeft assert metadata.sequence_number == 99 @@ -108,9 +101,7 @@ class TestFrameMetadataOakScenarios: """Tests for realistic OAK frame metadata scenarios.""" def test_first_frame(self): - metadata = FrameMetadataOak() - metadata.stream = StreamType.Color - metadata.sequence_number = 0 + metadata = FrameMetadataOak(stream=StreamType.Color, sequence_number=0) assert metadata.stream == StreamType.Color assert metadata.sequence_number == 0 @@ -118,17 +109,13 @@ def test_first_frame(self): def test_multi_stream(self): streams = [StreamType.Color, StreamType.MonoLeft, StreamType.MonoRight] for stream in streams: - metadata = FrameMetadataOak() - metadata.stream = stream - metadata.sequence_number = 5 + metadata = FrameMetadataOak(stream=stream, sequence_number=5) assert metadata.stream == stream def test_streaming_with_sequence_numbers(self): for i in range(5): - metadata = FrameMetadataOak() - metadata.stream = StreamType.Color - metadata.sequence_number = i + metadata = FrameMetadataOak(stream=StreamType.Color, sequence_number=i) assert metadata.stream == StreamType.Color assert metadata.sequence_number == i @@ -137,16 +124,12 @@ def test_streaming_with_sequence_numbers(self): class TestFrameMetadataOakEdgeCases: """Edge case tests for FrameMetadataOak table.""" - def test_overwrite_sequence_number(self): - metadata = FrameMetadataOak() - metadata.sequence_number = 10 - metadata.sequence_number = 20 - assert metadata.sequence_number == 20 + def test_zero_and_max_sequence_numbers_encode(self): + assert FrameMetadataOak(sequence_number=0).sequence_number == 0 + assert FrameMetadataOak(sequence_number=2**64 - 1).sequence_number == 2**64 - 1 def test_repr_with_all_fields(self): - metadata = FrameMetadataOak() - metadata.stream = StreamType.MonoRight - metadata.sequence_number = 7 + metadata = FrameMetadataOak(stream=StreamType.MonoRight, sequence_number=7) repr_str = repr(metadata) assert "FrameMetadataOak" in repr_str @@ -158,9 +141,7 @@ class TestFrameMetadataOakRecordTimestamp: def test_construction_with_timestamp(self): """Test FrameMetadataOakRecord carries DeviceDataTimestamp.""" - data = FrameMetadataOak() - data.stream = StreamType.MonoLeft - data.sequence_number = 42 + data = FrameMetadataOak(stream=StreamType.MonoLeft, sequence_number=42) ts = DeviceDataTimestamp(1000000000, 2000000000, 3000000000) record = FrameMetadataOakRecord(data, ts) diff --git a/src/core/schema_tests/python/test_full_body.py b/src/core/schema_tests/python/test_full_body.py index 4f472439b..6f55a7d47 100644 --- a/src/core/schema_tests/python/test_full_body.py +++ b/src/core/schema_tests/python/test_full_body.py @@ -1,9 +1,9 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Unit tests for FullBodyPoseT and related types in isaacteleop.schema. +"""Unit tests for FullBodyPose and related types in isaacteleop.schema. -FullBodyPoseT is a FlatBuffers table that represents full body pose data: +FullBodyPose is a FlatBuffers table that represents full body pose data: - joints: BodyJoints struct containing 24 BodyJointPose entries (XR_BD_body_tracking) BodyJoints is a struct with a fixed-size array of 24 BodyJointPose entries. @@ -12,7 +12,7 @@ - pose: The Pose (position and orientation) - is_valid: Whether this joint data is valid -Timestamps are carried by FullBodyPoseRecord, not FullBodyPoseT. +Timestamps are carried by FullBodyPoseRecord, not FullBodyPose. Joint indices follow XrBodyJointBD enum: 0: Pelvis, 1-2: Left/Right Hip, 3: Spine1, 4-5: Left/Right Knee, @@ -27,7 +27,7 @@ import pytest from isaacteleop.schema import ( - FullBodyPoseT, + FullBodyPose, FullBodyPoseRecord, BodyJoints, BodyJointPose, @@ -202,7 +202,7 @@ def test_views_write_through_to_schema(self): def test_views_keep_owner_alive(self): """A view outlives the last direct reference to the table it came from.""" - pose = FullBodyPoseT() + pose = FullBodyPose() positions = pose.joints.positions expected = np.arange(int(BodyJoint.NUM_JOINTS) * 3, dtype=np.float32).reshape( -1, 3 @@ -236,11 +236,11 @@ def test_repr(self): class TestFullBodyPoseTConstruction: - """Tests for FullBodyPoseT construction and basic properties.""" + """Tests for FullBodyPose construction and basic properties.""" def test_default_construction(self): - """Test default construction creates FullBodyPoseT with pre-populated joints.""" - body_pose = FullBodyPoseT() + """Test default construction creates FullBodyPose with pre-populated joints.""" + body_pose = FullBodyPose() assert body_pose is not None assert body_pose.joints is not None @@ -248,20 +248,20 @@ def test_default_construction(self): def test_parameterized_construction(self): """Test construction with joints.""" joints = BodyJoints() - body_pose = FullBodyPoseT(joints) + body_pose = FullBodyPose(joints) assert body_pose.joints is not None class TestFullBodyPoseTRepr: - """Tests for FullBodyPoseT __repr__ method.""" + """Tests for FullBodyPose __repr__ method.""" def test_repr_default(self): """Test __repr__ with default construction.""" - body_pose = FullBodyPoseT() + body_pose = FullBodyPose() repr_str = repr(body_pose) - assert "FullBodyPoseT" in repr_str + assert "FullBodyPose" in repr_str class TestBodyJointEnum: @@ -369,7 +369,7 @@ class TestFullBodyPoseRecordTimestamp: def test_construction_with_timestamp(self): """Test FullBodyPoseRecord carries DeviceDataTimestamp.""" - data = FullBodyPoseT() + data = FullBodyPose() ts = DeviceDataTimestamp(1000000000, 2000000000, 3000000000) record = FullBodyPoseRecord(data, ts) @@ -385,7 +385,7 @@ def test_default_construction(self): def test_timestamp_fields(self): """Test all three DeviceDataTimestamp fields are accessible.""" - data = FullBodyPoseT() + data = FullBodyPose() ts = DeviceDataTimestamp(111, 222, 333) record = FullBodyPoseRecord(data, ts) @@ -403,8 +403,7 @@ def test_aliases_resolve_and_warn(self): from isaacteleop import schema cases = [ - ("FullBodyPosePicoT", "FullBodyPoseT"), - ("FullBodyPosePicoTrackedT", "FullBodyPoseTrackedT"), + ("FullBodyPosePicoT", "FullBodyPose"), ("FullBodyPosePicoRecord", "FullBodyPoseRecord"), ("BodyJointsPico", "BodyJoints"), ("BodyJointPico", "BodyJoint"), diff --git a/src/core/schema_tests/python/test_hand.py b/src/core/schema_tests/python/test_hand.py index 34fd31209..dc63c26c4 100644 --- a/src/core/schema_tests/python/test_hand.py +++ b/src/core/schema_tests/python/test_hand.py @@ -1,9 +1,9 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Unit tests for HandPoseT and related types in isaacteleop.schema. +"""Unit tests for HandPose and related types in isaacteleop.schema. -HandPoseT is a FlatBuffers table that represents hand pose data: +HandPose is a FlatBuffers table that represents hand pose data: - joints: HandJoints struct with a fixed-size poses array (length HandJoint.NUM_JOINTS; OpenXR order) HandJoints is a struct with a fixed-size array of HandJointPose (length HandJoint.NUM_JOINTS). @@ -13,7 +13,7 @@ - is_valid: Whether this joint data is valid - radius: The radius of the joint (from OpenXR) -Timestamps are carried by HandPoseRecord, not HandPoseT. +Timestamps are carried by HandPoseRecord, not HandPose. """ import gc @@ -27,7 +27,7 @@ HandJointPose, HandJoints, HandPoseRecord, - HandPoseT, + HandPose, Point, Pose, Quaternion, @@ -216,7 +216,7 @@ def test_views_write_through_to_schema(self): def test_views_keep_owner_alive(self): """A view outlives the last direct reference to the table it came from.""" - pose = HandPoseT() + pose = HandPose() positions = pose.joints.positions expected = np.arange(int(HandJoint.NUM_JOINTS) * 3, dtype=np.float32).reshape( -1, 3 @@ -250,11 +250,11 @@ def test_repr(self): class TestHandPoseTConstruction: - """Tests for HandPoseT construction and basic properties.""" + """Tests for HandPose construction and basic properties.""" def test_default_construction(self): - """Test default construction creates HandPoseT with pre-populated joints.""" - hand_pose = HandPoseT() + """Test default construction creates HandPose with pre-populated joints.""" + hand_pose = HandPose() assert hand_pose is not None assert hand_pose.joints is not None @@ -262,27 +262,27 @@ def test_default_construction(self): def test_parameterized_construction(self): """Test construction with joints.""" joints = HandJoints() - hand_pose = HandPoseT(joints) + hand_pose = HandPose(joints) assert hand_pose.joints is not None class TestHandPoseTRepr: - """Tests for HandPoseT __repr__ method.""" + """Tests for HandPose __repr__ method.""" def test_repr_default(self): """Test __repr__ with default construction.""" - hand_pose = HandPoseT() + hand_pose = HandPose() repr_str = repr(hand_pose) - assert "HandPoseT" in repr_str + assert "HandPose" in repr_str def test_repr_with_values(self): """Test __repr__ with joints set.""" - hand_pose = HandPoseT(HandJoints()) + hand_pose = HandPose(HandJoints()) repr_str = repr(hand_pose) - assert "HandPoseT" in repr_str + assert "HandPose" in repr_str class TestHandPoseRecordTimestamp: @@ -290,7 +290,7 @@ class TestHandPoseRecordTimestamp: def test_construction_with_timestamp(self): """Test HandPoseRecord carries DeviceDataTimestamp.""" - data = HandPoseT(HandJoints()) + data = HandPose(HandJoints()) ts = DeviceDataTimestamp(1000000000, 2000000000, 3000000000) record = HandPoseRecord(data, ts) @@ -307,7 +307,7 @@ def test_default_construction(self): def test_timestamp_fields(self): """Test all three DeviceDataTimestamp fields are accessible.""" - data = HandPoseT() + data = HandPose() ts = DeviceDataTimestamp(111, 222, 333) record = HandPoseRecord(data, ts) diff --git a/src/core/schema_tests/python/test_head.py b/src/core/schema_tests/python/test_head.py index 4894edde8..c3378deff 100644 --- a/src/core/schema_tests/python/test_head.py +++ b/src/core/schema_tests/python/test_head.py @@ -1,13 +1,13 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Unit tests for HeadPoseT in isaacteleop.schema. +"""Unit tests for HeadPose in isaacteleop.schema. -HeadPoseT is a FlatBuffers table that represents head pose data: +HeadPose is a FlatBuffers table that represents head pose data: - pose: The Pose struct (position and orientation) - is_valid: Whether the head pose data is valid -Timestamps are carried by HeadPoseRecord, not HeadPoseT. +Timestamps are carried by HeadPoseRecord, not HeadPose. Note: Python code should only READ this data (created by C++ trackers), not modify it. """ @@ -15,7 +15,7 @@ import pytest from isaacteleop.schema import ( - HeadPoseT, + HeadPose, HeadPoseRecord, Pose, Point, @@ -25,11 +25,11 @@ class TestHeadPoseTConstruction: - """Tests for HeadPoseT construction and basic properties.""" + """Tests for HeadPose construction and basic properties.""" def test_default_construction(self): - """Test default construction creates HeadPoseT with default-initialized fields.""" - head_pose = HeadPoseT() + """Test default construction creates HeadPose with default-initialized fields.""" + head_pose = HeadPose() assert head_pose is not None assert head_pose.pose is not None @@ -38,7 +38,7 @@ def test_default_construction(self): def test_parameterized_construction(self): """Test construction with pose and is_valid.""" pose = Pose(Point(1.0, 2.0, 3.0), Quaternion(0.0, 0.0, 0.0, 1.0)) - head_pose = HeadPoseT(pose, True) + head_pose = HeadPose(pose, True) assert head_pose.pose.position.x == pytest.approx(1.0) assert head_pose.pose.position.y == pytest.approx(2.0) @@ -48,22 +48,22 @@ def test_parameterized_construction(self): class TestHeadPoseTRepr: - """Tests for HeadPoseT __repr__ method.""" + """Tests for HeadPose __repr__ method.""" def test_repr_default(self): """Test __repr__ with default construction.""" - head_pose = HeadPoseT() + head_pose = HeadPose() repr_str = repr(head_pose) - assert "HeadPoseT" in repr_str + assert "HeadPose" in repr_str def test_repr_with_values(self): """Test __repr__ with parameterized construction.""" pose = Pose(Point(1.0, 2.0, 3.0), Quaternion(0.0, 0.0, 0.0, 1.0)) - head_pose = HeadPoseT(pose, True) + head_pose = HeadPose(pose, True) repr_str = repr(head_pose) - assert "HeadPoseT" in repr_str + assert "HeadPose" in repr_str assert "is_valid=True" in repr_str @@ -73,7 +73,7 @@ class TestHeadPoseRecordTimestamp: def test_construction_with_timestamp(self): """Test HeadPoseRecord carries DeviceDataTimestamp.""" pose = Pose(Point(1.0, 2.0, 3.0), Quaternion(0.0, 0.0, 0.0, 1.0)) - data = HeadPoseT(pose, True) + data = HeadPose(pose, True) ts = DeviceDataTimestamp(1000000000, 2000000000, 3000000000) record = HeadPoseRecord(data, ts) @@ -90,7 +90,7 @@ def test_default_construction(self): def test_timestamp_fields(self): """Test all three DeviceDataTimestamp fields are accessible.""" - data = HeadPoseT() + data = HeadPose() ts = DeviceDataTimestamp(111, 222, 333) record = HeadPoseRecord(data, ts) diff --git a/src/core/schema_tests/python/test_oglo_tactile.py b/src/core/schema_tests/python/test_oglo_tactile.py index 0ce96bd00..3d06d4a9e 100644 --- a/src/core/schema_tests/python/test_oglo_tactile.py +++ b/src/core/schema_tests/python/test_oglo_tactile.py @@ -6,13 +6,12 @@ Tests the following FlatBuffers types: - OgloGloveSample: tactile glove sample (seq, device_time_us, 80 taxels, 6-axis IMU) - OgloGloveSampleRecord: record wrapper carrying DeviceDataTimestamp -- OgloGloveSampleTrackedT: tracked wrapper (data is None when inactive) +- OgloGloveSample: tracked wrapper (data is None when inactive) """ from isaacteleop.schema import ( - OgloGloveSample, OgloGloveSampleRecord, - OgloGloveSampleTrackedT, + OgloGloveSample, ) NUM_TAXELS = 80 @@ -28,12 +27,17 @@ def test_default_construction(self): assert list(s.taxels) == [] def test_field_round_trip(self): - s = OgloGloveSample() - s.seq = 12345 - s.device_time_us = 6_000_000 - s.taxels = list(range(NUM_TAXELS)) - s.accel_x, s.accel_y, s.accel_z = 100, -200, 4000 - s.gyro_x, s.gyro_y, s.gyro_z = 1, -2, 3 + s = OgloGloveSample( + seq=12345, + device_time_us=6_000_000, + taxels=list(range(NUM_TAXELS)), + accel_x=100, + accel_y=-200, + accel_z=4000, + gyro_x=1, + gyro_y=-2, + gyro_z=3, + ) assert s.seq == 12345 assert s.device_time_us == 6_000_000 @@ -46,16 +50,6 @@ def test_repr(self): assert "OgloGloveSample" in repr(OgloGloveSample()) -class TestOgloGloveSampleTrackedT: - """Tests for the tracked wrapper.""" - - def test_default_construction_inactive(self): - assert OgloGloveSampleTrackedT().data is None - - def test_repr_inactive(self): - assert "None" in repr(OgloGloveSampleTrackedT()) - - class TestOgloGloveSampleRecord: """Tests for the MCAP record wrapper.""" diff --git a/src/core/schema_tests/python/test_pedals.py b/src/core/schema_tests/python/test_pedals.py index ddcae524f..a6c55b904 100644 --- a/src/core/schema_tests/python/test_pedals.py +++ b/src/core/schema_tests/python/test_pedals.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 """Unit tests for Generic3AxisPedalOutput type in isaacteleop.schema. @@ -6,7 +6,7 @@ Tests the following FlatBuffers types: - Generic3AxisPedalOutput: Table with left_pedal, right_pedal, and rudder - Generic3AxisPedalOutputRecord: Record wrapper carrying DeviceDataTimestamp -- Generic3AxisPedalOutputTrackedT: Tracked wrapper (data is None when inactive) +- Generic3AxisPedalOutput: Tracked wrapper (data is None when inactive) Timestamps are carried by Generic3AxisPedalOutputRecord, not Generic3AxisPedalOutput. """ @@ -14,9 +14,8 @@ import pytest from isaacteleop.schema import ( - Generic3AxisPedalOutput, Generic3AxisPedalOutputRecord, - Generic3AxisPedalOutputTrackedT, + Generic3AxisPedalOutput, DeviceDataTimestamp, ) @@ -41,35 +40,29 @@ def test_repr(self): class TestGeneric3AxisPedalOutputPedals: - """Tests for Generic3AxisPedalOutput pedal properties.""" + """Tests that each pedal field round-trips through the encoding.""" - def test_set_left_pedal(self): - """Test setting left pedal value.""" - output = Generic3AxisPedalOutput() - output.left_pedal = 0.75 + def test_left_pedal(self): + """Test encoding left pedal value.""" + output = Generic3AxisPedalOutput(left_pedal=0.75) assert output.left_pedal == pytest.approx(0.75) - def test_set_right_pedal(self): - """Test setting right pedal value.""" - output = Generic3AxisPedalOutput() - output.right_pedal = 0.5 + def test_right_pedal(self): + """Test encoding right pedal value.""" + output = Generic3AxisPedalOutput(right_pedal=0.5) assert output.right_pedal == pytest.approx(0.5) - def test_set_rudder(self): - """Test setting rudder value.""" - output = Generic3AxisPedalOutput() - output.rudder = -0.33 + def test_rudder(self): + """Test encoding rudder value.""" + output = Generic3AxisPedalOutput(rudder=-0.33) assert output.rudder == pytest.approx(-0.33) - def test_set_all_pedal_values(self): - """Test setting all pedal values.""" - output = Generic3AxisPedalOutput() - output.left_pedal = 0.8 - output.right_pedal = 0.2 - output.rudder = 0.5 + def test_all_pedal_values(self): + """Test encoding all pedal values.""" + output = Generic3AxisPedalOutput(left_pedal=0.8, right_pedal=0.2, rudder=0.5) assert output.left_pedal == pytest.approx(0.8) assert output.right_pedal == pytest.approx(0.2) @@ -81,10 +74,7 @@ class TestGeneric3AxisPedalOutputCombined: def test_full_output(self): """Test with all fields set.""" - output = Generic3AxisPedalOutput() - output.left_pedal = 1.0 - output.right_pedal = 0.0 - output.rudder = -0.5 + output = Generic3AxisPedalOutput(left_pedal=1.0, right_pedal=0.0, rudder=-0.5) assert output.left_pedal == pytest.approx(1.0) assert output.right_pedal == pytest.approx(0.0) @@ -96,10 +86,7 @@ class TestGeneric3AxisPedalOutputScenarios: def test_full_forward_press(self): """Test full forward press on both pedals.""" - output = Generic3AxisPedalOutput() - output.left_pedal = 1.0 - output.right_pedal = 1.0 - output.rudder = 0.0 + output = Generic3AxisPedalOutput(left_pedal=1.0, right_pedal=1.0, rudder=0.0) assert output.left_pedal == pytest.approx(1.0) assert output.right_pedal == pytest.approx(1.0) @@ -107,28 +94,24 @@ def test_full_forward_press(self): def test_left_turn_with_rudder(self): """Test left turn using rudder.""" - output = Generic3AxisPedalOutput() - output.left_pedal = 0.5 - output.right_pedal = 0.5 - output.rudder = -1.0 # Full left rudder. + output = Generic3AxisPedalOutput( + left_pedal=0.5, right_pedal=0.5, rudder=-1.0 + ) # Full left rudder. assert output.rudder == pytest.approx(-1.0) def test_right_turn_with_rudder(self): """Test right turn using rudder.""" - output = Generic3AxisPedalOutput() - output.left_pedal = 0.5 - output.right_pedal = 0.5 - output.rudder = 1.0 # Full right rudder. + output = Generic3AxisPedalOutput( + left_pedal=0.5, right_pedal=0.5, rudder=1.0 + ) # Full right rudder. assert output.rudder == pytest.approx(1.0) def test_differential_braking(self): """Test differential braking scenario.""" - output = Generic3AxisPedalOutput() - output.left_pedal = 0.0 # Left brake applied. - output.right_pedal = 0.8 # Right pedal pressed. - output.rudder = 0.0 + # Left brake applied, right pedal pressed. + output = Generic3AxisPedalOutput(left_pedal=0.0, right_pedal=0.8, rudder=0.0) assert output.left_pedal == pytest.approx(0.0) assert output.right_pedal == pytest.approx(0.8) @@ -136,9 +119,6 @@ def test_differential_braking(self): def test_neutral_position(self): """Test neutral/idle position.""" output = Generic3AxisPedalOutput() - output.left_pedal = 0.0 - output.right_pedal = 0.0 - output.rudder = 0.0 assert output.left_pedal == pytest.approx(0.0) assert output.right_pedal == pytest.approx(0.0) @@ -150,10 +130,9 @@ class TestGeneric3AxisPedalOutputEdgeCases: def test_negative_pedal_values(self): """Test with negative pedal values (edge case).""" - output = Generic3AxisPedalOutput() - output.left_pedal = -0.5 - output.right_pedal = -0.25 - output.rudder = -1.0 + output = Generic3AxisPedalOutput( + left_pedal=-0.5, right_pedal=-0.25, rudder=-1.0 + ) assert output.left_pedal == pytest.approx(-0.5) assert output.right_pedal == pytest.approx(-0.25) @@ -161,68 +140,44 @@ def test_negative_pedal_values(self): def test_values_greater_than_one(self): """Test with pedal values exceeding typical range.""" - output = Generic3AxisPedalOutput() - output.left_pedal = 1.5 - output.right_pedal = 2.0 - output.rudder = 1.5 + output = Generic3AxisPedalOutput(left_pedal=1.5, right_pedal=2.0, rudder=1.5) assert output.left_pedal == pytest.approx(1.5) assert output.right_pedal == pytest.approx(2.0) assert output.rudder == pytest.approx(1.5) - def test_overwrite_left_pedal(self): - """Test overwriting left pedal value.""" - output = Generic3AxisPedalOutput() - output.left_pedal = 0.5 - output.left_pedal = 0.9 - - assert output.left_pedal == pytest.approx(0.9) - - def test_overwrite_right_pedal(self): - """Test overwriting right pedal value.""" - output = Generic3AxisPedalOutput() - output.right_pedal = 0.3 - output.right_pedal = 0.7 - - assert output.right_pedal == pytest.approx(0.7) - - def test_overwrite_rudder(self): - """Test overwriting rudder value.""" - output = Generic3AxisPedalOutput() - output.rudder = 0.2 - output.rudder = -0.8 - - assert output.rudder == pytest.approx(-0.8) + def test_encodings_are_independent(self): + """Test each encoding carries its own values, not a shared buffer's.""" + first = Generic3AxisPedalOutput(left_pedal=0.5, right_pedal=0.3, rudder=0.2) + second = Generic3AxisPedalOutput(left_pedal=0.9, right_pedal=0.7, rudder=-0.8) + assert first.left_pedal == pytest.approx(0.5) + assert first.right_pedal == pytest.approx(0.3) + assert first.rudder == pytest.approx(0.2) + assert second.left_pedal == pytest.approx(0.9) + assert second.right_pedal == pytest.approx(0.7) + assert second.rudder == pytest.approx(-0.8) -class TestGeneric3AxisPedalOutputTrackedT: - """Tests for Generic3AxisPedalOutputTrackedT tracked wrapper.""" - def test_default_construction_inactive(self): - """Default-constructed TrackedT has data=None (inactive).""" - tracked = Generic3AxisPedalOutputTrackedT() - assert tracked.data is None +class TestGeneric3AxisPedalOutputEncoding: + """Tests that an encoded payload reads back. - def test_construction_with_data(self): - """TrackedT constructed with data wraps the payload correctly.""" - output = Generic3AxisPedalOutput(0.8, 0.2, -0.5) - tracked = Generic3AxisPedalOutputTrackedT(output) + A tracker with no pedal data returns None rather than an empty payload, so + absence needs no case here; the source-node tests cover feeding None through. + """ - assert tracked.data is not None - assert tracked.data.left_pedal == pytest.approx(0.8) - assert tracked.data.right_pedal == pytest.approx(0.2) - assert tracked.data.rudder == pytest.approx(-0.5) + def test_encoded_payload_reads_back(self): + """An encoded payload gates as True and its fields read directly.""" + output = Generic3AxisPedalOutput(left_pedal=0.8, right_pedal=0.2, rudder=-0.5) - def test_repr_inactive(self): - """Repr of inactive TrackedT mentions None.""" - tracked = Generic3AxisPedalOutputTrackedT() - assert "None" in repr(tracked) + assert output + assert output.left_pedal == pytest.approx(0.8) + assert output.right_pedal == pytest.approx(0.2) + assert output.rudder == pytest.approx(-0.5) - def test_repr_active(self): - """Repr of active TrackedT mentions the payload type.""" - output = Generic3AxisPedalOutput() - tracked = Generic3AxisPedalOutputTrackedT(output) - assert "Generic3AxisPedalOutput" in repr(tracked) + def test_repr_present(self): + """Repr of a present payload names the type.""" + assert "Generic3AxisPedalOutput" in repr(Generic3AxisPedalOutput()) class TestGeneric3AxisPedalOutputRecordTimestamp: @@ -230,7 +185,7 @@ class TestGeneric3AxisPedalOutputRecordTimestamp: def test_construction_with_timestamp(self): """Test Generic3AxisPedalOutputRecord carries DeviceDataTimestamp.""" - data = Generic3AxisPedalOutput(0.8, 0.2, 0.5) + data = Generic3AxisPedalOutput(left_pedal=0.8, right_pedal=0.2, rudder=0.5) ts = DeviceDataTimestamp(1000000000, 2000000000, 3000000000) record = Generic3AxisPedalOutputRecord(data, ts) diff --git a/src/core/schema_tests/python/test_se3_tracker.py b/src/core/schema_tests/python/test_se3_tracker.py index 495849438..39f7ba8df 100644 --- a/src/core/schema_tests/python/test_se3_tracker.py +++ b/src/core/schema_tests/python/test_se3_tracker.py @@ -1,14 +1,14 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Unit tests for Se3TrackerPoseT in isaacteleop.schema. +"""Unit tests for Se3TrackerPose in isaacteleop.schema. -Se3TrackerPoseT is a FlatBuffers table for a generic SE3 (6-DoF) tracker device: +Se3TrackerPose is a FlatBuffers table for a generic SE3 (6-DoF) tracker device: - pose: The Pose struct (position and orientation) - is_valid: Whether the pose data is valid (False = producer streaming, tracking lost; pose contents are then unspecified) -Timestamps are carried by Se3TrackerPoseRecord, not Se3TrackerPoseT. +Timestamps are carried by Se3TrackerPoseRecord, not Se3TrackerPose. Note: Python code should only READ this data (created by C++ trackers), not modify it. """ @@ -16,8 +16,7 @@ import pytest from isaacteleop.schema import ( - Se3TrackerPoseT, - Se3TrackerPoseTrackedT, + Se3TrackerPose, Se3TrackerPoseRecord, Pose, Point, @@ -27,11 +26,11 @@ class TestSe3TrackerPoseTConstruction: - """Tests for Se3TrackerPoseT construction and basic properties.""" + """Tests for Se3TrackerPose construction and basic properties.""" def test_default_construction(self): - """Default construction creates Se3TrackerPoseT with default-initialized fields.""" - se3_pose = Se3TrackerPoseT() + """Default construction creates Se3TrackerPose with default-initialized fields.""" + se3_pose = Se3TrackerPose() assert se3_pose is not None assert se3_pose.pose is not None @@ -40,7 +39,7 @@ def test_default_construction(self): def test_parameterized_construction(self): """Construction with pose and is_valid.""" pose = Pose(Point(1.0, 2.0, 3.0), Quaternion(0.0, 0.0, 0.0, 1.0)) - se3_pose = Se3TrackerPoseT(pose, True) + se3_pose = Se3TrackerPose(pose, True) assert se3_pose.pose.position.x == pytest.approx(1.0) assert se3_pose.pose.position.y == pytest.approx(2.0) @@ -51,28 +50,27 @@ def test_parameterized_construction(self): def test_repr(self): """__repr__ includes the type name and is_valid.""" pose = Pose(Point(1.0, 2.0, 3.0), Quaternion(0.0, 0.0, 0.0, 1.0)) - se3_pose = Se3TrackerPoseT(pose, True) + se3_pose = Se3TrackerPose(pose, True) repr_str = repr(se3_pose) - assert "Se3TrackerPoseT" in repr_str + assert "Se3TrackerPose" in repr_str assert "is_valid=True" in repr_str -class TestSe3TrackerPoseTracked: - """Tests for the Se3TrackerPoseTrackedT wrapper.""" +class TestSe3TrackerPoseEncoding: + """Tests that an encoded pose reads back. - def test_default_construction_has_no_data(self): - """Default Tracked wrapper has no data (no sample yet / collection unavailable).""" - tracked = Se3TrackerPoseTrackedT() - assert tracked.data is None + A tracker with no sample returns None rather than an empty pose, so absence + needs no case here. + """ - def test_construction_with_data(self): - """Tracked wrapper carries the payload.""" + def test_encoded_payload_reads_back(self): + """An encoded pose gates as True and its fields read directly.""" pose = Pose(Point(1.0, 2.0, 3.0), Quaternion(0.0, 0.0, 0.0, 1.0)) - tracked = Se3TrackerPoseTrackedT(Se3TrackerPoseT(pose, True)) + data = Se3TrackerPose(pose, True) - assert tracked.data is not None - assert tracked.data.is_valid is True + assert data + assert data.is_valid is True class TestSe3TrackerPoseRecordTimestamp: @@ -81,7 +79,7 @@ class TestSe3TrackerPoseRecordTimestamp: def test_construction_with_timestamp(self): """Se3TrackerPoseRecord carries DeviceDataTimestamp (positional field order).""" pose = Pose(Point(1.0, 2.0, 3.0), Quaternion(0.0, 0.0, 0.0, 1.0)) - data = Se3TrackerPoseT(pose, True) + data = Se3TrackerPose(pose, True) ts = DeviceDataTimestamp(1000000000, 2000000000, 3000000000) record = Se3TrackerPoseRecord(data, ts) diff --git a/src/plugins/controller_se3_tracker/controller_se3_tracker_plugin.cpp b/src/plugins/controller_se3_tracker/controller_se3_tracker_plugin.cpp index 8b839e1a7..41761b00d 100644 --- a/src/plugins/controller_se3_tracker/controller_se3_tracker_plugin.cpp +++ b/src/plugins/controller_se3_tracker/controller_se3_tracker_plugin.cpp @@ -75,14 +75,16 @@ void ControllerSe3TrackerPlugin::update() m_deviceio_session->update(); - const core::ControllerSnapshotTrackedT& tracked = + const core::Serialized& tracked = m_use_left_hand ? m_controller_tracker->get_left_controller(*m_deviceio_session) : m_controller_tracker->get_right_controller(*m_deviceio_session); + const core::ControllerSnapshot* snapshot = tracked.get(); + core::Se3TrackerPoseT out; - if (tracked.data && tracked.data->grip_pose && tracked.data->grip_pose->is_valid()) + if (snapshot != nullptr && snapshot->grip_pose() != nullptr && snapshot->grip_pose()->is_valid()) { - out.pose = std::make_shared(tracked.data->grip_pose->pose()); + out.pose = std::make_shared(snapshot->grip_pose()->pose()); out.is_valid = true; } else diff --git a/src/plugins/controller_synthetic_hands/synthetic_hands_plugin.cpp b/src/plugins/controller_synthetic_hands/synthetic_hands_plugin.cpp index fded2b6ab..ebdaa6a23 100644 --- a/src/plugins/controller_synthetic_hands/synthetic_hands_plugin.cpp +++ b/src/plugins/controller_synthetic_hands/synthetic_hands_plugin.cpp @@ -69,8 +69,8 @@ void SyntheticHandsPlugin::worker_thread() while (m_running) { - core::ControllerSnapshotTrackedT left_tracked; - core::ControllerSnapshotTrackedT right_tracked; + core::Serialized left_tracked; + core::Serialized right_tracked; try { // Update DeviceIOSession (handles time and tracker updates) @@ -103,10 +103,10 @@ void SyntheticHandsPlugin::worker_thread() float left_target = 0.0f; float right_target = 0.0f; - if (left_tracked.data) - left_target = left_tracked.data->inputs->trigger_value(); - if (right_tracked.data) - right_target = right_tracked.data->inputs->trigger_value(); + if (left_tracked) + left_target = left_tracked->inputs()->trigger_value(); + if (right_tracked) + right_target = right_tracked->inputs()->trigger_value(); // Smoothly interpolate float curl_delta = CURL_SPEED * FRAME_TIME; @@ -133,12 +133,12 @@ void SyntheticHandsPlugin::worker_thread() // injector. A different plugin could choose a different policy — for example, a // plugin with independent joint data (e.g. a glove) could keep pushing joints // even when no controller pose is available. - if (m_left_enabled && left_tracked.data) + if (m_left_enabled && left_tracked) { bool grip_valid = false; bool aim_valid = false; - oxr_utils::get_grip_pose(*left_tracked.data, grip_valid); - XrPosef wrist = oxr_utils::get_aim_pose(*left_tracked.data, aim_valid); + oxr_utils::get_grip_pose(*left_tracked, grip_valid); + XrPosef wrist = oxr_utils::get_aim_pose(*left_tracked, aim_valid); if (grip_valid && aim_valid) { @@ -159,12 +159,12 @@ void SyntheticHandsPlugin::worker_thread() m_left_injector.reset(); } - if (m_right_enabled && right_tracked.data) + if (m_right_enabled && right_tracked) { bool grip_valid = false; bool aim_valid = false; - oxr_utils::get_grip_pose(*right_tracked.data, grip_valid); - XrPosef wrist = oxr_utils::get_aim_pose(*right_tracked.data, aim_valid); + oxr_utils::get_grip_pose(*right_tracked, grip_valid); + XrPosef wrist = oxr_utils::get_aim_pose(*right_tracked, aim_valid); if (grip_valid && aim_valid) { diff --git a/src/plugins/haptikos/haptikos_hands_plugin.cpp b/src/plugins/haptikos/haptikos_hands_plugin.cpp index 85c4f8f8a..37dd4db43 100644 --- a/src/plugins/haptikos/haptikos_hands_plugin.cpp +++ b/src/plugins/haptikos/haptikos_hands_plugin.cpp @@ -65,11 +65,11 @@ void HaptikosHandsPlugin::worker_thread() { auto frame_start = std::chrono::steady_clock::now(); - core::ControllerSnapshotTrackedT left_tracked; - core::ControllerSnapshotTrackedT right_tracked; + core::Serialized left_tracked; + core::Serialized right_tracked; - core::HandPoseTrackedT left_hand; - core::HandPoseTrackedT rigth_hand; + core::Serialized left_hand; + core::Serialized rigth_hand; try { @@ -101,11 +101,11 @@ void HaptikosHandsPlugin::worker_thread() bool rigth_published = false; - if (right_tracked.data) + if (right_tracked) { Haptikos::HandData right_data = m_client.GetData(true, Haptikos::GlobalToWrist, true, true, false); bool valid_wrist = false; - XrPosef rigth_controller = oxr_utils::get_aim_pose(*right_tracked.data, valid_wrist); + XrPosef rigth_controller = oxr_utils::get_aim_pose(*right_tracked, valid_wrist); if (right_data.IsValid() == 1 && valid_wrist) { @@ -130,11 +130,11 @@ void HaptikosHandsPlugin::worker_thread() bool left_published = false; - if (left_tracked.data) + if (left_tracked) { Haptikos::HandData left_data = m_client.GetData(false, Haptikos::GlobalToWrist, true, true, false); bool valid_wrist = false; - XrPosef left_controller = oxr_utils::get_aim_pose(*left_tracked.data, valid_wrist); + XrPosef left_controller = oxr_utils::get_aim_pose(*left_tracked, valid_wrist); if (left_data.IsValid() == 1 && valid_wrist) { diff --git a/src/plugins/manus/core/manus_hand_tracking_plugin.cpp b/src/plugins/manus/core/manus_hand_tracking_plugin.cpp index f653eb50c..c330e29be 100644 --- a/src/plugins/manus/core/manus_hand_tracking_plugin.cpp +++ b/src/plugins/manus/core/manus_hand_tracking_plugin.cpp @@ -103,12 +103,13 @@ void ManusTracker::update() for (const std::string_view endpoint : { std::string_view("left"), std::string_view("right") }) { const auto& tracked = m_haptic_reader->get_data(*m_deviceio_session, endpoint); - if (tracked.data && tracked.data->values.size() == kManusFingerCount) + const core::HapticCommand* command = tracked.get(); + if (command != nullptr && command->values() != nullptr && command->values()->size() == kManusFingerCount) { std::array powers{}; for (size_t i = 0; i < kManusFingerCount; ++i) { - powers[i] = tracked.data->values[i]; + powers[i] = command->values()->Get(i); } apply_haptic_command(endpoint == "left", powers); } @@ -989,13 +990,13 @@ bool ManusTracker::get_controller_wrist_pose(bool is_left, XrPosef& out_wrist_po const auto& tracked = is_left ? m_controller_tracker->get_left_controller(*m_deviceio_session) : m_controller_tracker->get_right_controller(*m_deviceio_session); - if (!tracked.data) + if (!tracked) { return false; } bool aim_valid = false; - XrPosef raw_pose = oxr_utils::get_aim_pose(*tracked.data, aim_valid); + XrPosef raw_pose = oxr_utils::get_aim_pose(*tracked, aim_valid); if (!aim_valid) { diff --git a/src/plugins/plugin_utils/wrist_pose_source.cpp b/src/plugins/plugin_utils/wrist_pose_source.cpp index 38e14ef2f..4c81e9ecf 100644 --- a/src/plugins/plugin_utils/wrist_pose_source.cpp +++ b/src/plugins/plugin_utils/wrist_pose_source.cpp @@ -336,13 +336,13 @@ bool WristPoseSource::query_controller(bool is_left, XrPosef& out_pose, bool& ou const auto& tracked = is_left ? m_controller_tracker->get_left_controller(*m_deviceio_session) : m_controller_tracker->get_right_controller(*m_deviceio_session); - if (tracked.data == nullptr) + if (!tracked) { return false; } bool aim_valid = false; - const XrPosef aim_pose = oxr_utils::get_aim_pose(*tracked.data, aim_valid); + const XrPosef aim_pose = oxr_utils::get_aim_pose(*tracked, aim_valid); if (!aim_valid) { return false; diff --git a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/controllers_source.py b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/controllers_source.py index f48c5b26c..ef41979f0 100644 --- a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/controllers_source.py +++ b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/controllers_source.py @@ -23,7 +23,7 @@ if TYPE_CHECKING: from isaacteleop.deviceio import ITracker - from isaacteleop.schema import ControllerSnapshot, ControllerSnapshotTrackedT + from isaacteleop.schema import ControllerSnapshot class ControllersSource(IDeviceIOSource): @@ -80,7 +80,7 @@ def poll_tracker(self, deviceio_session: Any) -> RetargeterIO: Returns: Dict with "deviceio_controller_left" and "deviceio_controller_right" - TensorGroups containing ControllerSnapshotTrackedT wrappers. + TensorGroups containing ControllerSnapshot wrappers. """ left_tracked = self._controller_tracker.get_left_controller(deviceio_session) right_tracked = self._controller_tracker.get_right_controller(deviceio_session) @@ -111,24 +111,24 @@ def output_spec(self) -> RetargeterIOType: def _compute_fn(self, inputs: RetargeterIO, outputs: RetargeterIO, context) -> None: """ - Convert DeviceIO ControllerSnapshotTrackedT to standard ControllerInput tensors. + Convert DeviceIO ControllerSnapshot to standard ControllerInput tensors. Calls ``set_none()`` on the output when the corresponding controller is inactive. Args: - inputs: Dict with "deviceio_controller_left" and "deviceio_controller_right" TrackedT wrappers + inputs: Dict with "deviceio_controller_left" and "deviceio_controller_right" Tracked wrappers outputs: Dict with "controller_left" and "controller_right" OptionalTensorGroups context: ComputeContext (unused by this converter node). """ - left_tracked: "ControllerSnapshotTrackedT" = inputs["deviceio_controller_left"][ + left_tracked: "ControllerSnapshot | None" = inputs["deviceio_controller_left"][ 0 ] - right_tracked: "ControllerSnapshotTrackedT" = inputs[ + right_tracked: "ControllerSnapshot | None" = inputs[ "deviceio_controller_right" ][0] - self._update_controller_data(outputs["controller_left"], left_tracked.data) - self._update_controller_data(outputs["controller_right"], right_tracked.data) + self._update_controller_data(outputs["controller_left"], left_tracked) + self._update_controller_data(outputs["controller_right"], right_tracked) def _update_controller_data( self, group: OptionalTensorGroup, snapshot: "ControllerSnapshot | None" diff --git a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/deviceio_tensor_types.py b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/deviceio_tensor_types.py index cb86f3bed..08c092a71 100644 --- a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/deviceio_tensor_types.py +++ b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/deviceio_tensor_types.py @@ -2,11 +2,11 @@ # SPDX-License-Identifier: Apache-2.0 """ -DeviceIO Tensor Types - Tracked wrapper objects from DeviceIO. +DeviceIO Tensor Types - payload handles from DeviceIO trackers. -These tensor types represent the TrackedT wrapper objects returned by DeviceIO trackers. -Each TrackedT always exists (never None) and contains a `.data` property that holds -the raw flatbuffer object (or None when the tracker is inactive). +These tensor types represent the encoded payloads returned by DeviceIO trackers. +Each carries a handle over the encoded payload, read directly through the schema +accessors. An inactive device arrives as None rather than as an empty handle. """ import warnings @@ -15,154 +15,89 @@ from ..interface.tensor_type import TensorType from ..interface.tensor_group_type import TensorGroupType from isaacteleop.schema import ( - HeadPoseTrackedT, - HandPoseTrackedT, - ControllerSnapshotTrackedT, - Generic3AxisPedalOutputTrackedT, - JointStateOutputTrackedT, - FullBodyPoseTrackedT, - MessageChannelMessagesTrackedT, + HeadPose, + HandPose, + ControllerSnapshot, + Generic3AxisPedalOutput, + JointStateOutput, + FullBodyPose, + MessageChannelMessagesTracked, ) -class HeadPoseTrackedType(TensorType): - """HeadPoseTrackedT wrapper type from DeviceIO HeadTracker.""" +def _payload_tensor_type(class_name: str, payload_cls: type, doc: str) -> type: + """Build the ``TensorType`` subclass carrying one DeviceIO payload handle. - def __init__(self, name: str) -> None: - super().__init__(name) - - def _check_instance_compatibility(self, other: TensorType) -> bool: - if not isinstance(other, HeadPoseTrackedType): - raise TypeError(f"Expected HeadPoseTrackedType, got {type(other).__name__}") - return True - - def validate_value(self, value: Any) -> None: - if not isinstance(value, HeadPoseTrackedT): - raise TypeError( - f"Expected HeadPoseTrackedT for '{self.name}', got {type(value).__name__}" - ) - - -class HandPoseTrackedType(TensorType): - """HandPoseTrackedT wrapper type from DeviceIO HandTracker.""" - - def __init__(self, name: str) -> None: - super().__init__(name) - - def _check_instance_compatibility(self, other: TensorType) -> bool: - if not isinstance(other, HandPoseTrackedType): - raise TypeError(f"Expected HandPoseTrackedType, got {type(other).__name__}") - return True - - def validate_value(self, value: Any) -> None: - if not isinstance(value, HandPoseTrackedT): - raise TypeError( - f"Expected HandPoseTrackedT for '{self.name}', got {type(value).__name__}" - ) - - -class ControllerSnapshotTrackedType(TensorType): - """ControllerSnapshotTrackedT wrapper type from DeviceIO ControllerTracker.""" - - def __init__(self, name: str) -> None: - super().__init__(name) + Every payload validates the same way, so the class name and the payload class + are the only things that vary between them. + """ def _check_instance_compatibility(self, other: TensorType) -> bool: - if not isinstance(other, ControllerSnapshotTrackedType): - raise TypeError( - f"Expected ControllerSnapshotTrackedType, got {type(other).__name__}" - ) + if not isinstance(other, cls): + raise TypeError(f"Expected {class_name}, got {type(other).__name__}") return True def validate_value(self, value: Any) -> None: - if not isinstance(value, ControllerSnapshotTrackedT): + # None is how an inactive device arrives; only a wrong type is an error. + if value is not None and not isinstance(value, payload_cls): raise TypeError( - f"Expected ControllerSnapshotTrackedT for '{self.name}', got {type(value).__name__}" + f"Expected {payload_cls.__name__} for '{self.name}', got {type(value).__name__}" ) - -class Generic3AxisPedalOutputTrackedType(TensorType): - """Generic3AxisPedalOutputTrackedT wrapper type from DeviceIO Generic3AxisPedalTracker.""" - - def __init__(self, name: str) -> None: - super().__init__(name) - - def _check_instance_compatibility(self, other: TensorType) -> bool: - if not isinstance(other, Generic3AxisPedalOutputTrackedType): - raise TypeError( - f"Expected Generic3AxisPedalOutputTrackedType, got {type(other).__name__}" - ) - return True - - def validate_value(self, value: Any) -> None: - if not isinstance(value, Generic3AxisPedalOutputTrackedT): - raise TypeError( - f"Expected Generic3AxisPedalOutputTrackedT for '{self.name}', got {type(value).__name__}" - ) + cls = type( + class_name, + (TensorType,), + { + "__doc__": doc, + "__module__": __name__, + "_check_instance_compatibility": _check_instance_compatibility, + "validate_value": validate_value, + }, + ) + return cls -class JointStateOutputTrackedType(TensorType): - """JointStateOutputTrackedT wrapper type from DeviceIO JointStateTracker.""" +HeadPoseTrackedType = _payload_tensor_type( + "HeadPoseTrackedType", HeadPose, "HeadPose wrapper type from DeviceIO HeadTracker." +) - def __init__(self, name: str) -> None: - super().__init__(name) +HandPoseTrackedType = _payload_tensor_type( + "HandPoseTrackedType", HandPose, "HandPose wrapper type from DeviceIO HandTracker." +) - def _check_instance_compatibility(self, other: TensorType) -> bool: - if not isinstance(other, JointStateOutputTrackedType): - raise TypeError( - f"Expected JointStateOutputTrackedType, got {type(other).__name__}" - ) - return True +ControllerSnapshotTrackedType = _payload_tensor_type( + "ControllerSnapshotTrackedType", + ControllerSnapshot, + "ControllerSnapshot wrapper type from DeviceIO ControllerTracker.", +) - def validate_value(self, value: Any) -> None: - if not isinstance(value, JointStateOutputTrackedT): - raise TypeError( - f"Expected JointStateOutputTrackedT for '{self.name}', got {type(value).__name__}" - ) +Generic3AxisPedalOutputTrackedType = _payload_tensor_type( + "Generic3AxisPedalOutputTrackedType", + Generic3AxisPedalOutput, + "Generic3AxisPedalOutput wrapper type from DeviceIO Generic3AxisPedalTracker.", +) +JointStateOutputTrackedType = _payload_tensor_type( + "JointStateOutputTrackedType", + JointStateOutput, + "JointStateOutput wrapper type from DeviceIO JointStateTracker.", +) -class FullBodyPoseTrackedType(TensorType): - """FullBodyPoseTrackedT wrapper type from DeviceIO FullBodyTracker. +FullBodyPoseTrackedType = _payload_tensor_type( + "FullBodyPoseTrackedType", + FullBodyPose, + """FullBodyPose wrapper type from DeviceIO FullBodyTracker. - Vendor-agnostic: the full-body tracker produces the same FullBodyPoseTrackedT + Vendor-agnostic: the full-body tracker produces the same FullBodyPose payload regardless of the live vendor (native XR, pushed tensor, ...). - """ - - def __init__(self, name: str) -> None: - super().__init__(name) - - def _check_instance_compatibility(self, other: TensorType) -> bool: - if not isinstance(other, FullBodyPoseTrackedType): - raise TypeError( - f"Expected FullBodyPoseTrackedType, got {type(other).__name__}" - ) - return True - - def validate_value(self, value: Any) -> None: - if not isinstance(value, FullBodyPoseTrackedT): - raise TypeError( - f"Expected FullBodyPoseTrackedT for '{self.name}', got {type(value).__name__}" - ) - - -class MessageChannelMessagesTrackedType(TensorType): - """MessageChannelMessagesTrackedT wrapper type from DeviceIO MessageChannelTracker.""" - - def __init__(self, name: str) -> None: - super().__init__(name) - - def _check_instance_compatibility(self, other: TensorType) -> bool: - if not isinstance(other, MessageChannelMessagesTrackedType): - raise TypeError( - f"Expected MessageChannelMessagesTrackedType, got {type(other).__name__}" - ) - return True + """, +) - def validate_value(self, value: Any) -> None: - if not isinstance(value, MessageChannelMessagesTrackedT): - raise TypeError( - f"Expected MessageChannelMessagesTrackedT for '{self.name}', got {type(value).__name__}" - ) +MessageChannelMessagesTrackedType = _payload_tensor_type( + "MessageChannelMessagesTrackedType", + MessageChannelMessagesTracked, + "MessageChannelMessagesTracked wrapper type from DeviceIO MessageChannelTracker.", +) class MessageChannelConnectionStatus(IntEnum): @@ -189,7 +124,9 @@ def _check_instance_compatibility(self, other: TensorType) -> bool: return True def validate_value(self, value: Any) -> None: - if not isinstance(value, MessageChannelConnectionStatus): + # Not a device payload: MessageChannelSource always assigns a status. None is + # tolerated only so an unset slot validates like the payload types above. + if value is not None and not isinstance(value, MessageChannelConnectionStatus): raise TypeError( f"Expected MessageChannelConnectionStatus for '{self.name}', got {type(value).__name__}" ) @@ -199,7 +136,7 @@ def DeviceIOHeadPoseTracked() -> TensorGroupType: """Tracked head pose from DeviceIO HeadTracker. Contains: - head_tracked: HeadPoseTrackedT wrapper (always set; .data is None when inactive) + head_tracked: HeadPose handle, or None when inactive """ return TensorGroupType("deviceio_head_pose", [HeadPoseTrackedType("head_tracked")]) @@ -208,7 +145,7 @@ def DeviceIOHandPoseTracked() -> TensorGroupType: """Tracked hand pose from DeviceIO HandTracker. Contains: - hand_tracked: HandPoseTrackedT wrapper (always set; .data is None when inactive) + hand_tracked: HandPose handle, or None when inactive """ return TensorGroupType("deviceio_hand_pose", [HandPoseTrackedType("hand_tracked")]) @@ -217,7 +154,7 @@ def DeviceIOControllerSnapshotTracked() -> TensorGroupType: """Tracked controller snapshot from DeviceIO ControllerTracker. Contains: - controller_tracked: ControllerSnapshotTrackedT wrapper (always set; .data is None when inactive) + controller_tracked: ControllerSnapshot handle, or None when inactive """ return TensorGroupType( "deviceio_controller_snapshot", @@ -229,7 +166,7 @@ def DeviceIOGeneric3AxisPedalOutputTracked() -> TensorGroupType: """Tracked pedal data from DeviceIO Generic3AxisPedalTracker. Contains: - pedal_tracked: Generic3AxisPedalOutputTrackedT wrapper (always set; .data is None when inactive) + pedal_tracked: Generic3AxisPedalOutput handle, or None when inactive """ return TensorGroupType( "deviceio_generic_3axis_pedal_output", @@ -241,7 +178,7 @@ def DeviceIOJointStateOutputTracked() -> TensorGroupType: """Tracked joint-state data from DeviceIO JointStateTracker. Contains: - joint_state_tracked: JointStateOutputTrackedT wrapper (always set; .data is None when inactive) + joint_state_tracked: JointStateOutput handle, or None when inactive """ return TensorGroupType( "deviceio_joint_state_output", @@ -253,7 +190,7 @@ def DeviceIOFullBodyPoseTracked() -> TensorGroupType: """Tracked full body pose data from DeviceIO FullBodyTracker. Contains: - full_body_tracked: FullBodyPoseTrackedT wrapper (always set; .data is None when inactive) + full_body_tracked: FullBodyPose handle, or None when inactive """ return TensorGroupType( "deviceio_full_body_pose", diff --git a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/full_body_source.py b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/full_body_source.py index abcd11e1b..a9626b33d 100644 --- a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/full_body_source.py +++ b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/full_body_source.py @@ -4,7 +4,7 @@ """ Full Body Source Node - DeviceIO to Retargeting Engine converter. -Converts raw FullBodyPoseT flatbuffer data to standard FullBodyInput tensor format. +Converts raw FullBodyPose flatbuffer data to standard FullBodyInput tensor format. """ import numpy as np @@ -22,15 +22,15 @@ if TYPE_CHECKING: from isaacteleop.deviceio import ITracker, TrackerVendor - from isaacteleop.schema import FullBodyPoseT, FullBodyPoseTrackedT + from isaacteleop.schema import FullBodyPose class FullBodySource(IDeviceIOSource): """ - Stateless converter: DeviceIO FullBodyPoseT -> FullBodyInput tensors. + Stateless converter: DeviceIO FullBodyPose -> FullBodyInput tensors. Inputs: - - "deviceio_full_body": Raw FullBodyPoseT flatbuffer + - "deviceio_full_body": Raw FullBodyPose flatbuffer Outputs (Optional — absent when body tracking is inactive): - "full_body": OptionalTensorGroup (check ``.is_none`` before access) @@ -78,7 +78,7 @@ def poll_tracker(self, deviceio_session: Any) -> RetargeterIO: Returns: Dict with "deviceio_full_body" TensorGroup containing raw - FullBodyPoseT data. + FullBodyPose data. """ body_pose = self._body_tracker.get_body_pose(deviceio_session) source_inputs = self.input_spec() @@ -103,17 +103,16 @@ def output_spec(self) -> RetargeterIOType: def _compute_fn(self, inputs: RetargeterIO, outputs: RetargeterIO, context) -> None: """ - Convert DeviceIO FullBodyPoseT to standard FullBodyInput tensors. + Convert DeviceIO FullBodyPose to standard FullBodyInput tensors. Calls ``set_none()`` on the output when body tracking is inactive. Args: - inputs: Dict with "deviceio_full_body" containing FullBodyPoseTrackedT wrapper + inputs: Dict with "deviceio_full_body" containing FullBodyPose wrapper outputs: Dict with "full_body" OptionalTensorGroup context: Shared ComputeContext for the current step (carries GraphTime). """ - tracked: "FullBodyPoseTrackedT" = inputs["deviceio_full_body"][0] - body_pose: "FullBodyPoseT | None" = tracked.data + body_pose: "FullBodyPose | None" = inputs["deviceio_full_body"][0] if body_pose is None: outputs["full_body"].set_none() diff --git a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/hands_source.py b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/hands_source.py index 5702ac679..3ba65a935 100644 --- a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/hands_source.py +++ b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/hands_source.py @@ -4,7 +4,7 @@ """ Hands Source Node - DeviceIO to Retargeting Engine converter. -Converts raw HandPoseT flatbuffer data to standard HandInput tensor format. +Converts raw HandPose flatbuffer data to standard HandInput tensor format. """ from typing import Any, TYPE_CHECKING @@ -22,16 +22,16 @@ if TYPE_CHECKING: from isaacteleop.deviceio import ITracker - from isaacteleop.schema import HandPoseT, HandPoseTrackedT + from isaacteleop.schema import HandPose class HandsSource(IDeviceIOSource): """ - Stateless converter: DeviceIO HandPoseT → HandInput tensors. + Stateless converter: DeviceIO HandPose → HandInput tensors. Inputs: - - "deviceio_hand_left": Raw HandPoseT flatbuffer for left hand - - "deviceio_hand_right": Raw HandPoseT flatbuffer for right hand + - "deviceio_hand_left": Raw HandPose flatbuffer for left hand + - "deviceio_hand_right": Raw HandPose flatbuffer for right hand Outputs (Optional — absent when tracking is inactive): - "hand_left": OptionalTensorGroup (check ``.is_none`` before access) @@ -79,7 +79,7 @@ def poll_tracker(self, deviceio_session: Any) -> RetargeterIO: Returns: Dict with "deviceio_hand_left" and "deviceio_hand_right" TensorGroups - containing HandPoseTrackedT wrappers. + containing HandPose wrappers. """ left_tracked = self._hand_tracker.get_left_hand(deviceio_session) right_tracked = self._hand_tracker.get_right_hand(deviceio_session) @@ -110,23 +110,23 @@ def output_spec(self) -> RetargeterIOType: def _compute_fn(self, inputs: RetargeterIO, outputs: RetargeterIO, context) -> None: """ - Convert DeviceIO HandPoseTrackedT to standard HandInput tensors. + Convert DeviceIO HandPose to standard HandInput tensors. Calls ``set_none()`` on the output when the corresponding hand is inactive. Args: - inputs: Dict with "deviceio_hand_left" and "deviceio_hand_right" HandPoseTrackedT wrappers + inputs: Dict with "deviceio_hand_left" and "deviceio_hand_right" HandPose wrappers outputs: Dict with "hand_left" and "hand_right" OptionalTensorGroups context: ComputeContext (unused by this converter node). """ - left_tracked: "HandPoseTrackedT" = inputs["deviceio_hand_left"][0] - right_tracked: "HandPoseTrackedT" = inputs["deviceio_hand_right"][0] + left_tracked: "HandPose | None" = inputs["deviceio_hand_left"][0] + right_tracked: "HandPose | None" = inputs["deviceio_hand_right"][0] - self._update_hand_data(outputs["hand_left"], left_tracked.data) - self._update_hand_data(outputs["hand_right"], right_tracked.data) + self._update_hand_data(outputs["hand_left"], left_tracked) + self._update_hand_data(outputs["hand_right"], right_tracked) def _update_hand_data( - self, group: OptionalTensorGroup, hand_data: "HandPoseT | None" + self, group: OptionalTensorGroup, hand_data: "HandPose | None" ) -> None: """Helper to convert hand data for a single hand.""" if hand_data is None: diff --git a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/head_source.py b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/head_source.py index 0c56ecabc..9de4c96c2 100644 --- a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/head_source.py +++ b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/head_source.py @@ -4,7 +4,7 @@ """ Head Source Node - DeviceIO to Retargeting Engine converter. -Converts raw HeadPoseT flatbuffer data to standard HeadInput tensor format. +Converts raw HeadPose flatbuffer data to standard HeadInput tensor format. """ import numpy as np @@ -23,15 +23,16 @@ if TYPE_CHECKING: from isaacteleop.deviceio import ITracker - from isaacteleop.schema import HeadPoseT, HeadPoseTrackedT + + from isaacteleop.schema import HeadPose class HeadSource(IDeviceIOSource): """ - Stateless converter: DeviceIO HeadPoseT → HeadInput tensor. + Stateless converter: schema HeadPose → HeadInput tensor. Inputs: - - "deviceio_head": Raw HeadPoseT flatbuffer object from DeviceIO + - "deviceio_head": Raw HeadPose flatbuffer object from DeviceIO Outputs (Optional — absent when head tracking is invalid): - "head": OptionalTensorGroup (check ``.is_none`` before access) @@ -70,7 +71,7 @@ def poll_tracker(self, deviceio_session: Any) -> RetargeterIO: deviceio_session: The active DeviceIO session. Returns: - Dict with "deviceio_head" TensorGroup containing HeadPoseTrackedT. + Dict with "deviceio_head" TensorGroup containing HeadPose. """ tracked = self._head_tracker.get_head(deviceio_session) source_inputs = self.input_spec() @@ -91,17 +92,16 @@ def output_spec(self) -> RetargeterIOType: def _compute_fn(self, inputs: RetargeterIO, outputs: RetargeterIO, context) -> None: """ - Convert DeviceIO HeadPoseTrackedT to standard HeadInput tensor. + Convert DeviceIO HeadPose to standard HeadInput tensor. Calls ``set_none()`` on the output when head tracking is inactive. Args: - inputs: Dict with "deviceio_head" containing HeadPoseTrackedT wrapper + inputs: Dict with "deviceio_head" containing HeadPose wrapper outputs: Dict with "head" OptionalTensorGroup context: ComputeContext (unused by this converter node). """ - tracked: "HeadPoseTrackedT" = inputs["deviceio_head"][0] - head_pose: "HeadPoseT | None" = tracked.data + head_pose: "HeadPose | None" = inputs["deviceio_head"][0] output = outputs["head"] if head_pose is None: diff --git a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/joint_state_source.py b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/joint_state_source.py index 96f8c76e9..992a15e26 100644 --- a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/joint_state_source.py +++ b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/joint_state_source.py @@ -24,7 +24,7 @@ if TYPE_CHECKING: from isaacteleop.deviceio import ITracker - from isaacteleop.schema import JointStateOutputTrackedT + from isaacteleop.schema import JointStateOutput class JointStateSource(IDeviceIOSource): @@ -91,12 +91,11 @@ def output_spec(self) -> RetargeterIOType: } def _compute_fn(self, inputs: RetargeterIO, outputs: RetargeterIO, context) -> None: - """Convert ``JointStateOutputTrackedT`` to a name-keyed joint-position group. + """Convert ``JointStateOutput`` to a name-keyed joint-position group. Calls ``set_none()`` on the output when the device is inactive. """ - tracked: "JointStateOutputTrackedT" = inputs["deviceio_joint_state"][0] - data = tracked.data + data: "JointStateOutput | None" = inputs["deviceio_joint_state"][0] out = outputs[self.JOINTS] if data is None: diff --git a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_config.py b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_config.py index 7e4eb8294..2dae023bc 100644 --- a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_config.py +++ b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_config.py @@ -13,7 +13,7 @@ import isaacteleop.deviceio as deviceio if TYPE_CHECKING: - from isaacteleop.schema import MessageChannelMessagesTrackedT + from isaacteleop.schema import MessageChannelMessagesTracked @dataclass @@ -39,7 +39,7 @@ def create_nodes(self) -> tuple[MessageChannelSource, MessageChannelSink]: self.max_message_size, ) # deque(maxlen=N) provides bounded queueing and drops oldest on overflow. - outbound_queue: deque["MessageChannelMessagesTrackedT"] = deque( + outbound_queue: deque["MessageChannelMessagesTracked"] = deque( maxlen=self.outbound_queue_capacity ) source = MessageChannelSource( diff --git a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_sink.py b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_sink.py index 55063660d..42c268f91 100644 --- a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_sink.py +++ b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_sink.py @@ -15,14 +15,14 @@ from .deviceio_tensor_types import MessageChannelMessagesTrackedGroup if TYPE_CHECKING: - from isaacteleop.schema import MessageChannelMessagesTrackedT + from isaacteleop.schema import MessageChannelMessagesTracked class MessageChannelSink(BaseRetargeter): """Sink node that enqueues outbound message channel payloads.""" def __init__( - self, name: str, outbound_queue: "deque[MessageChannelMessagesTrackedT]" + self, name: str, outbound_queue: "deque[MessageChannelMessagesTracked]" ) -> None: self._outbound_queue = outbound_queue super().__init__(name) diff --git a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_source.py b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_source.py index a65da7029..5f453e7ad 100644 --- a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_source.py +++ b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/message_channel_source.py @@ -4,7 +4,7 @@ """ Message channel source node. -Converts DeviceIO MessageChannelMessagesTrackedT wrapper data for graph use. +Converts DeviceIO MessageChannelMessagesTracked wrapper data for graph use. """ from collections import deque @@ -13,7 +13,7 @@ from .interface import IDeviceIOSource from ..interface.retargeter_core_types import RetargeterIO, RetargeterIOType from ..interface.tensor_group import TensorGroup -from isaacteleop.schema import MessageChannelMessages, MessageChannelMessagesTrackedT +from isaacteleop.schema import MessageChannelMessages, MessageChannelMessagesTracked from .deviceio_tensor_types import ( DeviceIOMessageChannelMessagesTracked, MessageChannelMessagesTrackedGroup, @@ -27,9 +27,14 @@ MessageChannelTracker, ) from isaacteleop.schema import ( - MessageChannelMessagesTrackedT, + MessageChannelMessagesTracked, ) +# Stands in for a frame that drained nothing. Encoding it is a full FlatBuffers build for +# a handful of constant bytes, and the buffer is immutable, so one instance serves every +# frame and every source. +_EMPTY_BATCH = MessageChannelMessagesTracked() + class MessageChannelSource(IDeviceIOSource): """Source node for reading message channel payloads from DeviceIO.""" @@ -38,13 +43,11 @@ def __init__( self, name: str, tracker: "MessageChannelTracker", - outbound_queue: "deque[MessageChannelMessagesTrackedT]", + outbound_queue: "deque[MessageChannelMessagesTracked]", ) -> None: self._tracker = tracker self._outbound_queue = outbound_queue - self._last_drained_messages_tracked: MessageChannelMessagesTrackedT | None = ( - None - ) + self._last_drained_messages_tracked: MessageChannelMessagesTracked | None = None self._last_status: MessageChannelConnectionStatus = ( MessageChannelConnectionStatus.UNKNOWN ) @@ -77,7 +80,7 @@ def poll_tracker(self, deviceio_session: Any) -> RetargeterIO: # Drop the delivered prefix so already-sent messages are # not re-delivered on the next flush attempt. if sent < len(batch.data): - self._outbound_queue[0] = MessageChannelMessagesTrackedT( + self._outbound_queue[0] = MessageChannelMessagesTracked( batch.data[sent:] ) else: @@ -93,10 +96,11 @@ def poll_tracker(self, deviceio_session: Any) -> RetargeterIO: result: RetargeterIO = {} for input_name, group_type in source_inputs.items(): tg = TensorGroup(group_type) - if self._last_drained_messages_tracked is None: - tg[0] = MessageChannelMessagesTrackedT() - else: - tg[0] = self._last_drained_messages_tracked + tg[0] = ( + _EMPTY_BATCH + if self._last_drained_messages_tracked is None + else self._last_drained_messages_tracked + ) result[input_name] = tg return result @@ -112,8 +116,9 @@ def output_spec(self) -> RetargeterIOType: } def _compute_fn(self, inputs: RetargeterIO, outputs: RetargeterIO, context) -> None: - if self._last_drained_messages_tracked is None: - outputs["messages_tracked"][0] = MessageChannelMessagesTrackedT() - else: - outputs["messages_tracked"][0] = self._last_drained_messages_tracked + outputs["messages_tracked"][0] = ( + _EMPTY_BATCH + if self._last_drained_messages_tracked is None + else self._last_drained_messages_tracked + ) outputs["status"][0] = self._last_status diff --git a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/pedals_source.py b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/pedals_source.py index 015e002b6..f70628023 100644 --- a/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/pedals_source.py +++ b/src/python/isaacteleop/retargeting_engine/deviceio_source_nodes/pedals_source.py @@ -21,7 +21,6 @@ from isaacteleop.deviceio import ITracker from isaacteleop.schema import ( Generic3AxisPedalOutput, - Generic3AxisPedalOutputTrackedT, ) # Default collection_id matching foot_pedal_reader / pedal_pusher and Generic3AxisPedalTracker. @@ -79,7 +78,7 @@ def poll_tracker(self, deviceio_session: Any) -> RetargeterIO: deviceio_session: The active DeviceIO session. Returns: - Dict with "deviceio_pedals" TensorGroup containing Generic3AxisPedalOutputTrackedT. + Dict with "deviceio_pedals" TensorGroup containing Generic3AxisPedalOutput. """ tracked = self._pedal_tracker.get_pedal_data(deviceio_session) tg = TensorGroup(DeviceIOGeneric3AxisPedalOutputTracked()) @@ -100,17 +99,16 @@ def output_spec(self) -> RetargeterIOType: def _compute_fn(self, inputs: RetargeterIO, outputs: RetargeterIO, context) -> None: """ - Convert DeviceIO Generic3AxisPedalOutputTrackedT to standard Generic3AxisPedalInput tensor. + Convert DeviceIO Generic3AxisPedalOutput to standard Generic3AxisPedalInput tensor. Calls ``set_none()`` on the output when pedal data is inactive. Args: - inputs: Dict with "deviceio_pedals" containing Generic3AxisPedalOutputTrackedT wrapper + inputs: Dict with "deviceio_pedals" containing Generic3AxisPedalOutput wrapper outputs: Dict with "pedals" OptionalTensorGroup context: Shared ComputeContext for the current step (carries GraphTime). """ - tracked: "Generic3AxisPedalOutputTrackedT" = inputs["deviceio_pedals"][0] - pedal: Generic3AxisPedalOutput | None = tracked.data + pedal: "Generic3AxisPedalOutput | None" = inputs["deviceio_pedals"][0] out = outputs["pedals"] if pedal is None: diff --git a/src/python/isaacteleop/schema/__init__.py b/src/python/isaacteleop/schema/__init__.py index cc1d3265c..2e04739ab 100644 --- a/src/python/isaacteleop/schema/__init__.py +++ b/src/python/isaacteleop/schema/__init__.py @@ -3,8 +3,13 @@ """Isaac Teleop Schema - FlatBuffer message types for teleoperation. -This module provides Python bindings for FlatBuffer-based message types -used in teleoperation, including poses, and controller data. +This module provides Python bindings for FlatBuffer-based message types used in +teleoperation, including poses and controller data. + +Each table is a read-only view over its encoded bytes: attribute reads go straight +into the buffer, and joint arrays come back as zero-copy NumPy views. To produce one, +call its constructor -- that encodes the arguments and hands back the view, which is +the only way to build these from Python. """ import warnings @@ -17,69 +22,72 @@ Quaternion, Pose, # Head-related types. - HeadPoseT, - HeadPoseTrackedT, + HeadPose, HeadPoseRecord, # Hand-related types. HandJoint, HandJointPose, HandJoints, - HandPoseT, - HandPoseTrackedT, + HandPose, HandPoseRecord, # Controller-related types. ControllerInputState, ControllerPose, ControllerSnapshot, - ControllerSnapshotTrackedT, ControllerSnapshotRecord, # Pedals-related types. Generic3AxisPedalOutput, - Generic3AxisPedalOutputTrackedT, Generic3AxisPedalOutputRecord, # OGLO tactile glove types. OgloGloveSample, - OgloGloveSampleTrackedT, OgloGloveSampleRecord, # Joint-state types (generic joint-space devices: leader arms, exoskeletons, ...). JointState, JointStateOutput, - JointStateOutputTrackedT, JointStateOutputRecord, # SE3 tracker types (generic 6-DoF pose sources: tracker pucks, mocap rigid bodies, ...). - # Record classes drop the T suffix in Python by family convention. - Se3TrackerPoseT, - Se3TrackerPoseTrackedT, + Se3TrackerPose, Se3TrackerPoseRecord, # Message channel types. MessageChannelMessages, - MessageChannelMessagesTrackedT, + MessageChannelMessagesTracked, MessageChannelMessagesRecord, # Haptic command types (vendor-neutral cross-process device output). HapticCommand, + HapticCommandRecord, pack_haptic_command, # Camera-related types. StreamType, FrameMetadataOak, - FrameMetadataOakTrackedT, FrameMetadataOakRecord, # Full body-related types. BodyJoint, BodyJointPose, BodyJoints, - FullBodyPoseT, - FullBodyPoseTrackedT, + FullBodyPose, FullBodyPoseRecord, ) -# Deprecated aliases for the renamed full-body schema types, resolved lazily via -# __getattr__ so accessing them emits a DeprecationWarning. Omitted from __all__. +# Deprecated aliases, resolved lazily via __getattr__ so accessing them emits a +# DeprecationWarning. Omitted from __all__. +# +# The `...T` spellings named the FlatBuffers object-API types these tables used to be +# bound to. Python no longer sees those at all, so the alias resolves to the encoded +# view: reads are unchanged, but the objects are immutable and are built by passing +# every field to the constructor rather than by assigning attributes afterwards. +# +# The Tracked wrappers have no alias: trackers now hand out the payload table itself +# and express "no data" with an empty view, so there is no object to redirect to. _DEPRECATED_ALIASES = { "BodyJointPico": "BodyJoint", "BodyJointsPico": "BodyJoints", - "FullBodyPosePicoT": "FullBodyPoseT", - "FullBodyPosePicoTrackedT": "FullBodyPoseTrackedT", + "FullBodyPosePicoT": "FullBodyPose", "FullBodyPosePicoRecord": "FullBodyPoseRecord", + "HeadPoseT": "HeadPose", + "HandPoseT": "HandPose", + "Se3TrackerPoseT": "Se3TrackerPose", + "MessageChannelMessagesTrackedT": "MessageChannelMessagesTracked", + "FullBodyPoseT": "FullBodyPose", } @@ -103,56 +111,48 @@ def __getattr__(name: str): "Quaternion", "Pose", # Head types. - "HeadPoseT", - "HeadPoseTrackedT", + "HeadPose", "HeadPoseRecord", # Hand types. "HandJoint", "HandJointPose", "HandJoints", - "HandPoseT", - "HandPoseTrackedT", + "HandPose", "HandPoseRecord", # Controller types. "ControllerInputState", "ControllerPose", "ControllerSnapshot", - "ControllerSnapshotTrackedT", "ControllerSnapshotRecord", # Pedals types. "Generic3AxisPedalOutput", - "Generic3AxisPedalOutputTrackedT", "Generic3AxisPedalOutputRecord", # OGLO tactile glove types. "OgloGloveSample", - "OgloGloveSampleTrackedT", "OgloGloveSampleRecord", # Joint-state types (generic joint-space devices). "JointState", "JointStateOutput", - "JointStateOutputTrackedT", "JointStateOutputRecord", # SE3 tracker types (generic 6-DoF pose sources). - "Se3TrackerPoseT", - "Se3TrackerPoseTrackedT", + "Se3TrackerPose", "Se3TrackerPoseRecord", # Message channel types. "MessageChannelMessages", - "MessageChannelMessagesTrackedT", + "MessageChannelMessagesTracked", "MessageChannelMessagesRecord", # Haptic command types. "HapticCommand", + "HapticCommandRecord", "pack_haptic_command", # Camera types. "StreamType", "FrameMetadataOak", - "FrameMetadataOakTrackedT", "FrameMetadataOakRecord", # Full body types. "BodyJointPose", "BodyJoint", "BodyJoints", - "FullBodyPoseT", - "FullBodyPoseTrackedT", + "FullBodyPose", "FullBodyPoseRecord", ]