Multi AOV rendering for OVRTX (isaac-sim#7204)

# Description

Add multi-AOV rendering for OVRTX.

Fixes # (issue)

<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->

## Type of change

<!-- As you go through the list, delete the ones that are not
applicable. -->

- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Breaking change (existing functionality will not work without user
modification)
- Documentation update

## Screenshots

Please attach before and after screenshots of the change if applicable.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

## Checklist

- [ ] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package (do **not** edit
`CHANGELOG.rst` or bump `extension.toml` — CI handles that)
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

Author: pbarejko

source/isaaclab_ov/changelog.d/ovrtx-enumerate-render-vars.minor.rst

@@ -0,0 +1,22 @@
+Fixed
+^^^^^
+
+* Fixed :class:`~isaaclab_ov.renderers.OVRTXRenderer` authoring only one pixel render var when a
+  camera requested several data types, which left every other requested output empty. The render
+  product now authors one render var per requested data type, so combinations such as ``rgb`` with
+  ``normals``, ``albedo``, ``motion_vectors``, segmentation, and depth are rendered together.
+* Fixed :class:`~isaaclab_ov.renderers.OVRTXRenderer` filling ``depth``,
+  ``distance_to_image_plane``, and ``distance_to_camera`` from a single depth render var, which
+  returned euclidean distance for the image-plane outputs (or the reverse) when they were requested
+  together. Each output is now extracted from the source that measures it.
+
+Changed
+^^^^^^^
+
+* **Breaking:** Changed :class:`~isaaclab_ov.renderers.OVRTXRenderer` to raise :class:`ValueError` when a camera
+  requests ``rgb`` or ``rgba`` together with a ``simple_shading_*`` data type, or more than one
+  distinct ``simple_shading_*`` data type. These outputs all read the ``LdrColor`` render var and
+  simple shading additionally requires the render product to be in RTX Minimal mode, so one render
+  product cannot serve them. Previously the conflict was resolved silently and produced wrongly
+  shaded or empty images. Request the conflicting outputs from separate cameras. Repeated identical
+  simple-shading requests still collapse to one render var.

source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py

@@ -26,7 +26,7 @@
 import re
 import sys
 import weakref
-from collections.abc import Iterator
+from collections.abc import Iterator, Sequence
 from pathlib import Path
 from typing import TYPE_CHECKING, Any, NoReturn, cast
 
@@ -110,6 +110,12 @@
 
 from isaaclab.renderers.camera_render_spec import CameraRenderSpec
 
+# Maps depth render-var sources to compatible output buffers.
+_DEPTH_VAR_BUFFER_KEYS: dict[str, tuple[str, ...]] = {
+    "DistanceToImagePlaneSD": ("depth", "distance_to_image_plane"),
+    "DistanceToCameraSD": ("distance_to_camera",),
+}
+
 # The resolved integer value is assigned to the ``omni:rtx:minimal:mode`` attribute of the render product.
 _RTX_MINIMAL_MODES = {
     RenderBufferKind.SIMPLE_SHADING_CONSTANT_DIFFUSE.value: 1,
@@ -1301,10 +1307,22 @@ def _extract_rgba_tiles(
         self._launch_extract_all_tiles(render_data, tiled_data, output_buffer)
 
     def _extract_depth_tiles(
-        self, render_data: OVRTXRenderData, tiled_depth_data: wp.array, output_buffers: dict
+        self,
+        render_data: OVRTXRenderData,
+        tiled_depth_data: wp.array,
+        output_buffers: dict,
+        buffer_keys: Sequence[str],
     ) -> None:
-        """Extract per-env depth tiles into output_buffers (single kernel launch)."""
-        for depth_type in ["depth", "distance_to_image_plane", "distance_to_camera"]:
+        """Extract per-env depth tiles into the given output buffers (one kernel launch each).
+
+        Args:
+            render_data: OVRTX render data for the current frame.
+            tiled_depth_data: Tiled depth data mapped from one depth render var.
+            output_buffers: Destination warp buffers, keyed by data type.
+            buffer_keys: Data types that this depth render var measures. Keys absent from
+                ``output_buffers`` are skipped.
+        """
+        for depth_type in buffer_keys:
             if depth_type in output_buffers:
                 self._launch_extract_all_tiles(render_data, tiled_depth_data, output_buffers[depth_type])
 
@@ -1359,16 +1377,17 @@ def _process_render_frame(self, render_data: OVRTXRenderData, frame, output_buff
                 with self._map_render_var_to_dlpack(frame.render_vars["LdrColor"]) as tiled_data:
                     self._extract_rgba_tiles(render_data, tiled_data, output_buffers, buffer_key)
 
-        for depth_var in ["DistanceToCameraSD", "DistanceToImagePlaneSD", "DepthSD"]:
+        for depth_var, buffer_keys in _DEPTH_VAR_BUFFER_KEYS.items():
             if depth_var not in frame.render_vars:
                 continue
+            if not any(buffer_key in output_buffers for buffer_key in buffer_keys):
+                continue
             with self._map_render_var_to_dlpack(frame.render_vars[depth_var]) as tiled_depth_data:
                 if tiled_depth_data.dtype == wp.uint32:
                     tiled_depth_data = wp.from_torch(
                         wp.to_torch(tiled_depth_data).view(torch.float32), dtype=wp.float32
                     )
-                self._extract_depth_tiles(render_data, tiled_depth_data, output_buffers)
-            break
+                self._extract_depth_tiles(render_data, tiled_depth_data, output_buffers, buffer_keys)
 
         if "DiffuseAlbedoSD" in frame.render_vars and "albedo" in output_buffers:
             with self._map_render_var_to_dlpack(frame.render_vars["DiffuseAlbedoSD"]) as tiled_albedo_data:

source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_usd.py

@@ -15,76 +15,121 @@
 logger = logging.getLogger(__name__)
 
 
-def get_render_var_config(data_types: list[str]) -> tuple[str, str, str]:
-    """Return (render_var_path, render_var_name, source_name) from data_types."""
-    use_depth = any(dt in ["depth", "distance_to_image_plane", "distance_to_camera"] for dt in data_types)
-    use_distance_to_camera = "distance_to_camera" in data_types and not any(
-        dt in ["depth", "distance_to_image_plane"] for dt in data_types
+# Maps camera data types to render-var configs. OVRTX frame vars are keyed by source name,
+# so shared sources use one config.
+_RENDER_VAR_BY_DATA_TYPE: dict[str, tuple[str, str, str]] = {
+    "rgb": ("/Render/Vars/LdrColor", "LdrColor", "LdrColor"),
+    "rgba": ("/Render/Vars/LdrColor", "LdrColor", "LdrColor"),
+    # Simple shading uses LdrColor in per-product RTX Minimal mode.
+    "simple_shading_constant_diffuse": ("/Render/Vars/LdrColor", "LdrColor", "LdrColor"),
+    "simple_shading_diffuse_mdl": ("/Render/Vars/LdrColor", "LdrColor", "LdrColor"),
+    "simple_shading_full_mdl": ("/Render/Vars/LdrColor", "LdrColor", "LdrColor"),
+    "rgb_hdr": ("/Render/Vars/HdrColor", "HdrColor", "HdrColor"),
+    "albedo": ("/Render/Vars/albedo", "albedo", "DiffuseAlbedoSD"),
+    "depth": ("/Render/Vars/depth", "depth", "DistanceToImagePlaneSD"),
+    "distance_to_image_plane": ("/Render/Vars/depth", "depth", "DistanceToImagePlaneSD"),
+    # This source requires a distinct render-var prim.
+    "distance_to_camera": ("/Render/Vars/DistanceToCameraSD", "DistanceToCameraSD", "DistanceToCameraSD"),
+    "normals": ("/Render/Vars/NormalSD", "NormalSD", "NormalSD"),
+    "motion_vectors": ("/Render/Vars/TargetMotionSD", "TargetMotionSD", "TargetMotionSD"),
+    "semantic_segmentation": ("/Render/Vars/semantic", "semantic", "SemanticSegmentation"),
+    "instance_segmentation": (
+        "/Render/Vars/NonStableInstanceSegmentation",
+        "NonStableInstanceSegmentation",
+        "NonStableInstanceSegmentation",
+    ),
+}
+
+# Data types produced by putting the whole render product into RTX Minimal mode.
+_SIMPLE_SHADING_DATA_TYPES = frozenset(
+    {
+        "simple_shading_constant_diffuse",
+        "simple_shading_diffuse_mdl",
+        "simple_shading_full_mdl",
+    }
+)
+
+_COLOR_DATA_TYPES = frozenset({"rgb", "rgba"})
+
+_DEFAULT_RENDER_VAR = _RENDER_VAR_BY_DATA_TYPE["rgb"]
+
+
+def _validate_data_type_combination(data_types: list[str]) -> None:
+    """Reject data type combinations that a single OVRTX render product cannot serve.
+
+    Args:
+        data_types: Requested camera data types.
+
+    Raises:
+        ValueError: If color and simple-shading data types are combined, or if more than one
+            simple-shading data type is requested.
+    """
+    simple_shading = list(
+        dict.fromkeys(data_type for data_type in data_types if data_type in _SIMPLE_SHADING_DATA_TYPES)
     )
-    use_albedo = "albedo" in data_types
-    use_semantic = "semantic_segmentation" in data_types
-    use_instance_seg = "instance_segmentation" in data_types
-    use_normals = "normals" in data_types
-    use_motion_vectors = "motion_vectors" in data_types
-    use_rgb = any(dt in ["rgb", "rgba"] for dt in data_types)
-    use_hdr = "rgb_hdr" in data_types
-
-    if use_depth and not (
-        use_rgb or use_albedo or use_semantic or use_instance_seg or use_normals or use_motion_vectors
-    ):
-        source = "DistanceToCameraSD" if use_distance_to_camera else "DistanceToImagePlaneSD"
-        return "/Render/Vars/depth", "depth", source
-    if use_albedo and not (use_rgb or use_semantic or use_instance_seg or use_normals or use_motion_vectors):
-        return "/Render/Vars/albedo", "albedo", "DiffuseAlbedoSD"
-    if use_semantic and not (use_rgb or use_albedo or use_normals or use_motion_vectors):
-        return "/Render/Vars/semantic", "semantic", "SemanticSegmentation"
-    if use_instance_seg and not (
-        use_rgb or use_albedo or use_semantic or use_normals or use_depth or use_hdr or use_motion_vectors
-    ):
-        return (
-            "/Render/Vars/NonStableInstanceSegmentation",
-            "NonStableInstanceSegmentation",
-            "NonStableInstanceSegmentation",
+    color = list(dict.fromkeys(data_type for data_type in data_types if data_type in _COLOR_DATA_TYPES))
+
+    if simple_shading and color:
+        raise ValueError(
+            f"OVRTX cannot render simple shading {simple_shading} together with {color} on one render product:"
+            " both read the 'LdrColor' render var, and simple shading additionally requires RTX Minimal mode."
+            " Request them from separate cameras."
+        )
+    if len(simple_shading) > 1:
+        raise ValueError(
+            f"OVRTX supports at most one simple shading data type per render product, got {simple_shading}."
+            " RTX Minimal mode is a per-render-product setting. Request them from separate cameras."
         )
-    if use_normals and not (
-        use_rgb or use_albedo or use_semantic or use_instance_seg or use_depth or use_motion_vectors
-    ):
-        return "/Render/Vars/NormalSD", "NormalSD", "NormalSD"
-    if use_motion_vectors and not (
-        use_rgb or use_albedo or use_semantic or use_instance_seg or use_depth or use_normals
-    ):
-        return "/Render/Vars/TargetMotionSD", "TargetMotionSD", "TargetMotionSD"
-    if use_hdr and not use_rgb:
-        return "/Render/Vars/HdrColor", "HdrColor", "HdrColor"
-    return "/Render/Vars/LdrColor", "LdrColor", "LdrColor"
+
+
+def get_render_var_config(data_types: list[str]) -> tuple[str, str, str]:
+    """Return the first supported render-var configuration for ``data_types``.
+
+    Args:
+        data_types: Requested camera data types.
+
+    Returns:
+        The render-var config, defaulting to ``LdrColor`` when no entry is supported.
+    """
+    return get_render_var_configs(data_types)[0]
 
 
 def get_render_var_configs(data_types: list[str]) -> list[tuple[str, str, str]]:
-    """Return render var configs needed for the requested data types.
+    """Return render-var configs for the requested camera data types.
 
-    Each config is a ``(render_var_path, render_var_name, source_name)`` tuple as defined by
-    :func:`get_render_var_config`. Always includes the single render var resolved by
-    :func:`get_render_var_config`, plus the following extras when applicable:
+    Shared sources are de-duplicated. Unsupported data types are logged and skipped; if no
+    supported type remains, ``LdrColor`` is used. Segmentation requests also add their ID-map vars.
 
-    * ``HdrColor`` — when both ``"rgb"`` (or ``"rgba"``) and ``"rgb_hdr"`` are requested, so
-      PPISP can consume the HDR AOV alongside the LDR destination on the same render product.
-    * ``SemanticIdMap`` — when ``"semantic_segmentation"`` is requested, so the
-      semantic-ID-to-label mapping can be decoded for ``camera.data.info``.
-    * ``StableIdSemanticIdMap``, ``StableIdMap``, ``SemanticIdMap`` — when
-      ``"instance_segmentation"`` is requested, so the instance-ID-to-prim-path
-      (``idToLabels``) and instance-ID-to-semantic (``idToSemantics``) mappings can be decoded.
+    Args:
+        data_types: Requested camera data types.
+
+    Returns:
+        Render-var configs to author on the render product.
 
-    Other multi-AOV combinations are not supported.
+    Raises:
+        ValueError: If ``data_types`` contains incompatible outputs.
     """
     data_types = data_types if data_types else ["rgb"]
-    render_vars: list[tuple[str, str, str]] = [get_render_var_config(data_types)]
-    use_rgb = any(dt in ["rgb", "rgba"] for dt in data_types)
-    if use_rgb and "rgb_hdr" in data_types:
-        render_vars.append(("/Render/Vars/HdrColor", "HdrColor", "HdrColor"))
-    # Author the ID-to-label map render vars needed to decode the segmentation info dicts. These are keyed off
-    # the requested data types (not the single AOV resolved by get_render_var_config) so they are still authored
-    # when segmentation is combined with other outputs. instance_segmentation needs StableIdSemanticIdMap +
-    # StableIdMap to resolve each pixel to a prim path.
+    _validate_data_type_combination(data_types)
+
+    render_vars: list[tuple[str, str, str]] = []
+    unsupported: list[str] = []
+    for data_type in data_types:
+        config = _RENDER_VAR_BY_DATA_TYPE.get(data_type)
+        if config is None:
+            unsupported.append(data_type)
+        elif config not in render_vars:
+            render_vars.append(config)
+
+    if unsupported:
+        logger.warning(
+            "OVRTX does not support the requested data type(s) %s; no render var is authored for them.", unsupported
+        )
+    if not render_vars:
+        render_vars.append(_DEFAULT_RENDER_VAR)
+
+    # Author the ID-to-label map render vars needed to decode the segmentation info dicts.
+    # instance_segmentation needs StableIdSemanticIdMap + StableIdMap to resolve each pixel to a prim path.
     if "instance_segmentation" in data_types:
         render_vars.append(("/Render/Vars/StableIdSemanticIdMap", "StableIdSemanticIdMap", "StableIdSemanticIdMap"))
         render_vars.append(("/Render/Vars/StableIdMap", "StableIdMap", "StableIdMap"))

source/isaaclab_ov/test/test_ovrtx_usd.py

@@ -133,9 +133,106 @@ def test_ovrtx_motion_vectors_uses_target_motion_render_var():
     )
 
 
-def test_ovrtx_motion_vectors_with_rgb_falls_back_to_rgb():
-    """OVRTX only supports one main AOV at a time; combining motion vectors with RGB keeps RGB."""
+def test_ovrtx_primary_render_var_follows_the_first_requested_data_type():
+    """The primary render var seeds the single-var arguments from the first requested data type."""
     assert get_render_var_config(["rgb", "motion_vectors"]) == ("/Render/Vars/LdrColor", "LdrColor", "LdrColor")
+    assert get_render_var_config(["motion_vectors", "rgb"]) == (
+        "/Render/Vars/TargetMotionSD",
+        "TargetMotionSD",
+        "TargetMotionSD",
+    )
+
+
+def test_ovrtx_authors_one_render_var_per_requested_data_type():
+    """Every requested AOV is authored, so combining them no longer drops any."""
+    data_types = [
+        "rgb",
+        "albedo",
+        "semantic_segmentation",
+        "instance_segmentation",
+        "depth",
+        "distance_to_camera",
+        "normals",
+        "motion_vectors",
+    ]
+
+    sources = [source for _, _, source in get_render_var_configs(data_types)]
+
+    assert sources == [
+        "LdrColor",
+        "DiffuseAlbedoSD",
+        "SemanticSegmentation",
+        "NonStableInstanceSegmentation",
+        "DistanceToImagePlaneSD",
+        "DistanceToCameraSD",
+        "NormalSD",
+        "TargetMotionSD",
+        "StableIdSemanticIdMap",
+        "StableIdMap",
+        "SemanticIdMap",
+    ]
+
+
+def test_ovrtx_data_types_sharing_a_source_author_one_render_var():
+    """``rgb``/``rgba`` and ``depth``/``distance_to_image_plane`` collapse onto one render var each."""
+    render_var_configs = get_render_var_configs(["rgb", "rgba", "depth", "distance_to_image_plane"])
+
+    assert render_var_configs == [
+        ("/Render/Vars/LdrColor", "LdrColor", "LdrColor"),
+        ("/Render/Vars/depth", "depth", "DistanceToImagePlaneSD"),
+    ]
+
+
+def test_ovrtx_depth_and_distance_to_camera_author_distinct_render_vars():
+    """Image-plane depth and distance-to-camera are different sources and get separate prims."""
+    render_scope = build_render_scope_usd(
+        camera_paths=["/World/envs/env_0/Camera"],
+        render_product_name="RenderProduct",
+        render_var_path="/Render/Vars/depth",
+        render_var_name="depth",
+        source_name="DistanceToImagePlaneSD",
+        tiled_width=16,
+        tiled_height=8,
+        render_var_configs=get_render_var_configs(["depth", "distance_to_camera"]),
+    )
+
+    assert "rel orderedVars = [</Render/Vars/depth>, </Render/Vars/DistanceToCameraSD>]" in render_scope
+    assert 'uniform string sourceName = "DistanceToImagePlaneSD"' in render_scope
+    assert 'uniform string sourceName = "DistanceToCameraSD"' in render_scope
+
+
+def test_ovrtx_unsupported_data_type_is_skipped_and_falls_back_to_ldr_color():
+    """Unsupported data types author no render var; an otherwise empty product keeps LdrColor."""
+    assert get_render_var_configs(["instance_id_segmentation_fast"]) == [
+        ("/Render/Vars/LdrColor", "LdrColor", "LdrColor")
+    ]
+    assert get_render_var_configs(["normals", "instance_id_segmentation_fast"]) == [
+        ("/Render/Vars/NormalSD", "NormalSD", "NormalSD")
+    ]
+
+
+def test_ovrtx_rejects_color_combined_with_simple_shading():
+    """Color and simple shading both read LdrColor, so one render product cannot serve both."""
+    with pytest.raises(ValueError, match="simple shading"):
+        get_render_var_configs(["rgb", "simple_shading_full_mdl"])
+
+
+def test_ovrtx_rejects_multiple_simple_shading_data_types():
+    """RTX Minimal mode is per render product, so only one simple shading output is possible."""
+    with pytest.raises(ValueError, match="at most one simple shading"):
+        get_render_var_configs(["simple_shading_constant_diffuse", "simple_shading_full_mdl"])
+
+
+def test_ovrtx_simple_shading_alone_uses_ldr_color():
+    """A lone simple shading request still reads LdrColor, shaded by RTX Minimal mode."""
+    assert get_render_var_configs(["simple_shading_diffuse_mdl"]) == [("/Render/Vars/LdrColor", "LdrColor", "LdrColor")]
+
+
+def test_ovrtx_duplicate_simple_shading_data_types_collapse():
+    """Repeated identical simple-shading requests share one LdrColor render var."""
+    assert get_render_var_configs(["simple_shading_full_mdl", "simple_shading_full_mdl"]) == [
+        ("/Render/Vars/LdrColor", "LdrColor", "LdrColor")
+    ]
 
 
 def test_render_product_initially_targets_only_the_resolvable_source_camera():