Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 58 additions & 6 deletions docs/source/references/camera_streaming.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ Requirements
:doc:`quick start </getting_started/quick_start>` steps :ref:`run-cloudxr-server` and
:ref:`connect-xr-headset`. No headset handy? ``--mode window`` renders to a desktop window
instead and only needs a local display.
- For Jetson Argus cameras, a Jetson image with libargus, CUDA, EGL, and the Jetson Multimedia API
headers installed. The optional Argus native source is built only when setup is run with
``--with-argus``.

Setup
-----
Expand All @@ -54,7 +57,9 @@ probes system packages (GStreamer plugins, cairo / girepository headers, JetPack
``ld.so`` wiring). When something is missing it prints the exact ``apt-get`` line and prompts
``[y/N]`` — answering ``n`` or running non-interactively aborts.

By default ``setup`` provisions everything except ZED support; flags trim or extend that:
By default ``setup`` provisions the portable sources and split-mode pieces. Jetson Argus and ZED
support are opt-in because they depend on vendor SDKs and headers that are not present on most
development hosts. Flags trim or extend setup:

.. list-table::
:header-rows: 1
Expand All @@ -69,6 +74,9 @@ By default ``setup`` provisions everything except ZED support; flags trim or ext
* - ``--no-rtp``
- Skip split-mode dependencies: the GStreamer system packages and the native NVENC/NVDEC
codec build. Direct mode still works.
* - ``--with-argus``
- Build the optional Jetson Argus native source (``type: argus``). Requires libargus, CUDA,
EGL, and Jetson Multimedia API headers on the machine.
* - ``--with-zed``
- Also build + install the ZED SDK's Python API (``pyzed``). Requires the ZED SDK on the
machine (default ``/usr/local/zed``; override with ``--zed-sdk PATH``).
Expand Down Expand Up @@ -128,6 +136,9 @@ The source kind is selected by the ``type`` field of each entry in the YAML ``ca
- OAK-D mono RGB / LEFT / RIGHT (stereo not yet wired).
* - ``zed``
- ZED 2 / Mini / X One; mono or ``stereo: true`` (per-eye SDK retrieve, zero-copy on the GPU).
* - ``argus``
- Jetson libargus cameras through CUDA EGLStream, including GMSL sensors exposed by
``nvargus-daemon``. Build setup with ``--with-argus`` first.
* - ``video``
- Video-file replay (anything OpenCV's FFmpeg backend reads). Loops by default;
``stereo: true`` splits side-by-side recordings into eyes (viewer only).
Expand All @@ -142,11 +153,40 @@ run with the matching config:

.. code-block:: bash

./camera_viz.sh run configs/v4l2.yaml # or oakd.yaml / zed.yaml
./camera_viz.sh run configs/v4l2.yaml # or oakd.yaml / zed.yaml / argus_shw5g.yaml

**You should see** the same startup lines as above with the camera's tag (``[v4l2]``,
``[oakd]``, ``[zed]``) and the live feed. Multiple entries in the ``cameras`` list render as one
plane each.
``[oakd]``, ``[zed]``, ``[argus]``) and the live feed. Multiple entries in the ``cameras`` list
render as one plane each.

Jetson Argus / GMSL cameras
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Argus support is for Jetson-local capture with minimum camera-to-display latency. It bypasses
``nvarguscamerasrc`` and consumes Argus EGLStreams directly:

.. code-block:: text

nvargus-daemon -> libargus EGLStream -> CUDA YUV-to-RGBA -> camera_viz

Install with the Argus backend enabled on the Jetson:

.. code-block:: bash

examples/camera_viz/camera_viz.sh setup --with-argus --jetson

Use ``nvargus_nvraw`` to confirm the Argus sensor indices before editing the YAML:

.. code-block:: bash

/usr/sbin/nvargus_nvraw --lps

The repository includes :code-file:`configs/argus_shw5g.yaml
<examples/camera_viz/configs/argus_shw5g.yaml>` for a SENSING SHW5G stereo pair on Jetson AGX
Thor. Its tested settings use independent Argus sessions for sensors ``0`` and ``1``,
``repeat_capture: true`` for lower request latency, ``pair_emit_mode: both`` to avoid duplicate
stereo submits, and ``swap_uv: true`` for the observed SHW5G color layout. Some SENSING driver
stacks also require running ``nvargus-daemon`` with ``enableCamInfiniteTimeout=1``.

Display modes
-------------
Expand Down Expand Up @@ -234,13 +274,17 @@ its own plane (and, in split mode, its own RTP port). Abbreviated:
cameras:
- name: cam
enabled: true
type: v4l2 # v4l2 | oakd | zed | video | synthetic
type: v4l2 # v4l2 | oakd | zed | argus | video | synthetic
width: 2560 # video: optional — defaults to the file's size
height: 720 # (required when source: rtp)
fps: 30
stereo: false # zed / video / synthetic — per-eye capture + SBS in XR
path: clip.mp4 # video only — file to replay, relative to this YAML
loop: true # video only — rewind at end of file
sensor_id: 0 # argus mono only; use sensor_id_left/right for stereo
sensor_mode: 0 # argus only
repeat_capture: true # argus only — use repeat requests when supported
pair_emit_mode: both # argus independent stereo only: either | both
rtp:
port: 5000 # left eye when stereo
port_right: 5001 # required when stereo + source: rtp
Expand Down Expand Up @@ -273,6 +317,11 @@ Troubleshooting
directory (``configs/``), not the directory you launched from.
- **A source fails asking for CuPy / CUDA** — check ``nvidia-smi`` works and setup completed;
all sources allocate their frame buffers on the GPU.
- **``type: argus`` is unavailable** — rerun setup on the Jetson with ``--with-argus`` and confirm
the Jetson Multimedia API Argus headers exist under ``/usr/src/jetson_multimedia_api/argus``.
- **Argus lists sensors but capture stalls** — confirm the selected ``sensor_id`` values with
``/usr/sbin/nvargus_nvraw --lps``. Some GMSL driver stacks need ``nvargus-daemon`` configured
with ``enableCamInfiniteTimeout=1``.
- **Split mode renders nothing** — check the sender is up (``./camera_viz.sh service-status``),
``$STREAMING_HOST`` was the workstation's IP at deploy time, and UDP ports (default 5000+)
aren't firewalled.
Expand All @@ -294,9 +343,10 @@ Televiz as the compositor at the end of the chain:
├── camera_streamer.py — robot-side RTP sender (per-camera supervisor)
├── pipeline/ — source ABC + threaded runner
├── placements/ — XR lock-mode strategies (world / head / lazy)
├── sources/ — V4L2 / OAK-D / ZED / video replay / synthetic / rtp_h264
├── sources/ — V4L2 / OAK-D / ZED / Argus / video replay / synthetic / rtp_h264
├── transports/ — RTP sender + receiver (native + GStreamer)
├── codec/ — native NVENC / NVDEC pybind module
├── argus/ — optional Jetson libargus / CUDA EGLStream pybind module
├── configs/ — one YAML per source kind
├── test_data/ — sample replay clip (Git LFS)
└── scripts/ — installer + systemd unit template
Expand All @@ -305,6 +355,8 @@ Televiz as the compositor at the end of the chain:
hand frames to a threaded runner in :code-dir:`pipeline/ <examples/camera_viz/pipeline>`. Each
source produces GPU frames where possible — e.g. the ZED source uses ``retrieve_image(MEM.GPU)`` so
BGRA8 stays in VRAM and a CUDA kernel channel-swaps into contiguous RGBA with no host round-trip.
The optional Argus source consumes Jetson EGLStreams with CUDA and publishes RGBA buffers without a
CPU readback.
- **The viewer** (:code-file:`camera_viz.py <examples/camera_viz/camera_viz.py>`) creates a
``VizSession`` and adds one ``QuadLayer`` per enabled camera, then submits each frame to its layer
and calls ``render()`` once per frame. Stereo cameras submit both eyes.
Expand Down
2 changes: 1 addition & 1 deletion examples/camera_viz/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ source examples/camera_viz/.venv/bin/activate

`setup` installs `isaacteleop` (which bundles Televiz) and every other Python dep from PyPI into `.venv/` via `uv` (no `--system-site-packages`), builds the native NVENC/NVDEC codec, and probes system packages (GStreamer plugins, cairo / girepository headers, JetPack `cuda-nvrtc` + ld.so wiring). If anything's missing it prints the exact `apt-get` line and prompts `[y/N]` — `n` or non-interactive aborts. No need to build IsaacTeleop from source.

Flags: `--no-{v4l2,oakd,rtp}`, `--with-zed`, `--sender-only`, `--jetson`. Pass `--venv PATH` to install into an existing venv (symlinks `.venv` → PATH so `run` / `loopback` pick it up too).
Flags: `--no-{v4l2,oakd,rtp}`, `--with-argus`, `--with-zed`, `--sender-only`, `--jetson`. Pass `--venv PATH` to install into an existing venv (symlinks `.venv` → PATH so `run` / `loopback` pick it up too).

> **Developing against a local build?** Pass `--wheel <path>` (e.g. `camera_viz.sh setup --wheel build/wheels/isaacteleop-*.whl`) to install a locally built wheel instead of the PyPI release. See the [build-from-source guide](../../docs/source/getting_started/build_from_source/index.rst).

Expand Down
115 changes: 115 additions & 0 deletions examples/camera_viz/argus/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Native Argus camera source for camera_viz. The .so lands next to
# __init__.py so the package imports without an install step.

cmake_minimum_required(VERSION 3.20)
project(camera_viz_argus LANGUAGES C CXX CUDA)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CUDA_STANDARD 17)
set(CMAKE_CUDA_STANDARD_REQUIRED ON)
set(CMAKE_POSITION_INDEPENDENT_CODE ON)

if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
set(CMAKE_CUDA_ARCHITECTURES "80;86;87;89;90")
endif()

find_package(CUDAToolkit REQUIRED)
find_package(Python3 REQUIRED COMPONENTS Interpreter Development.Module)

execute_process(
COMMAND ${Python3_EXECUTABLE} -c "import pybind11; print(pybind11.get_cmake_dir())"
OUTPUT_VARIABLE pybind11_DIR
OUTPUT_STRIP_TRAILING_WHITESPACE
RESULT_VARIABLE _pybind11_rc
)
if(NOT _pybind11_rc EQUAL 0)
message(FATAL_ERROR "pybind11 not found. `uv pip install pybind11` into the active venv.")
endif()
find_package(pybind11 REQUIRED CONFIG)

list(APPEND CMAKE_MODULE_PATH "/usr/src/jetson_multimedia_api/argus/cmake")
find_package(Argus QUIET)
find_package(EGL QUIET)

if(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64|arm64")
set(_tegra_lib_dirs
"/usr/lib/aarch64-linux-gnu/nvidia"
"/usr/lib/aarch64-linux-gnu/tegra"
"/usr/lib/${CMAKE_LIBRARY_ARCHITECTURE}/nvidia"
"/usr/lib/${CMAKE_LIBRARY_ARCHITECTURE}/tegra"
)
else()
set(_tegra_lib_dirs
"/usr/lib/x86_64-linux-gnu/nvidia"
"/usr/lib/${CMAKE_LIBRARY_ARCHITECTURE}/nvidia"
)
endif()

if(NOT ARGUS_INCLUDE_DIRS)
find_path(ARGUS_INCLUDE_DIRS Argus/Argus.h
PATHS "/usr/src/jetson_multimedia_api/argus/include"
"/usr/src/jetson_multimedia_api/include"
)
endif()
if(NOT ARGUS_LIBRARIES)
find_library(ARGUS_SOCKETCLIENT_LIBRARY
NAMES nvargus_socketclient
PATHS ${_tegra_lib_dirs}
)
find_library(ARGUS_LIBRARY
NAMES nvargus
PATHS ${_tegra_lib_dirs}
)
if(ARGUS_SOCKETCLIENT_LIBRARY)
set(ARGUS_LIBRARIES ${ARGUS_SOCKETCLIENT_LIBRARY})
elseif(ARGUS_LIBRARY)
set(ARGUS_LIBRARIES ${ARGUS_LIBRARY})
endif()
endif()
if(NOT ARGUS_INCLUDE_DIRS OR NOT ARGUS_LIBRARIES)
message(FATAL_ERROR "Argus headers/libs not found. Install Jetson Multimedia API / libargus.")
endif()

if(NOT EGL_INCLUDE_DIRS)
find_path(EGL_INCLUDE_DIRS EGL/egl.h)
endif()
if(NOT EGL_LIBRARIES)
find_library(EGL_LIBRARIES NAMES EGL)
endif()
if(NOT EGL_INCLUDE_DIRS OR NOT EGL_LIBRARIES)
message(FATAL_ERROR "EGL headers/libs not found.")
endif()

message(STATUS "camera_viz argus: ARGUS=${ARGUS_LIBRARIES}")
message(STATUS "camera_viz argus: EGL=${EGL_LIBRARIES}")

pybind11_add_module(_camera_viz_argus MODULE
argus_bindings.cpp
argus_camera.cpp
yuv_to_rgba.cu
)

target_include_directories(_camera_viz_argus PRIVATE
${CMAKE_CURRENT_SOURCE_DIR}
${ARGUS_INCLUDE_DIRS}
${EGL_INCLUDE_DIRS}
)

target_compile_options(_camera_viz_argus PRIVATE
$<$<COMPILE_LANGUAGE:CXX>:-Wall -Wextra -Wno-unused-parameter>
)

target_link_libraries(_camera_viz_argus PRIVATE
CUDA::cudart
CUDA::cuda_driver
${ARGUS_LIBRARIES}
${EGL_LIBRARIES}
)

set_target_properties(_camera_viz_argus PROPERTIES
LIBRARY_OUTPUT_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
)
19 changes: 19 additions & 0 deletions examples/camera_viz/argus/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Native Argus camera source - pybind11 module re-exports.

Build the .so with ``argus/build.sh``. This source is Jetson-only and
requires libargus, EGL, and CUDA from JetPack.
"""

from __future__ import annotations

try:
from ._camera_viz_argus import ArgusCamera, ArgusConfig, FrameView
except ImportError as e:
raise ImportError(
"camera_viz native Argus source not built. Run "
"`examples/camera_viz/argus/build.sh` on the Jetson."
) from e

__all__ = ["ArgusCamera", "ArgusConfig", "FrameView"]
58 changes: 58 additions & 0 deletions examples/camera_viz/argus/argus_bindings.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

#include "argus_camera.hpp"

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>

namespace py = pybind11;
using camera_viz::argus::ArgusCamera;
using camera_viz::argus::ArgusConfig;
using camera_viz::argus::FrameView;

PYBIND11_MODULE(_camera_viz_argus, m)
{
py::class_<ArgusConfig>(m, "ArgusConfig")
.def(py::init<>())
.def_readwrite("name", &ArgusConfig::name)
.def_readwrite("sensor_ids", &ArgusConfig::sensor_ids)
.def_readwrite("sensor_mode", &ArgusConfig::sensor_mode)
.def_readwrite("width", &ArgusConfig::width)
.def_readwrite("height", &ArgusConfig::height)
.def_readwrite("fps", &ArgusConfig::fps)
.def_readwrite("gpu_id", &ArgusConfig::gpu_id)
.def_readwrite("full_range", &ArgusConfig::full_range)
.def_readwrite("swap_uv", &ArgusConfig::swap_uv)
.def_readwrite("acquire_timeout_ms", &ArgusConfig::acquire_timeout_ms)
.def_readwrite("repeat_capture", &ArgusConfig::repeat_capture);

py::class_<FrameView>(m, "FrameView")
.def_readonly("left_ptr", &FrameView::left_ptr)
.def_readonly("left_pitch", &FrameView::left_pitch)
.def_readonly("right_ptr", &FrameView::right_ptr)
.def_readonly("right_pitch", &FrameView::right_pitch)
.def_readonly("width", &FrameView::width)
.def_readonly("height", &FrameView::height)
.def_readonly("timestamp_ns", &FrameView::timestamp_ns)
.def_readonly("sequence", &FrameView::sequence)
.def_readonly("stereo", &FrameView::stereo);

py::class_<ArgusCamera>(m, "ArgusCamera")
.def(py::init<const ArgusConfig&>())
.def("start", &ArgusCamera::start, py::call_guard<py::gil_scoped_release>())
.def("stop", &ArgusCamera::stop, py::call_guard<py::gil_scoped_release>())
.def("latest",
[](ArgusCamera& camera) -> py::object
{
auto frame = camera.latest();
if (!frame)
{
return py::none();
}
return py::cast(*frame);
})
.def_property_readonly("is_stereo", &ArgusCamera::is_stereo)
.def_property_readonly("width", &ArgusCamera::width)
.def_property_readonly("height", &ArgusCamera::height);
}
Loading
Loading