The simulation is configured to run at 1.0 RTF (100% real-time factor), meaning simulation time should match wall-clock time. If you're experiencing lower RTF, the following sections may help diagnose and resolve the issue.
If your machine has two GPUs (or a CPU with an integrated GPU), OpenGL may be using the integrated GPU for rendering, which causes RTF to be very low. To fix this, you may need to manually force it to use the discrete GPU.
To check if Open GL is using the discrete GPU, run glxinfo -B. The output should show the details of your discrete GPU. Additionally, you can verify GPU-specific process by running nvidia-smi. When the AIC sim is active, gz sim should appear in the process list.
If the wrong GPU is selected, run sudo prime-select nvidia.
Note: You must log out and log in again for the changes to take effect. Then, re-run glxinfo -B to verify that the discrete GPU is active.
You can also check out Problems with dual Intel and Nvidia GPU systems.
If your system doesn't have a dedicated GPU, you may experience poor real-time factor (RTF) performance. This is because Gazebo uses GlobalIllumination (GI) based rendering for the AIC scene, which requires GPU acceleration for optimal performance.
To improve simulation performance on systems without a GPU:
You can disable GlobalIllumination by editing aic.sdf and setting <enabled> to false in the global illumination configuration here and here. This will reduce rendering quality but may significantly improve RTF on CPU-only systems.
Warning
Disabling GI will change the visual appearance of the scene, which may affect vision-based policies.
When running the system, you may see warnings like:
WARN Watchdog Validator ThreadId(17) zenoh_shm::watchdog::periodic_task:
error setting scheduling priority for thread: OS(1), will run with priority 48.
This is not an hard error and it can be safely ignored under normal operating conditions.
This warning is harmless and can be safely ignored. It indicates that Zenoh's shared memory watchdog thread couldn't set a higher scheduling priority (which requires elevated privileges). The system will continue to work correctly.
Why it happens:
- The watchdog thread monitors shared memory health
- Setting higher priority requires
CAP_SYS_NICEcapability or root privileges - Without it, the thread runs at default priority (48)
When it might matter:
- Under extremely high CPU load, the watchdog may occasionally miss its deadlines
- This could cause rare timeouts in shared memory operations
- In practice, this is almost never an issue for typical workloads
To verify shared memory is working:
# Check for Zenoh shared memory files
ls -lh /dev/shm | grep zenoh
# Monitor network traffic (should be minimal)
sudo tcpdump -i lo port 7447 -vIf you see Zenoh files in /dev/shm and minimal traffic on port 7447, shared memory is functioning correctly despite the warning.
UserWarning:
NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.
If you want to use the NVIDIA GeForce RTX 5090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
The lerobot version in pixi.toml depends on an older version of pytorch (built for an older version of cuda).
pixi install will pull in that older version which does not support the newer sm_120 architecture for NVIDIA RTX 50xx cards.
We were able to run this policy on an Nvidia RTX 5090 by adding the following to pixi.toml:
[pypi-options.dependency-overrides]
torch = ">=2.7.1"
torchvision = ">=0.22.1"
See this LeRobot issue for details.
when running distrobox enter -r aic_eval, you might encounter the following error:
Error: no such container aic_evalBy default, distrobox uses podman but we are using docker in our setup. Make sure to have set the default container manager by exporting the DBX_CONTAINER_MANAGER environment variable:
export DBX_CONTAINER_MANAGER=dockerIf a policy passes the local verification test but fails on the portal with no error or stdout logs, the cause is often one of the following:
The task time limit is measured against simulation time (the ROS clock), not wall-clock time. A policy that relies on time.time() or time.sleep() may work locally (where sim time closely tracks wall time at ~1.0 RTF) but misbehave on the portal if sim and wall time diverge. Use the ROS clock for any time-based logic inside the policy.
When the policy module is loaded, all top-level code, including imports, runs within a 30-second model discovery budget (model_discovery_timeout_seconds). Importing large libraries such as torch at the top of the module can exceed this budget and cause the policy to be killed before it reports an error.
Move heavy imports into the policy class insert_cable callback instead.
It is important to not slow down the __init__ of the policy, as aic_engine will be periodically querying lifecycle state of aic_model, and the __init__ method of the Policy object will block responses to these queries.
This can cause aic_engine to assume the policy node has failed.
If you have large or slow-running imports, it is safest to run them inside the insert_cable callback of your policy.
The task board has up to five different NIC cards, even though only NIC cards 0 and 1 are used in the provided aic_engine sample_config.yaml .
We recommend running code locally also using NIC cards 2, 3, and 4, to ensure there are no hard-coded assumptions that cause exceptions when using these cards.
This can be done by modifying sample_config.yaml and rebuilding the eval container, or by using the pre-built eval container and following the launch parameter instructions as described in the Scene Description doc page.