Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Integration — Extended Reality (XR)

SRB’s optional XR path uses Isaac Sim’s OpenXR pipeline for headset viewing during teleoperation, demonstration collection, and scene inspection. It targets Isaac Sim 6.1.0 with Isaac Lab 3.0.0 and remains unverified until a headset run records the simulator pair, GPU, display, and OpenXR runtime.

Motivation

XR adds an immersive viewpoint to human-in-the-loop workflows.

Hardware & Software Requirements

XR mode relies on Isaac Sim’s OpenXR runtime.

Evidence status: unverified. No headset-specific test or run has exercised XR against SRB. Unit tests cover XR launch defaults, but do not establish headset compatibility; the compatibility statement below is inherited from Isaac Sim’s support matrix, not measured here. Read it as an expectation, not an SRB support claim — the same distinction the compatibility matrix draws for tasks.

Isaac Sim’s support matrix lists Meta Quest (Link/Air Link), Valve Index, HTC Vive, and Windows Mixed Reality devices. This is a vendor compatibility expectation, not SRB headset evidence. See the Isaac Sim 6.1 documentation for supported devices and runtime setup.

Enabling XR

Two CLI flags control XR behavior:

  • --xr — enables the XR session for VR/AR applications.
  • --rendering_mode xr — accepted for XR launches but applies no RTX preset: Isaac Lab removed its XR preset, and the XR Kit experience that --xr selects carries the XR rendering settings. performance, balanced and quality apply SRB’s copy of Isaac Lab’s former presets.

These flags work with simulation-based agent subcommands (zero, rand, teleop, ros, train, eval, collect). They affect rendering and display; actions still come from the subcommand’s configured source (--teleop_device for teleop).

Visualization of Trained Policies in XR

--xr can be combined with eval to observe a trained policy from inside the scene — a useful debugging aid when behavior is qualitatively odd:

srb agent eval --env peg_in_hole --algo sbx_ppo --xr --rendering_mode xr

Immersive Teleoperation

Pair --xr with a standard teleop device for an immersive viewpoint while controlling the robot through the existing input modality. The headset provides depth perception and natural head motion; the device drives the actions:

srb agent teleop --env peg_in_hole --xr --rendering_mode xr \
  --teleop_device spacemouse

Demonstration Collection in XR

The same flags apply to collect. Demonstration observations still come from the sensors configured for the environment; XR changes the display, not the observation stream:

srb agent collect --env peg_in_hole --xr --rendering_mode xr \
  --teleop_device spacemouse --num_episodes 50

Tips

  • Prefer env.num_envs=1 in XR — parallel environments multiply the rendering cost and can drop below the frame rate needed for comfort.
  • If stereo rendering causes motion sickness, lower env.num_envs or switch back to --rendering_mode performance and use the headset only for inspection.
  • XR collection records wall-clock time; use --horizon to cap collected episodes.
  • XR controller-as-teleop-input is not currently wired into SRB’s teleop device set; track upstream Isaac Sim XR additions if this is a future requirement.

How to Get Started?

See Installation, Basic Usage, and the XR launch examples.

See Also