Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Integration — Extended Reality (XR)

The Space Robotics Bench supports Extended Reality (XR) through Isaac Sim’s OpenXR pipeline. This enables immersive teleoperation, demonstration collection, and scene inspection using commodity VR/AR headsets, which is especially valuable for designing human-in-the-loop workflows for space robotics.

Motivation

  • Immersive teleoperation - Operators can place themselves inside the simulated scene, giving them accurate depth perception and natural six-DoF control — properties that are difficult to match with a screen-and-mouse setup.
  • High-fidelity demonstrations - Demonstrations collected in XR produce smoother, more physically plausible trajectories, which translates into better downstream Imitation Learning performance.
  • Mission rehearsal - Operators can walk through a Martian, Lunar, or orbital scene before mission execution to validate assumptions about scale, lighting, and hardware placement.
  • Outreach & education - Space robotics is inherently visual; XR lets non-expert audiences experience SRB environments at full fidelity.

Hardware & Software Requirements

XR mode relies on Isaac Sim’s OpenXR runtime. Any OpenXR-compliant headset supported by Isaac Sim should work — this notably includes Meta Quest (via Link/Air Link), Valve Index, HTC Vive, and Windows Mixed Reality devices. Consult the Isaac Sim documentation for the current XR support matrix and runtime configuration.

Enabling XR

Two CLI flags control XR behavior:

  • --xr — enables the XR session for VR/AR applications.
  • --rendering_mode xr — selects the dedicated XR rendering preset (optimized for VR frame rates and stereo rendering).

These flags are available for every simulation-based agent subcommand (zero, rand, teleop, ros, train, eval, collect). They control rendering and headset display only — action input still comes from whichever --teleop_device is configured (keyboard, spacemouse, ros, gamepad, haptic).

Visualization of Trained Policies in XR

--xr can be combined with eval to observe a trained policy from inside the scene — a useful debugging aid when behavior is qualitatively odd:

srb agent eval --env peg_in_hole --algo sbx_ppo --xr --rendering_mode xr

Immersive Teleoperation

Pair --xr with a standard teleop device for an immersive viewpoint while controlling the robot through the existing input modality. The headset provides depth perception and natural head motion; the device drives the actions:

srb agent teleop --env peg_in_hole --xr --rendering_mode xr \
  --teleop_device spacemouse

Demonstration Collection in XR

The same flags apply to collect, producing demonstrations whose observation stream reflects a fully visualized scene:

srb agent collect --env peg_in_hole --xr --rendering_mode xr \
  --teleop_device spacemouse --num_episodes 50 --success_only

Tips

  • Prefer env.num_envs=1 in XR — parallel environments multiply the rendering cost and can drop below the frame rate needed for comfort.
  • If stereo rendering causes motion sickness, lower env.num_envs or switch back to --rendering_mode performance and use the headset only for inspection.
  • XR sessions record wall-clock time, so long teleoperation bouts produce correspondingly long datasets — use --horizon to cap episode length.
  • XR controller-as-teleop-input is not currently wired into SRB’s teleop device set; track upstream Isaac Sim XR additions if this is a future requirement.

How to Get Started?

If you want to start using the Space Robotics Bench with Extended Reality, follow these steps:

  1. Installation
  2. Basic Usage
  3. Launch any simulation-based agent subcommand with the XR flags (see Enabling XR above)

See Also