Basic Usage

After installation, start the sample collection task:

srb agent teleop --env sample_collection

For Docker:

1. Start the simulator profile container:

   SRB_DOCKER_PROFILE=simulator ./.docker/run.bash
   

2. Inside the container, run the teleoperation command:

   srb agent teleop --env sample_collection
   

See Docker installation for setup.

Teleoperate your 1^st Robot across Diverse Domains

Reference: srb agent teleop — Teleoperate Agent

"$ISAACSIM_PYTHON" -m srb

sample_collection lets you control a Franka manipulator with the keyboard to collect sample tubes. Isaac Sim prints the controls in the terminal.

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃  Keyboard Scheme (focus the Isaac Sim window)  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Reset: [ L ]                                   │
│ Decrease Gain [ O ]   │ Increase Gain: [ P ]   │
│ Event: [ R / K ]                               │
├────────────────────────────────────────────────┤
│ Translation                                    │
│             [ W ] (+X)            [ Q ] (+Z)   │
│               ↑                     ↑          │
│               │                     │          │
│  (-Y) [ A ] ←─┼─→ [ D ] (+Y)        ┼          │
│               │                     │          │
│               ↓                     ↓          │
│             [ S ] (-X)            [ E ] (-Z)   │
├────────────────────────────────────────────────┤
│ Rotation                                       │
│       [ Z ] ←————————(±X)————————→ [ X ]       │
│                                                │
│       [ T ] ↻————————(±Y)————————↺ [ G ]       │
│                                                │
│       [ C ] ↺————————(±Z)————————↻ [ V ]       │
└────────────────────────────────────────────────┘

Note: Most tasks employ action spaces that support direct teleoperation (e.g. via Inverse Kinematics). However, some tasks such as locomotion_velocity_tracking rely on low-level control of individual joints. In this case, direct teleoperation is not supported, and you will need to provide a control policy that maps your teleoperation commands to low-level control signals. Further instructions are provided in the section for Teleoperation via Policy.

Move to a Different Domain

Reference: Environment Configuration
Reference: Environment Configuration — Domain

What if we want to collect natural samples on the Moon instead? Luckily, you can easily configure many aspects of the environment through Hydra. For instance, you can adjust the domain and configure the sample asset to spawn procedurally generated Lunar rock samples (moon_rock):

srb agent teleop --env sample_collection env.domain=moon env.sample=moon_rock

Observe Random Agents in Action

Reference: srb agent rand — Random Agent

Run a random policy in locomotion_velocity_tracking, which uses the Spot quadruped by default:

srb agent rand --env locomotion_velocity_tracking

Hint: Use --hide_ui option to disable most of the Isaac Sim UI, as shown in the video above.

Change the Robot

Reference: Environment Configuration — Robot

Selecting a different robot for any environment is as simple as adjusting the env.robot parameter. This particular environment accepts legged robots and humanoids, so let’s try the Unitree G1 humanoid:

srb agent rand --env locomotion_velocity_tracking env.robot=unitree_g1

Simulate Multiple Robots

Reference: Environment Configuration — Parallelism

Set env.num_envs to run parallel instances; this example uses 16 Cassie robots:

srb agent rand -e locomotion_velocity_tracking env.robot=cassie env.num_envs=16

For task defaults and stacking effects, see Environment Stacking; set SF_BAKER=0 to speed up generation.

SF_BAKER=0 srb agent rand -e locomotion_velocity_tracking env.stack=false env.num_envs=16

Explore & Experiment with Environment Templates

Reference: Assets
Reference: Environment Configuration — Robot

Environment templates provide starting points for custom scenarios:

This example combines a mobile Spot quadruped and Franka manipulator:

srb agent rand -e _ground_manipulation

Diversify Robot Configurations

Randomize configurations across parallel instances. Mobile manipulators use env.robot.mobile_base and env.robot.manipulator separately:

srb agent rand -e _ground_manipulation env.robot.mobile_base=random_unitree_quadruped env.robot.manipulator=random_ur_manipulator env.num_envs=6 env.stack=true

Customize Payloads & End Effectors

Reference: srb agent zero — Zero Agent

Configure mobile robot payloads and manipulator end effectors through env.robot or the corresponding mobile-base/manipulator fields. Join each base and attachment with +:

srb agent zero -e _ground_manipulation env.robot.mobile_base=anymal_d+cargo_bay env.robot.manipulator=unitree_z1+shadow_hand

Hint: Both sides of the + must be specified — e.g. anymal_d+cargo_bay or unitree_z1+shadow_hand. The shorthand forms +payload/+end_effector (omitting the base) and mobile_robot+/manipulator+ (omitting the attachment) are not supported and raise an error.

Valid combinations vary by task; sensor poses and action dimensions adjust automatically.

srb agent zero -e _aerial_manipulation env.robot.mobile_base=ingenuity env.robot.manipulator=franka+scoop

srb agent zero -e _orbital_manipulation env.robot.mobile_base=gateway env.robot.manipulator=canadarm3

Enable Visual Sensors

Most tasks disable visual sensors by default. Where supported, add visual to env.active_sensors to enable pre-configured cameras. Available image types depend on task configuration.

Let’s see a camera view of the Ingenuity helicopter on Mars:

srb agent teleop -e _aerial 'env.active_sensors=[proprioception,contact,transform,visual]'

Hint: You can also tweak env.camera_resolution (default (64, 64)), env.camera_data_types (default ("rgb", "depth")), env.camera_update_period, and env.camera_clipping_range (near and far planes in metres, filled only on cameras without an explicit range) to control the camera streams.

Optional: Browse Registered Assets & Environments

Reference: srb ls — List Assets and Environments

List registered assets, action groups, hardware interfaces, and environments:

srb ls --live

For cached and live behavior, see srb ls.

--live lists four registries: assets (sceneries, objects, robots), action groups, hardware interfaces, and environment templates and tasks. Hardware interface availability does not establish physical validation or readiness.

What’s Next?

Continue with:

• Workflows: ROS 2
• Workflows: Reinforcement Learning
• Contributing — New Assets
• Contributing — New Tasks