Introduction

Space Robotics Bench (SRB) provides modular environments and tasks for autonomous systems across extraterrestrial scenarios, robots, and research workflows.

Start: System Requirements → Installation → Basic Usage.

SRB targets Isaac Sim 6.1.0 and Isaac Lab 3.0.0; Docker is the tested installation path.

SimForge generates scenes and assets on demand.

System Requirements

Hardware Requirements

SRB targets Isaac Sim 6.1.0 (v6.1.0) and Isaac Lab 3.0.0 (release/3.0.0). Their commits are 7c206f75bdadd9e05fc457f19863ca4c3f0cb693 and fdc650c88dfd86aa158b9a1760b6537fef360555. Docker x86_64 is tested. Native, Apptainer/HPC, and arm64 are untested. The tables show SRB’s minimum requirements and NVIDIA’s tested baseline.

SRB declared minimum

SRB supports linux/amd64 with NVIDIA RTX. Linux arm64/aarch64 installers are untested and unsupported.

NVIDIA Isaac Sim 6.1 tested baseline

NVIDIA’s Isaac Sim 6.1 requirements specify this x86_64 baseline:

Training and large sensor scenes need more RAM and VRAM. GPUs without RT Cores, including A100/H100, are unsupported. NVIDIA aarch64 support is limited to DGX Spark (DGX OS 7, driver 580.159.03); SRB does not support it.

Software Requirements

SRB requires Linux and an NVIDIA driver appropriate for the GPU. Other operating systems are unsupported.

SRB builds Isaac Sim from source and installs Isaac Lab in that Python environment. Docker also supports NVIDIA’s NGC Isaac Sim image, with different paths, UID, and cache ownership; SRB workflows use the source image.

NVIDIA Driver

Install the NVIDIA driver using NVIDIA’s official guide or your distribution’s package manager.

Next: Installation

Installation

Clone the repository with submodules, then follow one installation guide:

git clone --recurse-submodules https://github.com/AndrejOrsula/space_robotics_bench.git
cd space_robotics_bench

Use Docker with a locally built image or a release image pinned by its full sha256 digest. Do not pipe a mutable branch’s setup script into a shell: clone or authenticate and inspect the runner first. See system requirements for platform support and simulator pins.

Installed wheel support

Wheels support data and CLI commands that need no simulator assets or repository files. CLI dependencies need pip install "srb[cli]"; the base wheel omits them but includes the srb entry point and hyperparameter resources. srb test, srb doc, and simulator-launching commands need a source checkout; simulator commands also work in an SRB container with assets. Clone tagged releases with submodules, e.g. git clone --recurse-submodules --branch 0.1.0 https://github.com/AndrejOrsula/space_robotics_bench.git. GitHub source archives omit submodules and simulator assets.

Wheel runs log to logs/ in the current directory. Cache defaults to $XDG_CACHE_HOME/srb (~/.cache/srb if XDG_CACHE_HOME is unset, empty, or relative); set SRB_CACHE_ROOT to change it.

Temporary Docker sessions

The runner uses --rm, so the container filesystem is discarded on exit. See the Docker installation guide for runner profiles and image requirements.

Next: Docker installation

Installation — Native

Native installs are untested and need manual work. Builds use Isaac Sim 6.1.0 and Isaac Lab 3.0.0 (system requirements). On Ubuntu 22.04, use uv Python 3.12; system Python is older. Use Isaac Sim’s python.sh for SRB and Isaac Lab.

1. Prepare the host

sudo apt-get update
sudo apt-get install -y git git-lfs gcc-11 g++-11 cmake build-essential curl unzip xz-utils python3 jq
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.12
uv python find 3.12

Without sudo, install git-lfs locally and add ~/.local/bin to PATH:

mkdir -p ~/.local/bin /tmp/git-lfs-install
curl -sSL https://github.com/git-lfs/git-lfs/releases/download/v3.6.1/git-lfs-linux-amd64-v3.6.1.tar.gz -o /tmp/git-lfs.tar.gz
tar xzf /tmp/git-lfs.tar.gz -C /tmp/git-lfs-install
cp /tmp/git-lfs-install/git-lfs-*/git-lfs ~/.local/bin/
rm -rf /tmp/git-lfs.tar.gz /tmp/git-lfs-install

2. Initialize Git LFS

Run this before cloning, including after the local fallback:

git lfs install

3. Clone SRB

Initialize submodules, then run commands from the repository root:

git clone https://github.com/AndrejOrsula/space_robotics_bench.git
cd space_robotics_bench
git submodule update --init --recursive

4. Build Isaac Sim

Accept NVIDIA’s license before setting ACCEPT_EULA=Y. The build requires sudo, substantial time, and disk space.

ACCEPT_EULA=Y \
SRC_BRANCH=release/6.1 \
SRC_COMMIT_SHA=7c206f75bdadd9e05fc457f19863ca4c3f0cb693 \
  ./.docker/install/install_isaacsim.bash "$HOME/isaac-sim"

5. Configure the Isaac Sim environment

Copy the installer’s export VK_DRIVER_FILES=... line. It detects or creates the Vulkan ICD path. Add all exports to shell config, source it, and keep ~/.local/bin on PATH for CLI wrappers.

Environment Variables

For bash/zsh, append with cat >> ~/.bashrc << 'EOF' or cat >> ~/.zshrc << 'EOF', close EOF, and source the file. In fish, set VK_DRIVER_FILES to the ICD path printed by the installer; the example above uses a common path. ISAACSIM_ASSET_ROOT defaults to $HOME/isaac-sim/assets; use the printed root and profile.

6. Install Isaac Lab

Install the pinned Isaac Lab source. Its script resolves NumPy conflicts and installs array_api_compat.

SRC_BRANCH=release/3.0.0 \
SRC_COMMIT_SHA=fdc650c88dfd86aa158b9a1760b6537fef360555 \
  ./.docker/install/install_isaaclab.bash "$HOME/isaaclab"

7. Install Blender and SimForge

Follow Blender’s official Linux instructions, then use SRB’s Blender installer for Blender 4.5 and SimForge:

./.docker/install/install_blender.bash "$HOME/blender"

Installer downloads Blender, links ~/.local/bin/blender, installs Docker-pinned simforge[dev] 0.4.0 and simforge-foundry 0.3.1 in Blender Python, checks exact pins before download, then applies SRB’s Isaac Lab compatibility patch (.docker/install/patch_simforge_isaaclab.bash). Avoid Snap: it blocks Python dependency integration.

8. Install optional RL frameworks

Source your shell configuration first; most scripts require ISAACSIM_PYTHON.

The TD-MPC2 script pins upstream and applies srb/integrations/tdmpc2/patches/, including the Q-ensemble initialization fix. It skips upstream’s dependency-pin commit to avoid downgrading Isaac Sim Torch/TensorDict/TorchRL. SRB rejects checkouts without the backport; use this script or the error’s pin, never plain git clone.

Set TDMPC2_PATH:

# Bash
echo "export TDMPC2_PATH='$HOME/tdmpc2/tdmpc2'" >> ~/.bashrc && source ~/.bashrc
# Zsh
echo "export TDMPC2_PATH='$HOME/tdmpc2/tdmpc2'" >> ~/.zshrc && source ~/.zshrc
# Fish
set -Ux TDMPC2_PATH "$HOME/tdmpc2/tdmpc2"

The robomimic checkout remains in place; no SRB extra includes it.

9. Install SRB

Install SRB with optional dependencies for all workflows:

"$ISAACSIM_PYTHON" -m pip install --editable ".[all]"

Adjust the all extra in pyproject.toml as needed. Published metadata rejects direct Git references, so extras omit Git-pinned upstreams; install robomimic in step 8. all omits the v2e event-camera backend and E2VID reconstruction operator. Install these pinned groups with bundled pip 25.1 or newer:

"$ISAACSIM_PYTHON" -m pip install --upgrade 'pip>=25.1'
"$ISAACSIM_PYTHON" -m pip install --editable ".[event-v2e,event-reconstruction]" --group source-event-v2e --group source-event-reconstruction

10. Set up the CLI

Put executable ISAACSIM_PYTHON and its destination (default ~/.local/bin) on PATH. Setup uses ISAACSIM_PYTHON or fallback python3 for SRB’s Hydra completion cache. register-python-argcomplete lookup checks Isaac Sim, system PATH, then Isaac Sim’s Python package.

./.docker/helpers/setup_cli.bash
# Regenerate completions only for these shells
./.docker/helpers/setup_cli.bash --completions-only bash zsh fish
srb completions generate
srb completions install
exec "$SHELL" -l

The setup script copies srb, simforge, and space_robotics_bench shims to ~/.local/bin and configures detected shells; it exits with an error when no requested shell gets completions. Hydra override completions need jq. If setup fails or shims are skipped, run "$ISAACSIM_PYTHON" -m srb.

11. Verify the installation

The first Isaac Sim launch takes longer while shaders compile. See Troubleshooting for failures.

Troubleshooting

“Failed to create any GPU devices”

Symptom: Isaac Sim cannot create a GPU device. Cause: VK_DRIVER_FILES does not point to a valid Vulkan ICD file. Fix: Check the configured path, then set it to the existing ICD file.

ls -la "$VK_DRIVER_FILES"
find /etc/vulkan /usr/share/vulkan -name "nvidia_icd.json" 2>/dev/null

Update VK_DRIVER_FILES in your shell configuration.

“Application failed to start”

Symptom: Isaac Sim fails to start. Cause: DISPLAY is unset. Fix: On a headless server, set it for the virtual display:

# Bash/Zsh
echo 'export DISPLAY=:99' >> ~/.bashrc  # or ~/.zshrc
# Fish
set -Ux DISPLAY :99

numpy version conflicts

Symptom: NumPy version errors return. Cause: Manual package changes reintroduced conflicts after the installer resolved them. Fix: Install compatible packages:

"$ISAACSIM_PYTHON" -m pip install "numpy>=2.1" array_api_compat

Extras

Development

See IDE configuration.

Next: Basic Usage

Installation — Docker (Recommended)

Docker isolates SRB. Its source image builds the Isaac Sim and Isaac Lab versions in system requirements. Headless runtime and task catalog are tested; GUI and WebRTC are untested.

1. Clone the Repository

git clone --recurse-submodules https://github.com/AndrejOrsula/space_robotics_bench.git
cd space_robotics_bench

2. Install Docker Engine & NVIDIA Container Toolkit

Follow the official Docker Engine, Linux post-install, and NVIDIA Container Toolkit instructions. Interactive use needs a working host NVIDIA driver (nvidia-smi). GUI forwarding also needs xauth (sudo apt-get install -y xauth) and an X11 session with DISPLAY set. Run the pinned installer:

./.docker/host/install_docker.bash

Start a new login session after adding the user to the docker group for passwordless Docker. Until then, SRB helpers use sudo docker ... as needed.

3. Run

Build the image and start the runner:

SRB_IMAGE_NAME=space_robotics_bench ./.docker/build.bash 6.1.0
export SRB_DOCKER_IMAGE='space_robotics_bench:6.1.0'
./.docker/run.bash

Local tags resolve to immutable image IDs; missing tags fail without remote fallback. Default safe is interactive, isolates networking/IPC, drops capabilities, and has no host devices or mounts. Use it for lightweight commands without Isaac Sim:

DOCKER_TTY=false ./.docker/run.bash srb --help

For remote releases, provide the complete registry digest; mutable tags are rejected:

SRB_DOCKER_IMAGE='<registry>/<image>@sha256:<64-hex-digest>' ./.docker/run.bash srb --help

Keep the image reference consistent across run.bash, dev.bash, and any join/HPC wrapper used together.

Isaac Sim requires GPU access and host networking for ROS 2 discovery and livestreaming. When an NVIDIA GPU is detected or force-enabled, simulator exposes all host-visible NVIDIA GPUs; the runner has no per-GPU selector. Select simulator:

SRB_DOCKER_PROFILE=simulator ./.docker/run.bash

Unprivileged simulator enables no-new-privileges, NVIDIA runtime when an NVIDIA GPU is detected or force-enabled, /dev/dri when available on the host, host networking/IPC, writable simulator caches, GUI forwarding/history, and writable /root/ws; it neither uses --privileged nor mounts full /dev. Disable host networking or use private IPC with explicit shared memory (default 8g):

# Bridge networking (no ROS 2 multicast discovery with host nodes)
SRB_DOCKER_PROFILE=simulator WITH_HOST_NETWORK=false ./.docker/run.bash

# Private IPC namespace with an explicit /dev/shm size (default 8g)
SRB_DOCKER_PROFILE=simulator WITH_HOST_IPC=false SRB_DOCKER_SHM_SIZE=16g ./.docker/run.bash

trusted-simulator enables --privileged, mounts all /dev, and prints a security warning. Use only with a locally trusted image or authenticated digest for hardware passthrough (cameras, serial adapters, joysticks):

SRB_DOCKER_PROFILE=trusted-simulator ./.docker/run.bash

Useful variants:

# Headless simulator session
SRB_DOCKER_PROFILE=simulator WITH_GUI=false ./.docker/run.bash

# Without GPU and GUI, for commands that do not launch Isaac Sim
SRB_DOCKER_PROFILE=simulator WITH_GPU=false WITH_GUI=false ./.docker/run.bash

# Simulator without writable source or shell history
SRB_DOCKER_PROFILE=simulator WITH_DEV_VOLUME=false WITH_HISTORY=false ./.docker/run.bash

# Force the NVIDIA runtime when GPU detection fails (diagnostics only)
SRB_DOCKER_PROFILE=simulator WITH_GPU_FORCE_NVIDIA=true ./.docker/run.bash

# Non-interactive command execution, for CI and scripts
DOCKER_TTY=false ./.docker/run.bash bash -lc "srb --help"

Pass variables with -e; bare -e NAME forwards secrets. Exported caller variables use a private temporary env file hidden from Docker’s command line and ps. Invalid entries exit 2: empty names, whitespace or leading # in names, invalid bare-name identifiers, or newline values. Errors expose only entry position and length. Docker applies -e before the entrypoint adds PYTHONPATH, LD_LIBRARY_PATH, PATH, and ROS entries; override them inside the command.

While the container runs, join from another terminal:

./.docker/join.bash

Both scripts accept a trailing command:

SRB_DOCKER_PROFILE=simulator ./.docker/run.bash bash
./.docker/join.bash 0 fish

4. Verify Installation

Verify inside the container; see Troubleshooting for help.

Isaac Sim

If using default safe, restart from the host with SRB_DOCKER_PROFILE=simulator ./.docker/run.bash, check GPU visibility, then launch Isaac Sim:

nvidia-smi

"$HOME/isaac-sim/isaac-sim.sh"

The first launch may take longer during shader compilation and environment setup.

Space Robotics Bench

srb --help
srb cache update

srb cache update launches Isaac Sim and builds the offline task catalog, which enables tab completion of task names and overrides. The simulator profile mounts the repository at /root/ws, so the catalog persists in .cache/ across containers.

GUI Forwarding

echo "$DISPLAY"
test -n "$XAUTHORITY" && ls -l "$XAUTHORITY"

Extras

Build a New Docker Image

Build a custom image with the script:

./.docker/build.bash

Build RUN steps use bridge networking and daemon DNS by default.

Symptom: Could not resolve host: github.com or uv reports operation timed out during a build.

Cause: VPN-only, split-horizon, or loopback stub DNS (such as systemd-resolved at 127.0.0.53) can block bridge DNS.

Fix: Auto mode probes bridge DNS and uses --network host with a warning only when lookup completes without a matching host; other failures stop. Set SRB_BUILD_NETWORK to any docker build --network value:

SRB_BUILD_NETWORK=default ./.docker/build.bash  # keep the bridge, fail loudly
SRB_BUILD_NETWORK=host ./.docker/build.bash     # skip the probe, always host

Forwarded --network takes precedence. For a durable fix, set reachable "dns" in /etc/docker/daemon.json and restart Docker.

Isaac Lab 3.0 pins runtime deps in root pyproject.toml; extensions declare none. Builds install extensions editable, then root deps via .docker/helpers/isaaclab_root_requirements.py (requirements/overrides). Documented omissions: Newton viewer (SRB is PhysX-only; PhysX articulations import newton); Isaac Lab’s transformers pin (SRB lock selects another via robomimic); rsl-rl-lib (already locked). RSL_RL_DEV=true replaces the locked library with an editable checkout at RSL_RL_COMMIT_SHA (off by default).

Image Dependencies

Dockerfile exports committed uv.lock with --frozen --all-extras --group image-runtime --group vcs-sources, then installs the editable project with --no-deps. image-runtime locks shared simulator/framework deps; vcs-sources pins Git robomimic, v2e, and e2vid, which PyPI metadata rejects. Neither is a published SRB extra. Registry artifacts retain lock hashes; Git sources use full commit refs. By default, ROBOMIMIC_DEV=true omits robomimic for the patched checkout; pinned checkouts are explicit build inputs. Arm64 omits unavailable usd-core and uses bundled USD.

The robomimic checkout lazily patches e10526b9a40c78b41f1e37e60041dc0ec0a5f60f; builds verify 9e88eb3288079c6014f912e8d805c76dcd40157a. Low-dimensional dataset imports/learning need no CLIP weights; language observations need the model and tokenizer.

After changing Isaac Sim or Isaac Lab pins, rebuild and remove old-pin caches with srb clean.

Optional NVIDIA NGC Isaac Sim Image

The ngc profile runs NVIDIA’s official Isaac Sim 6.1 image (nvcr.io/nvidia/isaac-sim), pinned to an architecture-specific digest:

SRB_DOCKER_PROFILE=ngc SRB_NGC_MODE=headless ./.docker/run.bash

Modes are SRB_NGC_MODE=headless (default), webrtc, and gui, with ngc-headless, ngc-webrtc, and ngc-gui shorthand profiles. The container runs as UID 1234:1234; Isaac Sim is at /isaac-sim, and the checkout is at /workspace. ACCEPT_EULA=Y accepts NVIDIA’s license unless overridden. Writable caches use SRB_NGC_CACHE_DIR (default ~/.cache/space_robotics_bench/ngc/isaac-sim-6.1/<arch>). WebRTC needs host networking; GUI mode needs a real display. SRB tasks, ROS 2, and Rust are untested.

NGC always requires an RTX-class GPU; other profiles opt in with SRB_REQUIRE_RTX=true. Preflight checks NVIDIA hardware ray tracing with vulkaninfo (VK_KHR_ray_tracing_pipeline, VK_KHR_acceleration_structure), not product names: L40S, A40, GB10 qualify; GTX/software devices do not. Errors distinguish missing/failed vulkaninfo, no device, or no ray-tracing device. GUI merges X credentials to user-owned ${TMPDIR:-/tmp}/srb-xauth-<project>-<user>/Xauthority.

Join a Running Container

Use the join script:

./.docker/join.bash

For multiple containers, choose a listed index. Set SRB_CONTAINER_NAME as for run.bash; only <name> and <name><digits> appear, excluding NGC. Set DOCKER_TTY=false in CI/scripts to avoid docker exec --tty failing with the input device is not a TTY.

DOCKER_TTY=false ./.docker/join.bash 0 bash -lc "srb --help"

Development

Simulator, trusted-simulator, and development profiles mount the workspace, so edits persist on the host; safe does not. For development, open a Dev Container.

Use ./.docker/dev.bash for development variables or bind mounts. It wraps run.bash with unprivileged simulator; SRB_DOCKER_PROFILE=trusted-simulator adds --privileged and /dev (security warning). LOG_LEVEL defaults to info; set LOG_LEVEL=debug ./.docker/dev.bash ... for verbose Isaac/SimForge startup logs.

The wrapper shell places mounted /root/ws before the ROS overlay on PYTHONPATH, so srb subprocesses find checkout modules without rebuilding. On exit, created checkout files (such as .cache/, logs/, outputs/) return to the host user. SRB_DOCKER_DEV_WRAPPER=false runs commands as given.

Rust caches persist across containers. Host Cargo registry, Git, and target caches default to ~/.cache/space_robotics_bench/docker. Container builds use CARGO_TARGET_DIR=/root/.cache/srb/cargo-target, preserving image artifacts in /root/.cargo/target. Override the host cache with SRB_DOCKER_CACHE_DIR=/path/to/cache ./.docker/dev.bash ... or the container target with SRB_CARGO_TARGET_DIR=/path/in/container ./.docker/dev.bash ...; do not set the latter to /root/.cargo/target.

Next: Basic Usage

Installation — Apptainer/Singularity

Build the .sif from the Docker image, which pins Isaac Sim and Lab (requirements), then transfer it to the cluster.

Setup (Local)

1. Clone the repository

Clone from the repository root:

git clone --recurse-submodules https://github.com/AndrejOrsula/space_robotics_bench.git
cd space_robotics_bench

2. Install Docker Engine and NVIDIA Container Toolkit

Install Docker Engine, follow its Linux post-install steps, and install NVIDIA Container Toolkit. Or run the pinned helper:

./.docker/host/install_docker.bash

The helper supports Ubuntu 22.04/24.04 amd64/arm64, verifies key fingerprints, and pins Docker Engine/CLI 29.8.0 and optional NVIDIA Container Toolkit 1.20.0. Docker’s guide covers version overrides and non-interactive group behavior.

3. Install Apptainer

Follow Apptainer installation instructions or install on Ubuntu:

sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update
sudo apt install -y apptainer

4. Build the Apptainer Image

Run the build script:

./.docker/hpc/build.bash 6.1.0

This builds a tagged Docker image and converts it to .docker/hpc/images/space_robotics_bench.sif.

5. Record image details

Before transfer, record its source digest and architecture, plus SIF metadata and checksum:

docker image inspect --format '{{.Id}} {{.Architecture}}' space_robotics_bench:6.1.0 # prefix '<user>/' if `docker info` lists a Username
apptainer inspect .docker/hpc/images/space_robotics_bench.sif
sha256sum .docker/hpc/images/space_robotics_bench.sif

A docker info username prefixes the image tag: <username>/space_robotics_bench:6.1.0.

Deployment (HPC Cluster)

1. Transfer repository and image

Choose one method:

scp -r /path/to/space_robotics_bench user@hpc-cluster:/path/to/space_robotics_bench

rsync -avr /path/to/space_robotics_bench user@hpc-cluster:/path/to/space_robotics_bench

2. Connect to the cluster

ssh user@hpc-cluster

3. Run the image

On a module node, adjust run.bash. The script loads tools/Singularity and calls singularity. Because singularity exec skips the Docker entrypoint, the script sources /entrypoint.bash (ROS Jazzy).

/path/to/space_robotics_bench/.docker/hpc/run.bash bash

For NGC images:

Use the target architecture’s <arch digest> for both NGC commands. From the repository root:

SRB_DOCKER_PROFILE=ngc IMAGE_PATH=X.sif ./.docker/hpc/run.bash bash

apptainer build X.sif docker://nvcr.io/nvidia/isaac-sim@sha256:<arch digest>
ref='nvcr.io/nvidia/isaac-sim@sha256:<arch digest>'
printf 'source=%s\nsif_sha256=%s\n' "$ref" "$(sha256sum X.sif | cut -d' ' -f1)" > X.sif.ngc-source

/path/to/space_robotics_bench/.docker/hpc/run.bash bash -lc '
source /opt/ros/jazzy/setup.bash
export ROS_DOMAIN_ID=0
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
printf "%s\n" "$ROS_DISTRO" "$AMENT_PREFIX_PATH" "$LD_LIBRARY_PATH" "$RMW_IMPLEMENTATION" "$ROS_DOMAIN_ID"
srb agent ros --headless -e peg_in_hole
'

HPC GUI and WebRTC sessions are untested.

4. Verify Installation

In the container, see Troubleshooting.

Isaac Sim

nvidia-smi
"${ISAACSIM_PATH:-$HOME/isaac-sim}/isaac-sim.sh"

Initial launch may take longer because Isaac Sim compiles shaders and prepares the environment.

Space Robotics Bench

Check the SRB cache and interpreter:

srb --help
srb cache update
"${ISAACSIM_PATH:-$HOME/isaac-sim}/python.sh" -c "import isaacsim; print(isaacsim.__file__)"
"${ISAACSIM_PATH:-$HOME/isaac-sim}/python.sh" -c "import srb; print(srb.__file__)"
srb --help

isaacsim and srb paths must share an installation; the last command runs the srb found on PATH. SRB has no --version command.

Extras

Schedule a SLURM Job

Adjust the cluster SLURM script:

/path/to/space_robotics_bench/.docker/hpc/submit.bash [CMD]

It writes under $HOME/jobs/ by default and submits with sbatch. Review the partition, constraint, memory, wall time, and GPU settings.

Next: Basic Usage

Basic Usage

After installation, list registries and run an environment:

"$ISAACSIM_PYTHON" -m srb --help

1. List Assets, Actions & Environments

Reference: srb ls — List Assets and Environments

srb ls --live

Use --live for headless Isaac Sim discovery and cache refresh; plain srb ls prints cached names. See the CLI reference for categories and options.

Live tables list assets (scenery, objects, robots), action groups, hardware interfaces, and environments. Hardware interface examples are not validated on hardware.

class MarsSurface(Terrain):
    ## Scenario - The asset is suitable for the Mars domain
    DOMAINS: ClassVar[Sequence[Domain]] = (Domain.MARS,)

    ## Source selector — opt in to real DEMs by passing `source="real"`.
    source: Literal["procedural", "real"] = "procedural"
    site: str = "jezero"

    ## Model - filled in model_post_init
    asset_cfg: AssetBaseCfg | None = None  # type: ignore[assignment]

    def _build_procedural_asset_cfg(self) -> AssetBaseCfg:
        return AssetBaseCfg(
            prim_path="{ENV_REGEX_NS}/mars_surface",
            ## Spawner procedurally generates SimForge models
            spawn=SimforgeAssetCfg(
                assets=[simforge_foundry.MarsSurface()],
                collision_props=CollisionPropertiesCfg(),
                # Static triangle-mesh collision is required; otherwise
                # particles and objects fall through the SimForge mesh.
                mesh_collision_props=MeshCollisionPropertiesCfg(
                    mesh_approximation="none"
                ),
            ),
        )

    def model_post_init(self, __context):
        # Build first so the parent forwards density, scale, and texture
        # settings to the newly constructed spawner config.
        self.asset_cfg = resolve_surface_asset_cfg(
            surface_name=self.__class__.__name__,
            body="mars",
            site=self.site,
            source=self.source,
            build_procedural_asset_cfg=self._build_procedural_asset_cfg,
            build_real_asset_cfg=build_real_terrain_asset_cfg,
        )
        super().model_post_init(__context)

class SampleTube(Object):
    ## Model - Rigid object affected by physics
    asset_cfg: RigidObjectCfg = RigidObjectCfg(
        prim_path="{ENV_REGEX_NS}/sample",
        ## Spawner loads a static USD file
        spawn=UsdFileCfg(
            usd_path=(
                SRB_ASSETS_DIR_SRB_OBJECT.joinpath("sample_tube.usdc").as_posix()
            ),
            collision_props=CollisionPropertiesCfg(),
            mesh_collision_props=MeshCollisionPropertiesCfg(
                mesh_approximation="convexDecomposition"
            ),
            rigid_props=RigidBodyPropertiesCfg(),
            mass_props=MassPropertiesCfg(density=1500.0),
        ),
    )

class Franka(SerialManipulator):
    ## Model - Articulation with several links connected by joints
    asset_cfg: ArticulationCfg = ArticulationCfg(
        prim_path="{ENV_REGEX_NS}/franka",
        ## Spawner loads a static USD file
        spawn=UsdFileCfg(
            usd_path=SRB_ASSETS_DIR_SRB_ROBOT.joinpath("manipulator")
            .joinpath("franka_arm.usdz")
            .as_posix(),
            ...
        ),
        ## Initial joint configuration of the robot
        init_state=ArticulationCfg.InitialStateCfg(
            joint_pos={
                "panda_joint1": 0.0,
                "panda_joint2": 0.0,
                "panda_joint3": 0.0,
                "panda_joint4": radians(-90.0),
                "panda_joint5": 0.0,
                "panda_joint6": radians(90.0),
                "panda_joint7": radians(45.0),
            },
        ),
        ...
    )
    ## End effector - The default hand is separate to allow for easy replacement
    end_effector: Tool | None = FrankaHand()

    ## Action modes — coded-in alternatives selectable via
    ## `env.robot.action_mode=<key>`. The first entry is the default.
    action_modes: ClassVar[dict[str, ActionGroup]] = {
        "diff_ik": InverseKinematicsActionGroup(
            DifferentialInverseKinematicsActionCfg(
                asset_name="robot",
                joint_names=["panda_joint[1-7]"],
                base_name="panda_link0",
                body_name="panda_link7",
                controller=DifferentialIKControllerCfg(
                    command_type="pose",
                    use_relative_mode=True,
                    ik_method="dls",
                ),
                scale=0.1,
                body_offset=DifferentialInverseKinematicsActionCfg.OffsetCfg(),
            ),
        ),
        "osc_fixed": OperationalSpaceControlActionGroup(
            OperationalSpaceControllerActionCfg(
                asset_name="robot",
                joint_names=["panda_joint[1-7]"],
                body_name="panda_link7",
                controller_cfg=OperationalSpaceControllerCfg(
                    target_types=["pose_rel"],
                    impedance_mode="fixed",
                    motion_stiffness_task=100.0,
                    motion_damping_ratio_task=1.0,
                    nullspace_control="position",
                    inertial_dynamics_decoupling=True,
                ),
                nullspace_joint_pos_target="center",
                position_scale=0.1,
                orientation_scale=0.1,
                body_offset=OperationalSpaceControllerActionCfg.OffsetCfg(),
            )
        ),
        "osc_var_stiffness": OperationalSpaceControlActionGroup(
            OperationalSpaceControllerActionCfg(
                asset_name="robot",
                joint_names=["panda_joint[1-7]"],
                body_name="panda_link7",
                controller_cfg=OperationalSpaceControllerCfg(
                    target_types=["pose_rel"],
                    impedance_mode="variable_kp",
                    motion_stiffness_limits_task=(10.0, 250.0),
                    motion_damping_ratio_task=1.0,
                    nullspace_control="position",
                    inertial_dynamics_decoupling=True,
                ),
                nullspace_joint_pos_target="center",
                position_scale=0.1,
                orientation_scale=0.1,
                stiffness_scale=120.0,
                body_offset=OperationalSpaceControllerActionCfg.OffsetCfg(),
            )
        ),
        "osc_var_full": OperationalSpaceControlActionGroup(
            OperationalSpaceControllerActionCfg(
                asset_name="robot",
                joint_names=["panda_joint[1-7]"],
                body_name="panda_link7",
                controller_cfg=OperationalSpaceControllerCfg(
                    target_types=["pose_rel"],
                    impedance_mode="variable",
                    motion_stiffness_limits_task=(10.0, 250.0),
                    motion_damping_ratio_limits_task=(0.5, 2.5),
                    nullspace_control="position",
                    inertial_dynamics_decoupling=True,
                ),
                nullspace_joint_pos_target="center",
                position_scale=0.1,
                orientation_scale=0.1,
                stiffness_scale=120.0,
                damping_ratio_scale=1.0,
                body_offset=OperationalSpaceControllerActionCfg.OffsetCfg(),
            )
        ),
        "joint_pos_rel": JointPositionRelativeActionGroup(),
    }
    ...
    ## Frames - Relevant frames for attaching the robot and mounting tool/sensors
    frame_base: Frame = Frame(prim_relpath="panda_link0")
    frame_flange: Frame = Frame(
        prim_relpath="panda_link7",
        offset=Transform(
            pos=(0.0, 0.0, 0.107),
            rot=rpy_to_quat(0.0, 0.0, -45.0),
        ),
    )

2. Teleoperate a Robot

Reference: srb agent teleop — Teleoperate Agent

Collect Mars Sample Return tubes in sample_collection with the Franka manipulator:

srb agent teleop --env sample_collection

Isaac Sim opens; focus its window to use the terminal controls.

Note: Direct teleoperation uses high-level actions. Low-level joint tasks (e.g., locomotion_velocity_tracking) need a policy to map commands to actions; see Teleoperation via Policy.

Move to a Different Domain

Reference: Environment Configuration
Reference: Environment Configuration — Domain

Collect procedural moon_rock samples on the Moon with Hydra overrides:

srb agent teleop --env sample_collection env.domain=moon env.sample=moon_rock

3. Run Random Actions

Reference: srb agent rand — Random Agent

Run locomotion_velocity_tracking with random actions (default Spot):

srb agent rand --env locomotion_velocity_tracking

Hint: --hide_ui hides most of the Isaac Sim UI.

Change the Robot

Reference: Environment Configuration — Robot

Set env.robot to a legged robot or humanoid such as Unitree G1:

srb agent rand --env locomotion_velocity_tracking env.robot=unitree_g1

Simulate Multiple Robots

Reference: Environment Configuration — Parallelism

Run 16 Cassie instances in parallel:

srb agent rand -e locomotion_velocity_tracking env.robot=cassie env.num_envs=16

Hint: locomotion_velocity_tracking shares terrain by default (other tasks may differ). Set env.stack=false for unique terrain geometry and materials per robot. Set SF_BAKER=0 to skip texture baking and speed generation; unbaked surfaces render with a magenta placeholder material:

SF_BAKER=0 srb agent rand -e locomotion_velocity_tracking env.stack=false env.num_envs=16

4. Customize Environment Templates

Reference: Assets
Reference: Environment Configuration — Robot

Start custom scenarios from a robot-category template; sample_collection and locomotion_velocity_tracking are goal-oriented tasks.

Try _ground_manipulation with the Spot quadruped and Franka manipulator:

srb agent rand -e _ground_manipulation

Diversify Robot Configurations

Randomize Unitree quadrupeds (random_unitree_quadruped) and Universal Robots manipulators (random_ur_manipulator, UR3e–UR30); select mobile-manipulator parts via env.robot.mobile_base and env.robot.manipulator.

srb agent rand -e _ground_manipulation env.robot.mobile_base=random_unitree_quadruped env.robot.manipulator=random_ur_manipulator env.num_envs=6 env.stack=true

Customize Payloads & End Effectors

Reference: srb agent zero — Zero Agent

Use + in env.robot or its mobile-base/manipulator fields to attach Cargo Bay to Anymal D and pair Unitree Z1 with Shadow Hand:

srb agent zero -e _ground_manipulation env.robot.mobile_base=anymal_d+cargo_bay env.robot.manipulator=unitree_z1+shadow_hand

Hint: Both sides of + are required (for example, anymal_d+cargo_bay); omitting either side (+payload, +end_effector, mobile_robot+, or manipulator+) raises an error.

Task and domain determine supported combinations; sensor poses and action-space dimensions adjust automatically.

srb agent zero -e _aerial_manipulation env.robot.mobile_base=ingenuity env.robot.manipulator=franka+scoop

srb agent zero -e _orbital_manipulation env.robot.mobile_base=gateway env.robot.manipulator=canadarm3

5. Enable Cameras

Most tasks disable visual sensors for performance; some enable them by default. Where supported, add visual to env.active_sensors.

View the Ingenuity helicopter on Mars:

srb agent teleop -e _aerial 'env.active_sensors=[proprioception,contact,transform,visual]'

Next: Workflows

Workflows

Develop and validate algorithms, and generate synthetic data.

Available Workflows

• ROS 2 - Connect robotics tools and libraries.
• Reinforcement Learning - Train and evaluate policies.
• Hyperparameter Tuning - Tune hyperparameters: srb agent tune.
• Imitation Learning - Collect demonstrations; train offline.
• Sim-to-Real Transfer - Prepare simulation-trained policies for hardware; operation is unvalidated.
• Terrain Landscaping Training - Train three landscaping tasks; review limitations.

Next: ROS 2 Workflow

ROS 2 Workflow

Connect SRB to ROS 2 or Space ROS for autonomy development and validation.

The active ROS 2 agent receives actions from ROS and steps the environment. The passive interface lets a workflow control actions while ROS publishes sensors and exposes services.

1. Start an Active ROS 2 Agent

Use srb agent ros with a batched Box action space:

srb agent ros -e peg_in_hole env.num_envs=2 'env.active_sensors=[proprioception,contact,transform,visual]'

For closed-loop control, --ros_sync awaits fresh actions after step one:

srb agent ros --ros_sync -e peg_in_hole env.num_envs=2 'env.active_sensors=[proprioception,contact,transform,visual]'

See the ROS agent reference for option details.

Debug with action and observation delays at zero:

srb agent ros --ros_sync -e waypoint_navigation env.action_delay_steps=0 env.observation_delay_steps=0

List Topics

Without local ROS 2, run .docker/join.bash, then list topics:

ros2 topic list

/clock
/parameter_events
/rosout
/srb/env0/action/cmd_vel
/srb/env0/action/event
/srb/env0/action/raw
/srb/env0/cam_base/camera_info
/srb/env0/cam_base/image_depth
/srb/env0/cam_base/image_rgb
/srb/env0/cam_base/pointcloud
/srb/env0/cam_scene/camera_info
/srb/env0/cam_scene/image_depth
/srb/env0/cam_scene/image_rgb
/srb/env0/cam_scene/pointcloud
/srb/env0/cam_wrist/camera_info
/srb/env0/cam_wrist/image_depth
/srb/env0/cam_wrist/image_rgb
/srb/env0/cam_wrist/pointcloud
/srb/env0/end_effector/joint_pos
/srb/env0/end_effector/joint_states
/srb/env0/reward
/srb/env0/reward/penalty_action_rate
/srb/env0/reward/penalty_undesired_robot_contacts
/srb/env0/reward/reward_align_peg_to_hole_primary
/srb/env0/reward/reward_align_peg_to_hole_secondary
/srb/env0/reward/reward_distance_ee_to_obj
/srb/env0/reward/reward_distance_peg_to_hole_bottom
/srb/env0/reward/reward_distance_peg_to_hole_entrance
/srb/env0/reward/reward_grasp
/srb/env0/reward/reward_lift
/srb/env0/robot/delta_twist
/srb/env0/robot/joint_states
/srb/env0/terminated
/srb/env0/truncated
/srb/env1/...          (mirrors env0)
/srb/envs/action/cmd_vel
/srb/envs/action/event
/srb/envs/action/raw
/srb/envs/end_effector/joint_pos
/srb/envs/robot/delta_twist
/tf
/tf_static

Namespaces:

• /srb/env[0-N]/... targets one environment; /srb/envs/... targets all.

With env.num_envs=1, prefixes vanish, e.g. /srb/reward and /srb/action/cmd_vel.

Subscribe to Topics

Echo env0’s reward:

ros2 topic echo /srb/env0/reward

data: 2.36441707611084
---
data: 2.3652756214141846
---
data: 2.36613130569458
---
data: 2.3669850826263428
---

Publish to Topics

Publish at 20 Hz:

ros2 topic pub --rate 20 /srb/envs/action/cmd_vel geometry_msgs/Twist "{linear: {z: -0.2}, angular: {z: 0.785}}"

2. Start the Passive ROS 2 Interface

Enable the passive interface with --interface ros. Run rand on eight multi_sample_collection instances with visual sensors:

srb agent rand -e multi_sample_collection env.num_envs=8 'env.active_sensors=[proprioception,contact,transform,visual]' --interface ros

--interface accepts multiple values; following Hydra key=value overrides forward automatically.

Call Services

Reset one via /srb/env{i}/reset, or all via /srb/envs/reset. Requests are queued and apply after the next step; with --ros_sync, that step waits for a fresh action.

ros2 service list

Reset one environment:

ros2 service call /srb/env0/reset std_srvs/srv/Empty

ros2 service call /srb/envs/reset std_srvs/srv/Empty

Next: Reinforcement Learning Workflow

Reinforcement Learning Workflow

1. Train your 1^st RL Agent

Reference: srb agent train — Train Agent

Train online with srb agent train; offline algorithms use separate workflows.

Watch SBX PPO on landing; omit --headless:

srb agent train --algo sbx_ppo --env landing env.num_envs=512 --hide_ui

Check terminal progress; checkpoints save regularly. Usually stop with Ctrl+C.

2. Evaluate your Agent

Reference: srb agent eval — Evaluate Agent

srb agent eval --algo sbx_ppo --env landing env.num_envs=16

srb agent eval --algo sbx_ppo --env landing env.num_envs=16 --model logs/landing/sbx_ppo/${TIMESTAMP}/ckpt/${CHECKPOINT}

Note: SB3/SBX .zip checkpoints live in ckpt/.

3. Try a Different Algorithm

skrl: Hyperparameter profiles are untuned starting points.

srb agent train --headless --algo <ALGO> --env landing env.num_envs=1024

Hint: --headless and more parallel environments can speed training.

4. Monitor Training Progress

Compare runs with TensorBoard; local artifacts and TensorBoard are on by default. W&B is opt-in:

tensorboard --logdir ./logs --bind_all

• DreamerV3: srb agent train ... 'agent.logger.outputs=[tensorboard,wandb]'
• SB3 & SBX: srb agent train ... +agent.wandb=true
• skrl: srb agent train ... agent.agent.experiment.wandb=true

Note: W&B logging requires an account and API key.

Each run records its provenance and metrics; see Run records.

5. Configure Hyperparameters

Reference: Agent Configuration

srb agent train --algo dreamer      agent.run.train_ratio=128  ...
srb agent train --algo sb3_ppo      agent.gamma=0.99           ...
srb agent train --algo sbx_sac      agent.learning_rate=0.0002 ...
srb agent train --algo skrl_ppo     agent.models.separate=True ...

Next: Hyperparameter Tuning

Hyperparameter Tuning

Tune an algorithm with Optuna through srb agent tune. Install the tuning extra first:

uv sync --extra tune

Study modes

1. Preflight a study. Run the check before a long search. It assembles a trial without Isaac. Omit --gpus; see CLI reference.

   srb agent tune check -e <env> --algo <algo> --trial-steps <n>
   

2. Choose a study mode and run it.

   Single task: Tune one task and export its task config.

   srb agent tune run --env sample_collection --algo skrl_ppo \
     --trial-steps 20_000_000 --n-trials 40 --n-workers 2 --gpus 0,1
   

   Portfolio: Tune tasks sequentially, one child per segment. Divergence and running aggregates prune weak trials. Export globally. See portfolio details.

   srb agent tune run --tasks sample_collection peg_in_hole landing --algo skrl_ppo \
     --trial-steps 10_000_000 --n-trials 30 --normalize baseline --aggregate median
   

Inspecting and exporting

srb agent tune status --study srb-skrl_ppo-state-sample_collection-v1
srb agent tune export --study srb-skrl_ppo-state-sample_collection-v1 --to task --dry-run
srb agent tune export --study srb-skrl_ppo-state-sample_collection-v1 --to task

TPE’s one-seed default can make best_trial a lucky outlier. Use validate to rerun top trials with fresh seeds and rank by mean; see CLI reference.

Use --seeds-per-trial K to average distinct seeds during search. See its cost and pruning behavior in the CLI reference.

export records provenance and restores production budget and W&B flag. --to defaults to task for single-task, global for portfolios, where --to task is rejected. Review and commit the diff; see CLI reference for overwrite and shadow warnings.

Picking a trial budget (wall-clock worksheet)

Set --trial-steps where reference runs reach 50–70% of converged return. Production YAMLs may train until stopped (for example, 2e8). Estimate:

hours_per_trial ≈ trial_steps / (env_steps_per_second) / 3600 × n_tasks
study_hours     ≈ n_trials × hours_per_trial × seeds_per_trial / n_workers

Runtime depends on throughput, pruning, and worker overhead. At 40,000 env-steps/s, 20_000_000 steps take ~8.3 min/trial; 40 trials with 2 workers take ~2.8 h. At 5,000 env-steps/s, expect ~67 min/trial or ~22 h. Project with --fps, cap with --max-wall-hours, pilot 10–20 trials.

Config layering

Under hyperparams/, configs resolve task-specific to global by modality:

Pass --obs {state,visual} to train, eval, and tune for modality config and log directory (…/<algo>/ or …/<algo>-visual/). It prevents cross-loading and preserves the algorithm label. Visual tuning is unsupported; see Agent Configuration.

How it works

1. The Isaac-free driver samples with study.ask() and launches train children in process groups: one per seed for single-task trials, and one per task segment per seed for portfolio trials. Each child boots Isaac.
2. It writes full agent: config and env: seed and num_envs to trial_cfg.yaml. Children report episode statistics as JSONL; the driver counts total environment steps.
3. Single-task studies report smoothed return at 20 rungs; portfolios report running aggregates at task-segment rungs. The terminal budget is reserved for the final objective and is not reported as a pruning rung. MedianPruner stops weak trials with killpg; crashes and timeouts FAIL, a non-finite reported return PRUNEs, and a missing or NaN final objective FAILs. The study continues; objective is the tail mean over the final training quarter.

Isolation contains VRAM failures and NaN physics. Worker processes across nodes share a journal.

Cross-task normalization

Trial user_attrs store each task’s raw and normalized score (seg/<task>/raw, seg/<task>/norm).

Parallelism and resuming

1. Set --n-workers N slots per driver; --gpus 0,1 round-robins GPUs. Starts stagger to avoid Isaac shader-cache races until metrics arrive.
2. Share logs/tune/<study>/journal.log across nodes; each SLURM array element is one worker, and Optuna assigns trials without scheduler integration.
3. Rerun to resume; terminal and RUNNING trials count toward --n-trials. Changes to search space, budget, objective, obs mode, or task set need a new --version.
4. Abrupt driver termination may leave trials RUNNING; handled cancellation marks active trials FAIL; repair with srb agent tune repair --study S --stale-hours H.
5. Pruned child logdirs are deleted, including checkpoints and TensorBoard events. --keep-artifacts retains them; trial_cfg.yaml and metrics.jsonl remain.

Reproducibility

Each logs/tune/<study>/trials/<n>/ preserves config, seed (seed-base + trial × seeds-per-trial + replicate), and metadata. Replicates use rep-<k>/trial_cfg.yaml. metadata.json records study/trial; logs/tune/<study>/driver.json records host, pid, and start time. Records aid reconstruction, not identical results. See Agent Configuration.

Next: Imitation Learning

Imitation Learning Workflow

Collect expert demonstrations, train offline, evaluate, then deploy a compatible policy. SRB supports compatible tasks, datasets, and algorithms. Robomimic requires one environment and compatible observations and actions.

1. Collect demonstrations

Teleoperate the task and keep successful demonstrations. Follow the success rules.

srb agent collect --env peg_in_hole --num_episodes 50

The keyboard is the default. A 6-DoF SpaceMouse gives higher fidelity. An XR headset adds an immersive view, not input (Extended Reality).

srb agent collect --env peg_in_hole --teleop_device spacemouse

For collection defaults, success-only behavior, and dataset paths, see the srb agent collect reference.

2. Train an IL policy

Train offline; Isaac Sim is not needed. Pass --dataset. A fresh learn run directory does not find earlier collect/ data and errors if no dataset is supplied. Verified private HDF5 copies stay open until workers, handles, and logging close. Configured allocation limits are checked before provider reads. See limits, provenance, and --normalize_actions.

srb agent learn --algo robomimic_bc --env peg_in_hole \
  --dataset logs/peg_in_hole/collect/<timestamp>/datasets/demos_<timestamp>.hdf5

For the RNN variant, use robomimic_bc_rnn:

srb agent learn --algo robomimic_bc_rnn --env peg_in_hole \
  --dataset logs/peg_in_hole/collect/<timestamp>/datasets/demos_<timestamp>.hdf5

HDF5 stores per-step dones, terminated, and truncated. dones = terminated OR truncated marks episode boundaries. Value learners bootstrap through time-limit and collector-horizon truncations; they stop only on terminated. Robomimic bcq, cql, iql, td3_bc, and iris (BCQ value component) read dones and treat truncations as terminal.

3. Evaluate the policy

Evaluate a learn checkpoint:

srb agent eval --algo robomimic_bc --env peg_in_hole env.num_envs=1

Robomimic uses one environment:

• Collection raises robomimic collection requires num_envs == 1 if configured otherwise.
• Evaluation rejects num_envs != 1 with integration.robomimic.vector_env.unsupported.
• Offline learn creates no environment.

Evaluation checks task success at rollout end. If the task has no success signal, it warns and reports success_rate: null, never 0%.

For normalized actions, checkpoint norm_factor_min and norm_factor_max restore the environment range. The caller cannot override this range. If the checkpoint lacks a range but the run directory has logs/normalization_params.txt, evaluation errors and requires retraining. Otherwise, it loads unscaled actions.

Nested observations use slash-joined training keys.

4. Deploy to hardware

Use IL for sim-to-real with a compatible task, checkpoint, and hardware interface. Hardware operation is unvalidated. See Sim-to-Real Transfer for real_agent gen and real_agent eval.

5. Optional: iterate

Augment with policy rollouts

Collect more demonstrations with a partially trained policy (DAgger-style). Apply the same --success_only rules as in step 1.

srb agent collect --algo robomimic_bc --env peg_in_hole \
  --num_episodes 200

Rerun srb agent learn on merged HDF5 files. It accepts one --dataset for union training.

Combine with RL

IL may not generalize to unseen conditions. Incompatible checkpoint formats prevent direct RL warm-starting. See the reinforcement learning workflow:

• Train RL from scratch on the same task while IL collects demos.
• Resume interrupted RL with srb agent train --continue_training or a same-framework --model.
• Prefill skrl_sac, skrl_ddpg, or skrl_td3 replay from canonical export with --demo-dataset, --demo-max-transitions (metadata, limits, resume).
• Prefill Dreamer sequence replay from the canonical format. See continuous-action requirements, persistence, capacity, and source-bound resume.

Next: Sim-to-Real Transfer

Sim-to-Real Transfer

Generate and verify a RealEnv bridge before running a simulation-trained policy with srb real_agent. This page covers adapter wiring and evaluation, not validated hardware operation; operators remain responsible for physical safety.

Runtime and reporting limits:

• RealEnv checks every adapter before actions and post-action observations. Degraded fallback frames fail.
• Transports are not physically atomic. A later send failure cannot undo earlier sends, so RealEnv latches and attempts safe stops.
• Kortex reports show submitted commands (Cartesian metres/radians per step; joint velocity radians/s), not measured motion or controller acknowledgements.

1. Train a policy in simulation

Train a stable checkpoint; evaluation depends on algorithm, backend, and preflight gates. Dreamer’s train_real fine-tuning is experimental. Example: Dreamer on waypoint_navigation, Leo Rover, one environment, Earth gravity:

srb agent train --headless --algo dreamer --env waypoint_navigation env.robot=leo_rover env.num_envs=1 env.domain=earth

See the Reinforcement Learning Workflow.

2. Generate and verify the bridge

srb real_agent gen inspects a simulated Gymnasium environment and writes a lightweight hardware counterpart independent of the simulation backend. --hardware selects HardwareInterface drivers; for Leo Rover, ros_cmd_vel sends velocity, ros_tf supplies pose, and ros_mw provides pause/resume. Specify the robot so generation can extract settings such as action scaling.

srb real_agent gen --env waypoint_navigation env.robot=leo_rover --hardware ros_cmd_vel ros_tf ros_mw

Generation starts a temporary headless SRB session and replaces a Python module in sim_to_real/env. The module defines RealEnv with selected hardware IDs as defaults and registers under srb_real/; shipped waypoint code defaults to dummy. Find IDs with srb real_agent gen --help or browse sim_to_real/hardware.

Regenerate the waypoint module with your robot and hardware IDs before evaluation; shipped code uses cadre and dummy. Its deployment profile, configure_waypoint_navigation_hardware in srb/interfaces/sim_to_real/deploy_profiles/multi_robot.py, configures ros_tf for robot→target with 2D position and yaw, and ros_cmd_vel with the drive filter and simulator-scale clamps. The waypoint profile sets required_dynamic_frame_pairs to robot→target; these required pairs also declare observation keys. Bare ros_tf declares none. Verify profiles on the target robot.

Registration is lazy: importing srb or srb.interfaces.sim_to_real.env registers no srb_real/ ID. register_all() sweeps generated modules, and srb real_agent commands call it. srb_real/ tasks therefore do not appear in srb ls env or environment pages, which list only srb/. Shipped modules register srb_real/excavation, srb_real/formation_following, srb_real/terrain_landscaping, srb_real/terrain_landscaping_crater, and srb_real/waypoint_navigation. Agent verbs (not validate) generate a missing module in a subprocess, then exit and ask for a re-run.

See the srb real_agent gen reference.

3. Deploy and evaluate on hardware

Run policy evaluation

Check bridge routes and the production profile, then run generated RealEnv on configured hardware without launching simulation.

srb real_agent eval --env waypoint_navigation --algo dreamer

The regenerated profile routes actions through ros_cmd_vel/RosCmdVelInterface and supplies next-step pose observations through ros_tf/RosTfInterface; ros_mw provides middleware services.

Runtime cleanup and validation

Runtime verbs (eval, rand, zero, validate) close the environment on Ctrl-C, SIGTERM, errors, or normal exit while keeping ROS alive. Cleanup latches the session, submits final stops, then releases an owned context. Repeated signals cannot interrupt cleanup, but a blocking transport can delay it. Successful SIGTERM cleanup exits 143. Failed stops retain environment/context for retry; submission proves neither controller receipt nor that the robot stopped.

Validation cleans up the environment, abort watcher, and telemetry in one scope: environment closes before telemetry; ROS context releases only after successful closure. Failed closes retain resources for retry; completed closes are not repeated. ROS initializes immediately before environment construction, not on dry runs or earlier preflight failures. Compatible signal ownership also protects preflight resources. Ctrl-C/SIGTERM during rollout propagates after cleanup without guaranteeing an interrupted-rollout report. Ordinary failures report and exit 4 if cleanup succeeds; persistent cleanup failure raises and retains resources, so a report alone does not prove shutdown.

Embedding processes must invoke the CLI runtime on the main thread. An initialized default ROS context is allowed only with native handlers disabled (SignalHandlerOptions.NO); conflicts are rejected before hardware construction. Borrowed contexts remain active and Python handlers are restored on exit. Library-created RealEnv leaves process signal/context handling to the caller.

ROS policy input

srb real_agent ros accepts flat policy vectors on the environment’s action/raw topic, using generated action order and hardware mapping without simulator configuration. Missing, invalid, or stale input aborts through hardware stop methods. See the ROS input contract for message bounds, startup/freshness timeouts, and cleanup.

Dataset collection

Nonhuman real_agent collect sources (random, zero, or loaded policy) use RealEnv scalar steps and write datasets with single-environment batch axes and the real split. Without a success term, success-only collection uses explicit terminal-step boolean metadata; collector time limits do not make intermediate metadata a final outcome.

Native teleoperation and human collection

Native direct teleop and human collection need generated COMMAND_MAPPING (declared by shipped modules); ROS is default, with SpaceMouse or haptic supported. Keyboard/gamepad input and teleop --algo/--model are rejected. See the generation contract and native input details.

4. Advanced workflows

Zero and random agents

Use only after hardware validation and operator-controlled bring-up. zero sends zero policy input, which may actuate after mapping; it is not an emergency stop. rand sends random actions, not a safety check.

srb real_agent zero --env waypoint_navigation

srb real_agent rand --env waypoint_navigation

Dreamer real-data fine-tuning (srb real_agent train --algo dreamer --continue) remains experimental.

5. Additional task examples

5.1 Excavation

excavation targets a Husky + Kinova mobile manipulator with scoop payload. OSC and joint-velocity depend on the robot motion-generation stack. Deployment is unsupported: both commands raise route ValueError before ROS acquisition. OSC needs ros_optitrack for proprio/base_pose_world_optitrack (missing from generated HARDWARE), and no shipped adapter supplies state/ee_pose_world. Joint-velocity cannot route robot/delta_twist, accepted only by ros_kortex_cartesian (alias) and moveit_servo. The excavation profile configures only ros_depth_heightmap to the training grid: 16×16 cells at 3.75 cm, in kinova_base_link, centered on the default scoop target, invalid cells at 1.5 m. Shipped code omits this profile.

OSC (operational-space control):

srb real_agent eval --env excavation --algo skrl_ppo \
  --hardware ros_depth_heightmap ros_tf ros_imu ros_kortex_cartesian ros_scoop_target ros_mw

Joint-velocity variant:

srb real_agent eval --env excavation --algo skrl_ppo \
  --hardware ros_depth_heightmap ros_tf ros_imu ros_kortex_joint_vel ros_scoop_target ros_mw

For simulation rollouts, monitor success_rate, terminal_reward, scoop_particle_count, and action_smoothness. Hardware RealEnv does not emit scoop_particle_count because it has no particle simulation.

5.2 Terrain Landscaping

Canonical deployment uses RaphRover + prismatic-velocity RaphShovel and one external heightmap. It has no onboard depth camera, IMU observation, blade-pitch channel (measured or synthetic), previous-action feedback, or Leo Rover profile.

terrain_landscaping_mound is simulation-only, with no generated module. General landscaping requires its hashed per-episode mission manifest to construct deployment RealEnv.

Regenerate checked-in bridge modules in place, then compare each with a fresh render:

# Regenerate (writes modules in place)
srb real_agent gen --env terrain_landscaping_crater
srb real_agent gen --env terrain_landscaping

# Check byte-identical fresh renders
srb real_agent gen --env terrain_landscaping_crater --check
srb real_agent gen --env terrain_landscaping --check

See --check for the drift check.

Generation derives modules through env.srb_spec from the frozen IO contract; it emits no np.finfo dtype-extrema bounds. The schema has 3 normalized velocity actions: robot/cmd_vel [0:2] (linear, angular) and payload/joint_vel [2:3]; plus 7 finite-bound actor observation leaves (1287 float32 values, in contract order): proprio_dyn/heightmap_current_global, proprio_dyn/heightmap_target_global, proprio_dyn/heightmap_current_local, proprio_dyn/heightmap_target_local, proprio/mission_pose, proprio/base_velocity, proprio_dyn/heightmap_age_s. SINGLE_OBSERVATION_SPACE assembles four map outputs and one packed vector, not seven top-level outputs. Deployment identity constants are ENV_ID, IO_SCHEMA_FINGERPRINT, TARGET_MANIFEST_SHA256, REQUIRES_RUNTIME_TARGET_MANIFEST, and DEPLOYMENT_REQUIREMENTS; target_map_sha256, reset_layout_sha256, manifest_sha256, and io_schema_fingerprint are distinct hashes.

Capability preflight blocks deployment. Before ROS acquisition, hardware start, or policy loading, canonical landscaping RealEnv checks the union of tags claimed by hardware interfaces and capability providers. All four are required: raph.drive_velocity, raph.shovel_prismatic_velocity, landscaping.external_heightmap_batch, and landscaping.observable_task_evaluator. srb.interfaces.sim_to_real.validation.landscaping.LandscapingRealEvaluator is the only shipped claimant (evaluator tag). Pass it as RealEnv(..., task_evaluator=...); it grades accepted external frames against the verified target manifest, owns reward, and rejects model artifacts with mismatched task ID, IO-schema fingerprint, target manifest, reset layout, or normalization policy before ROS, hardware start, or inference. Other tags have no shipped claimants, so construction raises typed DeploymentNotReadyError listing them. Deployment needs Raph ROS drive/shovel adapters and a live external heightmap mapper claiming those tags; SRB ships none.

hyperparams/validation/terrain_landscaping_crater.yaml declares no hardware and fails capability preflight (exit 4). Metrics are success_rate, final_grading_mae_m, final_fraction_within_tolerance, action_smoothness, and map_age_s; final-step info adds central-ROI/local MAE, sustained success, four distinct hashes, and calibration version. See Terrain Landscaping Training for learners, observation contract, and limitations. TD-MPC2 is unsupported for both IDs and fails before expensive side effects.

6. Validation Workflow

6.1 Capture a simulation baseline

Evaluate each checkpoint once in simulation:

srb agent eval --env excavation --algo skrl_ppo --model <ckpt> \
    env.num_envs=1

Use --model and Hydra override env.num_envs=1 for this simulator baseline; the validation spec does not configure the run. It configures hardware n_episodes and .rrd recording; srb real_agent validate --wandb enables W&B. The harness reads baseline scalars (success_rate, terminal_reward, action_smoothness, etc.) from sim_baseline.metrics_json or, failing that, a W&B run ID (see Validation spec YAML). srb agent eval does not write this file; provide a flat JSON object of finite numbers for every tracked metric (real-only metrics such as success_rate need no baseline).

6.2 Validation spec YAML

The spec defines the contract among the operator, generated RealEnv, and harness. CLI flags override selected runtime fields; YAML holds task, policy, hardware, criteria, telemetry, baseline, and drift settings.

checkpoint accepts a raw checkpoint or SRB model artifact path; notes is copied to the spec snapshot.

Lower-is-better real/sim ratios cap at 1/sim_real_ratio_min; add metric names to lower_is_better_metrics.

6.3 Run validation on the real robot

srb real_agent validate \
    --spec hyperparams/validation/excavation.yaml \
    --checkpoint <real-or-shared-ckpt> \
    --episodes 10 \
    --storage-root /tmp/srb_validation

Flags: --spec, --checkpoint, --episodes, --wandb, --no-tensorboard, --no-rerun-web, --storage-root, --post-status, --dry-run. Artifacts and TensorBoard are local by default; W&B is opt-in.

For graceful abort from another process, touch the exact .abort path logged at information level. It is in the session’s unique workspace beside private .staging:

touch '/exact/session/workspace/.abort'

Programmatic callers can read ValidationHarness.abort_request_path; information logging exposes the CLI path. Checks occur between reset, policy, and step calls: external_abort latches, stops later episodes, and records an incomplete failed session; an active operation may finish first. This is not an emergency stop or controller watchdog. Cleanup removes unpublished .abort. Inherited SRB_ABORT=1 also cancels; exporting it in another shell cannot affect a running process.

Dry runs never pass: --dry-run skips environment and policy construction, then finalizes with pass_overall: false, hardware_evidence: false, exit_code: 4, metrics.n_episodes: 0; all three criteria are skipped, the badge is red with non-hardware evidence, and the session is excluded from drift history.

The harness starts Rerun at http://localhost:9090, runs --episodes rollouts through generated RealEnv, writes session and storage-root artifacts, and checks:

1. Threshold: success floor and sim/real ratios pass. The success floor applies without a baseline; with a positive success_rate_min, zero successes cannot pass.
2. Crash: no policy_nan, hardware_event, disk_full, or safety abort (sensor_stale, sensor_incomplete, calibration_mismatch, manifest_mismatch).
3. Drift: tracked-metric z-scores are at most thresholds.drift_z_max over the last K sessions.

Horizon endings are time_limit_truncation with final metrics, not wall-clock timeout. Out-of-MDP abort reasons/counts (sensor_stale, sensor_incomplete, calibration_mismatch, manifest_mismatch, operator_abort) are neither task termination nor truncation. Operator stop fails the success floor, not crash criterion.

Exit codes:

Validation runs only generated srb_real/<task> environments; unresolved IDs fail with exit 4, never falling back to srb/<task> Isaac tasks. Shipped specs:

6.4 Validation artifacts

logs/real_validation/
+-- _index.jsonl
+-- index.html
+-- <task>/
    +-- <uuid>_<utc-timestamp>/
        +-- current.json
        +-- generations/
            +-- <generation-uuid>/
                +-- spec.yaml
                +-- episodes.jsonl
                +-- rollout.rrd
                +-- tensorboard/
                +-- summary.json
                +-- report.md
                +-- badge.json

Each run regenerates logs/real_validation/index.html; trend charts group by (task, algo, metric). Compare the failed session’s simulation ghost (transparent) with its real run (solid), aligned by episode:

srb real_agent replay-twin \
    --session logs/real_validation/excavation/<uuid>_<ts>/generations/<generation-uuid> \
    --baseline baselines/excavation/<sha>.rrd

If a crash leaves summary.json missing, rebuild it from spec.yaml and episodes.jsonl:

srb real_agent rebuild-summary logs/real_validation/excavation/<uuid>_<ts>/generations/<generation-uuid>

The harness records estop, joint-limit, and contact-force events for criterion (b), but does not enforce safety or replace deadman switches, joint-limit watchdogs, or estop circuitry.

7. Creating custom hardware interfaces

Add a Python file under sim_to_real/hardware, subclass HardwareInterface, declare supported_action_spaces/supported_observation_spaces (used for routing), and implement apply_action (send commands), sync (refresh readings), and observation; override start/close (calling super()) to connect and clean up. _template.py is an example. Discovery exposes it to --hardware.

Next: Terrain Landscaping Training

Terrain Landscaping Training

No trained policy ships.

Tasks

See Mobile Robotics for the observation contract.

Targets: checked-in crater_target_manifest.json, mound_target_manifest.json; general missions are hashed and mass-balanced.

Documented success target (all tasks): at least 90% of observable grading cells within 0.02 m of the target for 20 consecutive accepted frames, reported at truncation. This target has not been validated against a physical system.

Choose env.external_heightmap.profile: ideal (default, no error), randomized (up to 0.1 s delay, 5% dropped frames, 2 mm elevation noise, small registration and pose errors), or failure_stale (stale-frame abort). Seeded defaults do not represent measured lab errors.

Training

Crater and mound auto-load task-local DreamerV3 and skrl PPO profiles, shadowing global profiles; general uses global profiles. See hyperparams/task/terrain_landscaping_{crater,mound}/ for these files.

srb agent train --headless --algo dreamer -e terrain_landscaping_mound env.num_envs=1
srb agent train --headless --algo skrl_ppo -e terrain_landscaping_crater env.num_envs=1

• Dreamer: strict merge rejects unknown or dropped keys. Profiles use 10,000 steps and train_ratio: 512.0; gradient updates equal steps × train_ratio ÷ (batch_size × batch_length).

• One environment: Dreamer landscaping rejects env.num_envs > 1; mixed batches cannot selectively reset. Stacked multi-environment particle scenes are unsupported; multi-environment particle training has not been validated.

• TD-MPC2 disabled: all three tasks fail before creating logs, config, replay, or model:

  TD-MPC2 landscaping is disabled: velocity-controlled shovel extension is unobserved and TD-MPC2 keeps no recurrent or history state to infer it.
  

• GPU required: PhysX particles need a GPU; tasks read them through CPU-facing USD (sim.device = "cpu"). “Force CPU” disables particles; see Particles.

Dreamer checkpoints

Resume is not bitwise; it fails when ckpt_generation.json lacks replay chunks. Loads verify task ID, IO-schema fingerprint, observation layout, target manifest, measurement profile, and normalization against the live environment. SRB_ALLOW_UNVERIFIED_LANDSCAPING_CHECKPOINT allows missing metadata, not mismatches.

Results and limitations

• Crater: no shipped controller meets the success criterion; a pushing blade disturbs the graded surface about as fast as it fills the crater.
• Mound: the scripted reference clears 1.75 L.
• General task: the scripted reference does not improve on zero action.
• Leaving the mission-pose (±2.0 m) or body-velocity envelope terminates the episode. Out-of-envelope elevations raise ExternalHeightmapFrameError (see Modelling Envelope).
• Particle behavior is simulation-only and not calibrated against regolith.

Deployment

See Sim-to-Real for bridge status, capabilities, and crater-spec validation. Validation Workflow covers dry runs.

Next: Environments

Environments

SRB environments:

• Manipulation - Stationary manipulation skills and sequences in various contexts
• Mobile Robotics - Extraterrestrial traversal and navigation
• Mobile Manipulation - Combined mobility and manipulation
• Digital Twins - Facility-inspired simulations; not validated against physical facilities

Namespaces

These pages list the simulated Gymnasium srb/ environments enumerated by srb ls env and srb doc docgen.

srb_real/ contains generated ROS 2 RealEnv deployment interfaces for srb real_agent. They are not validated on physical robots and have no simulated scenario. Real-task metrics require an explicitly configured task evaluator. Sim-to-Real Transfer lists shipped IDs, generation, and registration.

Environments – Manipulation (Static)

Fixed-base robot arm manipulation environments.

Manipulation (_manipulation)

Template environment for fixed-base manipulation tasks. Configure via env.robot=<manipulator>.

Robots

Canadarm3 | Franka | Kinova J2N6S | Kinova J2N7S | Kinova Gen3N7 | Levion Arms | So Arm100 D5 | So Arm100 D7 | Unitree Z1 | Ur3 | Ur3E | Ur5 | Ur5E | Ur10 | Ur10E | Ur16E | Ur20 | Ur30 | Random Ur Manipulator | Vispa | Zeroglab Arms

Debris Capture (debris_capture)

Capture and securement of floating debris in microgravity with a fixed-base manipulator

Robots

Canadarm3 | Franka | Kinova J2N6S | Kinova J2N7S | Kinova Gen3N7 | Levion Arms | So Arm100 D5 | So Arm100 D7 | Unitree Z1 | Ur3 | Ur3E | Ur5 | Ur5E | Ur10 | Ur10E | Ur16E | Ur20 | Ur30 | Random Ur Manipulator | Vispa | Zeroglab Arms

• distance_debris_too_far applies penalty_debris_too_far only; episodes end by horizon (truncated).
• Episode info omits success; bounded evaluation reports return only.

Excavation (excavation)

Extraction of granular material (regolith)

Robots

Kinova Gen3N7

• Heightmap bins live regolith positions onto the observation grid at runtime; PhysX particle sets cannot be raycast.
• imu_pedestal is active only at env.num_envs=1; otherwise it is disabled with a warning and returns zero observations.
• Episode-end info includes success; bounded evaluation reports success rate and return.
• Wrist event camera: add events to env.active_sensors.

Excavation Demo (excavation_demo)

Manual excavation demo with presentation-oriented lunar regolith defaults

Robots

Kinova Gen3N7

Modular Assembly (modular_assembly)

Pick-and-place modular elements into an assembly configuration

Robots

Canadarm3 | Franka | Kinova J2N6S | Kinova J2N7S | Kinova Gen3N7 | Levion Arms | So Arm100 D5 | So Arm100 D7 | Unitree Z1 | Ur3 | Ur3E | Ur5 | Ur5E | Ur10 | Ur10E | Ur16E | Ur20 | Ur30 | Random Ur Manipulator | Vispa | Zeroglab Arms

Multi Peg In Hole (multi_peg_in_hole)

A variation of peg-in-hole assembly with multiple pegs and holes

Robots

Canadarm3 | Franka | Kinova J2N6S | Kinova J2N7S | Kinova Gen3N7 | Levion Arms | So Arm100 D5 | So Arm100 D7 | Unitree Z1 | Ur3 | Ur3E | Ur5 | Ur5E | Ur10 | Ur10E | Ur16E | Ur20 | Ur30 | Random Ur Manipulator | Vispa | Zeroglab Arms

Multi Sample Collection (multi_sample_collection)

A variation of sample collection with multiple samples

Robots

Canadarm3 | Franka | Kinova J2N6S | Kinova J2N7S | Kinova Gen3N7 | Levion Arms | So Arm100 D5 | So Arm100 D7 | Unitree Z1 | Ur3 | Ur3E | Ur5 | Ur5E | Ur10 | Ur10E | Ur16E | Ur20 | Ur30 | Random Ur Manipulator | Vispa | Zeroglab Arms

Peg In Hole (peg_in_hole)

Precision assembly task requiring the robot to insert a peg into a hole

Robots

Canadarm3 | Franka | Kinova J2N6S | Kinova J2N7S | Kinova Gen3N7 | Levion Arms | So Arm100 D5 | So Arm100 D7 | Unitree Z1 | Ur3 | Ur3E | Ur5 | Ur5E | Ur10 | Ur10E | Ur16E | Ur20 | Ur30 | Random Ur Manipulator | Vispa | Zeroglab Arms

Evaluation

• Episode info has no success; bounded evaluation reports return only.

Sample Collection (sample_collection)

Grasping and collection of samples on planetary surfaces

Robots

Canadarm3 | Franka | Kinova J2N6S | Kinova J2N7S | Kinova Gen3N7 | Levion Arms | So Arm100 D5 | So Arm100 D7 | Unitree Z1 | Ur3 | Ur3E | Ur5 | Ur5E | Ur10 | Ur10E | Ur16E | Ur20 | Ur30 | Random Ur Manipulator | Vispa | Zeroglab Arms

• obj_floor_z drives penalty_obj_fallen; falls do not terminate. Horizon ends as truncated.
• Wrist event camera: add events to env.active_sensors.
• Episode info omits success; bounded evaluation reports return only.

Screwdriving (screwdriving)

Fastening of a pre-inserted bolt

Robots

Canadarm3 | Franka | Kinova J2N6S | Kinova J2N7S | Kinova Gen3N7 | Levion Arms | So Arm100 D5 | So Arm100 D7 | Unitree Z1 | Ur3 | Ur3E | Ur5 | Ur5E | Ur10 | Ur10E | Ur16E | Ur20 | Ur30 | Random Ur Manipulator | Vispa | Zeroglab Arms

Evaluation

• Episode info has no success; bounded evaluation reports return only.

Solar Panel Assembly (solar_panel_assembly)

Complex assembly sequence involving solar panel components

Robots

Canadarm3 | Franka | Kinova J2N6S | Kinova J2N7S | Kinova Gen3N7 | Levion Arms | So Arm100 D5 | So Arm100 D7 | Unitree Z1 | Ur3 | Ur3E | Ur5 | Ur5E | Ur10 | Ur10E | Ur16E | Ur20 | Ur30 | Random Ur Manipulator | Vispa | Zeroglab Arms

Evaluation

• Episode info has no success; bounded evaluation reports return only.

Environments – Mobile Robotics

Mobile robot navigation and locomotion tasks for Isaac Sim 6.1.0 with Isaac Lab 3.0.0.

Aerial (_aerial)

Template environment for aerial mobile robots. Configure via env.robot=<aerial_robot>.

Simulation only; no trained policy ships.

Robots

Crazyflie | Ingenuity

Ground (_ground)

Template environment for ground mobile robots. Configure via env.robot=<ground_robot>.

Robots

Anymal C | Anymal D | Random Anymal Quadruped | Cadre | Cassie | Husky | Leo Rover | Mars Rover | Nifty | Open Source Rover | Perseverance | Pragyan | Raph Rover | Spot | Unitree A1 | Unitree Go1 | Unitree Go2 | Random Unitree Quadruped

Orbital (_orbital)

Template environment for orbital mobile robots. Configure via env.robot=<orbital_robot>.

Robots

Floating Platform | Int Ball1 | Int Ball2 | Apollo Lander | Peregrine Lander | Vikram Lander | Resilience Lander | Random Lander | Iss | Gateway | Cubesat | Satellite Mockup | Venus Express | Starship | Super Heavy

Camera clipping. Set env.camera_clipping_range=[near,far] to override the render far plane and uint8 depth quantization; depth beyond far saturates and geometry is omitted from RGB.

Aerial Navigation (aerial_navigation)

Aerial navigation to a dynamic waypoint with terrain-clearance tracking

Simulation only; no trained policy ships.

Robots

Crazyflie | Ingenuity

Bounded evaluation. Aerial Navigation and linked tasks omit success; bounded evaluation reports return only.

Beam Transport (beam_transport)

Two-robot collaborative transport of a rigid beam payload to a target point

Two Spot quadrupeds carry a rigid beam to a target on procedural MoonSurface. One shared policy observes robots in fixed order and splits one joint action across them, as in formation following.

No trained policy ships. Train and evaluate with:

srb agent train --algo dreamer --env beam_transport
srb agent eval --algo dreamer --env beam_transport --model <path/to/checkpoint>

Rollouts report min_beam_distance_to_target (m), success-hold streak, and success_rate. At zero success, minimum distance shows partial progress.

Robots

Anymal C | Anymal D | Random Anymal Quadruped | Cadre | Cassie | Husky | Leo Rover | Mars Rover | Nifty | Open Source Rover | Perseverance | Pragyan | Raph Rover | Spot | Unitree A1 | Unitree Go1 | Unitree Go2 | Random Unitree Quadruped

Success limits. Hold target pose for success_hold_steps=8 steps (0.32 s); target distance ≤ target_zone_radius_xy=0.20 m; beam-to-midpoint and support-slot alignment ≤ beam_attachment_radius_xy=0.30 m; planar speed ≤ target_speed_threshold_xy=0.35 m/s. Free-beam mode also requires 1 - |cos θ| ≤ beam_tilt_tolerance=0.20 (θ ≲ 37° from vertical). Holding success terminates the episode; otherwise it truncates at 30 s. Resets preserve robot spacing with a shared planar offset within ±1.5 m and shared yaw over the full circle; the mounted beam follows the transformed pair.

Policy spaces. Observations: per-robot planar position, velocity, and beam offset; beam position and 6D rotation; beam-target offset and distance; assignment-invariant support-slot error; beam planar/angular velocity; pair midpoint. Positions are environment-relative; inter-body offsets are unchanged. Actions drive both Spot quadrupeds. The reward table includes per-robot penalties for joint torque/acceleration, vertical velocity, roll/pitch rate, and projected-gravity alignment.

Episode-end info publishes success; bounded evaluation reports success rate and return.

Flight (flight)

Generic powered aerial flight profile with takeoff, transfer, and final landing

Flight: 90 s Martian mission—ascend to cruise altitude, transfer to landing pad, guarded touchdown approach. Mars is default because rotorcraft need atmosphere; see Modelling envelope. Aerial configuration’s domain-selected scenery owns default Mars spawn and clearance.

Set env.landing_only=true to start in final descent phase 2, skipping ascent and transfer. This is the aerial Flight variant; orbital landing is separate.

Robots

Crazyflie | Ingenuity

Termination/success. Crash terminates below termination_min_clearance or above termination_max_tilt. Pad arrival latches success in episode_outcome_info (crashed, on_pad, mission_phase), but episode runs to the limit. Success requires reaching the pad without crashing on the final step. At default weights, reward_mission_progress pays 60 per completed phase each step, more than other terms combined per step.

Formation Following (formation_following)

Multi-robot formation tracking behind a moving formation center

Three Cadre-class rovers follow a drifting center in triangular slots (1.5, 0), (-0.75, 1.3), (-0.75, -1.3) m. Center follows a smooth random walk or ROS formation_center TF.

One shared policy: concatenate observations in fixed rover order, split one joint action across rovers, return one team reward per environment. Observation/action sizes scale linearly with team size. Small teams avoid decentralized communication and role assignment while keeping checkpoints reproducible. Companion centralized surface task: beam transport.

No trained policy ships.

srb agent train --algo dreamer --env formation_following
srb agent eval --algo dreamer --env formation_following --model <path/to/checkpoint>

Rollouts report mean per-rover slot-distance error (formation_distance_error, m), mean heading error (formation_orientation_error, rad), success-hold streak, and success_rate. At zero success, errors show partial progress.

Robots

Anymal C | Anymal D | Random Anymal Quadruped | Cadre | Cassie | Husky | Leo Rover | Mars Rover | Nifty | Open Source Rover | Perseverance | Pragyan | Raph Rover | Spot | Unitree A1 | Unitree Go1 | Unitree Go2 | Random Unitree Quadruped

Success limits. All rovers stay within formation_success_position_tolerance_xy=0.20 m of their slots and formation_success_yaw_tolerance≈0.349 rad (20°) in heading for success_hold_steps=8 (0.32 s). Holding success terminates the episode (terminate_on_success=true; formation_following_decentralized keeps running); otherwise it truncates at 60 s. Infinite-horizon (is_finite_horizon=false).

Policy spaces. Each rover observes planar slot offset and heading-to-slot (sin, cos); policy observes center and rover planar velocities. Fixed episodic reset noise: 1 cm position, 2.5° heading. Actions: planar linear/angular velocity.

ROS center. ROS replaces random walk with formation_center TF. First post-reset pose re-anchors center; reset jump is not velocity. Not hardware-validated.

Episode-end info publishes success; bounded evaluation reports success rate and return.

Formation Following Decentralized (formation_following_decentralized)

Single-rover policy view for decentralized formation following

Robots

Anymal C | Anymal D | Random Anymal Quadruped | Cadre | Cassie | Husky | Leo Rover | Mars Rover | Nifty | Open Source Rover | Perseverance | Pragyan | Raph Rover | Spot | Unitree A1 | Unitree Go1 | Unitree Go2 | Random Unitree Quadruped

Formation Following Orbital (formation_following_orbital)

Orbital formation tracking in microgravity

Robots

Floating Platform | Int Ball1 | Int Ball2 | Apollo Lander | Peregrine Lander | Vikram Lander | Resilience Lander | Random Lander | Iss | Gateway | Cubesat | Satellite Mockup | Venus Express | Starship | Super Heavy

Slot assignment. randomize_formation_assignment=false assigns environment i to slot i % N (N formation positions), including partial resets. When true (default), each reset draws a random slot.

Landing (landing)

Landing of spacecraft on planetary surfaces with varying terrain and obstacles

Robots

Floating Platform | Int Ball1 | Int Ball2 | Apollo Lander | Peregrine Lander | Vikram Lander | Resilience Lander | Random Lander | Iss | Gateway | Cubesat | Satellite Mockup | Venus Express | Starship | Super Heavy

See evaluation note.

Event cameras. Add events to env.active_sensors for the onboard (landing, moonfall) or front (velocity_tracking, waypoint_navigation) event camera.

Locomotion Velocity Tracking (locomotion_velocity_tracking)

Locomotion with legged robots and humanoids while tracking desired velocity commands

Simulation only; no trained policy ships.

Robots

Anymal C | Anymal D | Random Anymal Quadruped | Cassie | Spot | Unitree A1 | Unitree Go1 | Unitree Go2 | Random Unitree Quadruped | Humanoid21 | Humanoid28 | Unitree H1 | Unitree G1

Termination matches Locomotion Waypoint Navigation: undesired non-foot contact, fall (planar projected gravity above 0.85), or non-finite state. Time limits truncate.

See evaluation note.

Locomotion Waypoint Navigation (locomotion_waypoint_navigation)

Navigation to a dynamic waypoint with legged robots and humanoids

Simulation only; no trained policy ships.

Robots

Anymal C | Anymal D | Random Anymal Quadruped | Cassie | Spot | Unitree A1 | Unitree Go1 | Unitree Go2 | Random Unitree Quadruped | Humanoid21 | Humanoid28 | Unitree H1 | Unitree G1

Episodes terminate if non-foot contact exceeds its configured undesired-contact threshold, projected gravity’s planar norm exceeds 0.85 (fall), or contact/gravity state is non-finite (fail-closed). Time limits truncate, not terminate.

See evaluation note.

Moonfall (moonfall)

Heavy-weight lunar descent with deployment stage and multi-stage landing phases

Robots

Floating Platform | Int Ball1 | Int Ball2 | Apollo Lander | Peregrine Lander | Vikram Lander | Resilience Lander | Random Lander | Iss | Gateway | Cubesat | Satellite Mockup | Venus Express | Starship | Super Heavy

See event camera options.

See evaluation note.

Orbital Evasion (orbital_evasion)

Navigation of spacecraft in microgravity while avoiding obstacles

Robots

Floating Platform | Int Ball1 | Int Ball2 | Apollo Lander | Peregrine Lander | Vikram Lander | Resilience Lander | Random Lander | Iss | Gateway | Cubesat | Satellite Mockup | Venus Express | Starship | Super Heavy

Termination. max_target_distance affects penalty_out_of_bounds, not terminated. Collision terminates when robot center of mass enters an obstacle keep-out sphere (0.5 * obstacle_scale * 1.4; 3.5 m for default 5 m asteroids), i.e. clearance falls below termination_min_obstacle_distance (0 m). Robot is a point: large robots may touch obstacles without termination, and about 7% of asteroids extend beyond the sphere on one axis. Obstacle centers spawn at z ∈ [-43, -7] m, at least 7 m from robot spawn and target. Time limits set truncated.

See evaluation note.

Orbital Waypoint Navigation (orbital_waypoint_navigation)

Navigation to a dynamic waypoint in orbital microgravity

Robots

Floating Platform | Int Ball1 | Int Ball2 | Apollo Lander | Peregrine Lander | Vikram Lander | Resilience Lander | Random Lander | Iss | Gateway | Cubesat | Satellite Mockup | Venus Express | Starship | Super Heavy

See evaluation note.

Rendezvous (rendezvous)

Maneuvering of spacecraft in microgravity to rendezvous with an uncooperative target

Robots

Floating Platform | Int Ball1 | Int Ball2 | Apollo Lander | Peregrine Lander | Vikram Lander | Resilience Lander | Random Lander | Iss | Gateway | Cubesat | Satellite Mockup | Venus Express | Starship | Super Heavy

Termination. Exceeding max_target_distance incurs penalty_too_far but leaves terminated false so the agent can recover. Only the time limit sets truncated.

See evaluation note.

Terrain Landscaping (terrain_landscaping)

Terrain landscaping with a ground mobile robot reshaping a particle bed

No trained policy ships. Dreamer requires env.num_envs=1; TD-MPC2 is unsupported; Raph hardware validation is incomplete. See Terrain Landscaping Training.

Setup: RaphRover + articulated RaphShovel (env.robot=raph_rover+raph_shovel). Filtered actions, in order: rover linear, rover angular, shovel extension velocity (terms robot/cmd_vel = linear, angular; payload/joint_vel = prismatic). Shovel resets raised at its 0.2 m joint limit; zero velocity holds position via actuator damping, keeping blade above settled particles until policy lowers it.

Reset: rover within ±1.1 m of bed center, random yaw. Blade reaches about 0.53 m from rover origin and starts clear of ±2.0 m walls. Rear-axle Ackermann steering limits yaw rate; full C/V turn gives about half the command.

Teleoperate (one environment):

srb agent teleop -e terrain_landscaping env.num_envs=1

Each episode deforms the flat reset bed with a hashed, mass-balanced mission: cut/fill balance within 1e-4 m³; cuts stay above available material, fills below containment ceiling, significant deformation in reachable zone. Seed (env.general_mission_seed, env_id, episode_index) reproduces missions across processes. Manifest records generator version, seed, frame, geometry, physical bounds, target-map, reset-layout, and manifest hashes. Infeasible missions redraw; exhausted attempts raise. Configure env.general_mission_profile.*.

This task shares only reward implementation and observation contract with terrain_landscaping_crater; targets, manifests, and curriculum stages are separate.

Robots

Raph Rover

Observation contract. Actor uses external heightmaps, not depth; PhysX particles cannot be raycast. Each policy step publishes one accepted-or-held 10 Hz live-regolith frame. Seven ordered leaves: global current/target (16×16), rover-frame local current/target (24×16), mission pose (x,y,yaw), body velocity (vx,vy,wz), frame age; flattened to 1287 float32. Elevations use mission-frame +z, matching deployment adapters. No validity mask: all 146×146 source cells finite; incomplete/out-of-envelope frames rejected whole. Unobserved: shovel extension, previous action, particle positions, simulator world pose, privileged current-minus-target maps. env.external_heightmap.profile: ideal, randomized, failure_stale.

Reward/episode. Sole reward 1 / (1 + mae / 0.02) on fixed mission grading cells, shared with crater. MDP continues (is_finite_horizon=false); time limit truncates. Success is evaluation-only, never termination or bonus.

Canonical tasks require RaphRover and RaphShovel; other robots or payloads fail before scene binding. Actor observations and canonical deployment profile omit shovel state.

Frame validity. Map, pose, and velocity share a timestamp; local crop uses that pose. Simulation resets capture one synchronous frame; deployment resets wait up to 2.0 s. A missing frame raises (no zero substitute). Frames older than 0.5 s stale-abort outside MDP. Dropped updates repeat the last complete map and increase frame age.

TD-MPC2 is unsupported because shovel extension is unobserved; it fails with an actionable error before creating logs, config, replay, or model. See Terrain Landscaping Training.

Terrain Landscaping Crater (terrain_landscaping_crater)

Separate deterministic baseline: fill one shallow crater from a nearby regolith pile

No trained policy ships; Dreamer supports only env.num_envs=1; TD-MPC2 is unsupported; the task is not validated on Raph hardware. See Terrain Landscaping Training.

Separate deterministic task from terrain_landscaping: no curriculum, scheduler, or automatic switching. Layered bed: one shallow bowl crater and one loose pile; crater particles reused, preserving particle count/mass. RaphRover starts fixed with both forward. Visible 4 m slab below physical ground, dark neutral terrain, and light regolith keep the bed readable against lunar skydome in viewport/headless captures.

Static analytic flat-bed target: crater_target_manifest.json; config fails if crater, pile, or spawner settings cannot reproduce hashed reset layout. Reward: 1 / (1 + mae / 0.02) on fixed manifest grading cells using oracle particle truth. Success: 90% of cells within tolerance for 20 consecutive newly accepted mapper frames; report at time-limit truncation, never termination/bonus. Continuing MDP (is_finite_horizon=false): the time limit truncates; leaving the rover pose/velocity envelope terminates. Physics perturbations/lighting randomization disabled. Action/observation schemas match terrain_landscaping.

Inspect by teleoperation:

srb agent teleop -e terrain_landscaping_crater env.num_envs=1

Robots

Raph Rover

Override under env.*. Out-of-bed, overlapping, or under-capacity layouts fail before particle spawn.

Observation contract. Same heightmap and frame contract as terrain_landscaping; schema flattens to 1287 float32.

TD-MPC2 has the same unsupported setup and fails before logs, config, replay, or model. PhysX PBD particles require a CUDA-capable NVIDIA GPU; particle state uses CPU readback (sim.device = "cpu") for heightmap, reward, and rendering. See Terrain Landscaping Training.

Terrain Landscaping Mound (terrain_landscaping_mound)

Separate deterministic baseline: level one regolith mound on a flat bed, graded by excess volume

No trained policy ships, Dreamer runs with env.num_envs=1 only, TD-MPC2 is not supported, and the task is not validated on Raph hardware. See Terrain Landscaping Training.

Robots

Raph Rover

Traversal (traversal)

Straight-line path following at a target velocity

Robots

Anymal C | Anymal D | Random Anymal Quadruped | Cadre | Cassie | Husky | Leo Rover | Mars Rover | Nifty | Open Source Rover | Perseverance | Pragyan | Raph Rover | Spot | Unitree A1 | Unitree Go1 | Unitree Go2 | Random Unitree Quadruped

Termination. Exceeding max_tilt_rad incurs penalty_rollover but leaves terminated false so the rover can recover. The time limit sets truncated.

See evaluation note.

Velocity Tracking (velocity_tracking)

Basic tracking of desired velocity commands

Robots

Anymal C | Anymal D | Random Anymal Quadruped | Cadre | Cassie | Husky | Leo Rover | Mars Rover | Nifty | Open Source Rover | Perseverance | Pragyan | Raph Rover | Spot | Unitree A1 | Unitree Go1 | Unitree Go2 | Random Unitree Quadruped

Commands (vx, vy, wz) are sampled uniformly in ±1 m/s and rad/s. Wheeled drives use their own limits instead and get no lateral command.

See evaluation note.

See event camera options.

Waypoint Navigation (waypoint_navigation)

Navigation to a dynamic waypoint

Robots

Anymal C | Anymal D | Random Anymal Quadruped | Cadre | Cassie | Husky | Leo Rover | Mars Rover | Nifty | Open Source Rover | Perseverance | Pragyan | Raph Rover | Spot | Unitree A1 | Unitree Go1 | Unitree Go2 | Random Unitree Quadruped

See evaluation note.

See event camera options.

Environments – Mobile Manipulation

Environments combining mobility and manipulation.

Aerial Manipulation (_aerial_manipulation)

Template environment for aerial mobile manipulators. Configure via env.robot.mobile_base=<aerial_robot> and env.robot.manipulator=<manipulator>.

Simulation only; no trained policy ships.

Robots

Generic Aerial Manipulator

Ground Manipulation (_ground_manipulation)

Template environment for ground mobile manipulators. Configure via env.robot.mobile_base=<ground_robot> and env.robot.manipulator=<manipulator>.

Robots

Generic Ground Manipulator

Orbital Manipulation (_orbital_manipulation)

Template environment for orbital mobile manipulators. Configure via env.robot.mobile_base=<orbital_robot> and env.robot.manipulator=<manipulator>.

Robots

Generic Orbital Manipulator

Mobile Debris Capture (mobile_debris_capture)

Capture of floating debris in microgravity with a floating-base mobile manipulator

Robots

Generic Orbital Manipulator

distance_debris_too_far activates penalty_debris_too_far; it does not terminate the episode. The finite horizon reports truncated. Episode info has no success, so bounded evaluation reports return only.

Spacewalk (spacewalk)

Humanoid spacewalk: rail-aligned waypoint traversal near spacecraft structure

Simulation only; no trained policy ships. Thruster-less humanoid traverses waypoints and station-keeps by pushing off spacecraft structure.

Aliases spacewalk_eva / spacewalk_iva make benchmark/launch configuration explicit. Built-in EVA waypoints clear the station; custom paths must too, or humanoid, rails, and cameras can enter station meshes. Set env.use_waypoint_path=false for one random target within ±target_pos_range m of spacecraft origin, a simpler reaching ablation.

Train and evaluate.

The task checks only the current waypoint each control step and advances at most one waypoint. Passing through an intermediate waypoint between samples does not count as reaching it.

srb agent train --algo dreamer --env spacewalk_eva
srb agent eval --algo dreamer --env spacewalk_eva --model <path/to/checkpoint>

No task-specific DreamerV3 profile exists; the global preset (deter = 8192, train_ratio = 32) is not tuned for contact locomotion. Evaluation reports:

Diagnose near-zero success. Read min_distance_to_target with waypoint_progress:

• Low progress + large distance: never nears first waypoint.
• Low progress + small distance: nears a waypoint, cannot advance reliably.
• Progress near 1 + zero success: inspect min_distance_to_final_target; above success_radius means goal region not reached, at/below means required hold failed.

EVA scenery. ISSGatewayScenery:

Success: final waypoint within success_radius (0.5 m) for success_stability_steps (10 control steps). Episode-sticky flag records reaching and briefly holding, not ending there. Resets randomize position in a ±0.5 m box at EVA path start or Kibou origin, orientation fully, linear velocity ±0.1 m/s, angular velocity ±5°/s.

Observation/action. DreamerV3 vector fields: tf_rot6d_robot (6D orientation), body-frame target vector tf_pos_robot_to_target, path-mean dist_to_spacecraft, nearest-rail distance, normalized progress, body-frame linear/angular velocity, joint positions/velocities, IMU linear acceleration/ angular velocity. No contact state; infer contact from IMU acceleration. Actions: per-joint position targets; dimension follows actuated USD articulation joints at runtime. JointPositionToLimits bounds physical limits. Raw observations: metre-scale distances, [0,1] progress, [-1,1] rot6d. DreamerV3 symlog handles this range; normalize for other algorithms. No real-hardware validation spec, real environment generation, or joint-position actuation adapter.

Robots

Humanoid21 | Humanoid28 | Unitree H1 | Unitree G1

Waypoints reach ~6 m from work-volume centre, within the drift penalty’s 9 m knee, barely penalizing traversal. Leaving structure is effectively unrecoverable without a push surface; the penalty discourages escape without pulling to origin. Reward favors reaching/holding the goal.

Shaping. Distance/reaching rewards follow the current waypoint, advancing immediately on arrival. No advance bonus: observe/report waypoint_progress, never reward it. Intermediate advances transiently drop per-step reward by about that leg’s distance/reaching rewards. Final goal stays optimal; agents must cross the local dip instead of camping outside each radius.

Horizon/contact. Truncate at 20 s / 500 steps; no failure termination and termination always false. Finite horizon (is_finite_horizon=true), so RSL-RL does not bootstrap at the time limit. Rewards cover traversal/station-keeping only; rigid station, Gateway, and handrail contact are neither rewarded nor penalized. No grasp reward or penalty for brushing structure.

Locomotion/routes. Without thrusters/ground, internal joint motion cannot change centre-of-mass velocity; translation needs station/rail push-offs. Randomized initial drift (±0.1 m/s) covers only part of a multi-metre leg. Rails cue distance and provide propulsion; at EVA endpoint, brace to arrest velocity. Rewards measure approach velocity/progress/distance, not contact; infer push-offs from downstream effects. EVA: ~12.5 m exterior rail route with lateral turn; 20 s needs ~0.6 m/s average and efficient push-off chaining. IVA: short, confined Kibou; waypoint spacing comparable to shared 0.5 m success_radius, favoring final acquisition/station-keeping over long-range traversal.

Spacewalk Eva (spacewalk_eva)

Humanoid EVA traversal along exterior spacecraft handrails

Simulation only; no trained policy ships. See Spacewalk for training and evaluation.

Robots

Humanoid21 | Humanoid28 | Unitree H1 | Unitree G1

Evaluation. Episode-end info publishes success; bounded evaluation reports success rate and return.

Spacewalk Iva (spacewalk_iva)

Humanoid IVA traversal inside the Kibou module

Simulation only; no trained policy ships. See Spacewalk for training and evaluation.

Robots

Humanoid21 | Humanoid28 | Unitree H1 | Unitree G1

Environments – Digital Twins

Facility-inspired simulator environments, not validated against their physical counterparts.

Lunalab (lunalab)

Digital twin of the LunaLab planetary analogue testbed

Robots

Generic Ground Manipulator

Zerog Lab (zerog_lab)

Digital twin of a zero-gravity laboratory testbed

Robots

Generic Orbital Manipulator

Assets

Robots, objects, and scenery available in the Space Robotics Bench.

• Robot Arms & Manipulators
• Mobile Robots
• Mobile Manipulators
• End Effectors & Tools
• Objects
• Scenery

Robot Arms & Manipulators

Serial and dual-arm manipulators registered in the Space Robotics Bench for dexterous manipulation, assembly, and maintenance across terrestrial and space environments.

Franka

Franka (franka)

7-DoF torque-controlled research arm from Franka Robotics

Kinova

Kinova Gen3N7 (kinova_gen3n7)

7-DoF lightweight research arm from Kinova Robotics

Kinova J2N6S (kinova_j2n6s)

Kinova J2N7S (kinova_j2n7s)