Reinforcement Learning Workflow

Reinforcement Learning (RL) is one of the primary focus areas of the Space Robotics Bench. While there are several RL frameworks with their unique peculiarities, SRB offers a unified interface for training and evaluating policies across a diverse set of space robotics tasks.

1. Train your 1^st RL Agent

Reference: srb agent train — Train Agent

The fastest way to get started with training an RL agent is by using the srb agent train command, which provides a streamlined interface for all integrated RL frameworks. In general, you want to specify the RL algorithm to use, the environment to train on, and the number of parallel environment instances used for rollout collection.

Let’s start with a simple landing environment using the sbx_ppo algorithm (PPO implementation of SBX). For now, omit the --headless flag so that you can observe the convergence in real time:

srb agent train --algo sbx_ppo --env landing env.num_envs=512 --hide_ui

As you begin to observe the training process, you can also monitor the progress in your terminal. After about 25M timesteps, you will see that the agent found a stable policy that successfully solves the task. Checkpoints are saved regularly, so you are free to stop the training process at any point by sending an interrupt signal (Ctrl+C in most terminals).

2. Evaluate your Agent

Reference: srb agent eval — Evaluate Agent

Once training is complete, you can evaluate your agent with the srb agent eval command:

srb agent eval --algo sbx_ppo --env landing env.num_envs=16

By default, the latest checkpoint from the training run is loaded for evaluation. However, you might want to run the evaluation for a checkpoint specified via --model:

srb agent eval --algo sbx_ppo --env landing env.num_envs=16 --model space_robotics_bench/logs/landing/sbx_ppo/${TIMESTAMP}/ckpt/${CHECKPOINT}

Note: Each training run writes to its own timestamped directory (logs/<env>/<algo>/<timestamp>/), and SB3/SBX checkpoints are saved under its ckpt/ subdirectory (.zip files).

3. Try a Different Algorithm

SRB directly supports several popular RL algorithms from different frameworks:

Algorithm Type	DreamerV3	TD-MPC2	Stable-Baselines3	SBX	RSL-RL	skrl
Model-based	dreamer	tdmpc2

On-Policy			sb3_a2c		rsl_rl_ppo	skrl_a2c
			sb3_ppo	sbx_ppo		skrl_ppo
			sb3_ppo_lstm			skrl_ppo_rnn
						skrl_rpo
			sb3_trpo			skrl_trpo

Off-Policy			sb3_ddpg	sbx_ddpg		skrl_ddpg
			sb3_td3	sbx_td3		skrl_td3
			sb3_sac	sbx_sac		skrl_sac
			sb3_crossq	sbx_crossq
			sb3_tqc	sbx_tqc

Evolutionary			sb3_ars
						skrl_cem

Imitation-based						skrl_amp

Note: ACME is intentionally absent from SRB algorithm choices. SRB integrations run in the same Python process as Isaac Sim, while current ACME dependencies do not resolve in the Python 3.12 Docker runtime used by SRB.

Known gap — skrl: skrl_sac, skrl_ppo_rnn and skrl_amp are not usable. SquashedGaussianMixin, SequencedMemory and PPO_RNN exist in no released skrl (checked 1.4.3 and 2.1.0), and nothing passes a motion dataset to skrl’s AMP agent, so its discriminator never sees the reference motions. Each needs code in SRB, not a config change. The remaining skrl entries are written against the pinned release and are exercised by tests/unit/test_skrl_hyperparams.py, which also records the three gaps — but note that no skrl profile has a trained policy behind it in this repository, so treat their hyperparameters as untuned starting points.

This time, you can train another agent using an algorithm of your choice:

srb agent train --headless --algo <ALGO> --env landing env.num_envs=1024

Hint: Use --headless mode with more parallel environments for faster convergence.

4. Monitor Training Progress

While training, you might be interested in monitoring the progress and comparing different runs through a visual interface. Local run artifacts and TensorBoard are the accepted defaults; W&B is an optional explicit opt-in and is not required for acceptance evidence. TensorBoard logs are saved in the space_robotics_bench/logs directory. You can start TensorBoard to visualize the training progress:

tensorboard --logdir ./logs --bind_all

You can enable Weights & Biases (wandb) logging with framework-specific flags:

DreamerV3: srb agent train ... 'agent.logger.outputs=[tensorboard,wandb]'
SB3 & SBX: srb agent train ... +agent.wandb=true
skrl: srb agent train ... agent.agent.experiment.wandb=true

Note: Logging to Weights & Biases requires an account and API key.

5. Configure Hyperparameters

Reference: Agent Configuration

The default hyperparameters for all algorithms and environments are available under the space_robotics_bench/hyperparams directory. Similar to the environment configuration, you can adjust the hyperparameters of the selected RL algorithm through Hydra. However, the available hyperparameters and their structure is specific to each framework and algorithm.

Here are some examples (consult hyperparameter configs for more details):

srb agent train --algo dreamer      agent.run.train_ratio=128  ...
srb agent train --algo sb3_ppo      agent.gamma=0.99           ...
srb agent train --algo sbx_sac      agent.learning_rate=0.0002 ...
srb agent train --algo skrl_ppo     agent.models.separate=True ...

Keyboard shortcuts