Reward & Normalization Configuration

In reinforcement learning, the design and shaping of the reward function are critical to policy convergence and stability. The Space Robotics Bench (SRB) provides a modular, declarative, and robust reward and normalization configuration system built on Pydantic models.

This guide explains how rewards are defined, the available reward term types, and how auto-normalization scales returns across diverse task horizons.

1. Defining a Reward Configuration

Each task exposes a specific RewardCfg class inheriting from BaseRewardCfg (or domain-specific subclasses like GroundMobileRewardCfg or LocomotionRewardCfg).

You usually don’t need to edit Python. To retune an existing reward term, pass a CLI override — e.g. env.reward.penalty_action_rate.weight=-0.2. To inspect a task’s default reward terms, read its RewardCfg class in source. The class below shows how those defaults are declared.

Here is an example structure:

from srb.core.reward import BaseRewardCfg, LinearRewardTerm, GaussianRewardTerm

class RewardCfg(BaseRewardCfg):
    # Smoothness & efficiency penalties
    penalty_action_rate: LinearRewardTerm = LinearRewardTerm(weight=-0.5, limit=4.0)

    # Task specific target rewards
    reward_target_approach: GaussianRewardTerm = GaussianRewardTerm(
        weight=4.0, scale=0.5
    )

At every environment step:

The environment’s extract_step_return() computes step metrics (e.g., current action variance or distance to target).
The metrics are passed to their corresponding reward terms inside the task dictionary.
SRB automatically aggregates all non-ignored RewardTerm instances, sums their computed values, and applies optional normalization before returning the final reward to the RL agent.

2. Types of Reward Terms

Most reward term classes are defined in srb.core.reward.terms (a few, such as TukeyRewardTerm, live in srb.core.reward.normalize); all are exported from srb.core.reward. They generally inherit from RewardTerm and are designed to shape physical metrics mathematically.

📐 Linear and Clamped Terms

LinearRewardTerm: Computes a linear function of the absolute value of the input metric, clamped by a specified maximum/minimum reward limit. (For a sign-preserving variant, use ScaledRewardTerm.) $$\text{Reward} = \text{clamp}(\text{weight} \times |\text{metric}|, -\text{limit}, \text{limit})$$
QuadraticRewardTerm: Applies a quadratic scaling to penalize larger errors exponentially.
OffsetLinearRewardTerm / OffsetQuadraticRewardTerm: Introduces a deadband threshold. No penalty is applied until the metric exceeds the specified threshold.

🎯 Kernel and Tracking Terms

GaussianRewardTerm: A radial basis function that provides a smooth bell-shaped curve. Highly effective for rewarding precise positioning. $$\text{Reward} = \text{weight} \times \exp\left(-\frac{\text{metric}^2}{2 \, \text{scale}^2}\right)$$
GaussianTrackingRewardTerm: Extends the Gaussian reward to specifically track a dynamic target value, matching a given command.
GaussianZoneTrackingRewardTerm: Tracking reward with a flat tolerance band (deadband zone) around the target value.

🛡️ Barrier Terms

HyperbolicBarrierRewardTerm / GaussianBarrierRewardTerm: Triggers a steep asymptotic penalty when a metric crosses a safety threshold (e.g., ground clearance or joint limit violations).

3. Reward Normalization

SRB features a robust normalization system to prevent specific tasks or custom configurations from dominating the reward budget. This ensures stable learning across very different environments.

Under the hood, BaseRewardCfg automatically compiles the theoretical worst and best step returns by summing the signed reward_bounds (worst, best) extremes of all registered reward terms. For one-sided terms this reduces to bucketing each limit by the sign of its weight; sign-preserving terms (e.g. ScaledRewardTerm, SmoothMaxRewardTerm) contribute to both sides:

$$R_{\text{worst}} = \sum_{T} \text{worst}(T) \qquad R_{\text{best}} = \sum_{T} \text{best}(T)$$

Normalization Modes

You can control how scaling behaves via the normalize parameter on the reward configuration:

Normalization Mode	Value	Behavior & Mathematics	Recommended For
`NONE`	`"none"`	sum is returned exactly as-is.	Custom manual debugging or classic benchmarks.
`PER_STEP`	`"per_step"`	Divided by the maximum step scale: $S = \max(R_{\text{best}},	R_{\text{worst}}
`EPISODE`	`"episode"`	Divided by the full episode scale: $S = \max(R_{\text{best}},	R_{\text{worst}}
`AUTO` (Default)	`"auto"`	Automatically resolves to `EPISODE` for finite-horizon tasks, and `PER_STEP` otherwise.	Standard default for all environments.

4. Normalization Troubleshooting & Best Practices

Tip

Always verify that your reward terms are balanced! If one term has a extremely large limit (e.g., limit=1000.0), it will distort the scale factor, rendering other smaller terms completely negligible under normalization.

To inspect and validate your task’s active bounds and normalized scale, you can initialize your task and print the bound introspection metrics:

reward_cfg = YourTaskRewardCfg()
worst, best = reward_cfg.get_bounds()
print(f"Worst per-step: {worst}, Best per-step: {best}")

This bounds check helps ensure that no single term dominates and that the normalization range is tight and uniform.

Keyboard shortcuts

Space Robotics Bench