Hyperparameter Tuning

srb agent tune optimizes reinforcement-learning hyperparameters with Optuna. It answers two needs at once:

• One tuned set per algorithm, shared across all tasks — the default (matches the global hyperparams/<framework>/<algo>.yaml files), with a separate set for state-based vs visual observations.
• Optional per-task overrides tuned for a single task, which take precedence for that task only.

Install the optional dependency first:

uv sync --extra tune

How it works

The tuner is Isaac-free. It owns an Optuna study and, for each trial, spawns one srb agent train subprocess:

1. study.ask() samples a hyperparameter set from the algorithm’s search space.
2. The driver writes a single trial_cfg.yaml — a complete agent: section plus a minimal env: section (seed, num_envs) — and launches srb agent train --cfg trial_cfg.yaml --headless in its own process group.
3. A lightweight reporter wrapper inside the child writes episode statistics to a JSONL file; the driver tails it, counting total environment steps.
4. Smoothed return is reported at 20 fixed rungs; a MedianPruner kills unpromising trials (killpg on the whole group). Crashes and timeouts FAIL the trial and divergence (NaN) records it as PRUNED — neither takes down the study.
5. study.tell() records the tail-mean of the objective over the last quarter of training.

Because each trial is a separate process, one bad hyperparameter set (VRAM blow-up, NaN physics) cannot corrupt the study, and multiple workers/nodes can share one journal file.

Config layering

Configs resolve task-specific → global, in the requested observation modality:

hyperparams/
  skrl/ppo.yaml                     # L0  — global, state observations (exists today)
  skrl/ppo_visual.yaml              # L0v — global, visual observations
  task/<task>/skrl/ppo.yaml         # L1  — task-specific, state
  task/<task>/skrl/ppo_visual.yaml  # L1v — task-specific, visual

--obs {state,visual} (on train, eval, and tune) selects the modality. It only rewrites the config entry key and splits the logdir workflow name (…/<algo>/ vs …/<algo>-visual/) so state and visual checkpoints never cross-load — the algorithm label is unchanged. Visual studies additionally require sensor profiles and arrive in a later phase; until then --obs visual errors with the file to add.

Study modes

Before committing to a long study, preflight the exact command with srb agent tune check (swap run→check). It resolves the algorithm, search space, and base config, checks your --pins, and dry-assembles one trial doc in memory — no Isaac boot, no study written — so a typo or a missing portfolio baseline fails in milliseconds instead of after the first trial spins up.

Single-task (default) tunes one task and exports to its task-specific slot:

srb agent tune run --env sample_collection --algo skrl_ppo \
  --trial-steps 20_000_000 --n-trials 40 --n-workers 2 --gpus 0,1

Multi-task (portfolio) studies tune one hyperparameter set across a task set and export it to the global slot:

srb agent tune run --tasks sample_collection peg_in_hole landing --algo skrl_ppo \
  --trial-steps 10_000_000 --n-trials 30 --normalize baseline --aggregate median

Each trial trains the sampled set sequentially on every task (one child per task segment); a diverging segment prunes the whole trial, and the running aggregate is reported at each segment so weak candidates are cut between tasks.

Cross-task normalization

Tasks have different reward scales, so raw returns are not directly comparable. --normalize maps each task’s score onto a common scale before --aggregate (median default, or mean/min) combines them:

• baseline — (score − floor) / (ref − floor), where floor is a random policy and ref a reference run. Capture both first with srb agent tune baseline --tasks … --algo … --budget <trial-steps>, which writes hyperparams/tune/baselines.yaml. A missing baseline fails fast; a stale one (the source config’s git blob SHA changed since capture) warns.
• rank — per-task fractional rank across completed trials; needs no baselines but is non-stationary early, so it is off by default.
• none (default) — aggregates raw returns; only meaningful when the tasks already share a scale, and the driver warns for a portfolio study otherwise.

Per-task raw and normalized scores are recorded in each trial’s user_attrs (seg/<task>/raw, seg/<task>/norm).

Picking a trial budget (wall-clock worksheet)

--trial-steps is total environment steps, identical across frameworks. Do not copy the production budget (those yamls train until stopped, e.g. 2e8). Instead pick the step count at which a reference run reaches ~50–70 % of its converged return, then:

hours_per_trial ≈ trial_steps / (env_steps_per_second) / 3600
study_hours     ≈ n_trials × hours_per_trial / n_workers      (pruning cuts this 30–50 %)

Worked example. A task sustaining 40 000 env-steps/s with --trial-steps 20_000_000 → ~8.3 min/trial → 40 trials on 2 workers ≈ 2.8 h. A heavier task at 5 000 env-steps/s → the same budget is ~67 min/trial → ≈ 22 h. Pass --fps to have tune run print this projection, and --max-wall-hours to stop asking new trials past a cap. Start with a 10–20-trial smoke study before committing to 40–150 trials.

Inspecting and exporting

srb agent tune status --study srb-skrl_ppo-state-sample_collection-v1
srb agent tune export --study srb-skrl_ppo-state-sample_collection-v1 --to task --dry-run
srb agent tune export --study srb-skrl_ppo-state-sample_collection-v1 --to task

Before exporting, consider srb agent tune validate --study S --top-k 5 --seeds 4 as a noise arbiter: TPE scores each trial on one seed, so best_trial can be a seed-lucky outlier. validate re-runs the top-K trials on several fresh seeds and re-ranks them by the multi-seed mean, printing the robust winner (and flagging when it differs from best_trial). It is advisory — nothing is written into the study — so you then export --trial <robust winner>.

To attack that same seed noise during the search instead of after it, pass --seeds-per-trial K to run: each trial trains K children on distinct seeds and the sampler is told their mean. Pruning runs on the first replicate only, so bad configs still cost one child while good ones cost ~K×. It trades compute for a cleaner objective; validate trades nothing but only re-ranks the survivors. Use one, the other, or both.

export writes the winning trial’s tuned hyperparameters to the L0/L1 slot with a provenance header (study, trial, value, objective, seeds, date, git SHA, Optuna version). The trial’s tiny budget and forced-off W&B flag are reset to the production defaults. --to defaults to the task slot for single-task studies and the global slot for portfolio studies (--to task on a portfolio study is ambiguous and errors). It refuses to overwrite without --force, and warns about shadowing in both directions: a global export that existing task-specific files will keep overriding, or a task export that shadows the current global set for that task. Adoption is ordinary git review — inspect the diff and commit.

Parallelism and resuming

• --n-workers N runs N concurrent trial slots in one driver; --gpus 0,1 round-robins GPUs; child starts are staggered to avoid concurrent Isaac first-boot shader-cache races (the stagger stops automatically once the first child has booted).
• Multi-node: run the same tune run command on each node against the shared logs/tune/<study>/journal.log. A ready-to-adapt SLURM array job is in scripts/tune_slurm.sbatch — each array element is one worker sharing the journal; Optuna coordinates trial hand-out with no scheduler integration.
• Re-running the same command resumes (until --n-trials finished trials exist). A changed search space, trial budget, objective, obs mode, or task set is refused with a suggestion to bump --version.
• Interrupted RUNNING trials are left alone (safe across nodes); clean them up explicitly with srb agent tune repair --study S --stale-hours H.
• Pruned trials’ child logdirs (checkpoints, TensorBoard events) are deleted after the prune to keep disk growth bounded; pass --keep-artifacts to keep them. trial_cfg.yaml and metrics.jsonl always remain.

Reproducibility

Every trial is fully reproducible from its logs/tune/<study>/trials/<n>/ directory: the exact trial_cfg.yaml, the seed recorded in it (seed-base + trial × seeds-per-trial + replicate; replicate seeds live in rep-<k>/trial_cfg.yaml), and the child’s own run metadata. Each child’s metadata.json also records tune: {study, trial}, so any checkpoint can be traced back to its trial, and the study directory carries a driver.json (host, pid, start time) identifying the active driver. See Agent Configuration for where tuned files live and how precedence works.