srb agent tune — Hyperparameter Optimization

The srb agent tune command optimizes reinforcement-learning hyperparameters with Optuna. It runs without launching Isaac Sim itself — instead it drives an Optuna study that spawns one srb agent train subprocess per trial, so a crashing or diverging trial never takes down the study. Install the optional dependency with uv sync --extra tune.

See the Hyperparameter Tuning workflow for concepts, the wall-clock worksheet, and adoption guidance.

Verbs

srb agent tune check    [options]   # preflight a run config without training (Isaac-free)
srb agent tune run      [options]   # run or resume a study (single-task or portfolio)
srb agent tune status   --study S   # summarize a study
srb agent tune validate --study S   # re-run top-K trials on fresh seeds, re-rank (noise arbiter)
srb agent tune export   --study S   # write a tuned config to hyperparams/
srb agent tune repair   --study S   # fail stale RUNNING trials
srb agent tune baseline [options]   # capture normalization baselines (portfolio)

run Options

The objective field must actually be emitted by the task: a --objective success_rate study on a task that never reports success FAILs each trial early with reason success_rate_unavailable instead of training the full budget.

Averaging seeds during a study (--seeds-per-trial)

A single seed per trial gives the TPE sampler a noisy objective. --seeds-per-trial K trains K children per trial — each on a distinct seed (seed-base + trial*K + replicate) — and tells the sampler their mean, so the search sees a lower-variance signal. To keep the cost bounded, pruning runs on replicate 0 only: a config that looks bad on its first seed is pruned after one child, while surviving configs pay ~K× the compute. A replicate that crashes is dropped from the mean rather than failing the trial (n_seeds/n_seeds_ok are recorded on the trial). Replicate 0 keeps the canonical trials/NNN/ directory; the confirmation replicates write to trials/NNN/rep-K/ and their checkpoints are discarded. K=1 (the default) is the original single-seed behavior.

This is the during-optimization complement to validate, the post-hoc noise arbiter: use --seeds-per-trial to steer the search with a cleaner objective (at K× the budget), or run a cheap single-seed study and let validate re-rank the top handful on fresh seeds afterward.

Resuming an existing study re-validates the frozen manifest: a changed search space, --trial-steps, --objective, --obs, or task set is refused with a suggestion to bump --version (mixed budgets/objectives would corrupt the study history). Only seed-base and Optuna-version drift are tolerated.

check Options

check runs every cheap check run performs before its first trial — resolve the algorithm adapter and its search space, load the base agent config, check --pin format and reserved-key overlap — and then dry-assembles a single trial doc in memory (exercising the search-space sampler, config merge, pin↔sampled-key collisions, and budget conversion). It launches no study and no srb agent train child, so a misconfigured multi-hour run fails in milliseconds instead of after the first Isaac boot. It prints one [OK]/[WARN]/[FAIL] line per check and exits non-zero if any check is a hard error.

It accepts the config-shaping subset of run’s options: -e/--env/--task or --tasks, --algo (REQUIRED), --obs, --trial-steps (REQUIRED), --pin, --trial-num-envs, --seed-base, --normalize, --aggregate, --objective, --fps, --n-trials, --n-workers, --version, --logdir. For a --normalize baseline portfolio it also verifies a floor/ref baseline is present and fresh for every task. --fps adds the same wall-clock projection run prints.

srb agent tune check --env sample_collection --algo skrl_ppo --trial-steps 10_000_000 --fps 50000
srb agent tune check --tasks sample_collection peg_in_hole --algo skrl_ppo \
  --trial-steps 10_000_000 --normalize baseline   # also checks baselines.yaml

validate Options (noise arbiter)

TPE optimizes on a single seed per trial, so best_trial can be a seed-lucky outlier — a config that scored high by chance at the trial budget, not because it is genuinely better. validate arbitrates that noise: it re-runs the top-K completed trials with several fresh seeds each (disjoint from the training seeds) and re-ranks them by the multi-seed mean objective. It is advisory — it never writes into the study; results go to <study>/validation.json and a printed table, and you export the robust winner yourself.

Each candidate is re-run faithfully from its recorded trials/NNN/trial_cfg.yaml (the exact trained config, pins included); only the seed changes, and the budget is recomputed for --validate-steps. A portfolio candidate re-runs every task segment per seed and re-aggregates with the study’s own normalization. A candidate must COMPLETE at least --min-ok seeds (default: a strict majority) to be eligible as the winner — a config that crashes on most fresh seeds is not robust, however high its surviving-seed mean, and is flagged (below min-ok) in the table.

# Re-run the 5 best trials on 4 fresh seeds each; flag if the ranking flips.
srb agent tune validate --study srb-skrl_ppo-state-sample_collection-v1 \
  --top-k 5 --seeds 4 --gpus 0,1

The printout ranks candidates by validated mean and names the robust winner; when that differs from the study’s best_trial, it says so and prints the export command for the robust choice.

status / export / repair Options

baseline Options

Portfolio normalization (--normalize baseline) needs a per-task floor/ref pair. srb agent tune baseline captures them into hyperparams/tune/baselines.yaml.

Each entry records floor (random-policy mean return), ref (reference-run tail-mean), the source config path, and a git blob SHA of that config file. tune run and tune check warn when the recorded SHA no longer matches the current file — a signal to re-run tune baseline before trusting normalization.

Registered search spaces

State-observation search spaces are registered for online RL labels with global hyperparameter files. Search-space presence does not prove runtime support:

If you pass an --algo without a registered space, the driver lists the available ones and exits. A registered but unavailable label still fails its capability preflight. Visual-observation spaces are phased in later.

DreamerV3 / TD-MPC2 caveats

These two frameworks consume their config differently from the plain-YAML frameworks, so two limits apply:

• Global (L0) tuning only. Their integrations apply the global config and silently ignore the per-task blocks inside dreamerv3.yaml / tdmpc2.yaml, so a task-specific (L1) base is not honored for them yet. --to task still writes a task-scoped export (at hyperparams/task/<task>/dreamerv3.yaml / tdmpc2.yaml — the slot the config catalog resolves), but the framework will not read it until the integration learns to extract task blocks. The plain-YAML frameworks support L1 fully.
• Budget calibration. --trial-steps is passed through as the framework’s native total-env-step budget (dreamer’s nested run.steps, tdmpc2’s steps); those keys are driver-owned — export resets them to the production base, and a --pin on them errors, exactly as for the other frameworks. With num_envs > 1 the framework’s internal step counter and the reporter’s env-step count can differ by a constant factor — calibrate --trial-steps against a short reference run before trusting absolute wall-clock projections.

The DreamerV3 space tunes run.train_ratio and a model_size capacity preset (the same rssm/depth/units ladder documented in dreamerv3.yaml). The TD-MPC2 space tunes the optimizer, MPPI-planner, and loss-coefficient knobs; the architecture dims are left to model_size (TD-MPC2 derives them at load).

Portfolio (Multi-Task) Studies

Passing --tasks A B C … tunes one hyperparameter set against a whole task set: each trial trains that set sequentially on every task (fixed order), one srb agent train child per task segment under trials/NNN/seg-<task>/. The per-task scores are normalized and aggregated into a single objective; a diverging segment prunes the whole trial, and the running aggregate is reported at each segment so MedianPruner can cut weak candidates between tasks.

Because tasks have different reward scales, aggregating raw returns (--normalize none, the default) is only meaningful when they already share a scale — the driver warns otherwise. Two normalizers make scores comparable:

• --normalize baseline — (score − floor) / (ref − floor) per task, from hyperparams/tune/baselines.yaml (capture it with tune baseline first). A missing baseline fails fast; a stale one (config SHA drift) warns.
• --normalize rank — per-task fractional rank across completed trials; needs no baselines but is non-stationary early, so it is off by default.

--aggregate median (default) is robust to one outlier task; min is the robustness aggregate (one broken task sinks the trial); mean is the plain average. A portfolio study exports to the global L0 slot by default.

# 1. Capture floor/ref baselines for the task set.
srb agent tune baseline --tasks sample_collection peg_in_hole landing \
  --algo skrl_ppo --budget 10_000_000

# 2. Run the portfolio study, normalized against those baselines.
srb agent tune run --tasks sample_collection peg_in_hole landing --algo skrl_ppo \
  --trial-steps 10_000_000 --n-trials 30 --normalize baseline --aggregate median

# 3. Export the winner to the global per-algorithm config.
srb agent tune export --study srb-skrl_ppo-state-multi-<hash8>-v1   # --to global by default

Examples

Single-task study on sample_collection with a smoke budget, two workers on two GPUs:

srb agent tune run --env sample_collection --algo skrl_ppo \
  --trial-steps 20_000_000 --n-trials 40 --n-workers 2 --gpus 0,1

Inspect progress, then export the best trial to the task-specific slot:

srb agent tune status --study srb-skrl_ppo-state-sample_collection-v1
srb agent tune export --study srb-skrl_ppo-state-sample_collection-v1 --to task --dry-run
srb agent tune export --study srb-skrl_ppo-state-sample_collection-v1 --to task

Pin a value out of the search (and fix a task-side override):

srb agent tune run --env peg_in_hole --algo skrl_ppo --trial-steps 10_000_000 \
  --pin agent.agent.grad_norm_clip=1.0 env.robot=franka

Output

Study artifacts live under logs/tune/<study-name>/:

logs/tune/srb-skrl_ppo-state-sample_collection-v1/
  journal.log        # Optuna JournalStorage (NFS-safe; shared by parallel workers)
  study.yaml         # frozen manifest (search-space hash, budget, objective, seed base)
  trials/000/        # per-trial: trial_cfg.yaml, metrics.jsonl, logs/

Exported configs land in hyperparams/<framework>/<algo>.yaml (global) or hyperparams/task/<task>/<framework>/<algo>.yaml (task-specific) with a provenance header and are adopted via ordinary git review.

See Also

• Hyperparameter Tuning Workflow
• srb agent train — the per-trial training subprocess
• Agent Configuration — config layering and tuned-file locations