The reinforcement-learning evidence¶
This page connects published findings about reinforcement-learning (RL) tuning to atune's design. It is not evidence of atune's comparative quality; see Benchmarks for measurements and Prior art for capability citations.
Why RL is a different problem¶
RL tuning combines noisy, costly evaluations with settings that may need to change during training; divergence and NaN are ordinary outcomes. A single configuration score therefore gives an incomplete basis for search decisions.
What the literature establishes¶
Eimer, Lindauer and Raileanu (Hyperparameters in Reinforcement Learning and How To Tune Them, ICML 2023) report a 64-run DEHB search beating an 810-run grid, and find multi-fidelity and population methods useful under RL noise. Their seed-overfitting result also motivates separating tuning seeds from a final held-out evaluation; see the multi-seed protocol. Agarwal et al.'s rliable work recommends IQM over fragile point estimates; at three seeds or fewer, atune's IQM trims nothing and equals the mean.
Population-based training (Jaderberg et al., 2017) and PB2 (Parker-Holder et al., NeurIPS 2020) adapt settings during a run; FIRE-PBT (Dalibard & Jaderberg, 2021) highlights the risk of greedy exploitation. Successive halving (Li et al., 2020) and BOHB (Falkner et al., 2018) establish multi-fidelity search, while noisy RL curves make early stopping error-prone. These findings motivate the scheduling and noise-aware pruning paths described in Prune and schedule.
ARLBench (arXiv:2409.18827) finds that a
small number of hyperparameters carry much of the variance, supporting
importance analysis. ULTHO (arXiv:2503.06101)
adapts parameters inside one run, where low-cost decisions matter. DyHPO (2022)
and ifBO (ICML 2024) resume paused trials when their predicted curves improve;
pause and resume are prerequisites for that class of method. atune's
atune::OnlineTuner provides the low-cost decision seam for
in-run adaptation.
One number, and what it is not¶
Illustrative, and not reproducible from this repository
A cheap soft actor-critic recipe on CartPole, at a fixed budget, returned mean scores of roughly 183, 86 and 119 on seeds 0, 1 and 2 — same code, same configuration, same budget.
That measurement comes from an internal harness driving an external RL implementation, so it cannot be reproduced from this repository. It is here to show the size of the seed effect, not to say anything about atune's quality and not to compare atune with anything. Every number that does support a comparison lives on Benchmarks with the command that reproduces it.
Two configurations compared on one seed apiece, at that budget, are being compared on noise. That is the whole argument for the multi-seed protocol in one line.
What has not been measured here¶
No reproducible RL benchmark result is published from this repository; current measurements cover synthetic single-objective problems (Benchmarks). The cited results support methods, not atune's implementation. The runnable protocol demonstrates machinery, not that seed-overfitting has been re-measured.
Where to go next¶
| If you want to | Go to |
|---|---|
| The protocol as a lesson, on a runnable example | Tuning an RL agent |
| The protocol as a recipe | Multi-seed RL protocol |
| Schedules rather than constants | Population-based training |
| Pruning that survives noise | Prune and schedule |
| What ships, and behind which feature | Catalog |
| Measurements with reproduction commands | Benchmarks |