Benchmarks¶
This page gives context for the committed benchmark artifacts and their limits. It does not claim that these measurements establish a general ranking.
The suite also writes a self-contained interactive page — every chart inline, no
external asset — and a site deployment publishes it alongside these tables:
the interactive benchmark page. It is the same
crates/atune_bench/bench/benchmark.html that is committed in the repository, so
a clone opens it without a browser ever touching this site.
What is measured, and what is not¶
The kurobako suite compares samplers on synthetic single-objective problems. It does not measure multi-fidelity methods, reinforcement-learning workloads, or the Python and CLI surfaces. The benchmark harness and reproduction details live in the crate benchmark README; Python/Rust parity has its own gate.
Open-space growth and calibration are separate test-gated measurements, not published results; their tests print the current numbers.
The kurobako suite¶
The tables below reflect committed artifacts, not a fresh run. Their source
and run provenance appear in the generated table headers; the historical run
does not record a complete environment. The run predates the current sampler
defaults: its atune-tpe, atune-sobol, atune-gp, and atune-cmaes arms ran
the earlier defaults (univariate TPE, sorted-name Sobol, GP-EI's random-candidate
search, CMA-ES without Optuna initialization), which the suite now runs as the
-legacy arms. The benchmark README
owns the reproduction recipe and candidate-review process.
Cross-problem aggregate¶
The aggregate uses normalized scores, tied mean ranks and seed-paired win rates; see the benchmark README for the calculation rules.
| Arm | normalized (0 = best) | mean rank | win rate vs kurobako-random |
win rate vs optuna-tpe |
note |
|---|---|---|---|---|---|
atune-gp |
0.032 | 1.55 | 100% | 94% | |
atune-auto |
0.076 | 1.95 | 100% | 89% | |
optuna-cmaes |
0.236 | 4.64 | 90% | 63% | |
atune-tpe |
0.250 | 4.50 | 93% | 59% | |
atune-cmaes |
0.265 | 4.73 | 88% | 59% | |
optuna-tpe |
0.267 | 4.73 | 90% | — | |
optuna-qmc |
0.674 | 8.00 | 54% | 11% | |
atune-sobol |
0.692 | 8.55 | 48% | 11% | deterministic (ignores the seed) |
kurobako-random |
0.700 | 9.00 | — | 10% | |
optuna-random |
0.729 | 9.09 | 45% | 8% | |
atune-random |
0.730 | 9.27 | 49% | 8% |
Two problems are excluded from that aggregate, and both exclusions are structural rather than chosen after seeing the result:
ln(sigopt/evalset/Ackley(dim=2))— kurobako'slnwrapper evaluates the same objective atln x, so it is protocol coverage rather than a second landscape. Measured agreement: 214 of 220 (arm, seed) cells reached an identical final value.sigopt/evalset/Sphere(dim=6, int=[0, 1, 2, 3, 4, 5])—atune-cmaesdeclines a space with no continuous axis, so scoring the problem would average the other ten arms over a subset one arm never ran.
One problem in full¶
sigopt/evalset/Ackley(dim=6), final best value, lower is better. It illustrates
a case where atune's TPE, CMA-ES and random arms lose to their Optuna counterparts.
| Arm | best (mean ± sd) | repeats | solver time / study |
|---|---|---|---|
atune-auto |
3.34236 ± 1.15 | 20 | 8122.0 ms |
atune-gp |
3.34236 ± 1.15 | 20 | 7320.8 ms |
optuna-tpe |
8.36072 ± 1.47 | 20 | 585.9 ms |
optuna-cmaes |
8.95716 ± 1.73 | 20 | 118.5 ms |
atune-cmaes |
8.96967 ± 1.14 | 20 | 3.7 ms |
atune-tpe |
10.257 ± 1.93 | 20 | 39.5 ms |
optuna-random |
13.9797 ± 1.55 | 20 | 55.9 ms |
atune-random |
14.5717 ± 1.74 | 20 | 2.7 ms |
kurobako-random |
15.2465 ± 1.51 | 20 | 0.0 ms |
atune-sobol |
15.6586 ± 0 | 20 | 2.1 ms |
optuna-qmc |
15.6586 ± 0 | 20 | 89.6 ms |
The other twelve per-problem tables are in the same artifact. Four readings the table does not make for you:
- A
± 0means unseeded, not reliable. An unscrambled Sobol sequence ignores the seed, so those 20 repeats are one sample shown 20 times. - Solver time is reported, never ranked on. The Optuna arms pay a Python round-trip per protocol message that the native arms do not, and the two Gaussian-process arms pay a model fit per suggestion.
- The win-rate baseline is
kurobako-random, which shares no code with atune, rather than atune's own random arm. - Optuna's
GPSampleris not an arm. It needstorch, which the benchmark environment does not carry, so atune's GP arm has no direct counterpart here.
The in-repo sampler oracle¶
A separate self-contained oracle runs in the ordinary test gate on standard
problems and three samplers. Its recorded values are also available in
oracle.json.
The command is documented in the crate README. Cells are means over eight seeds; the tool reports no dispersion. Final gap to the known optimum, lower is better:
| Problem | random |
sobol |
tpe |
|---|---|---|---|
| sphere, 2-D | 3.0121e-1 | 0.0000e0 | 5.9029e-4 |
| sphere, 5-D | 5.8348e0 | 0.0000e0 | 4.9250e-1 |
| shifted sphere, 2-D | 3.3623e-1 | 8.2421e-2 | 3.7727e-4 |
| shifted sphere, 5-D | 8.8523e0 | 9.3999e0 | 6.9139e-1 |
| Rastrigin, 2-D | 5.4198e0 | 0.0000e0 | 4.0237e-1 |
| Ackley, 2-D | 9.0886e0 | 0.0000e0 | 9.9977e-1 |
| Branin | 3.4350e-1 | 3.0654e-1 | 5.6728e-2 |
Sobol's 0.0000e0 cells are a property of the fixture rather than a result:
those four problems are centred on the origin, and an unscrambled Sobol sequence
evaluates the domain centre early, so the sampler lands on the optimum exactly
before any search has happened. The shifted variants exist to move the optimum
off that grid — and there Sobol loses to random search in 5-D (bold above),
while on Branin its margin is 3.6962e-2. That is why the crate's own quality
gate asserts the Sobol ordering only where it holds.
The same command ends with a multi-fidelity block: budget 1440 fidelity units, mean of 8 seeds, quality measured as the true value of the best configuration re-evaluated at the full budget.
| Problem | dehb |
random |
tpe |
|---|---|---|---|
| sphere 4-D, ladder [1, 3, 9] | 4.1093e-1 (400 evaluations) | 2.8038e0 (160) | 1.1051e-1 (160) |
| Rastrigin 3-D, ladder [1, 3, 9] | 3.7679e0 (400) | 1.4250e1 (160) | 2.2134e0 (160) |
| sphere 4-D, ladder [1, 3, 9, 27] | 1.1144e0 (252) | 6.5478e0 (54) | 7.5370e-1 (54) |
DEHB loses to full-budget TPE on all three of these surfaces. The other number in the same cells is a count rather than a quality: at a matched fidelity budget DEHB evaluated two to five times as many configurations as the other two arms. Both are stated; neither is a verdict.
Where to go next¶
| If you want to | Go to |
|---|---|
| The reasoning behind the RL-oriented features | RL evidence |
| Which tool has which capability, with citations | Prior art |
| What atune does that is unusual | Why atune |
| The samplers and schedulers themselves | Catalog |
| To run these gates yourself | Contributing |