Module cmaes_bench 

The M4.3 CMA-ES oracle’s driver: prove Cmaes earns its slot as the continuous black-box optimizer — and prove it where the covariance is what does the earning (docs/design/09-implementation.md §13).

This is to CMA-ES what dehb_bench is to DEHB and pbt_bench is to PBT. The comparison axis here is the plain one — matched trial count — because CMA-ES is a single-fidelity sampler: every searcher gets the same number of evaluations of the same objective, and the winner is whoever ends lower.

What is actually being claimed

“Beats random search on a smooth bowl” is table stakes; TPE has cleared that bar since M2.2. CMA-ES’s distinguishing claim is narrower and much sharper:

it adapts a full covariance, so it solves an ill-conditioned problem whose valley runs at an angle to the coordinate axes — the case an axis-aligned searcher (uniform random, or a TPE that models each dimension with its own 1-D density) has no representation for.

Landscape::rotated_ellipsoid is that problem, and it is built so the two ingredients can be separated rather than asserted as a bundle:

Fixture	Conditioning	Frame	What it isolates
Landscape::sphere	1	—	the baseline every sampler must clear
Landscape::ellipsoid	1e4	axis-aligned	conditioning alone — an axis-aligned searcher can still solve it coordinate by coordinate
Landscape::rotated_ellipsoid	1e4	rotated	conditioning off the coordinate frame — the covariance is the only way in

If CMA-ES could not beat an axis-aligned searcher on the third row, its covariance update would be wrong; that is a finding, not a tuning problem, and cmaes_oracle is where it would surface.

The fixtures are shifted, on purpose

Every landscape’s minimizer is displaced from the centre of its box (Landscape::shift). CMA-ES starts its mean at the centre of the unit hypercube — which is the centre of the box — so a centred optimum would hand it the answer before the first generation and make every margin below a measurement of the fixture rather than of the sampler. The shift is irrational-looking and never lands on a Sobol node, for the same reason Problem::shifted_sphere exists.

How the claims are stated

Win-rates over a seed set, with thresholds taken from a measured 64-seed sweep, never per-seed universals — the M4.2 lesson (docs/design/STATUS.md, DEHB notes: “the oracle asserted a ~94 %-true property as universal”). Structural properties of the fixtures (the condition number, the orthogonality of the rotation, the location of the optimum) stay hard assertions, because those really are universal.

Determinism

Every run is single-worker with a fixed study seed, a ManualClock and a pure objective, so a whole run is reproducible (run is a pure function of (searcher, landscape, budget, seed)). CMA-ES is stateful and history-dependent, so that is single-worker determinism plus replayability, never trial-number-indexed determinism (docs/design/09-implementation.md §5).

Structs

Landscape

A continuous minimization problem: a boxed domain, a displaced minimizer, an optional rotation, and a base function.

RunReport

One searcher’s run over one landscape.

Enums

Searcher

Which searcher a run drives.

Constants

CONDITION

The condition number every ill-conditioned fixture uses.

Functions

rayleigh

The Rayleigh quotient uᵀ C u / uᵀu — how much variance the covariance C carries along the direction u.

rotation_matrix

A fixed orthogonal d × d rotation that mixes every axis into every other, row-major: the DCT-IV matrix R[k][j] = √(2/d)·cos(π(2k+1)(2j+1)/(4d)).

run

Runs searcher on landscape for budget trials under study seed seed.

run_cmaes

run for CMA-ES, handing back the sampler so a test can inspect the distribution it learned (generation count, step size, covariance).