Module cartpole 

CartPole-v1 and the Augmented Random Search learner behind the cartpole problem (D113).

The environment follows Gymnasium’s CartPole-v1: gravity 9.8, a cart of mass 1.0, a pole of mass 0.1 and half-length 0.5, a push of 10 newtons to either side, and Euler integration at tau = 0.02 seconds. An episode terminates when the cart leaves |x| ≤ 2.4 or the pole leaves |θ| ≤ 12°, and is truncated at 500 steps; its return is its length.

The learner is basic Augmented Random Search (Mania, Guy and Recht 2018, V1) on a linear deterministic policy w ∈ R⁵: it pushes right iff w · (x, ẋ, θ, θ̇, 1) > 0. Each iteration draws DIRECTIONS Rademacher directions δ ∈ {−1, +1}⁵, runs one episode at w + νδ and one at w − νδ from a shared start state, and moves w ← w + α / (N σ_R) · Σ (r₊ − r₋) δ, with σ_R the standard deviation of the 2N returns (no update when it is 0). An epoch is ITERATIONS iterations followed by EVAL_EPISODES evaluation episodes from fixed start states; the epoch’s shortfall is 500 minus their mean length.

A run reports, after epoch k, the mean shortfall over its epochs 1..=k: the area above its learning curve so far, which rewards learning fast as well as learning well. After crate::training::STEPS epochs that mean is the objective, so a scheduler’s rung at epoch k compares runs over the same epochs. 0 is a lower bound: it would mean balancing the pole for all 500 steps of every evaluation episode from the first epoch on. A checkpoint carries the shortfall sum with the weights, so a paused run resumes its own mean, and a PBT or PB2 fork inherits its parent’s: the child’s report at epoch k averages the parent’s epochs up to the fork and its own after it.

A linear policy only reads the sign of w · s, so plain ARS from w = 0 is scale-equivariant: scaling α and ν together scales every iterate, and the landscape would depend on ν / α alone. Training therefore starts from the fixed weights INITIAL_WEIGHTS, which sets a scale: a step size far below it barely moves the policy in crate::training::STEPS epochs, and a noise far below it rarely changes an action.

Everything is integer hashing (the SplitMix64 finalizer of crate::training::unit_noise) and basic float arithmetic: sine and cosine are fixed polynomials (sin, cos), 10^x is pow10, and the only other operations are + − × ÷, sqrt and floor, which IEEE 754 rounds exactly. The host and the browser therefore train bit for bit the same policies.

Structs

Hyper

A training run’s hyperparameters.

Learner

A training run’s state: what a checkpoint holds.

Constants

DIRECTIONS

ARS directions per iteration (N).

EVAL_EPISODES

Evaluation episodes after each epoch.

EVAL_SEED

The seed of the fixed evaluation start states.

FORCE

The push to either side, N.

GRAVITY

Gravitational acceleration, m/s².

HALF_LENGTH

Half the pole’s length, m.

INITIAL_WEIGHTS

The weights every training run starts from (see the module docs): a policy that drops the pole within a few dozen steps.

ITERATIONS

ARS iterations per training epoch.

MASS_CART

The cart’s mass, kg.

MASS_POLE

The pole’s mass, kg.

MAX_STEPS

The step at which an episode is truncated, and the best return.

START_SPREAD

The half-width of the uniform start-state distribution.

TAU

The integration step, s: 50 steps per second of simulated time.

THETA_LIMIT

The pole angle beyond which an episode terminates: 12°, in radians.

TRAIN_SEED

The seed of the training draws (directions and start states). Every trial trains from it, so a trial’s value is a pure function of its point.

WEIGHTS

The policy’s weights: one per observation, then the bias.

X_LIMIT

The cart position beyond which an episode terminates, m.

Functions

act

The policy’s action in state: push right iff w · (s, 1) > 0.

advance

One Euler step of the dynamics under a push to the right (true) or left.

cos

cos x for |x| ≤ 0.25 by its Taylor polynomial through x¹⁰: the truncation error is below 0.25¹² / 12! < 1.3e-16 (see sin).

episode

The length of one episode of weights from start: the steps until termination (the terminating step included), at most MAX_STEPS.

eval_start

The indexth fixed evaluation start state.

evaluate

The shortfall of weights: 500 minus the mean length of the EVAL_EPISODES evaluation episodes.

objective

The objective at (x, y) = (log10 α, log10 ν): a fresh run’s mean shortfall over its STEPS epochs. The heatmap shows it; an unscheduled trial reports it.

pow10

10^x from basic arithmetic: x = n + f with n = ⌊x⌋, 10^f as exp(f ln 10 / 16)¹⁶ by a degree-11 Taylor polynomial, and 10^|n| by exact multiplication (for |n| ≤ 22).

replay

Every state of one episode of weights from the first evaluation start, the start included: what the policy panel replays.

sin

sin x for |x| ≤ 0.25 by its Taylor polynomial through x⁹.

terminated

Whether state ends the episode: the cart or the pole is out of bounds.

train

A fresh run’s STEPS epochs at (log10 α, log10 ν): the final learner and its value.

Type Aliases

State

(x, ẋ, θ, θ̇): cart position and velocity, pole angle (0 is upright, positive leans right) and angular velocity.