Skip to content

Resume and scale a study

Goal: survive an interrupted run, and add workers — threads, processes or machines — without a coordinator.

Both halves follow from one idea, explained in Storage: a study lives in storage, so a file outlives the process and workers sharing it are one study.

Give the study a file

An in-memory study dies with the process. A journal-backed one does not: it is an append-only file that records every operation, so a later handle re-reads it, continues the trial numbering, restores whatever state the sampler and scheduler persisted, and carries on.

/// Runs the study in two halves, with the handle dropped in between.
///
/// The second handle knows only the file and the study id. Everything else — the
/// seed, the direction, the declared space, the seed protocol, the trial
/// numbering, and any state the sampler or scheduler persisted — comes back out
/// of storage, which is why a resumed study cannot silently disagree with the one
/// it continues.
fn run_in_two_halves(path: &Path) -> Result<FrozenTrial> {
    let spec = path.to_string_lossy().into_owned();

    // First handle: create the study in the journal file and spend half the
    // budget. Dropping it at the end of this scope is the "interruption" — a
    // crashed worker, a pre-empted spot instance, a laptop closing.
    {
        let study = Study::builder()
            .storage(atune::open_storage(&spec)?)
            .budget(Budget::new_with_trials(SEGMENT))
            .create(StudyConfig::new(STUDY).with_seed(SEED))?;
        study.optimize(evaluate)?;
        expect_count(&study, SEGMENT, "the first handle")?;
    }

    // Second handle: a fresh `open_storage` on the same path, then `load` rather
    // than `create`. The budget counts the trials already in the study, so
    // `trials(TRIALS)` means "run up to TRIALS", not "run TRIALS more".
    let study = Study::builder()
        .storage(atune::open_storage(&spec)?)
        .budget(Budget::new_with_trials(TRIALS))
        .load(ONLY_STUDY)?;
    study.optimize(evaluate)?;
    expect_count(&study, TRIALS, "the resumed handle")?;

    // The numbering continued rather than restarting: trial `SEGMENT` is the
    // first one the second handle ran, and it exists exactly once.
    let view = study.view()?;
    let first_after = TrialNumber::new(SEGMENT);
    if view.get(first_after).is_none() {
        return Err(Error::Conflict(format!(
            "the resumed study has no trial #{SEGMENT}: the numbering restarted"
        )));
    }

    study
        .best_trial()?
        .ok_or_else(|| Error::NotFound("the resumed study has no best trial".to_owned()))
}

def run_in_two_halves(path: str) -> atune.FrozenTrial:
    """Runs the study in two halves, with the handle dropped in between.

    The second handle knows only the file. Everything else — the seed, the
    direction, the declared space, the seed protocol, the trial numbering, and any
    state the sampler or scheduler persisted — comes back out of storage, which is
    why a resumed study cannot silently disagree with the one it continues.
    """
    # First handle: create the study in the journal file and spend half the
    # budget. Dropping it at the end of this scope is the "interruption" — a
    # crashed worker, a pre-empted spot instance, a laptop closing.
    study = atune.create_study(
        direction="minimize", storage=path, name=STUDY, seed=SEED
    )
    study.optimize(objective, n_trials=SEGMENT)
    expect_count(study, SEGMENT, "the first handle")
    del study

    # Second handle: `load_study` on the same path. `n_trials` counts the trials
    # this call runs, so the study reaches TRIALS in total.
    study = atune.load_study(path, name=STUDY)
    study.optimize(objective, n_trials=TRIALS - SEGMENT)
    expect_count(study, TRIALS, "the resumed handle")

    # The numbering continued rather than restarting: trial `SEGMENT` is the first
    # one the second handle ran, and it exists exactly once.
    numbers = [trial.number for trial in study.trials]
    if numbers.count(SEGMENT) != 1:
        msg = (
            f"the resumed study has no single trial #{SEGMENT}: the numbering restarted"
        )
        raise AssertionError(msg)

    best = study.best_trial
    if best is None:
        msg = "the resumed study has no best trial"
        raise AssertionError(msg)
    return best

That is the whole of resuming from the caller's side — open the same storage, keep going. In Rust the trial budget counts trials in the study, so a study of 100 trials that stopped at 60 needs the same budget of 100, not 40; Python's optimize(n_trials=…) counts the trials that call runs, so the resumed handle asks for the 40 that are left. Python can also leave the count out entirely — optimize(objective, n_trials=None, timeout=1800) runs for half an hour and then stops, and Study.stop() from inside an objective or an optimize(callbacks=…) callback ends the run at the trial that decides it has seen enough. Both mean "start no new trial after this": the trial in flight always finishes and is recorded, so a resumed study never inherits a half-written one.

Reopening it, in either language

Rust reopens with open_storage(spec) + builder().load(id) — the listing passes the constant id 0 a single-study journal always has, and StudyLocator is the lookup for storages that hold more; Python with atune.load_study(spec), or create_study(..., load_if_exists=True) when the same script both creates and resumes. Both adapters keep one study per spec.

Prove the resume was exact

A resume that silently loses state is worse than a crash, because you keep the number. So the example does not illustrate resuming — it checks it, by running the same study both ways and comparing:

/// Fails unless two trials are the same answer, to the bit.
///
/// Values and parameters are compared as `f64::to_bits`, not with `==`: two
/// numbers that print identically can still differ in the last place, and "the
/// resumed run agrees to six decimals" is not the claim being made.
fn require_identical(resumed: &FrozenTrial, control: &FrozenTrial) -> Result<()> {
    if resumed.number != control.number {
        return Err(Error::Conflict(format!(
            "the resumed study's best is #{}, the uninterrupted study's is #{}",
            resumed.number.get(),
            control.number.get()
        )));
    }
    let (left, right) = (value_of(resumed)?, value_of(control)?);
    if left.to_bits() != right.to_bits() {
        return Err(Error::Conflict(format!(
            "best values differ: resumed {left:?} (bits {:#018x}) vs uninterrupted {right:?} \
             (bits {:#018x})",
            left.to_bits(),
            right.to_bits()
        )));
    }
    for (name, value) in &resumed.params {
        let other = control.params.get(name).ok_or_else(|| {
            Error::Conflict(format!("the uninterrupted study's best has no `{name}`"))
        })?;
        if !same_bits(*value, *other) {
            return Err(Error::Conflict(format!(
                "parameter `{name}` differs: resumed {value} vs uninterrupted {other}"
            )));
        }
    }
    Ok(())
}

/// `true` when two parameter values are identical — floats by their bits.
fn same_bits(left: ParamValue, right: ParamValue) -> bool {
    match (left, right) {
        (ParamValue::F64(left), ParamValue::F64(right)) => left.to_bits() == right.to_bits(),
        (left, right) => left == right,
    }
}

def require_identical(resumed: atune.FrozenTrial, control: atune.FrozenTrial) -> None:
    """Raises unless two trials are the same answer, to the bit.

    Values and parameters are compared through ``float.hex()``, not with ``==``:
    two numbers that print identically can still differ in the last place, and
    "the resumed run agrees to six decimals" is not the claim being made. It is
    the same comparison the Rust arm makes with ``f64::to_bits``.
    """
    if resumed.number != control.number:
        msg = (
            f"the resumed study's best is #{resumed.number}, "
            f"the uninterrupted study's is #{control.number}"
        )
        raise AssertionError(msg)
    left, right = value_of(resumed), value_of(control)
    if left.hex() != right.hex():
        msg = f"best values differ: resumed {left!r} ({left.hex()}) vs uninterrupted {right!r} ({right.hex()})"
        raise AssertionError(msg)
    for name, value in resumed.params.items():
        if name not in control.params:
            msg = f"the uninterrupted study's best has no `{name}`"
            raise AssertionError(msg)
        other = control.params[name]
        if not same_bits(value, other):
            msg = (
                f"parameter `{name}` differs: resumed {value} vs uninterrupted {other}"
            )
            raise AssertionError(msg)


def same_bits(left: object, right: object) -> bool:
    """``True`` when two parameter values are identical — floats by their bits."""
    if isinstance(left, float) and isinstance(right, float):
        return left.hex() == right.hex()
    return left == right

Twenty trials, reopen, twenty more, against forty uninterrupted in memory. The comparison is on f64::to_bits, not ==: two numbers that print identically can still differ in the last place, and "agrees to six decimals" is not the claim.

This is worth stealing for your own storage backend. If you implement Storage, the conformance suite is the real test — but a resume-equals-uninterrupted check is a small sanity assertion on top of it.

Add workers

Three levels, in increasing order of how much can go wrong:

Threads in one process. parallelism(n) on the builder. Same storage, same process, n objectives evaluated at once; Determinism says when the result is still reproducible trial for trial.

Processes on one machine. Point each at the same journal file. Trial numbers are assigned race-safely by storage, so two workers never claim the same number. The CLI does this natively — several atune run invocations against one --study path are one study.

Machines. One process runs atune serve --storage <spec>, and the others use an atune://host:port storage spec. The wire protocol mirrors the Storage trait operation for operation, and the proof it does is that the same conformance suite that tests the local backends passes over TCP unchanged.

atune serve is plaintext TCP with no authentication and no TLS; a peer that can reach it can read and modify every study. Before binding beyond loopback, follow the security policy deployment controls. The feature reference describes the remote feature.

Structural costs to plan for

Journal write amplification and observer refresh costs are covered in Storage.

Finalization recovery has a separate rule: it may reclaim a held claim only after the exact claim remains unchanged for three seconds of local monotonic observation, with nominal 100 ms polling. This recovery grace does not use Journal's heartbeat setting or wall-clock timestamps. Direct finalizer admission remains nonblocking; a terminal call's pending-work drain can invoke the same recovery path. A live but stalled callback can lose its ownership fence after the grace; its later guarded writes are rejected. Recovery does not retry a failed acquisition. If it returns LeaseLost, recovery reads the claim once more and reports Backpressure only when that read confirms a different, held claim. An unchanged or vacant claim, or a failed read-back, preserves LeaseLost; other acquisition errors remain unchanged. In builds without system, including browser builds, recovery leaves a held claim in place and fails closed.

Finalization callback effects may run more than once across a crash or claim takeover. Give external effects stable identities and make them idempotent; exactly-once external effects are not promised.

What breaks when you scale

  • Determinism explains history-dependent sampling under parallel workers.
  • A lost worker leaves its in-flight trials Running until a caller runs fail_over_stale, which marks them Failed and, when the policy enables requeue, enqueues a Waiting replacement.
  • atune serve is a single point of failure while it is up. It is a convenience for sharing a study, not a replicated store.

Next