Write a storage backend¶
Storage is the only thing workers coordinate through — there is no coordinator process — so its invariants are the ones a bug in your backend would corrupt a study through. They are subtle, mostly concurrent, and easy to almost get right.
Which is why the strongest thing this page has to tell you is not a walkthrough: the same conformance suite that validates the in-memory, journal, SQLite and TCP backends will validate yours, unchanged. The contract is not left as prose for you to interpret. It is a battery of executable checks, written once against a factory, and you can run it before you have written a single test of your own.
What you are signing up to¶
The base CRUD/read-model methods and a contract with four load-bearing promises: trial numbers are contiguous and race-safe; a study's configuration comes back exactly as it went in; every state change has exactly one winner; a finished trial is immutable. Reads are incremental through a cursor, and side-channel state is a typed, versioned blob rather than a junk drawer. Atomic lifecycle and study-catalog behavior are opt-in capabilities, so a backend can implement the base seam without pretending it can safely provide either.
The base Storage trait alone is not enough to run the normal study loop.
It supports direct CRUD and observation, but every Study-driven trial requires
the atomic TrialLifecycleStorage capability returned by
Storage::trial_lifecycle. Without it, ask/optimize fails with
Error::Unsupported; never reconstruct reserve, claim, renew, or fenced writes
from separate CRUD calls. A third-party backend intended for optimization must
implement that capability as well as the base trait.
If essential backend state is protected by a mutex and that mutex is poisoned, return a typed error. Do not inspect or repair the protected state, or substitute an empty or default value.
Lifecycle writes and staged imports use ParamWrite with named name,
distribution, and value fields. This changes Rust construction from a tuple
to a struct literal, but its serialized form remains the historical three-item
tuple. The base Storage::write_params method remains tuple-based. Match
FencedMutation with a wildcard arm: if an adapter cannot represent a future
operation, return Error::Unsupported before changing state.
Storage explains all of that from the user's side, and
it is worth reading first — this page assumes it. One property is worth repeating
here because it is a requirement on your code rather than a feature of the
system: storage never reads the operating-system clock implicitly. Caller
timestamps are handed in, and storage-owned timestamps come from an injected
Clock, which is what lets a test drive a time-bounded study deterministically.
Snapshot compatibility¶
Implementing the existing storage methods without overriding
Storage::read_snapshot remains source-compatible. The default result is
SnapshotConsistency::BestEffort, not an atomic observation. Per pass it does
one sync trial-list read, one get_state call per concrete requested scope,
and, when outbox reads are requested and the lifecycle capability is present,
bounded list_outbox_page reads followed by one get_trial call per listed
record. If paging is unsupported or a record exceeds the page budget, this
best-effort full-snapshot fallback uses list_outbox; its assembled result can
therefore be unbounded. Named-prefix state cannot be enumerated by this fallback.
StudySnapshot may make one
additional pass when returned trial parameters reveal policy names not present
in the immutable configuration; it does not retry discovery indefinitely.
Exact built-in backends (Memory, Journal, and SQLite) provide the one-epoch
Atomic result. A remote wrapper can retain that guarantee only when it wraps
an exact backend; a custom backend using the fallback remains BestEffort.
StudyReader::user_attrs() is a separate metadata read and is not covered by
the snapshot epoch.
Backends used for writable study load or finalization recovery must implement
the finalization claim operations (observe, acquire, renew and release) and the
guarded lifecycle operations. Recovery also requires bounded outbox paging.
Unsupported paging and records too large for the requested page fail closed;
they do not fall back to the unbounded list_outbox method. The full-snapshot
fallback above is a separate, best-effort observation path.
Finalization effects are at-least-once across crashes or claim takeover. Make external effects idempotent using stable effect identities. A live but stalled owner can lose its fence after recovery observes an unchanged claim for the recovery grace, so later guarded writes from that owner are rejected.
The suite is the guide¶
The suite's own module documentation is the step-by-step walkthrough, its code
examples are compiled and run by the repository's gate, and it lives at
crates/atune_core/src/storage/conformance.rs.
Rather than restate it, here is what it costs you and what it gives back.
Add atune_core with the conformance feature under [dev-dependencies], so
the suite is not included in release builds. Use a path or git dependency until
the first release, as Install explains:
atune_core = { version = "0.1", features = ["conformance"] }.
Name atune_core even if your crate otherwise depends on the facade. The facade's
conformance feature re-exports the module as atune::storage::conformance, so
run_all and the individual checks are reachable through it — but the macro below
is exported at atune_core's root and atune::storage_conformance_tests! does not
resolve. The facade's own SQLite suite names atune_core for that reason.
Pass a factory returning a fresh, empty Box<dyn Storage> to
atune_core::storage_conformance_tests!(factory). The macro expands to thirty-four
tests named for their invariants; put it in its own module because it defines
fixed item names. Each check gets a fresh backend and reports the invariant and
observed result on failure. run_all runs the same checks as one smoke test.
A full Study backend runs four suites¶
storage_conformance_tests! checks the base Storage trait only. A backend a
Study drives must also return its TrialLifecycleStorage from
Storage::trial_lifecycle, and three more macros, defined in
crates/atune_core/src/storage/lifecycle_conformance.rs,
check that capability. They come from the same conformance feature and are
exported at atune_core's root as well. The facade re-exports neither these
macros nor the lifecycle_conformance module, so name atune_core here too.
Every lifecycle check panics at its first call if trial_lifecycle returns
None.
| Capability | Suite | Tests |
|---|---|---|
The base trait: CRUD, contiguous numbering, cursor reads, state blobs, and StudyCatalog when present |
atune_core::storage_conformance_tests! |
34 |
Budgeted reservation, claim and heartbeat renewal, abandon, fail_stale, fenced mutations, terminal writes, pause and epoch-advancing reclaim, and the state-compatibility matrix |
atune_core::lifecycle_conformance_tests! |
10 |
Pause that atomically replaces both seam states (FencedMutation::PauseWithState) |
atune_core::lifecycle_state_conformance_tests! |
1 |
The terminal-finalization outbox: terminal_with_intent, advance_outbox, outbox reads, create_replacement, put_finalized_state, and fail_stale outbox atomicity |
atune_core::outbox_conformance_tests! |
4 |
Every built-in backend runs all four. A Study reaches each row: it reserves,
claims, and fences every trial, replaces both seam states atomically when a
scheduler pauses one, and finalizes every trial through the outbox. The
lifecycle macros are separate so an adapter can land ownership first and opt
into the outbox once it is durable, but such an adapter cannot yet run a
Study to completion. The four macros' test names do not collide, so they can
share one module, kept apart from your own tests.
atune_core::storage::lifecycle_conformance::run_all runs all fifteen
lifecycle checks as one smoke test.
The four suites are not the whole contract. They do not call the finalization
claim operations (observe_finalization, acquire_finalization,
renew_finalization, release_finalization), the owned and operation-tagged
writes the study loop issues (terminal_with_intent_owned, mutate_owned,
advance_outbox_owned, and the *_with_operation methods), or bounded
list_outbox_page reads. A Study acquires a finalization claim before it
finalizes a trial, and recovery needs paging, so cover those methods with your
backend's own tests, as each built-in backend does.
Four backends already run it¶
| Backend | How it runs the suite |
|---|---|
| In memory | The reference implementation, and the oracle the suite is calibrated against |
| Journal file | The same battery over a real file, with its own extra tests for what the trait cannot express |
| SQLite | The suite unchanged, through the exported macro, exactly the way an out-of-tree backend would run it |
| Remote | The suite unchanged, over a real loopback TCP connection, against a fresh embedded server per check |
The remote row is the strongest evidence that the suite is genuinely backend-agnostic: every invariant is asserted across a socket, including the typed error cases and the non-finite float round trips, which are precisely what a subtly lossy translation layer would break.
The concurrent checks, and how to know they ran¶
Four checks exercise concurrent catalog creation (when supported), trial
numbering, transitions, and cursor reads. Barriers make their ordering
deterministic. Their bodies are no-ops on wasm32 and in
--no-default-features builds; assert runs_concurrent_checks() in any build
where concurrency is part of your release gate.
What the suite deliberately does not check¶
The suite does not cover durability, cross-process visibility or crash recovery;
those are backend-specific, and the journal backend has separate tests. It also
does not require unique study names. If the backend opts into StudyCatalog, it
must return a typed conflict for duplicates and atomically create into an empty
catalog. Consumers such as the CLI can then enforce their own one-study-per-file
convention without guessing an ID.
How a user reaches your backend¶
By handing it over directly: the study builder takes an Arc<dyn Storage>, and
yours goes exactly where a built-in would.
What will not pick it up is a spec string. :memory:, study.atj,
study.db/study.sqlite/study.sqlite3 and atune://host:port are matched by a closed
function in the facade. The CLI's --study flag goes through that function;
Python's create_study(storage=…) accepts the in-memory, journal and SQLite
forms, but the Python wheel does not enable the facade's remote feature and
therefore does not accept atune://host:port. So an out-of-tree backend is
usable from Rust and is not reachable from the CLI or from Python — worth
knowing before you plan a deployment around it.
Where to go next¶
| If you want to | Go to |
|---|---|
| Understand what storage is responsible for | Storage |
| See what the built-in backends buy and cost | Storage, and Resume and scale |
| Know what is written into a journal file | Journal format |
| Know which feature carries which backend | Feature reference |
| Ship it as a crate someone can depend on | Publish a plugin crate |
| Read the trait, method by method | Rust API |