Struct JournalStorage 

pub struct JournalStorage { /* private fields */ }

A resumable study shared between processes without a server.

Journal v6 stores a versioned header and bounded head at <path>. Operations retain their JSON Lines encoding in immutable, checksummed extents under <path>.d/. Each extent names its predecessor; the head names the committed tip and records total operation count and logical bytes. Readers apply only extents reachable from that head, walking backward to their previously applied tip and then replaying forward.

Each writer instance owns one exclusively created append-only data file, protected by the handle’s in-process mutex. A mutation writes one extent plus a bounded head, independent of history length. Failed or uncertain appends seal that file; later writes use a new file and never repair bytes. The renewable lease serializes writers, and its generation-bound ticket rename fences publication after takeover.

Writable v1-v5 journals migrate once under the lease. Stop all older handles before upgrading writers. Read-only legacy opens keep the original format. Snapshots and cursors retain virtual record-stream offsets and sequence semantics. Compaction and garbage collection are not implemented.

Files

Path	What
<path>	authoritative header and bounded head
<path>.d/	immutable extents in per-writer append-only files
<path>.lock	renewable append-lease directory
<path>.lock.append-<generation>	ephemeral generation-bound head ticket
<path>.snapshot	optional replayed-state cache

Deleting a study requires deleting its head, data directory, and sidecars after all handles have stopped. Unreferenced extents are ignored, including torn tails beyond the published tip. Missing, short, or corrupt committed extents are hard errors.

Replay trusts committed records

Opening a journal, and catching up on records other handles appended, folds each committed record into the study state through the same step a live write takes once its own checks have passed, without running those checks again. Replay refuses what it cannot fold consistently, such as a gap or repeat in the record sequence, an extent whose checksum disagrees, an unknown study or trial, an identity that does not match the one replay derives, or a state transition from a state the trial is not in or that the state machine forbids. A well-formed record that a live write would have rejected is otherwise applied as written, so a hand-edited or corrupt journal can carry operations no handle accepts: a parameter recorded under a distribution incompatible with the one already recorded for it, or with a value outside its own distribution, and a parameter batch, report, error message or heartbeat recorded against a finished or lifecycle-managed trial. Nothing re-validates such records on open, because that would cost every open.

Durability

Both settings sync extent data and the staged head before publication. A new file’s first referencing head also waits for data-directory and parent- directory syncs. Durability::Record requests another parent-directory sync after the rename; Durability::Os leaves that publication metadata durability to the operating system. Readers see either complete head. Snapshots retain atomic synced publication. These requests do not establish a power-loss guarantee on an arbitrary filesystem or NFS mount. journal_bytes_staged counts extent plus head bytes, while journal_canonical_size measures the bounded canonical file.

Concurrency

Threads of one process share one journal handle and are serialised by an in-process mutex; separate processes are serialised by the lock file. Reads take the in-process mutex but never the file lock: they catch up on the tail they have not read yet, which is how a worker learns what the other workers did. (Because a write holds the mutex across acquiring the file lock, an in-process read can wait behind a concurrent writer — up to that writer’s lock-acquisition time — but it never contends for the file lock itself.) A Cursor is therefore durable and portable — it is a study-local sequence number derived from the log, identical in every process that replays the same file, not a memory offset that resets when a worker restarts.

use atune_core::study::StudyConfig;
use atune_core::storage::{JournalStorage, Storage};
use atune_core::trial::TrialState;

// One directory per process, so concurrent doctest runs never share it.
let dir = std::env::temp_dir().join(format!("atune-journal-doc-basic-{}", std::process::id()));
std::fs::create_dir_all(&dir).expect("create the study directory");
let path = dir.join("study.jsonl");

let storage = JournalStorage::try_new(&path)?;
let study = storage.create_study(StudyConfig::new("demo").with_seed(7))?;
let trial = storage.create_trial(study, None)?;
storage.try_transition(trial.id, TrialState::Waiting, TrialState::Running)?;
storage.set_values(trial.id, &[0.25])?;
storage.try_transition(trial.id, TrialState::Running, TrialState::Complete)?;
drop(storage);

// A second process — here, a second handle — resumes by replaying.
let resumed = JournalStorage::try_new(&path)?;
assert_eq!(resumed.get_trial(trial.id)?.objective_value(), Some(&[0.25][..]));
assert_eq!(resumed.study_config(study)?.seed(), 7);

Implementations

impl JournalStorage

pub fn try_new(path: impl AsRef<Path>) -> Result<Self>

Opens (creating if necessary) the journal at path, with the defaults.

Errors

Error::Storage if the file cannot be created or read, if it is not a journal, or if it is corrupt in a way replay cannot explain (a gap in the record sequence, an identity that does not match the one replay derives). A torn legacy final record or unpublished v6 extent is ignored. A torn committed extent is corruption. Error::Incompatible is returned for a newer journal format. Error::ResourceLimit is returned when the finite byte, operation, snapshot, or record-entry limits are exceeded.

pub fn try_new_with_options( path: impl AsRef<Path>, options: JournalOptions, ) -> Result<Self>

Opens the journal at path with options.

Errors

As try_new.

pub fn try_new_read_only(path: impl AsRef<Path>) -> Result<Self>

Opens the journal at path for read-only queries.

This is the path a pure reader — atune best, atune trials, any command that only inspects a study — should take. Unlike try_new it neither creates the file nor takes the append lock, so it will not leave an empty journal behind on a mistyped path and it succeeds in a directory the process cannot write (where taking the .lock would fail with a permission error).

The returned handle serves every read of the Storage trait — study_config, sync, get_trial, get_state. A mutating call will fail because this handle is read-only; use try_new for anything that writes. This is an inherent method, not part of the frozen Storage trait.

Errors

Error::Storage if the file does not exist or cannot be read, if it is not a journal. Error::Incompatible is returned for a newer journal format. Error::ResourceLimit is returned when the finite byte, operation, snapshot, or record-entry limits are exceeded. A missing file is reported rather than created — the caller distinguishes “no study yet” by checking existence first if it wants a friendlier message.

pub fn try_new_read_only_with_options( path: impl AsRef<Path>, options: JournalOptions, ) -> Result<Self>

Opens the journal read-only with explicit finite admission options.

Unlike try_new_with_options, this never creates the journal or takes its append lock. The options still govern every bounded read and snapshot admission.

Errors

Returns Error::Storage when the path or its journal image cannot be opened, read, or validated under the supplied options, Error::Incompatible for a newer journal format, or Error::ResourceLimit when a finite option cap is exceeded.

pub fn path(&self) -> &Path

The journal’s path.

pub fn lock_path(&self) -> &Path

The append lock’s path, beside the resolved target for a journal symlink.

pub fn snapshot_path(&self) -> &Path

The snapshot side-file’s path, beside the resolved target for a journal symlink.

pub const fn worker(&self) -> WorkerId

This handle’s worker identity.

pub fn ops(&self) -> Result<u64>

How many op records this handle has replayed.

Catches up first, so it counts what is in the file, including what other processes have written.

Errors

Error::Storage if the tail cannot be read.

pub fn snapshot(&self) -> Result<()>

Writes a snapshot now, whatever the configured cadence.

Useful before a long pause, and the way to find out why automatic snapshots are failing: the periodic ones are best-effort and silent (a mutation that succeeded must not be reported as failed because an optimization could not be written), this one reports.

Errors

Error::Storage if the tail cannot be read or the snapshot cannot be written, synced or published. Error::ResourceLimit is returned when the finite snapshot byte or collection-entry cap is exceeded.

pub fn fail_over_stale( &self, study: StudyId, now_millis: u64, ) -> Result<Vec<FailedOver>>

Fails over every trial of study whose heartbeat has gone quiet.

A Running trial whose last heartbeat is older than the grace period (JournalOptions::stale_after_millis, by default twice the heartbeat interval) belongs to a worker that is not coming back: the node was preempted, the job hit its wall clock, somebody kill -9ed it. Any worker may take it over — that is what “workers coordinate only through storage” means when a worker dies.

The trial is failed with an explanatory message, and — if JournalOptions::requeue_on_failover is on — its parameters are re-enqueued as a fresh trial, so the configuration is still evaluated. A trial that never heartbeat at all is judged on its creation time, because “started long ago and never said anything” is the same evidence as “said something long ago”.

The shared core-side mechanism is kept out of any one backend: its implementation uses the public TrialLifecycleStorage capability to observe and conditionally fail the exact owner epoch. This handle keeps the unchanged-observation proof across calls; now_millis therefore must come from one monotonic sweeper clock. Owner heartbeat wall-clock values are not compared with it.

Races

Two workers may decide to fail the same trial over at the same moment. Exactly one wins the Running -> Failed transition; the loser reports nothing for that trial, so a caller can treat the returned list as “trials I failed over”.

Errors

Error::NotFound if the study does not exist; Error::Storage on a backend fault.

Trait Implementations

impl AtomicImportStorage for JournalStorage

Applies one import in a shadow journal while the source generation lease is held, then copies its suffix into one extent and publishes the new head once.

fn import_batch(&self, batch: &ImportBatch) -> Result<ImportBatchResult>

Publishes one completely staged batch at one backend commit point.

impl Debug for JournalStorage

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Storage for JournalStorage

fn sync_limited( &self, study: StudyId, cursor: Cursor, max_trials: usize, ) -> Result<SyncPage>

Stops reading after the page and one look-ahead entry instead of materializing the whole post-cursor feed and truncating it, so paging a dense study costs each page, not the remainder.

fn backing_paths(&self) -> Option<Vec<PathBuf>>

Local files that an exporter must never replace, including fixed sidecars.

fn create_study(&self, cfg: StudyConfig) -> Result<StudyId>

Creates a study and returns its identity.

fn study_config(&self, study: StudyId) -> Result<StudyConfig>

Reads a study’s configuration back.

fn study_user_attrs(&self, study: StudyId) -> Result<UserAttrs>

Returns all user-owned JSON metadata attached to a study.

fn set_study_user_attr( &self, study: StudyId, key: &str, value: Value, ) -> Result<()>

Inserts or replaces one study user attribute.

fn create_trial( &self, study: StudyId, template: Option<TrialTemplate>, ) -> Result<TrialRecord>

Creates a trial, optionally pre-loaded from a template.

fn try_transition( &self, trial: TrialId, from: TrialState, to: TrialState, ) -> Result<bool>

Atomically moves a trial from from to to.

fn write_params( &self, trial: TrialId, batch: &[(String, Distribution, ParamValue)], ) -> Result<()>

Records sampled parameters together with the distributions they came from.

fn report(&self, trial: TrialId, step: u64, values: &[f64]) -> Result<()>

Appends an intermediate report.

fn set_values(&self, trial: TrialId, values: &[f64]) -> Result<()>

Records a trial’s final objective values.

fn set_error(&self, trial: TrialId, message: &str) -> Result<()>

Records why a trial failed.

fn heartbeat(&self, trial: TrialId, now_millis: u64) -> Result<()>

Records that the trial’s owner is still alive at now_millis.

fn set_trial_user_attr( &self, trial: TrialId, key: &str, value: Value, ) -> Result<()>

Inserts or replaces one user-owned JSON attribute on an unfinished trial and marks that trial changed for sync.

fn put_state(&self, scope: Scope, blob: StateBlob) -> Result<()>

Writes the state blob of scope, replacing any previous one.

fn trial_lifecycle(&self) -> Option<&dyn TrialLifecycleStorage>

Returns the backend’s atomic trial-lifecycle capability, if available.

fn atomic_import(&self) -> Option<&dyn AtomicImportStorage>

Returns the backend’s whole-import atomic-publication capability, if available.

fn study_catalog(&self) -> Option<&dyn StudyCatalog>

Returns the backend’s study-catalog capability, if available.

fn get_state(&self, scope: Scope) -> Result<Option<StateBlob>>

Reads the state blob of scope, if one was ever written.

fn read_snapshot(&self, request: &SnapshotRequest) -> Result<StorageSnapshot>

Reads trial deltas, requested side state, and optional finalization records for one observation.

fn sync( &self, study: StudyId, cursor: Cursor, ) -> Result<(Vec<TrialDelta>, Cursor)>

Returns everything that happened in study after cursor.

fn get_trial(&self, trial: TrialId) -> Result<FrozenTrial>

Reads one trial’s snapshot.

impl StudyCatalog for JournalStorage

fn create_if_empty(&self, cfg: StudyConfig) -> Result<StudyCreateOutcome>

Atomically creates cfg only when the catalog has no studies.

fn list_studies(&self) -> Result<Vec<StudySummary>>

Lists study identities in deterministic ascending StudyId order.

fn find_by_name(&self, name: &str) -> Result<StudyLoadOutcome>

Resolves an exact name without selecting arbitrarily among duplicates.

fn list(&self) -> Result<Vec<StudySummary>>

Short alias for list_studies.

impl TrialLifecycleStorage for JournalStorage

fn observe_finalization( &self, study: StudyId, ) -> Result<FinalizationObservation>

Reads the study-wide finalization ownership observation.

fn acquire_finalization( &self, request: FinalizationAcquireRequest, ) -> Result<FinalizationToken>

Atomically acquires or reclaims the study-wide finalization claim.

fn renew_finalization( &self, token: FinalizationToken, ) -> Result<FinalizationObservation>

Atomically renews a held study-wide finalization claim.

fn release_finalization(&self, token: FinalizationToken) -> Result<()>

Atomically releases a held study-wide finalization claim.

fn reserve_and_create_if_budget( &self, request: ReserveAndCreateRequest, ) -> Result<ReservationOutcome>

Reserves and creates a waiting trial if the durable-record budget allows it. Created consumes a trial number; Exhausted consumes nothing.

fn abandon(&self, request: AbandonRequest) -> Result<bool>

Conditionally abandons an as-yet-unclaimed reservation.

fn observe(&self, trial: TrialId) -> Result<LeaseObservation>

Reads the complete current lease observation for a trial.

fn claim(&self, request: ClaimRequest) -> Result<TrialOwnerToken>

Atomically claims a reservation or paused trial, writes its initial parameters/state, and starts a new owner epoch.

fn renew(&self, request: RenewRequest) -> Result<LeaseObservation>

Renews a running owner and returns the new observation.

fn fail_stale(&self, request: FailStaleRequest) -> Result<bool>

Conditionally fails a running trial whose complete observed lease the caller proved unchanged for the request’s grace interval. Backends must compare the complete observation atomically, but must not compare the sweeper timestamp to the owner’s heartbeat timestamp: those clocks may belong to different nodes. Returns true for the sole race winner. The winner atomically commits the failed outcome and an Initial outbox record keyed by the observed trial and epoch, including the request’s finalization intent. A race loss commits neither.

fn fenced_mutate(&self, request: FencedMutationRequest) -> Result<()>

Applies one mutation only if the owner/epoch token is still exact.

fn mutate_owned( &self, owner: FinalizationToken, request: FencedMutationRequest, ) -> Result<()>

Applies one lifecycle mutation under a study-wide finalization owner.

fn terminal(&self, request: TerminalRequest) -> Result<()>

Validates and commits a terminal outcome only if the owner/epoch token is still exact.

fn terminal_with_intent( &self, request: TerminalRequest, intent: FinalizationIntent, ) -> Result<OutboxKey>

Commits a terminal outcome together with its durable initial outbox intent.

fn terminal_with_intent_owned( &self, owner: FinalizationToken, request: TerminalRequest, intent: FinalizationIntent, ) -> Result<OutboxKey>

Commits a terminal result and initial outbox intent under a study-wide finalization owner.

fn advance_outbox(&self, request: OutboxAdvanceRequest) -> Result<OutboxPhase>

Atomically persists callback state and discovered intents while moving one outbox record through its explicit phases.

fn advance_outbox_owned( &self, owner: FinalizationToken, request: OutboxAdvanceRequest, _operation: OperationId, ) -> Result<OutboxPhase>

Advances one outbox phase under a study-wide finalization owner.

fn get_outbox(&self, key: OutboxKey) -> Result<Option<OutboxRecord>>

Reads one durable outbox record, including its replay cursor and completed effect keys.

fn list_outbox(&self) -> Result<Vec<OutboxRecord>>

Lists durable outbox records in key order, including fully-done records.

fn list_outbox_page(&self, request: OutboxPageRequest) -> Result<OutboxPage>

Reads one bounded page of study-scoped durable outbox records.

fn complete_outbox_effect(&self, request: OutboxEffectCompletion) -> Result<()>

Idempotently acknowledges an externally executed effect.

fn create_replacement( &self, effect: OutboxEffectKey, ) -> Result<ReplacementOutcome>

Idempotently creates the replacement described by a durable effect.

fn put_finalized_state(&self, request: FinalizedStateRequest) -> Result<()>

Replaces one state blob on a terminal lifecycle-managed trial.

fn advance_outbox_with_operation( &self, request: OutboxAdvanceRequest, _operation: OperationId, ) -> Result<OutboxPhase>

Advances one outbox phase with the caller’s lifecycle operation correlation.

fn complete_outbox_effect_with_operation( &self, request: OutboxEffectCompletion, _operation: OperationId, ) -> Result<()>

Acknowledges one externally executed effect with caller correlation.

fn create_replacement_with_operation( &self, effect: OutboxEffectKey, _operation: OperationId, ) -> Result<ReplacementOutcome>

Creates one replacement with caller correlation.

fn put_finalized_state_with_operation( &self, request: FinalizedStateRequest, _operation: OperationId, ) -> Result<()>

Replaces finalized state with caller correlation.