Journal format¶
The on-disk format of a .atj study file, record by record.
A journal is the log of the mutations that produced a study, so resuming is a
replay. Since format v6 it is a small file at the study path, holding the header
and a bounded head, plus a <journal>.d/ directory of append-only data files,
each created and written by one writer, whose operations are JSON Lines grouped
into immutable extents. A writer appends one extent to its own data file and
then publishes it by
replacing the head through a generation-bound ticket rename under a renewable
lease; nothing already published is rewritten, and bytes past the last
published extent, such as a torn append, are never read (see
Storage). The operations stay readable with the tools
you already have — tail -f on a data file follows a writer, and
grep '"kind":"set-values"' across <journal>.d/ lists finished trials — but a
data file can hold records no head has published yet, so only a reader that
walks the head's back-pointers sees exactly the study.
You do not need this page to use atune. It is here for the cases where the file itself is the interface: reading a study from another language, auditing what a worker wrote, or writing a tool over the log. Nothing on the page is a promise about the values — the format version at the top is the compatibility contract, and a reader must refuse a version higher than it knows.
The record types and their field names are generated from
crates/atune_core/src/storage/journal/record.rs, including the serde attributes
that decide the wire spelling, and the worked example at the end is the byte golden
a test compares the writer against — quoted, not retyped.
Generated file. Do not edit between the pragmas below — run
cargo dev generate-all --mode write instead. Everything outside them is
hand-written and is never touched.
The file¶
Journal v6 publishes a header and bounded head at the canonical path. Operations retain JSON Lines encoding inside immutable extents in <journal>.d/. Each writer exclusively creates its own append-only data file. The head names the tip extent and totals; each extent names its predecessor. Readers walk only published back-pointers and apply forward. A raw data-file grep can include unpublished records.
This build writes format version 6 (FORMAT_VERSION in crates/atune_core/src/storage/journal/record.rs). A journal whose header names a higher version is refused rather than guessed at; a lower one still opens read-only. Writable v1-v5 opens migrate once under the lease; stop all older handles before upgrading writers. Logical offsets remain positions in the virtual header-and-operation stream. Failed or uncertain appends seal the data file; missing or short committed extents are corruption. Each append writes one extent plus a bounded head, committed through the generation-bound ticket rename. New-file data and both directories are synced before the first referencing head. No compaction or garbage collection is implemented.
The header line¶
"record": "header", then:
format— The on-disk format version —FORMAT_VERSIONwhen this build wrote it.atune— Theatune_coreversion that created the journal, for forensics.created_at_millis— When the journal was created, from the creator'sClock.created_by— Which worker created it.
An operation line¶
"record": "op", then:
seq— The record's position in the file: 1 for the first op.worker— The worker that appended it.at_millis— The writer's clock reading when it did.op— What happened.
An extent pointer¶
file— The safe basename of one writer's data file under<journal>.d/.offset— The physical byte offset of the immutable extent in that file.len— The complete physical extent length, metadata newline plus payload.
The bounded head line¶
tip— The committed tip pointer, or null for a header-only history.ops— The total committed operation sequence count.bytes— The total logical bytes in the virtual header-and-operation stream.
An immutable extent header¶
magic— The literal format markeratune.extent/6.previous— The previous committed extent, or null at the beginning of history.first— The first operation sequence in this payload, inclusive.last— The last operation sequence in this payload, inclusive.start— The payload's starting logical byte offset in the virtual stream.payload_len— The exact encoded JSONL payload length following this metadata line.checksum— The unsigned FNV-1a 64-bit checksum of the payload bytes.
Head and extent metadata lines are capped at 2,048 bytes including their newline. Extent payloads retain the configured journal and per-record admission caps. FNV-1a detects accidental corruption and torn writes; it is not authentication.
The operations¶
There is exactly one variant per mutating Storage method, and the kebab-case kind is that method's name. A state-machine audit must follow the published head; raw data files can include unpublished operations. 22 of them:
"kind": "create-study"¶
A study was created.
Op::CreateStudy. Fields:
study— The identity assigned by the writer (derived from replay position).config— The stored configuration.
"kind": "create-trial"¶
A trial was created.
Op::CreateTrial. Fields:
study— The owning study.trial— The identity assigned by the writer.number— The per-study ordinal assigned by the writer.template— The template it was created from, if any.
"kind": "transition"¶
A trial changed state. Only ever written by the caller that won the transition; replay still refuses one whose from is not the trial's current state, whose move the state machine forbids, or that names a lifecycle-managed trial.
Op::Transition. Fields:
trial— The trial.from— The state it was in.to— The state it is in now.
"kind": "params-batch"¶
Sampled parameters were recorded, all-or-nothing.
Op::ParamsBatch. Fields:
trial— The trial.params— The whole batch, in the order it was written.
"kind": "report"¶
An intermediate value was reported.
Op::Report. Fields:
trial— The trial.step— The step, in the study's resource unit.values— The reported values.
"kind": "set-values"¶
Final objective values were recorded.
Op::SetValues. Fields:
trial— The trial.values— The values, one per direction.
"kind": "set-error"¶
A failure message was recorded.
Op::SetError. Fields:
trial— The trial.message— The message, verbatim.
"kind": "heartbeat"¶
A heartbeat was recorded.
Op::Heartbeat. Fields:
trial— The trial.at_millis— The caller's timestamp, never the writer's clock.
"kind": "set-study-user-attr"¶
A study user attribute was inserted or replaced.
Op::SetStudyUserAttr. Fields:
study— The owning study.key— The user key; theatune:namespace is reserved.value— The portable JSON value.
"kind": "set-trial-user-attr"¶
A raw trial user attribute was inserted or replaced.
Op::SetTrialUserAttr. Fields:
trial— The owning trial.key— The user key; theatune:namespace is reserved.value— The portable JSON value.
"kind": "put-state"¶
A state blob was written.
Op::PutState. Fields:
scope— What it belongs to.blob— The blob.
"kind": "lifecycle-reserve"¶
A lifecycle-managed trial was reserved and created under the budget.
Op::LifecycleReserve. Fields:
study— The owning study.trial— The identity assigned by the writer.number— The per-study ordinal assigned by the writer.template— The template it was created from, if any.managed— Explicit schema marker distinguishing lifecycle records from rawcreate-trialrecords.
"kind": "lifecycle-abandon"¶
A reserved lifecycle trial was conditionally abandoned.
Op::LifecycleAbandon. Fields:
trial— The reserved trial.reason— The durable failure reason.
"kind": "lifecycle-claim"¶
A reservation or paused trial was claimed and started at a new epoch.
Op::LifecycleClaim. Fields:
trial— The trial being claimed.from— The state required by the claim target.worker— The new owner.heartbeat_millis— The first heartbeat supplied by the owner.epoch— The new ownership epoch.heartbeat_seq— The first heartbeat sequence in the new epoch.params— Parameters written atomically with the claim.state— Optional trial-owned state written atomically with the claim.
"kind": "lifecycle-renew"¶
A lifecycle owner renewed its heartbeat.
Op::LifecycleRenew. Fields:
trial— The owned trial.worker— The owner fence.epoch— The ownership epoch.heartbeat_millis— The caller's new heartbeat timestamp.heartbeat_seq— The resulting heartbeat sequence.
"kind": "lifecycle-fail-stale"¶
A stale trial was conditionally failed from a complete observation.
Op::LifecycleFailStale. Fields:
trial— The observed trial.observed_state— The complete observed state.owner— The complete observed owner marker.epoch— The complete observed epoch.heartbeat_millis— The complete observed heartbeat timestamp.heartbeat_seq— The complete observed heartbeat sequence.now_millis— The timestamp used for the stale comparison.grace_millis— The stale grace threshold.reason— The durable failure reason.intent— The initial outbox intent, absent on pre-v4 records.
"kind": "lifecycle-fenced"¶
One ownership-fenced lifecycle mutation.
Op::LifecycleFenced. Fields:
trial— The owned trial.worker— The owner fence.epoch— The ownership epoch.mutation— The mutation payload.
"kind": "lifecycle-terminal"¶
One ownership-fenced terminal transition.
Op::LifecycleTerminal. Fields:
trial— The owned trial.worker— The owner fence.epoch— The ownership epoch.outcome— The terminal payload, including values/error and state.intent— The initial outbox intent, absent on pre-v4 records.
"kind": "outbox-advance"¶
Atomically persists callback state/effects and advances one outbox.
Op::OutboxAdvance. Fields:
key— The durable terminal-finalization key.phase— The phase being committed.sampler_state— Replacement sampler seam state, if supplied.scheduler_state— Replacement scheduler seam state, if supplied.effects— Effects discovered at this callback boundary.
"kind": "outbox-effect-complete"¶
Durably acknowledges one externally executed outbox effect.
Op::OutboxEffectComplete. Fields:
key— The effect being acknowledged.
"kind": "outbox-replacement"¶
Atomically creates a replacement and marks its effect complete.
Op::OutboxReplacement. Fields:
effect— The idempotency key of the replacement effect.study— The study receiving the replacement.trial— The identity assigned by the writer.number— The per-study ordinal assigned by the writer.template— The replacement's fixed template, if any.attempt— The retry attempt metadata.retry_of— The failed source trial when this is a retry.
"kind": "lifecycle-finalized-state"¶
Replaces a state blob on a terminal lifecycle trial.
Op::LifecycleFinalizedState. Fields:
trial— The terminal trial receiving the state.scope— The exact trial-owned scope.blob— The replacement blob.
Why the floats are special¶
A trial may report NaN or ±inf — a diverging run is a real observation
(Storage::report stores values verbatim, non-finite ones included). JSON
has no such literals, and serde_json silently writes null for them,
which then fails to deserialize back into an f64. Every float this module
writes therefore goes through JsonF64, which encodes the three
non-finite values as the strings "NaN", "Infinity" and "-Infinity"
and everything else as a plain JSON number.
A whole study, byte for byte¶
crates/atune_core/tests/goldens/journal_full_study_v6.jsonl — the journal of a 5-trial study, committed and compared byte for byte by crates/atune_core/tests/journal_goldens.rs, which also replays it back into the study that produced it. Regenerate it deliberately with ATUNE_BLESS_JOURNAL_GOLDEN=1 cargo test -p atune_core --test journal_goldens; the only normalised values are the writer's atune version and fixed-width writer filename. Extent bytes, checksums, offsets, lengths, and back-pointers remain pinned.
{"record":"header","format":6,"atune":"0.0.0","created_at_millis":1000000,"created_by":0}
{"tip":{"file":"data-0000000000000000-00000000-0000000000000000","offset":9160,"len":331},"ops":27,"bytes":4166}
{"magic":"atune.extent/6","previous":null,"first":1,"last":1,"start":90,"payload_len":239,"checksum":11805423599770331537}
{"record":"op","seq":1,"worker":0,"at_millis":1000000,"op":{"kind":"create-study","study":0,"config":{"name":"golden","directions":["Minimize","Maximize"],"seed":167,"space":null,"resource_unit":"Steps","metric_names":["loss","reward"]}}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":0,"len":362},"first":2,"last":2,"start":329,"payload_len":132,"checksum":6273483532505398269}
{"record":"op","seq":2,"worker":0,"at_millis":1000000,"op":{"kind":"set-study-user-attr","study":0,"key":"team","value":"control"}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":362,"len":330},"first":3,"last":3,"start":461,"payload_len":115,"checksum":17949059053767199002}
{"record":"op","seq":3,"worker":0,"at_millis":1000000,"op":{"kind":"create-trial","study":0,"trial":0,"number":0}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":692,"len":316},"first":4,"last":4,"start":576,"payload_len":124,"checksum":8124123066316402708}
{"record":"op","seq":4,"worker":0,"at_millis":1000000,"op":{"kind":"transition","trial":0,"from":"Waiting","to":"Running"}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":1008,"len":324},"first":5,"last":5,"start":700,"payload_len":552,"checksum":12161201457292993943}
{"record":"op","seq":5,"worker":0,"at_millis":1000000,"op":{"kind":"params-batch","trial":0,"params":[{"name":"lr","dist":{"Float":{"low":0.00001,"high":0.1,"log":true,"step":null}},"value":{"F64":0.01}},{"name":"opt","dist":{"Cat":{"choices":{"version":1,"choices":[{"label":"adam","value":{"type":"string","value":"adam"}},{"label":"sgd","value":{"type":"string","value":"sgd"}}]}}},"value":{"Cat":0}},{"name":"layers","dist":{"Int":{"low":1,"high":8,"log":false,"step":1}},"value":{"I64":4}},{"name":"use_bn","dist":"Bool","value":{"Bool":true}}]}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":1332,"len":754},"first":6,"last":6,"start":1252,"payload_len":152,"checksum":14057393430921405142}
{"record":"op","seq":6,"worker":0,"at_millis":1000000,"op":{"kind":"set-trial-user-attr","trial":0,"key":"dataset","value":{"fold":2,"split":"train"}}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":2086,"len":355},"first":7,"last":7,"start":1404,"payload_len":111,"checksum":12025456614441431174}
{"record":"op","seq":7,"worker":0,"at_millis":1000050,"op":{"kind":"heartbeat","trial":0,"at_millis":1000050}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":2441,"len":314},"first":8,"last":8,"start":1515,"payload_len":116,"checksum":1938700201154457292}
{"record":"op","seq":8,"worker":0,"at_millis":1000050,"op":{"kind":"report","trial":0,"step":1,"values":[0.9,0.1]}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":2755,"len":318},"first":9,"last":9,"start":1631,"payload_len":116,"checksum":18421679176321400045}
{"record":"op","seq":9,"worker":0,"at_millis":1000050,"op":{"kind":"report","trial":0,"step":2,"values":[0.5,0.4]}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":3073,"len":319},"first":10,"last":10,"start":1747,"payload_len":114,"checksum":11561302113925031518}
{"record":"op","seq":10,"worker":0,"at_millis":1000050,"op":{"kind":"set-values","trial":0,"values":[0.42,0.58]}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":3392,"len":319},"first":11,"last":11,"start":1861,"payload_len":126,"checksum":3587209604265566495}
{"record":"op","seq":11,"worker":0,"at_millis":1000050,"op":{"kind":"transition","trial":0,"from":"Running","to":"Complete"}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":3711,"len":330},"first":12,"last":12,"start":1987,"payload_len":116,"checksum":2163934545002085884}
{"record":"op","seq":12,"worker":0,"at_millis":1001050,"op":{"kind":"create-trial","study":0,"trial":1,"number":1}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":4041,"len":320},"first":13,"last":13,"start":2103,"payload_len":125,"checksum":9701496543177914555}
{"record":"op","seq":13,"worker":0,"at_millis":1001050,"op":{"kind":"transition","trial":1,"from":"Waiting","to":"Running"}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":4361,"len":329},"first":14,"last":14,"start":2228,"payload_len":208,"checksum":10180553164403044043}
{"record":"op","seq":14,"worker":0,"at_millis":1001050,"op":{"kind":"params-batch","trial":1,"params":[{"name":"lr","dist":{"Float":{"low":0.00001,"high":0.1,"log":true,"step":null}},"value":{"F64":0.05}}]}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":4690,"len":413},"first":15,"last":15,"start":2436,"payload_len":118,"checksum":1606924981939238496}
{"record":"op","seq":15,"worker":0,"at_millis":1001050,"op":{"kind":"report","trial":1,"step":1,"values":[1.2,0.05]}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":5103,"len":322},"first":16,"last":16,"start":2554,"payload_len":124,"checksum":1264567823944544473}
{"record":"op","seq":16,"worker":0,"at_millis":1001050,"op":{"kind":"transition","trial":1,"from":"Running","to":"Pruned"}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":5425,"len":328},"first":17,"last":17,"start":2678,"payload_len":116,"checksum":12436364596112988900}
{"record":"op","seq":17,"worker":0,"at_millis":1002050,"op":{"kind":"create-trial","study":0,"trial":2,"number":2}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":5753,"len":321},"first":18,"last":18,"start":2794,"payload_len":125,"checksum":9187280388936404744}
{"record":"op","seq":18,"worker":0,"at_millis":1002050,"op":{"kind":"transition","trial":2,"from":"Waiting","to":"Running"}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":6074,"len":329},"first":19,"last":19,"start":2919,"payload_len":143,"checksum":5644263864551661327}
{"record":"op","seq":19,"worker":0,"at_millis":1002050,"op":{"kind":"set-error","trial":2,"message":"objective panicked: NaN loss at step 3"}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":6403,"len":347},"first":20,"last":20,"start":3062,"payload_len":124,"checksum":8386915286371558673}
{"record":"op","seq":20,"worker":0,"at_millis":1002050,"op":{"kind":"transition","trial":2,"from":"Running","to":"Failed"}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":6750,"len":328},"first":21,"last":21,"start":3186,"payload_len":188,"checksum":7314121843354688030}
{"record":"op","seq":21,"worker":0,"at_millis":1003050,"op":{"kind":"create-trial","study":0,"trial":3,"number":3,"template":{"fixed":{"lr":{"F64":0.001},"opt":{"Cat":1}},"parent":null}}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":7078,"len":392},"first":22,"last":22,"start":3374,"payload_len":177,"checksum":17075817932011471418}
{"record":"op","seq":22,"worker":0,"at_millis":1003050,"op":{"kind":"put-state","scope":{"Sampler":0},"blob":{"kind":"tpe","version":1,"data":{"gamma":0.25,"observations":3}}}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":7470,"len":382},"first":23,"last":23,"start":3551,"payload_len":116,"checksum":17695267736029406871}
{"record":"op","seq":23,"worker":0,"at_millis":1004050,"op":{"kind":"create-trial","study":0,"trial":4,"number":4}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":7852,"len":321},"first":24,"last":24,"start":3667,"payload_len":125,"checksum":10493196373587185347}
{"record":"op","seq":24,"worker":0,"at_millis":1004050,"op":{"kind":"transition","trial":4,"from":"Waiting","to":"Running"}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":8173,"len":330},"first":25,"last":25,"start":3792,"payload_len":126,"checksum":11923258870654222687}
{"record":"op","seq":25,"worker":0,"at_millis":1004050,"op":{"kind":"report","trial":4,"step":0,"values":["NaN","Infinity"]}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":8503,"len":331},"first":26,"last":26,"start":3918,"payload_len":122,"checksum":6756855385206689664}
{"record":"op","seq":26,"worker":0,"at_millis":1004050,"op":{"kind":"set-values","trial":4,"values":["NaN","-Infinity"]}}
{"magic":"atune.extent/6","previous":{"file":"data-0000000000000000-00000000-0000000000000000","offset":8834,"len":326},"first":27,"last":27,"start":4040,"payload_len":126,"checksum":17168217925120672172}
{"record":"op","seq":27,"worker":0,"at_millis":1004050,"op":{"kind":"transition","trial":4,"from":"Running","to":"Complete"}}