Journal format

The on-disk format of a .atj study file, record by record.

A journal is a plain append-only text file, one JSON object per line, and that is the whole design: a study is the log of the mutations that produced it, so resuming is a replay and there is nothing to migrate, lock or repair. It also means the file is readable with the tools you already have — wc -l counts the records, tail -f follows a run, grep '"kind":"set-values"' lists the finished trials.

You do not need this page to use atune. It is here for the cases where the file itself is the interface: reading a study from another language, auditing what a worker wrote, or writing a tool over the log. Nothing on the page is a promise about the values — the format version at the top is the compatibility contract, and a reader must refuse a version higher than it knows.

The record types and their field names are generated from crates/atune_core/src/storage/journal/record.rs, including the serde attributes that decide the wire spelling, and the worked example at the end is the byte golden a test compares the writer against — quoted, not retyped.

Generated file. Do not edit between the pragmas below — run cargo dev generate-all --mode write instead. Everything outside them is hand-written and is never touched.

The file

A journal is JSON Lines: one JSON object per line, appended and never rewritten. The first line is a header; every line after it is one operation. Both carry a record field naming which they are, so head -1 identifies a file and grep finds a record kind in it without a parser.

This build writes format version 5 (FORMAT_VERSION in crates/atune_core/src/storage/journal/record.rs). A journal whose header names a higher version is refused rather than guessed at; a lower one still opens.

The header line

"record": "header", then:

• format — The on-disk format version — FORMAT_VERSION when this build wrote it.
• atune — The atune_core version that created the journal, for forensics.
• created_at_millis — When the journal was created, from the creator's Clock.
• created_by — Which worker created it.

An operation line

"record": "op", then:

• seq — The record's position in the file: 1 for the first op.
• worker — The worker that appended it.
• at_millis — The writer's clock reading when it did.
• op — What happened.

The operations

There is exactly one variant per mutating Storage method, and the kebab-case kind is that method's name — so grep '"kind":"transition"' is a complete audit of a study's state machine. 22 of them:

"kind": "create-study"

A study was created.

Op::CreateStudy. Fields:

• study — The identity assigned by the writer (derived from replay position).
• config — The stored configuration.

"kind": "create-trial"

A trial was created.

Op::CreateTrial. Fields:

• study — The owning study.
• trial — The identity assigned by the writer.
• number — The per-study ordinal assigned by the writer.
• template — The template it was created from, if any.

"kind": "transition"

A trial changed state. Only ever written by the caller that won the transition, so replay applies it unconditionally.

Op::Transition. Fields:

• trial — The trial.
• from — The state it was in.
• to — The state it is in now.

"kind": "params-batch"

Sampled parameters were recorded, all-or-nothing.

Op::ParamsBatch. Fields:

• trial — The trial.
• params — The whole batch, in the order it was written.

"kind": "report"

An intermediate value was reported.

Op::Report. Fields:

• trial — The trial.
• step — The step, in the study's resource unit.
• values — The reported values.

"kind": "set-values"

Final objective values were recorded.

Op::SetValues. Fields:

• trial — The trial.
• values — The values, one per direction.

"kind": "set-error"

A failure message was recorded.

Op::SetError. Fields:

• trial — The trial.
• message — The message, verbatim.

"kind": "heartbeat"

A heartbeat was recorded.

Op::Heartbeat. Fields:

• trial — The trial.
• at_millis — The caller's timestamp, never the writer's clock.

"kind": "set-study-user-attr"

A study user attribute was inserted or replaced.

Op::SetStudyUserAttr. Fields:

• study — The owning study.
• key — The user key; the atune: namespace is reserved.
• value — The portable JSON value.

"kind": "set-trial-user-attr"

A raw trial user attribute was inserted or replaced.

Op::SetTrialUserAttr. Fields:

• trial — The owning trial.
• key — The user key; the atune: namespace is reserved.
• value — The portable JSON value.

"kind": "put-state"

A state blob was written.

Op::PutState. Fields:

• scope — What it belongs to.
• blob — The blob.

"kind": "lifecycle-reserve"

A lifecycle-managed trial was reserved and created under the budget.

Op::LifecycleReserve. Fields:

• study — The owning study.
• trial — The identity assigned by the writer.
• number — The per-study ordinal assigned by the writer.
• template — The template it was created from, if any.
• managed — Explicit schema marker distinguishing lifecycle records from raw create-trial records.

"kind": "lifecycle-abandon"

A reserved lifecycle trial was conditionally abandoned.

Op::LifecycleAbandon. Fields:

• trial — The reserved trial.
• reason — The durable failure reason.

"kind": "lifecycle-claim"

A reservation or paused trial was claimed and started at a new epoch.

Op::LifecycleClaim. Fields:

• trial — The trial being claimed.
• from — The state required by the claim target.
• worker — The new owner.
• heartbeat_millis — The first heartbeat supplied by the owner.
• epoch — The new ownership epoch.
• heartbeat_seq — The first heartbeat sequence in the new epoch.
• params — Parameters written atomically with the claim.
• state — Optional trial-owned state written atomically with the claim.

"kind": "lifecycle-renew"

A lifecycle owner renewed its heartbeat.

Op::LifecycleRenew. Fields:

• trial — The owned trial.
• worker — The owner fence.
• epoch — The ownership epoch.
• heartbeat_millis — The caller's new heartbeat timestamp.
• heartbeat_seq — The resulting heartbeat sequence.

"kind": "lifecycle-fail-stale"

A stale trial was conditionally failed from a complete observation.

Op::LifecycleFailStale. Fields:

• trial — The observed trial.
• observed_state — The complete observed state.
• owner — The complete observed owner marker.
• epoch — The complete observed epoch.
• heartbeat_millis — The complete observed heartbeat timestamp.
• heartbeat_seq — The complete observed heartbeat sequence.
• now_millis — The timestamp used for the stale comparison.
• grace_millis — The stale grace threshold.
• reason — The durable failure reason.
• intent — The initial outbox intent, absent on pre-v4 records.

"kind": "lifecycle-fenced"

One ownership-fenced lifecycle mutation.

Op::LifecycleFenced. Fields:

• trial — The owned trial.
• worker — The owner fence.
• epoch — The ownership epoch.
• mutation — The mutation payload.

"kind": "lifecycle-terminal"

One ownership-fenced terminal transition.

Op::LifecycleTerminal. Fields:

• trial — The owned trial.
• worker — The owner fence.
• epoch — The ownership epoch.
• outcome — The terminal payload, including values/error and state.
• intent — The initial outbox intent, absent on pre-v4 records.

"kind": "outbox-advance"

Atomically persists callback state/effects and advances one outbox.

Op::OutboxAdvance. Fields:

• key — The durable terminal-finalization key.
• phase — The phase being committed.
• sampler_state — Replacement sampler seam state, if supplied.
• scheduler_state — Replacement scheduler seam state, if supplied.
• effects — Effects discovered at this callback boundary.

"kind": "outbox-effect-complete"

Durably acknowledges one externally executed outbox effect.

Op::OutboxEffectComplete. Fields:

• key — The effect being acknowledged.

"kind": "outbox-replacement"

Atomically creates a replacement and marks its effect complete.

Op::OutboxReplacement. Fields:

• effect — The idempotency key of the replacement effect.
• study — The study receiving the replacement.
• trial — The identity assigned by the writer.
• number — The per-study ordinal assigned by the writer.
• template — The replacement's fixed template, if any.
• attempt — The retry attempt metadata.
• retry_of — The failed source trial when this is a retry.

"kind": "lifecycle-finalized-state"

Replaces a state blob on a terminal lifecycle trial.

Op::LifecycleFinalizedState. Fields:

• trial — The terminal trial receiving the state.
• scope — The exact trial-owned scope.
• blob — The replacement blob.

Why the floats are special

A trial may report NaN or ±inf — a diverging run is a real observation (Storage::report stores values verbatim, non-finite ones included). JSON has no such literals, and serde_json silently writes null for them, which then fails to deserialize back into an f64. Every float this module writes therefore goes through JsonF64, which encodes the three non-finite values as the strings "NaN", "Infinity" and "-Infinity" and everything else as a plain JSON number.

A whole study, byte for byte

crates/atune_core/tests/goldens/journal_full_study_v5.jsonl — the journal of a 5-trial study, committed and compared byte for byte by crates/atune_core/tests/journal_goldens.rs, which also replays it back into the study that produced it. Regenerate it deliberately with ATUNE_BLESS_JOURNAL_GOLDEN=1 cargo test -p atune_core --test journal_goldens; the only field normalised is the writer's atune version, which would otherwise force a regeneration on every release.

crates/atune_core/tests/goldens/journal_full_study_v5.jsonl

{"record":"header","format":5,"atune":"0.0.0-GOLDEN","created_at_millis":1000000,"created_by":0}
{"record":"op","seq":1,"worker":0,"at_millis":1000000,"op":{"kind":"create-study","study":0,"config":{"name":"golden","directions":["Minimize","Maximize"],"seed":167,"space":null,"resource_unit":"Steps","metric_names":["loss","reward"]}}}
{"record":"op","seq":2,"worker":0,"at_millis":1000000,"op":{"kind":"set-study-user-attr","study":0,"key":"team","value":"control"}}
{"record":"op","seq":3,"worker":0,"at_millis":1000000,"op":{"kind":"create-trial","study":0,"trial":0,"number":0}}
{"record":"op","seq":4,"worker":0,"at_millis":1000000,"op":{"kind":"transition","trial":0,"from":"Waiting","to":"Running"}}
{"record":"op","seq":5,"worker":0,"at_millis":1000000,"op":{"kind":"params-batch","trial":0,"params":[{"name":"lr","dist":{"Float":{"low":0.00001,"high":0.1,"log":true,"step":null}},"value":{"F64":0.01}},{"name":"opt","dist":{"Cat":{"choices":{"version":1,"choices":[{"label":"adam","value":{"type":"string","value":"adam"}},{"label":"sgd","value":{"type":"string","value":"sgd"}}]}}},"value":{"Cat":0}},{"name":"layers","dist":{"Int":{"low":1,"high":8,"log":false,"step":1}},"value":{"I64":4}},{"name":"use_bn","dist":"Bool","value":{"Bool":true}}]}}
{"record":"op","seq":6,"worker":0,"at_millis":1000000,"op":{"kind":"set-trial-user-attr","trial":0,"key":"dataset","value":{"fold":2,"split":"train"}}}
{"record":"op","seq":7,"worker":0,"at_millis":1000050,"op":{"kind":"heartbeat","trial":0,"at_millis":1000050}}
{"record":"op","seq":8,"worker":0,"at_millis":1000050,"op":{"kind":"report","trial":0,"step":1,"values":[0.9,0.1]}}
{"record":"op","seq":9,"worker":0,"at_millis":1000050,"op":{"kind":"report","trial":0,"step":2,"values":[0.5,0.4]}}
{"record":"op","seq":10,"worker":0,"at_millis":1000050,"op":{"kind":"set-values","trial":0,"values":[0.42,0.58]}}
{"record":"op","seq":11,"worker":0,"at_millis":1000050,"op":{"kind":"transition","trial":0,"from":"Running","to":"Complete"}}
{"record":"op","seq":12,"worker":0,"at_millis":1001050,"op":{"kind":"create-trial","study":0,"trial":1,"number":1}}
{"record":"op","seq":13,"worker":0,"at_millis":1001050,"op":{"kind":"transition","trial":1,"from":"Waiting","to":"Running"}}
{"record":"op","seq":14,"worker":0,"at_millis":1001050,"op":{"kind":"params-batch","trial":1,"params":[{"name":"lr","dist":{"Float":{"low":0.00001,"high":0.1,"log":true,"step":null}},"value":{"F64":0.05}}]}}
{"record":"op","seq":15,"worker":0,"at_millis":1001050,"op":{"kind":"report","trial":1,"step":1,"values":[1.2,0.05]}}
{"record":"op","seq":16,"worker":0,"at_millis":1001050,"op":{"kind":"transition","trial":1,"from":"Running","to":"Pruned"}}
{"record":"op","seq":17,"worker":0,"at_millis":1002050,"op":{"kind":"create-trial","study":0,"trial":2,"number":2}}
{"record":"op","seq":18,"worker":0,"at_millis":1002050,"op":{"kind":"transition","trial":2,"from":"Waiting","to":"Running"}}
{"record":"op","seq":19,"worker":0,"at_millis":1002050,"op":{"kind":"set-error","trial":2,"message":"objective panicked: NaN loss at step 3"}}
{"record":"op","seq":20,"worker":0,"at_millis":1002050,"op":{"kind":"transition","trial":2,"from":"Running","to":"Failed"}}
{"record":"op","seq":21,"worker":0,"at_millis":1003050,"op":{"kind":"create-trial","study":0,"trial":3,"number":3,"template":{"fixed":{"lr":{"F64":0.001},"opt":{"Cat":1}},"parent":null}}}
{"record":"op","seq":22,"worker":0,"at_millis":1003050,"op":{"kind":"put-state","scope":{"Sampler":0},"blob":{"kind":"tpe","version":1,"data":{"gamma":0.25,"observations":3}}}}
{"record":"op","seq":23,"worker":0,"at_millis":1004050,"op":{"kind":"create-trial","study":0,"trial":4,"number":4}}
{"record":"op","seq":24,"worker":0,"at_millis":1004050,"op":{"kind":"transition","trial":4,"from":"Waiting","to":"Running"}}
{"record":"op","seq":25,"worker":0,"at_millis":1004050,"op":{"kind":"report","trial":4,"step":0,"values":["NaN","Infinity"]}}
{"record":"op","seq":26,"worker":0,"at_millis":1004050,"op":{"kind":"set-values","trial":4,"values":["NaN","-Infinity"]}}
{"record":"op","seq":27,"worker":0,"at_millis":1004050,"op":{"kind":"transition","trial":4,"from":"Running","to":"Complete"}}