Run trials through spacedeck

atune::exec::SpacedeckExecutor evaluates each trial on another machine by handing it to spacedeck's run command. The study, its storage and its sampler stay in the process that drives them; only the objective travels. It is the supported way to spread an atune study over a fleet of hosts that spacedeck manages.

The direction matters. atune is spacedeck's client: the driving process asks spacedeck to run one command, and spacedeck places it, runs it and reports back. The reverse — atune run workers on remote hosts joining a study that atune serve exposes over atune:// — is a different topology with no worker mode today, and would need its own design decision before it exists.

What one trial costs

Each trial is one synchronous spacedeck --json run --on <target> process (with --pick and its flags when spacedeck picks the host). The executor builds its argument vector, waits for spacedeck's spacedeck/v1 JSON envelope, and requires exactly one host result in it. A study that runs trials in parallel runs that many spacedeck processes at once; nothing is pooled or kept warm between trials.

So every trial pays a fixed overhead before the objective starts: spacedeck's own start-up, its capacity reads of the candidate hosts, and the round trip to the chosen one. That is small next to a training job that runs for minutes or hours, and it can dominate an objective that finishes in seconds — batch such work into fewer, longer trials, or keep it on the subprocess executor.

Set it up

The executor lives in atune::exec behind the system feature. It is built from three things:

Input	What it is
The declared space	Every {name} placeholder in the command must be a parameter of it; the constructor refuses one that is not.
A target selector	spacedeck's --on argument, naming where the trial runs. It must be non-empty and contain no NUL byte.
A atune::exec::CommandTemplate	The program and its argument templates, exactly as for the subprocess executor.

The command reaches spacedeck as an argument vector after --; the executor adds no shell, and spacedeck quotes each argument for the remote shell, so a sampled value arrives as one literal argument. Its run method checks that every parameter it names is in the study's declared space, then drives the study's ordinary optimize loop, so retries, budgets, parallelism and resume behave as they do for any other objective.

spacedeck forwards only the environment variables its --env flags name. The executor names every variable it sets: the trial handshake (ATUNE_STUDY and ATUNE_TRIAL_NUMBER; ATUNE_TRIAL, ATUNE_TRIAL_WORKER and ATUNE_TRIAL_EPOCH while the trial is owned by a live lease; ATUNE_SEED; ATUNE_CHECKPOINT for a forked or resumed trial; the ATUNE_RESTART_* variables for a restarted one; and ATUNE_EVALUATE and ATUNE_RECIPE in a final evaluation stage) and every sampled parameter under the ATUNE_PARAM_ prefix, so each parameter name must form a valid variable name there. The environment reference describes each one.

Which spacedeck binary

The executor runs spacedeck from PATH unless SPACEDECK_CLI names another executable. The variable is read once, when the executor is built; SpacedeckExecutor::cli_path overrides it for that executor.

Timeouts

SpacedeckExecutor::timeout passes a per-host bound to spacedeck's --timeout, in whole milliseconds rounded up, so spacedeck stops the remote command and reports a timeout result itself; a bound under one millisecond is sent as 1ms, never as spacedeck's unbounded 0ms. The executor also keeps a local deadline of that timeout plus one minute: if the spacedeck process has not answered by then, it is killed and the trial fails. Without a timeout, or with a zero one (spacedeck's "no bound"), the executor waits for spacedeck indefinitely.

On Unix the spacedeck process runs in its own process group, which is killed when it exits or overruns, so nothing it started outlives the trial on the driving host. Elsewhere only the spacedeck process itself is killed. The envelope it may print is bounded too: one host's retained output, JSON escaped, and no more.

Let spacedeck pick the host

SpacedeckExecutor::pick runs each trial on one host that spacedeck chooses from the target set, through spacedeck's run --pick (its decision D-095), instead of on every host the target names. The target stays the candidate set, so @all considers every host in spacedeck's inventory. A atune::exec::SpacedeckPick carries the optional constraints, each passed as the run flag of the same name:

Builder method	spacedeck flag	Effect
window("Europe/Luxembourg,22:00,06:00")	--window	A daily window, TZ,HH:MM,HH:MM; outside it no host is admitted and none is contacted
min_disk_avail("500G")	--min-disk-avail	The free disk a host needs where the command runs
unattended_idle_hours(2)	--unattended-idle-hours	Excludes a host where another account's interactive session has been idle for less than that many hours; 0 excludes any such session
prefer(SpacedeckPrefer::MostDisk)	--prefer	Which of equally free hosts to take: atune::exec::SpacedeckPrefer

spacedeck checks the values and ranks the hosts it admits: those with the fewest free slots first, then the preference among them (the most available disk, the lowest one-minute load average, the coolest hottest GPU, or the fewest leased slots), then the host name. It names the chosen host and every excluded one on its stderr and in the envelope's placement; the executor reads only the host result. pick itself refuses an empty window or size. A pick that admits no host runs nothing: spacedeck refuses it with the code no_eligible_host, and the trial fails with that code and spacedeck's reason. spacedeck's run holds no slot, so trials picked at the same time can choose the same host; lower the study's parallelism when they must not share one.

How a result becomes a trial state

The executor reads the payload's exit status from the envelope, never spacedeck's own exit code, which belongs to spacedeck's namespace:

Envelope	Trial
Payload exited 0	Complete, with the objective parsed from the payload's standard output by the subprocess executor's rule — the last non-empty line, a number or JSON — including an optional checkpoint reference
Payload exited 42	Pruned; unlike the subprocess executor, the code is not configurable here
Payload exited with any other code	Failed, with the remote stderr in the message
Payload reported no exit	Failed, with spacedeck's error, or its result status when it gives none, in the message
A refusal, an unknown host, invalid JSON, a schema other than spacedeck/v1, a cut-off envelope, or other than one host result	Failed

DGX Spark hosts

On the NVIDIA GB10 (DGX Spark) hosts of this fleet, nvidia-smi reports GPU memory as [N/A]. A check that decides a host is idle from free GPU memory therefore sees nothing; judge idleness from GPU utilization and the compute-apps list instead, before choosing a host as a target.

What this page rests on

• The envelope (decision D92). The executor accepts only spacedeck's spacedeck/v1 envelope; a refusal, which carries a code and an error but no results, fails the trial and names its code. Real output of spacedeck 0.2.0, recorded on 2026-10-19, is kept per schema version in crates/atune_core/tests/fixtures/xc-12-spacedeck-run-v1.json, and the executor's tests replay it, so a change to the envelope fails them before it reaches a study.
• The downstream build. Oniro's oniro_tuning, the atune consumer that compiles atune on these hosts whenever it trains there, passes cargo check --locked against this atune revision (2026-10-19, x86_64). On a GB10 node itself (aarch64, Rust 1.92.0, 2026-10-19), atune passes cargo check --locked with the system and pb2 features oniro_tuning enables, and its facade contract tests pass. This is checked by hand once per release candidate, because no aarch64 CI runner exists.

Where to go next

If you want to	Go to
Write the objective program the remote host runs	The subprocess contract
Run parallel workers against one study file instead	Resume and scale
Read every variable a trial receives	Environment reference
Read the executor's API, method by method	Rust API