Skip to main content

evaluate_config_on_test_seeds

Function evaluate_config_on_test_seeds 

pub fn evaluate_config_on_test_seeds(
    cfg: &SpikeConfig,
    config_toml: &str,
) -> Result<f64>
Expand description

Evaluates a single, already-complete RunConfig TOML on the protocol’s disjoint test seeds and returns the same aggregate the tune/test split reports for a tuned trial.

This is the honest baseline half of the M3 exit criterion (04-rl-and-oniro.md §A.1, “tuned oniro PPO beats the default config on test seeds”): to compare fairly, the default config must be judged by the exact yard-stick the study’s re-evaluation stage judged the tuned winner (FinalReport::best) by. So this function

  1. derives the test seeds with the same SeedProtocol::test_seed derivation the re-evaluation stage uses. Under the spike’s default Paired pairing the test seed is trial-number-independent, so these are byte-for-byte the seeds every tuned trial — the winner included — was re-evaluated on; the baseline and the tuned best are therefore judged on the identical seeds;
  2. for each test seed runs the exact child invocation and objective a trial replicate runs — evaluate_child: materialize the config with the seed written to run.seed, oniro_cli run, then the greedy oniro_cli eval mean_return (or meta.json’s final_metric for the Meta source);
  3. aggregates the per-seed values with the protocol’s own aggregate (SpikeConfig::aggregate, IQM by default) — the same Aggregate::apply the re-evaluation stage applies.

config_toml is a complete oniro RunConfig (for the exit criterion: the base default-config with train.algo = "ppo" and PPO’s own defaults, no tuned overrides). It is passed to evaluate_child as the base with an empty overlay, so nothing is tuned — the config is evaluated exactly as given, only its run.seed and run directory changing per test seed.

Unlike the re-evaluation stage — which re-ranks many configs and so records a diverged replicate as a NaN sub-result rather than let one failure abort the stage — this single-config baseline propagates a diverged or faulted replicate as an error, so a caller sees a real failure instead of a silently NaN baseline.

Host-only: it spawns oniro processes, so it is not part of the wasm surface.

§Errors

SpikeError::Oniro if the protocol declares no test seeds; otherwise whatever evaluate_child returns (a materialize fault, an I/O fault, or a non-zero oniro exit / missing objective) or Aggregate::apply returns (an empty value set).