Function evaluate_config_on_test_seeds
pub fn evaluate_config_on_test_seeds(
cfg: &SpikeConfig,
config_toml: &str,
) -> Result<f64>Expand description
Evaluates a single, already-complete RunConfig TOML on the protocol’s
disjoint test seeds and returns the same aggregate the tune/test split
reports for a tuned trial.
This is the honest baseline half of the M3 exit criterion
(04-rl-and-oniro.md §A.1, “tuned oniro PPO beats the default config on test
seeds”): to compare fairly, the default config must be judged by the exact
yard-stick the study’s re-evaluation stage
judged the tuned winner (FinalReport::best) by. So this function
- derives the test seeds with the same
SeedProtocol::test_seedderivation the re-evaluation stage uses. Under the spike’s defaultPairedpairing the test seed is trial-number-independent, so these are byte-for-byte the seeds every tuned trial — the winner included — was re-evaluated on; the baseline and the tuned best are therefore judged on the identical seeds; - for each test seed runs the exact child invocation and objective a
trial replicate runs —
evaluate_child: materialize the config with the seed written torun.seed,oniro_cli run, then the greedyoniro_cli evalmean_return(ormeta.json’sfinal_metricfor theMetasource); - aggregates the per-seed values with the protocol’s own aggregate
(
SpikeConfig::aggregate, IQM by default) — the sameAggregate::applythe re-evaluation stage applies.
config_toml is a complete oniro RunConfig (for the exit criterion:
the base default-config with train.algo = "ppo" and PPO’s own defaults, no
tuned overrides). It is passed to evaluate_child as the base with an
empty overlay, so nothing is tuned — the config is evaluated exactly as
given, only its run.seed and run directory changing per test seed.
Unlike the re-evaluation stage — which re-ranks many configs and so records
a diverged replicate as a NaN sub-result rather than let one failure abort
the stage — this single-config baseline propagates a diverged or faulted
replicate as an error, so a caller sees a real failure instead of a silently
NaN baseline.
Host-only: it spawns oniro processes, so it is not part of the wasm surface.
§Errors
SpikeError::Oniro if the protocol declares no test seeds; otherwise
whatever evaluate_child returns (a materialize fault, an I/O fault, or a
non-zero oniro exit / missing objective) or
Aggregate::apply returns (an empty
value set).