Skip to main content

overfit_gap

Function overfit_gap 

pub fn overfit_gap(
    direction: Direction,
    tune_value: f64,
    test_value: f64,
) -> f64
Expand description

The signed tune-vs-test gap, positive when a configuration looked better on tune seeds than it held up on test seeds — the ICML-2023 overfitting the protocol guards against.

It is direction-aware so that “positive means overfit” holds for both directions: for a Maximize study a higher value is better, so overfit is tune - test; for Minimize a lower value is better, so overfit is test - tune.