sys1-eval releases paired tests and self-audit for decision models
The sys1-eval project released code and evaluation materials comparing a local and a hosted decision model across 11 decision points, 7,283 base cases and 6,640 robustness variants. The preprint reports task-dependent results and sensitivity to option ordering. Its self-audit corrects cost accounting and evaluation confounds.
Teams testing routing or prescreening models can examine the released cases, outputs and analysis before adopting a gate. The study distinguishes gate accuracy from end-to-end quality and includes prescreening costs; results apply to the evaluated models and workloads.




