How is the baseline evaluation score aggregated across runs? (single-seed vs multi-seed / best-of-N)

[lingyoumax]

Hi, thanks for the great work. I'm trying to reproduce your reported baseline numbers with the qwen backend and want to confirm the exact eval protocol before I blame my setup.

Reading through engine/trainer.py, I see that the baseline is a single call:

engine/trainer.py:1001-1002

baseline_dir = os.path.join(out_root, "selection_eval_baseline")
baseline_results = adapter.rollout(sel_env, skill_init, baseline_dir)

And for livemathematicianbench, configs/livemathematicianbench/default.yaml sets max_turns: 1, so each item gets a single LLM call. qwen_backend.py also defaults to temperature=0.7, so per-run stochasticity is non-trivial.

A few questions:

1. Are the baseline numbers you report from a single seed / single run, or aggregated across N runs?
2. If aggregated, is it mean, median, or best-of-N?
3. Is there any code path (that I might have missed) where a skill is rolled out multiple times and the max/mean is taken?

Thanks!

[LESdylan]

Not a maintainer, but I can answer the code-level part (your Q3) from the released source, since it's checkable:

Q3 — no, there is no multi-rollout / best-of-N path in skillopt/. Every evaluation is a single pass:

• The baseline you found is a single adapter.rollout(sel_env, skill_init, ...) into selection_eval_baseline — no repetition, no aggregation.
• Selection-set scores are then cached by skill hash (sel_cache[skill_hash(...)] = (hard, soft) in engine/trainer.py), so a given skill document is never even re-rolled when it's revisited later in training — the opposite of best-of-N.
• scripts/eval_only.py is likewise one rollout over the chosen split.
• The only repeated-rollout machinery in the repo is dream_rollouts in skillopt_sleep/dream.py, which belongs to the separate SkillOpt-Sleep tool, not the paper's training/eval pipeline.

So with the qwen backend at its default temperature=0.7, run-to-run variance on a single-seed protocol is real and expected — for what it's worth, docs/sleep/RESULTS.md describes single-seed (42) baselines with ±1–2 pt run-to-run variance on SearchQA, and train.seed defaults to 42.

Q1/Q2 (what protocol was used for the reported paper numbers) only the authors can confirm.

[raytrun]

Thank you for your interest and for the detailed code check.

To confirm your questions:

Q1 / Q2 (baseline protocol): The numbers reported in our paper are based on a single run with a fixed random seed (the default train.seed = 42). They are not aggregated across multiple seeds, nor are they median or best-of-N. This is why run‑to‑run stochasticity is indeed present.

Q3 (multi‑rollout / best‑of‑N): As you correctly observed from the code, there is no repeated rollout or aggregation in the training/eval pipeline. The baseline is a single adapter.rollout, and skills are cached by hash to avoid re‑evaluation.

We are aware that a single‑seed evaluation may not fully capture the model's performance stability. To address this, we have already run experiments with multiple random seeds and are currently analyzing the results.