assign_splits in skillopt_sleep/mine.py says right in its own comment "Guarantee val (the gate) is non-empty when we have >=2 real tasks", but there's a gap: _promote_one only pulls from train (to top up val) or from val (to top up train), never from test. If every real task's hash bucket lands in [val_cut, test_cut), both train and val start empty, so both guarantee calls have nothing to promote and silently no-op. A nightly batch mines N real tasks, all go to test, and run_sleep_cycle finishes with gate_action='reject', edits=0, no error, no warning, holdout_leaked doesn't flag it either.
Not contrived: for a small nightly batch (2-5 tasks is realistic for a solo user's session), any test_fraction above roughly 0.5 makes this a matter of when, not if. Confirmed on 79124b37: 5 tasks with val_fraction=0.10, test_fraction=0.80 all land in test, val and train both come back empty.
I think I see why it's shaped this way. #235 replaced the old unconditional real[-1].split = "val" fallback (append-order-unstable, could demote an already hash-assigned test task) with the current from-train/from-val-only _promote_one, to stop reassigning hash-assigned test tasks per that review. That fixed the instability and reopened this as a side effect.
An additive fix, only reach into test for the promotion when train and val are both empty, and log it the way consolidate.py already logs holdout_leaked, would keep the stability guarantee for the normal case and only touch the pathological one. Wanted to check that's the right shape before sending a PR, since it's touching the val/test tradeoff #235 just settled.
assign_splitsinskillopt_sleep/mine.pysays right in its own comment "Guarantee val (the gate) is non-empty when we have >=2 real tasks", but there's a gap:_promote_oneonly pulls fromtrain(to top upval) or fromval(to top uptrain), never fromtest. If every real task's hash bucket lands in[val_cut, test_cut), bothtrainandvalstart empty, so both guarantee calls have nothing to promote and silently no-op. A nightly batch mines N real tasks, all go totest, andrun_sleep_cyclefinishes withgate_action='reject',edits=0, no error, no warning,holdout_leakeddoesn't flag it either.Not contrived: for a small nightly batch (2-5 tasks is realistic for a solo user's session), any
test_fractionabove roughly 0.5 makes this a matter of when, not if. Confirmed on79124b37: 5 tasks withval_fraction=0.10, test_fraction=0.80all land in test, val and train both come back empty.I think I see why it's shaped this way. #235 replaced the old unconditional
real[-1].split = "val"fallback (append-order-unstable, could demote an already hash-assigned test task) with the current from-train/from-val-only_promote_one, to stop reassigning hash-assigned test tasks per that review. That fixed the instability and reopened this as a side effect.An additive fix, only reach into
testfor the promotion whentrainandvalare both empty, and log it the wayconsolidate.pyalready logsholdout_leaked, would keep the stability guarantee for the normal case and only touch the pathological one. Wanted to check that's the right shape before sending a PR, since it's touching the val/test tradeoff #235 just settled.