Skip to content

fix(benchmark): disclose accepted EdgeBench result recovery - #6180

Merged
huangruiteng merged 1 commit into
mainfrom
codex/edgebench-result-recovery-guidance
Oct 11, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/edgebench-result-recovery-guidance

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

The LoopX native task wrapper replaces the provider prompt but omitted accepted-result recovery, leaving an interrupted submit/poll without a documented way to retrieve its result. The shared startup/AGENTS policy now explains --result ID --details, lost-ID history reconciliation and older-client/session limitations, while distinguishing lookup from a new --details submission.

Validation: 41 prompt/worker-policy tests passed across all five profiles and all feedback modes. Commands extracted from the policy passed two isolated real-HTTP/generated-Bash lost-response scenarios with synthetic grading; result retrieval preserved the accepted submission and cooldown. Existing native clauses and restricted-mode output were independently checked for parity. Ruff and diff checks passed.

No model or Docker grading jobs were launched. Client result lookup requires a compatible pinned provider; this change does not qualify restart persistence, resource fairness or model adoption. The existing shared policy owner serves both surfaces; no additional policy abstraction was needed.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

作者自检:共享 native 策略现同时在启动提示和任务 AGENTS 中说明已接受提交的结果恢复。41 项 worker/prompt 检查和两项隔离真实 HTTP+生成 Bash 的丢回包验证通过;读回不增加提交或重启冷却,原有策略条款保留,blind/best-only 输出不变。旧客户端先检查帮助,缺失结果仍记为未恢复。未运行模型或 Docker 评分,不宣称持续公平性、重启持久性或模型采用。沿用现有共享策略生成入口,没有新增决策源。待维护者独立 review/merge。

English verdict: APPROVE for the bounded disclosure change based on the focused source/HTTP evidence above. This is an author self-review on this exact head, not independent acceptance or permission to merge. Client support, installed experiment adoption, sustained fairness and provider restart limitations remain separate qualification boundaries.

@loopx-agent loopx-agent left a comment •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | xhigh

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

Exact head: ece91b680676dc5e68a28c669b150e271fbc0e74. APPROVE for the bounded reviewed slice; merge authority and GitHub aggregate approval remain separate.

动机

使用原生 EdgeBench 反馈的实验执行者会遇到提交回复或后续轮询中断。 旧任务说明只教新提交和历史列表;已被服务器接受的候选若没收到结果,执行者无法从任务说明找到恢复命令,容易把新提交误当成读取,浪费提交机会或撞上冷却限制。

新版任务说明给出先核对客户端支持、再按原 ID 恢复的路径;真实 HTTP 和生成 Bash 的两种丢响应测试均取得原结果,重复读取不新增历史、提交计数或接受时钟。 本次只补共享 native 任务说明和文档;不改变评分、评估调度、权限或 blind/best-only 模式,也不证明服务器重启持久化、模型实际采用或长期公平性。

这是实际调用路径中的有界维护结果。未把作者测试计数或一个成功回执当成完整交付;独立验证覆盖后续重复读取/创建、拒绝和恢复。未更改的外部依赖与更大迁移目标仍保留原资格边界。

改动思路

复用现有共享任务说明即可把恢复操作送到实际执行者;新增服务、重试调度或修改评分都会超出这个已验证的说明缺口。 交付边界是现有 native startup/AGENTS 说明及支持恢复选项的 pinned provider 客户端;provider 持久化与真实模型采用另有资格边界。

只有--details新提交与--list列表说明;原有cooldown、feedback、source-checkpoint、network与stop义务仍存在。 全部旧说明行保留,新增help条件与--result恢复、lost-ID列表/checkpoint对账、不盲目重提、missing不等于未接受。 规则与效果分开:既有owner负责事实/准入,说明或Python桥接只传输结果;可选observer或外部依赖不产生额外义务。plain --details仍是新提交,恢复需当前token-visible history,help里有选项也不授予hidden-test访问。

具体改动

依据改动前 benchmark/edgebench/README.md,spec_revision f15d071249782f2732d3def3d5b4caaeba6432fc。原文未编号,criterion_id使用下面已接受条款的描述;新说明不作为自己的验收oracle。

  • Native submission accounting:Accepted submissions own cooldown/budget; --details creates a new submission.;已实现于native_task_instructions and generated client recovery,验证:native-http16 unchanged acceptance counters/clock and details cooldown。
  • Native public feedback:Public full feedback remains visible including ties/regressions, not score-only or permission for hidden answers.;已实现于native_task_instructions unchanged feedback clauses,验证:prompt-worker41 and native-http16。
  • Shared task policy delivery:All native workers receive the same policy at startup and in workspace AGENTS; retain task-owned instructions.;已实现于native_task_prompt / SForgeWorker._install_workspace_instructions,验证:prompt-worker41 actual Bash/files/retry/resume。
  • Provider lifetime qualification:In-memory provider session/cooldown and campaign fairness remain unqualified; adapter does not repair provider accounting.;已实现于README.md and conditional help/token-visible recovery clauses,验证:old provider lacks --result; current pinned client rejects unavailable/foreign result。

全差异3文件+38/-2:prompt增加恢复条款,README说明兼容/生命周期限制,测试覆盖startup/AGENTS和恢复说明,移除不用的import。公开provider客户端提交 bb956867提供被说明的操作;测试provider pin为ebbcb7ed516d163fa45bc214a8ead034e21277fb。

关键代码讲解

  • native_task_instructions(benchmark/edgebench/prompts.py:10):声明新提交与已有结果读取的差别,并规定丢ID先核对历史及checkpoint。
  • native_task_prompt(benchmark/edgebench/prompts.py:85):startup复用同一说明,保留任务query、objective与网络/冷却约束。
  • SForgeWorker._install_workspace_instructions(benchmark/runtime/sforge.py:318):现有AGENTS安装路径保留原文件并幂等追加共享native说明;真实Bash文件读回覆盖全部五种profile。

native/五profile的startup与AGENTS是改变的实际入口,restricted modes不新增恢复说明;无新frontend/Lark设置。

对主干的风险

最强回归候选是恢复操作重新消耗提交机会、读错当前候选,或把找不到结果当成未接受。丢失接受回复后先读列表;删掉当前候选目录,仍按原ID两次读到公共诊断和Score7。未知ID/无效token失败,历史与session计数/接受时钟不变;error/missing report/not found不写零分缓存。

benchmark/tests/test_edgebench_prompts.py 41通过;test_edgebench_feedback.py/test_edgebench_feedback_hook.py95通过。独立提取新说明的命令,运行两种响应丢失、重复恢复和未知ID/token负例,再运行provider原有11例,共16通过。worker环境执行真实Bash文件安装,但后端transport/模型入口受控;HTTP fixture仅替换task/评分,server/client/session/history是真实路径。 最初私有测试用了错误的fixture参数名,并把完整文本增量误写成删除一行,产生3个harness失败;修正后保持全部有效断言,当前16例通过。错误参数/断言与产品无关,未改生产源码或放宽规则。一次邻接测试命令路径拼错无测试运行,随后用实际现有两文件95通过。

初次premerge所有技术检查通过,但没有本次worktree精确scope质量回执,overall gate为receipt_missing;已按当前scope独立评审并记录读回回执。 4direct+19selected技术检查通过;benchmark-sensitive manual hold仍要求维护者review/merge。

受限模式函数源码/签名与base相同;五profile的实际文件读回验证feature-off隔离。其余native旧义务逐行保留,未压缩checkpoint、public-data-not-instruction、cooldown、停止或权限条件。

本次 wait_for_ci=false,未获取、轮询或等待远程CI。Synthetic grading replaces evaluator workload; real HTTP, generated Bash and state are exercised. Worker transport/model entry is controlled; no Docker/model job, restart/fairness or installed live campaign adoption claim. Provider tested source pin ebbcb7ed516d163fa45bc214a8ead034e21277fb includes public client commit bb9568672af53034fe28fc3328f29022128e3d6e; help support remains mandatory. 评审结论不自动满足平台Code Owner/last-push资格;COMMENTED批准也不等于平台APPROVED。

我的整体评价

APPROVE。这次有界任务目标达到:新版任务说明给出先核对客户端支持、再按原 ID 恢复的路径;真实 HTTP 和生成 Bash 的两种丢响应测试均取得原结果,重复读取不新增历史、提交计数或接受时钟。 长期执行与用户体验判断分别为improved/improved(仅丢响应恢复和命令发现),其证据和未验证维度如上。

复用现有共享任务说明即可把恢复操作送到实际执行者;新增服务、重试调度或修改评分都会超出这个已验证的说明缺口。 交付边界是现有 native startup/AGENTS 说明及支持恢复选项的 pinned provider 客户端;provider 持久化与真实模型采用另有资格边界。 未来相关重构检查确认现有共享说明已足够,没有必要增加新helper、重试服务或评分owner。没有发现本head的阻断问题;provider升级、产品安装采用和合并须按各自owner/readiness另行处理。

English verdict: APPROVE — ece91b680676dc5e68a28c669b150e271fbc0e74. 41+95+16 source/provider cases, focused Ruff; premerge4 direct +19 selected all technical checks passed; benchmark maintainer hold remains explicit. Conditional accepted-result recovery retains opportunity/full public feedback; restricted modes unchanged. Synthetic grading/model boundaries and provider restart/fairness remain unqualified. No CI consulted; author-owned COMMENTED is distinct from GitHub formal approval and merge authority.

@huangruiteng
huangruiteng merged commit 63707bf into main Oct 11, 2026
13 of 23 checks passed
@huangruiteng
huangruiteng deleted the codex/edgebench-result-recovery-guidance branch October 11, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants