Repository navigation
fix(benchmark): disclose accepted EdgeBench result recovery - #6180
Conversation
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
作者自检:共享 native 策略现同时在启动提示和任务 AGENTS 中说明已接受提交的结果恢复。41 项 worker/prompt 检查和两项隔离真实 HTTP+生成 Bash 的丢回包验证通过;读回不增加提交或重启冷却,原有策略条款保留,blind/best-only 输出不变。旧客户端先检查帮助,缺失结果仍记为未恢复。未运行模型或 Docker 评分,不宣称持续公平性、重启持久性或模型采用。沿用现有共享策略生成入口,没有新增决策源。待维护者独立 review/merge。
English verdict: APPROVE for the bounded disclosure change based on the focused source/HTTP evidence above. This is an author self-review on this exact head, not independent acceptance or permission to merge. Client support, installed experiment adoption, sustained fairness and provider restart limitations remain separate qualification boundaries.
There was a problem hiding this comment.
Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | xhigh
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
Exact head: ece91b680676dc5e68a28c669b150e271fbc0e74. APPROVE for the bounded reviewed slice; merge authority and GitHub aggregate approval remain separate.
动机
使用原生 EdgeBench 反馈的实验执行者会遇到提交回复或后续轮询中断。 旧任务说明只教新提交和历史列表;已被服务器接受的候选若没收到结果,执行者无法从任务说明找到恢复命令,容易把新提交误当成读取,浪费提交机会或撞上冷却限制。
新版任务说明给出先核对客户端支持、再按原 ID 恢复的路径;真实 HTTP 和生成 Bash 的两种丢响应测试均取得原结果,重复读取不新增历史、提交计数或接受时钟。 本次只补共享 native 任务说明和文档;不改变评分、评估调度、权限或 blind/best-only 模式,也不证明服务器重启持久化、模型实际采用或长期公平性。
这是实际调用路径中的有界维护结果。未把作者测试计数或一个成功回执当成完整交付;独立验证覆盖后续重复读取/创建、拒绝和恢复。未更改的外部依赖与更大迁移目标仍保留原资格边界。
改动思路
复用现有共享任务说明即可把恢复操作送到实际执行者;新增服务、重试调度或修改评分都会超出这个已验证的说明缺口。 交付边界是现有 native startup/AGENTS 说明及支持恢复选项的 pinned provider 客户端;provider 持久化与真实模型采用另有资格边界。
只有--details新提交与--list列表说明;原有cooldown、feedback、source-checkpoint、network与stop义务仍存在。 全部旧说明行保留,新增help条件与--result恢复、lost-ID列表/checkpoint对账、不盲目重提、missing不等于未接受。 规则与效果分开:既有owner负责事实/准入,说明或Python桥接只传输结果;可选observer或外部依赖不产生额外义务。plain --details仍是新提交,恢复需当前token-visible history,help里有选项也不授予hidden-test访问。
具体改动
依据改动前 benchmark/edgebench/README.md,spec_revision f15d071249782f2732d3def3d5b4caaeba6432fc。原文未编号,criterion_id使用下面已接受条款的描述;新说明不作为自己的验收oracle。
- Native submission accounting:Accepted submissions own cooldown/budget; --details creates a new submission.;已实现于native_task_instructions and generated client recovery,验证:native-http16 unchanged acceptance counters/clock and details cooldown。
- Native public feedback:Public full feedback remains visible including ties/regressions, not score-only or permission for hidden answers.;已实现于native_task_instructions unchanged feedback clauses,验证:prompt-worker41 and native-http16。
- Shared task policy delivery:All native workers receive the same policy at startup and in workspace AGENTS; retain task-owned instructions.;已实现于native_task_prompt / SForgeWorker._install_workspace_instructions,验证:prompt-worker41 actual Bash/files/retry/resume。
- Provider lifetime qualification:In-memory provider session/cooldown and campaign fairness remain unqualified; adapter does not repair provider accounting.;已实现于README.md and conditional help/token-visible recovery clauses,验证:old provider lacks --result; current pinned client rejects unavailable/foreign result。
全差异3文件+38/-2:prompt增加恢复条款,README说明兼容/生命周期限制,测试覆盖startup/AGENTS和恢复说明,移除不用的import。公开provider客户端提交 bb956867提供被说明的操作;测试provider pin为ebbcb7ed516d163fa45bc214a8ead034e21277fb。
关键代码讲解
native_task_instructions(benchmark/edgebench/prompts.py:10):声明新提交与已有结果读取的差别,并规定丢ID先核对历史及checkpoint。native_task_prompt(benchmark/edgebench/prompts.py:85):startup复用同一说明,保留任务query、objective与网络/冷却约束。SForgeWorker._install_workspace_instructions(benchmark/runtime/sforge.py:318):现有AGENTS安装路径保留原文件并幂等追加共享native说明;真实Bash文件读回覆盖全部五种profile。
native/五profile的startup与AGENTS是改变的实际入口,restricted modes不新增恢复说明;无新frontend/Lark设置。
对主干的风险
最强回归候选是恢复操作重新消耗提交机会、读错当前候选,或把找不到结果当成未接受。丢失接受回复后先读列表;删掉当前候选目录,仍按原ID两次读到公共诊断和Score7。未知ID/无效token失败,历史与session计数/接受时钟不变;error/missing report/not found不写零分缓存。
benchmark/tests/test_edgebench_prompts.py 41通过;test_edgebench_feedback.py/test_edgebench_feedback_hook.py95通过。独立提取新说明的命令,运行两种响应丢失、重复恢复和未知ID/token负例,再运行provider原有11例,共16通过。worker环境执行真实Bash文件安装,但后端transport/模型入口受控;HTTP fixture仅替换task/评分,server/client/session/history是真实路径。 最初私有测试用了错误的fixture参数名,并把完整文本增量误写成删除一行,产生3个harness失败;修正后保持全部有效断言,当前16例通过。错误参数/断言与产品无关,未改生产源码或放宽规则。一次邻接测试命令路径拼错无测试运行,随后用实际现有两文件95通过。
初次premerge所有技术检查通过,但没有本次worktree精确scope质量回执,overall gate为receipt_missing;已按当前scope独立评审并记录读回回执。 4direct+19selected技术检查通过;benchmark-sensitive manual hold仍要求维护者review/merge。
受限模式函数源码/签名与base相同;五profile的实际文件读回验证feature-off隔离。其余native旧义务逐行保留,未压缩checkpoint、public-data-not-instruction、cooldown、停止或权限条件。
本次 wait_for_ci=false,未获取、轮询或等待远程CI。Synthetic grading replaces evaluator workload; real HTTP, generated Bash and state are exercised. Worker transport/model entry is controlled; no Docker/model job, restart/fairness or installed live campaign adoption claim. Provider tested source pin ebbcb7ed516d163fa45bc214a8ead034e21277fb includes public client commit bb9568672af53034fe28fc3328f29022128e3d6e; help support remains mandatory. 评审结论不自动满足平台Code Owner/last-push资格;COMMENTED批准也不等于平台APPROVED。
我的整体评价
APPROVE。这次有界任务目标达到:新版任务说明给出先核对客户端支持、再按原 ID 恢复的路径;真实 HTTP 和生成 Bash 的两种丢响应测试均取得原结果,重复读取不新增历史、提交计数或接受时钟。 长期执行与用户体验判断分别为improved/improved(仅丢响应恢复和命令发现),其证据和未验证维度如上。
复用现有共享任务说明即可把恢复操作送到实际执行者;新增服务、重试调度或修改评分都会超出这个已验证的说明缺口。 交付边界是现有 native startup/AGENTS 说明及支持恢复选项的 pinned provider 客户端;provider 持久化与真实模型采用另有资格边界。 未来相关重构检查确认现有共享说明已足够,没有必要增加新helper、重试服务或评分owner。没有发现本head的阻断问题;provider升级、产品安装采用和合并须按各自owner/readiness另行处理。
English verdict: APPROVE — ece91b680676dc5e68a28c669b150e271fbc0e74. 41+95+16 source/provider cases, focused Ruff; premerge4 direct +19 selected all technical checks passed; benchmark maintainer hold remains explicit. Conditional accepted-result recovery retains opportunity/full public feedback; restricted modes unchanged. Synthetic grading/model boundaries and provider restart/fairness remain unqualified. No CI consulted; author-owned COMMENTED is distinct from GitHub formal approval and merge authority.
The LoopX native task wrapper replaces the provider prompt but omitted accepted-result recovery, leaving an interrupted submit/poll without a documented way to retrieve its result. The shared startup/AGENTS policy now explains
--result ID --details, lost-ID history reconciliation and older-client/session limitations, while distinguishing lookup from a new--detailssubmission.Validation: 41 prompt/worker-policy tests passed across all five profiles and all feedback modes. Commands extracted from the policy passed two isolated real-HTTP/generated-Bash lost-response scenarios with synthetic grading; result retrieval preserved the accepted submission and cooldown. Existing native clauses and restricted-mode output were independently checked for parity. Ruff and diff checks passed.
No model or Docker grading jobs were launched. Client result lookup requires a compatible pinned provider; this change does not qualify restart persistence, resource fairness or model adoption. The existing shared policy owner serves both surfaces; no additional policy abstraction was needed.