Skip to content

[Task]: Implement computer_use_runtime protocol as provider-neutral schema builders #3110

Description

@KarlLeen

Task ID or area

area: computer-use runtime connector (docs/reference/protocols/computer-use-runtime-connector-v0.md)

Intent

I am proposing a new task

Summary

Context: I came across a Twitter thread where people discussed wiring Grok
4.5 to a computer-use-style QA loop, and huangruiteng publicly welcomed a
loopx PR for Grok support. After digging into it, I found xAI does not ship
a dedicated computer-use primitive -- a real Grok-driven loop would mean
building a screenshot -> Grok judgment -> action loop myself, which is more
than I can take on right now. Scoping this task down to the schema/builder
layer instead, so a full provider loop (Grok or otherwise) has a tested
contract to build against later.

docs/reference/protocols/computer-use-runtime-connector-v0.md defines a full
provider-neutral protocol for a computer-use runtime connector (capability
profile, action plan, receipt, etc.), but it is currently docs-only. I
checked and found no Python implementation anywhere in the repo, no smoke
test, and no open issue/PR covering it.

I'd like to implement the core request/plan/receipt builders as plain,
provider-neutral Python so any future computer-use runtime (browser or
desktop automation, any model behind it) can be wired against a tested
contract instead of a prose spec. This is infrastructure only -- no
specific provider, model, or live automation is included.

Proposed scope

In scope:

  • New module loopx/computer_use_runtime.py with schema-version constants
    and pure builder/validator functions for computer_use_capability_profile_v0,
    computer_use_action_plan_v0, and computer_use_action_receipt_v0
  • Enforce the guardrails already named in the protocol doc's "Smoke
    Expectations": capability-profile builder rejects raw screenshots,
    cookies, credentials, and private bodies; action-plan builder refuses
    external-write-class actions without forbidden_actions/gate coverage;
    receipt builder distinguishes completed / stopped_at_gate /
    blocked_by_unknown_modal outcomes
  • One synthetic example smoke (examples/computer-use-runtime-contract-smoke.py)
    using fixture data only -- no real screenshots, browser/desktop session,
    or model call
  • Register the new smoke in loopx/canary/planner.py with an appropriate
    tier, mirroring the existing worker-bridge-install-contract-smoke.py entry

Out of scope (future slices, not this task):

  • computer_use_session_v0, computer_use_observation_v0,
    computer_use_replay_handle_v0, computer_use_handoff_gate_v0 records
  • Wiring into status/dashboard projection or the concrete user-gate question
  • Quota spend integration
  • Any specific provider/runtime adapter (browser automation harness, any
    model's vision+tool-calling loop, etc.)
  • A new row in the main runtime-connector-catalog.md table -- that only
    makes sense once something is actually runnable

Relevant files or commands

  • docs/reference/protocols/computer-use-runtime-connector-v0.md
  • docs/integrations/runtime-connector-catalog.md
  • loopx/worker_bridge.py (closest existing pattern for a non-capability
    connector module)
  • examples/worker-bridge-install-contract-smoke.py (closest existing smoke
    pattern)
  • loopx/canary/planner.py (central smoke registry consumed by
    loopx canary premerge; the new smoke needs an entry here, mirroring
    how examples/worker-bridge-install-contract-smoke.py is registered)
  • New: loopx/computer_use_runtime.py,
    examples/computer-use-runtime-contract-smoke.py

Validation plan

  • python3 -m py_compile loopx/computer_use_runtime.py
  • python3 examples/computer-use-runtime-contract-smoke.py
  • python -m ruff check loopx tests examples
  • python -m mypy
  • loopx check --scan-path loopx/computer_use_runtime.py --scan-path examples/computer-use-runtime-contract-smoke.py --scan-path docs/reference/protocols/computer-use-runtime-connector-v0.md

Public/private boundary

  • This issue does not include private benchmark traces, verifier output, credentials, internal document links, raw agent sessions, or local runtime state.
  • I will not run or duplicate maintainer-owned benchmark cases unless a maintainer explicitly splits out a public task.

Activity

  1. huangruiteng commented on Aug 14, 2026

    @huangruiteng
    Collaborator

    感谢这个 issue,调研很扎实。但直接实现 docs-only 协议的三个 builder 会撞上仓库的 scope-fit 规则:没有 active call site 的协议不应先变成生产代码;而且仓库里已经有 content_ops(social_browser_x、connector_packets.py)和 value_connectors(planner.py)两套真实在用的 browser-backed packet 契约。再独立建一套 computer_use_* 词汇,将来要么做映射层,要么废弃一边。建议改成下面这个结合方案:

    1. A — 把协议变成 machine-checkable 契约

    • 为 computer_use_capability_profile_v0 / action_plan_v0 / receipt_v0 增加 JSON Schema(或等价的 typed schema),放在协议文档旁;
    • 把文档里的 “Smoke Expectations” 固化成正反例 fixtures(拒绝 raw screenshot/cookie/credential、拒绝无 gate 的 external-write、区分 completed / stopped_at_gate / blocked_by_unknown_modal);
    • smoke 从 schema + 文档语义推导,而不是从实现自证。

    2. B — 以真实契约做提取,提取结果必须被真实代码调用

    • 通用 builder 从 content_ops/connector_packets.py(capability profile ↔ source_profile / connector_trial / install_check)、social_browser_x.py(write_gate / truth_contract)、value_connectors/planner.py(approval gate / forbidden actions / URL 与敏感文本清洗)的真实 shape 提取;
    • 验收条件:提取出的 builder 必须被 content_ops / value_connectors 或新的 consumer 真实调用,不许先抽象后找用户;
    • 放置上先归最近 owner(content_ops/value_connectors 内部),等出现第二个真实 consumer(如 Grok/桌面 runtime)再提升为共享模块。

    3. C — 真实 consumer

    • 新增 loopx computer-use plan / receipt CLI(或复用 value-connectors plan 接线),让 plan 从 LoopX todo/gate 状态生成、receipt 进入 user gate 投影;
    • quota spend 复用已有 loopx turn run-once 的 validated writeback,不做新的计费逻辑。

    4. CUA showcase(合入门槛,核心)

    • 一个 hermetic 端到端 demo:本地 fixture 页面(无真实账号) + 浏览器驱动(Playwright/ego-browser,或 fake runtime 模式)+ mock/deterministic 判断,完整走通:
      loopx turn run-once → CUA host adapter(用新的 builders 从 Turn envelope 生成 action plan)→ runtime 执行 → compact receipt → 独立 validator → LoopX writeback + quota spend → 幂等回放;
    • 至少两个场景:draft-until-gate(填表后停在 submit,receipt=stopped_at_gate,投影出具体用户问题)和 unknown modal(返回 blocked_by_unknown_modal blocker,不硬点);
    • 真实浏览器模式由贡献者本地跑通并贴证据;CI 跑 hermetic 模式;
    • 原始截图/DOM/凭据绝不进 LoopX state,只有 compact facts + evidence handle;
    • showcase 跑通后再补 catalog 行。

    合入验收清单

    • py_compile / ruff / mypy / loopx check 通过;
    • schema fixtures + builder smoke 通过;
    • content_ops / value_connectors 现有 smoke 在提取后不回归(证明 B 没有破坏真实契约);
    • CUA showcase hermetic e2e 在 CI 绿,真实浏览器模式有本地验证记录;
    • 无凭据、无真实账号、无原始会话数据入库。

    参考实现:scripts/dsh_turn_host_adapter.py(adapter 骨架)、examples/loopx-turn-dsh-real-e2e-smoke.py(hermetic e2e 模式)。如果整个范围太大,可以拆两个 PR:PR1 = A+B+C(无新 runtime 行为),PR2 = showcase;但 showcase 是最终合入的前置条件。如果你只想做契约层,可以只做 A(docs + fixtures),但那不进 loopx/ 生产代码。


    Thanks for the well-researched issue. Directly implementing the three builders for a docs-only protocol would violate the repo's scope-fit rule: no production module without an active call site. The repo already ships two real browser-backed contract families (content_ops and value_connectors), so we'd rather combine all three directions:

    1. A — Machine-checkable contract: JSON Schema + positive/negative fixtures next to the protocol doc; smokes derive expectations from the schema and doc semantics, not from the implementation.
    2. B — Extraction anchored on real contracts: derive the generic computer_use_* builders from the actual shapes in content_ops and value_connectors, and require that they are actually called by those code paths or by a new consumer — no abstraction before a user.
    3. C — Real consumer: a loopx computer-use plan / receipt CLI (or value-connectors wiring) that generates plans from LoopX todo/gate state and routes receipts into user-gate projection; quota spend reuses the existing validated loopx turn run-once writeback.
    4. CUA showcase (merge gate): a hermetic end-to-end demo on a local fixture page — loopx turn run-once → CUA host adapter → runtime execution → compact receipt → independent validator → writeback + quota spend → idempotent replay. At least two scenarios: draft-until-gate (stopped_at_gate with a concrete user question) and unknown-modal (blocked_by_unknown_modal, no clicking through). Real-browser mode is run locally by the contributor with evidence; CI runs the hermetic mode. Raw screenshots/DOM/credentials never enter LoopX state — only compact facts and evidence handles.

    Reference patterns: scripts/dsh_turn_host_adapter.py (adapter skeleton) and examples/loopx-turn-dsh-real-e2e-smoke.py (hermetic e2e). If the scope is too large, split into PR1 (A+B+C) and PR2 (showcase), but the showcase stays a precondition for the final merge. If you only want the contract layer, A alone (docs + fixtures) is fine but stays out of loopx/ production code.

  2. huangruiteng commented on Aug 14, 2026

    @huangruiteng
    Collaborator

    更正上一条回复的第 4 点:showcase 不必基于 loopx turn run-once + CUA host adapter。

    上一条把 showcase 限定在 Turn 协议上,这是我过度指定了。修正后的验收是:使用标准 LoopX 使用姿势跑通一个 CUA 场景即可,不需要为 showcase 造新的 turn 协议或 adapter。

    具体来说:

    • 标准 host 姿势:Codex App 里 /loopx <goal text> → loopx start-goal,LoopX 照常投影 goal/todo/claim/gate/quota/evidence;未来的 Grok build 如果作为 host,走同样的标准接线即可。
    • CUA runtime(ego-browser / Playwright / 未来 Grok computer-use)只是 host 的工具/执行层:agent 在 goal 下通过工具调用驱动浏览器,LoopX 不接管像素级循环。
    • 演示场景保持两个:draft-until-gate(填表后停在 submit,LoopX 投影出具体用户问题)和 unknown modal(返回 blocker,不硬点)。
    • 约束不变:本地 fixture 页面、无真实账号、无凭据、原始截图/DOM 不进 LoopX state;CI 跑 hermetic 版本,真实浏览器模式本地验证并贴证据。

    这样 A/B/C 不变(machine-checkable 契约、从 content_ops/value_connectors 真实 packet 提取并被真实调用、真实 consumer),只是 showcase 的宿主从“自定义 turn adapter”改成“标准 host + 工具层”。可参考 docs/reference/protocols/codex-app-host-command-registry-v0.md 的标准接入姿势,以及 content_ops/social_browser_x.py 里 ego-browser 作为现成 browser-backed 工具的先例。


    Correction to point 4 of my previous reply: the showcase does not need to be built on loopx turn run-once + a CUA host adapter. That was over-specified on my part.

    The revised acceptance is: run a CUA scenario through the standard LoopX usage posture, without inventing a new turn protocol for the showcase.

    • Standard host posture: /loopx <goal text> in Codex App → loopx start-goal; LoopX projects goal/todo/claim/gate/quota/evidence as usual. A future Grok build would wire in as a host the same way.
    • The CUA runtime (ego-browser / Playwright / a future Grok computer-use) is just the host's tool/execution layer: the agent drives the browser through tool calls under a LoopX goal; LoopX does not own the pixel-level loop.
    • Keep the two demo scenarios: draft-until-gate (fill the form, stop at submit, LoopX projects a concrete user question) and unknown modal (return a blocker, do not click through).
    • Constraints stay the same: local fixture page, no real accounts, no credentials, raw screenshots/DOM never enter LoopX state; hermetic version runs in CI, real-browser mode is validated locally with evidence.

    A/B/C stay as described (machine-checkable contract; extraction anchored on and actually called by content_ops/value_connectors; a real consumer). Only the showcase host changes from "custom turn adapter" to "standard host + tool layer". Reference: docs/reference/protocols/codex-app-host-command-registry-v0.md, and content_ops/social_browser_x.py as the existing ego-browser tool precedent.

  3. KarlLeen commented on Aug 16, 2026

    @KarlLeen
    ContributorAuthor

    感谢这么详细拆解,这几天我认真评估了一下 A+B+C+showcase 整体的实现难度。

    我感觉按照目前我对这个仓库的熟悉程度来说,这个 scope 有点挑战,我担心自己接下来完不成,尤其是 CUA showcase 这一块。我去看了一下仓库现状,发现目前完全没有 Playwright(或任何浏览器自动化)依赖,也没有任何本地 fixture 页面的先例可以参考, social_browser_x.py 其实也不是真正驱动浏览器的代码,它只是一个描述外部工具 ego-browser 能力的 descriptor。也就是说 showcase 里"浏览器自动化基础设施"这部分基本要从零搭,还要保证它在 CI 里能 hermetic 地跑起来,同时正确接入标准 host 姿势(/loopx → start-goal → 工具调用驱动浏览器)下的 todo/gate/quota 投影。 这些我目前还不是很熟悉。虽然 showcase 用 mock/deterministic 判断替掉了真实 AI 判断这部分,但"截图循环"里我原本担心的浏览器驱动+落地验证的工作量基本还在。

    所以我想先只做 A(JSON Schema + 正反例 fixture,不进 loopx/ 生产代码)。B/C/showcase 我打算等自己对 LoopX 的核心机制(todo/gate/quota、host 接入姿势)更熟悉之后再挑战,或者如果有其他贡献者想先接手也完全没问题,我不想占着位置不动。

    Thanks for the thorough, well-reasoned breakdown. I spent the past few days assessing what it would actually take to implement A+B+C+showcase end to end.

    Honestly, given how unfamiliar I still am with this codebase, this scope feels like a real stretch, and I'm worried I won't be able to see it through, especially the CUA showcase. I looked into the current state of the repo and found there is no Playwright (or any browser-automation) dependency anywhere yet, and no existing local fixture-page pattern to build on social_browser_x.py isn't actually driving a browser either; it's just a descriptor for an external tool (ego-browser). So the "browser-automation infrastructure" part of the showcase would essentially need to be built from scratch, made to run hermetically in CI, and correctly wired into the standard host posture (/loopx -> start-goal -> tool calls driving the browser) with todo/gate/quota projection I'm not yet familiar with. The showcase replaces real AI judgment with a mock/deterministic decision, but the browser-driving and end-to-end verification work I was originally worried about in the "screenshot loop" is still largely there.

    So I'd like to start with just A (JSON Schema + positive/negative fixtures, staying out of loopx/ production code), which you mentioned is an acceptable starting point. I plan to come back to B/C/showcase once I'm more familiar with LoopX's core mechanics (todo/gate/quota, host integration posture) -- or if another contributor wants to pick those up first, that's completely fine, I don't want to sit on them.

  4. huangruiteng commented on Aug 16, 2026

    @huangruiteng
    Collaborator

    也行,主要是我感觉 A 的设计需要经过 B、C 使用的打磨,类似于跑通一个 loopx 支持下的复杂 CUA 长程研究。
    我觉得不一定要给 loopx 补基础设施,基于 ego lite browser 等外部依赖也行

  5. KarlLeen commented on Aug 16, 2026

    @KarlLeen
    ContributorAuthor

    刚跑了个本地 spike 验证机制:一个 fixture 表单页,agent 用浏览器工具填完约定字段后停在 submit 前,产出 stopped_at_gate receipt;另一个页面模拟意外的支付类弹窗,agent 不点确认也不点关闭,直接停下产出 blocked_by_unknown_modal receipt。两次都是用的就是 agent 自带的工具调用能力,跟 ego-browser 暴露给 agent 的能力是一类东西。

    这证明了不需要往 loopx 里加任何浏览器自动化基础设施,"判断+停在正确位置"这层能力本来就有,loopx 只需要在外面包 plan/receipt/gate 这层状态追踪。

    我打算按 A → B → C → showcase 走完整条链路,不只是停在A。
    CI 里的 hermetic 部分打算参考 loopx/experiments/planner_worker/runtime.py 的 Protocol + Fake 实现模式,不依赖真实浏览器。

    动手前有两个想提前对齐的问题:

    B 提炼出来的公共 builder,先放在 content_ops 还是 value_connectors 内部比较合适?
    C 里"标准 host 姿势"具体希望我对接到哪个现有 CLI 命令族,是新开 loopx computer-use plan/receipt,还是复用 value-connectors plan 加个 connector kind?

    如果方向 OK 我就按这个顺序开始了

  6. huangruiteng commented on Aug 16, 2026

    @huangruiteng
    Collaborator

    这个 spike 很有价值:它证明了 LoopX 不需要自建 browser driver 或 pixel loop,host 已经可以提供足够的 computer-use primitive;LoopX 真正需要补的是边界协议、状态归约、gate 和可验证回执。

    但我想同时纠正一个容易越做越重的抽象方向:CUA 不等同于 value connector,目前也不应直接注册成一个 LoopX product capability。 它首先是一类 provider/runtime execution surface。

    Maintainer 结论

    可以继续按 A → B → C → showcase 推进,但需要按下面的 ownership 重新定义 B 和 C:

    outcome capability
      -> bounded action request
    CUA provider/runtime
      -> observation + typed ACK/receipt
    capability-local reducer
      -> proposed transition
    LoopX Kernel
      -> accepted todo/gate/evidence/quota state
    

    四层职责应当分开:

    层 应负责 不应负责
    Capability(如 content-ops、explore、issue-fix) 用户可感知的 outcome、domain policy、允许的 effect、如何解释 provider ACK 浏览器安装、像素循环、某个 provider 的 session 细节
    CUA provider/runtime(如 host browser tool、ego-browser、Playwright provider) session、低层 action、observation、replay/evidence handle、typed stop reason 直接完成 LoopX todo、创建 gate、决定 quota/writeback
    Extension/package 可选 provider 的 install/doctor/enable/disable/upgrade/compatibility 发明一个没有 caller outcome 的“能力”
    Kernel durable todo、gate、quota、evidence、recovery 与最终 transition authority 执行浏览器动作或理解某个 UI 的领域语义

    这里是多对多关系:一个 CUA provider 可以服务 content-ops、explore 等多个 capability;同一个 capability 也可以选择 API、CLI、CUA 或人工 provider。value-connectors 当前是兼容 facade,不应成为新的通用 CUA owner。

    另外要区分两种经常都叫 “capability” 的东西:

    • LoopX product capability:稳定、provider-neutral、面向 caller outcome 的契约;
    • provider-advertised capability:某个 runtime 声明自己支持 screenshot、click、type、accessibility tree 等 primitive。

    因此协议里的 computer_use_capability_profile_v0 容易造成误解。我建议在落地前改成 computer_use_runtime_profile_v0(或 provider_profile_v0),并优先使用 runtime_id / provider_id,不要让 connector_id 暗示它属于 value-connectors。协议名称也可以从 computer-use-runtime-connector-v0 收敛成 computer-use-runtime-v0。

    对当前 protocol draft 的关键修正

    现在最需要调整的是 receipt ownership。provider 可以报告事实,但不能替 LoopX 决定状态迁移。

    也就是说,receipt 可以包含:

    • 尝试了什么 bounded action;
    • 观察到了什么;
    • completed、stopped_at_gate、blocked_by_unknown_modal、failed 等 typed stop reason;
    • compact evidence / replay handle;
    • provider-side idempotency key 和 session reference。

    但不应包含由 provider 决定的 next_loopx_writeback.complete_todo、create_user_gate 或类似命令。正确的语义是:

    1. provider 返回 ACK/事实;
    2. owning capability 的 domain-local reducer 结合原 action request、gate binding 和 domain policy 解释 ACK;
    3. reducer 提出 todo/gate/evidence transition;
    4. Kernel 校验 revision、authority、合法 transition 和 quota 后接受或拒绝。

    这也是我对 “reducer/ACK” 的判断:它是 capability/domain-local 的,不应做成一个全局 CUA reducer。 unknown_modal 在内容发布、支付、账号设置和研究浏览中的后续动作可能完全不同;CUA 层只需忠实报告事实。

    安全边界也不要主要依赖 forbidden_actions 里的自然语言,或复用 value_connectors/planner.py 中的字符串 denylist。建议最少有 typed 字段表达:

    • effect_class: read / draft / external_write / credential_use 等;
    • write_scope 与 validation target;
    • gate_binding: gate id、revision、status;
    • action/result/stop reason;
    • receipt/replay identity。

    自然语言可以补充说明,但 machine-enforced obligation 必须由 schema 和 transition validator 承担。

    A → B → C → showcase 的建议落地

    A. 先把 provider boundary 做成 machine-checkable contract

    保留正向与负向 fixtures,覆盖至少:

    • 在 external write 前准确停止;
    • 未知 modal 不被擅自确认或关闭;
    • receipt 不携带 credential、cookie、原始 DOM、私有页面正文等 durable state;
    • stale gate revision、缺少 write authority、重复 replay 被拒绝或幂等处理;
    • provider 不能直接构造 Kernel writeback。

    这里可以使用 Protocol + Fake,但 Fake 是测试替身,不是产品实现。测试 oracle 应独立判断 fixture 的 UI state 和合法 transition,不能让 Fake 直接回放测试预期,否则会变成 self-proving test。

    A 阶段可以提供 schema/validator/fixtures;在真实 consumer 出现前,不建议仅为了放 schema 就新增一个宽泛的生产模块 loopx/computer_use_runtime.py。

    B. 第一个 adapter/reducer 放在真实 outcome owner 内

    对你当前 “填写内容并停在 submit 前” 的场景:

    • 如果它表达 draft/publish 工作流,第一条 vertical slice 放在 content-ops;
    • 如果主场景是长程网页研究与证据采集,则放在 explore;
    • 如果 fixture 只是任意表单,它本身还不足以决定一个新 capability owner。

    所以对第一个具体问题,答案不是在 content_ops 和 value_connectors 之间选一个“永久公共 builder”:首个 domain adapter/reducer 放在 content-ops(若采用当前发布前 gate 场景),不要放进 value-connectors;共享 provider SPI 暂缓提炼。

    stopped_at_gate 和 blocked_by_unknown_modal 是同一个 consumer 的两个结果分支,不算两个独立 consumer。等第二个真实 outcome domain(例如 explore 或另一个非内容发布路径)也需要相同 request/receipt 语义,再提取窄的共享 CUA provider protocol。这样提取的是已被两个调用点证明的共同知识,而不是未来假设。

    C. CLI 应由 outcome 领航,而不是 execution mechanism 领航

    对第二个具体问题,我建议:

    • 不新增 value-connectors plan --connector-kind computer_use;
    • 暂时也不新增顶层 loopx computer-use plan/receipt;
    • 先通过 owning capability 的现有 CLI/API surface 发起,例如内容发布场景归 content-ops,研究场景归 explore;如果现有命令不够,再增加 capability-owned 的窄命令。

    只有当我们已经证明存在一个跨 domain、对 caller 独立有用的 outcome(例如 “bounded and verified UI task”),并且它有真实入口、至少两个 provider/consumer 约束和 focused validation,才考虑把它提升为独立 product capability。即使届时提升,名称也应描述 caller outcome,而不是 delivery mechanism computer-use。

    provider 自身的 readiness/doctor/profile 命令可以属于 host integration 或 extension lifecycle;那是诊断 provider,不等于新建 product capability。

    Showcase. 证明 LoopX 控制面,而不是证明浏览器库

    showcase 使用标准 /loopx host posture:目标与 todo 在 LoopX,agent 使用 host 已提供的 browser/CUA tool,provider 返回 compact receipt,capability reducer 决定是否提出 gate/transition。

    建议保留两个 fixture 页面:

    1. 正常填写后停在 submit 前,形成带精确问题的 external-write gate;
    2. 遇到未知支付类 modal,保持页面不变并 hand off。

    本地可以用真实 browser tool 作为补充证据;CI 用 hermetic Fake 验证协议与 reducer。不要把真实账户、凭据、原始截图/DOM、完整页面正文写入 LoopX durable/public-safe state。showcase 的成功标准是 stop/gate/recovery 可验证且 replay 幂等,不是 LoopX 自己驱动了多少像素动作。

    Extension 什么时候出现

    • 如果 CUA 已经是 Codex/Claude/其他 host 原生暴露的 tool,LoopX 只消费 host declaration 和 receipt,不需要为了它创建 extension;
    • 如果 LoopX 要负责安装、doctor、启停、升级、权限声明、版本兼容和 dispatch 某个可选 provider,再把该 provider 做成 extension/package;
    • 如果是 “Grok 负责 reasoning + browser driver 负责动作”,它是 composite provider/host wiring,Grok 本身并不会因为能看图就自动成为 CUA provider。

    extension 是交付与生命周期轴,不是 runtime 数据流里的第五层,也不应为了“可安装”而反向制造一个空的 capability。

    建议拆分与 acceptance

    为了让 diff 可审查,我建议至少拆成两阶段:

    1. Contract PR:修订 research protocol、ownership table、schema/fixtures/validator;重命名歧义字段;移除 provider-authored Kernel writeback;不加 speculative core abstraction。
    2. Vertical-slice PR:一个 outcome-owned request + domain reducer + Fake provider + 标准 host showcase;覆盖 gate、unknown modal、stale revision、idempotent replay 和 public/private boundary。

    第二个真实 outcome consumer 出现后,再做第三个 extraction PR,决定共享 provider SPI 和 extension packaging。无需为了宣称 “A→B→C 全做了” 在首个 PR 里提前固化整个横向框架。

    我会按以下条件判断这条实现链是否完成:

    • CUA provider 不能直接 complete todo 或 create gate;
    • external write 必须绑定可验证的 live gate/revision;
    • compact receipt 足够 reducer 做决定,但不泄露 raw/private browser state;
    • replay/retry 可幂等,未知 UI 默认停下而非猜测;
    • validator 与 Fake 输出独立,负向 fixture 能证明非法 transition 被拒绝;
    • preview/readiness 不改变 durable work state,真实 writeback 先经 Kernel validation;
    • 不扩大默认权限,不改变现有 value-connectors compatibility contract。

    所以总体方向是认可 spike、继续推进,但将目标从“实现一个 generic computer-use builder/CLI”调整为:先完成一个 outcome-owned vertical slice,证明 provider ACK → domain reducer → Kernel transition;等第二个真实 domain 再抽共享 CUA provider contract。


    English summary

    The spike validates the right execution model: LoopX does not need to own a browser driver or pixel loop. However, CUA is primarily a provider/runtime surface, not a value connector and not yet a standalone LoopX product capability.

    • Keep outcome policy in content-ops, explore, or another real capability.
    • Let the CUA provider return observations and typed ACKs only; it must not author todo/gate writebacks.
    • Interpret the ACK in a capability-local reducer, then let the Kernel validate and commit the transition.
    • Do not add CUA to the value-connectors compatibility facade.
    • Do not add a top-level loopx computer-use CLI until a provider-neutral caller outcome is proven.
    • Put the first publish-before-submit slice in content-ops; extract a shared provider SPI only after a second distinct outcome consumer exists.
    • Use an extension only when LoopX owns optional provider lifecycle such as install/doctor/enable/upgrade. A host-native browser tool needs no extension merely to be consumed.
    • Use the real host tool for local evidence and a hermetic Fake for CI, while ensuring the validator is independent of the Fake's expected output.

    This preserves the useful A → B → C → showcase plan without prematurely turning an execution mechanism into a broad core abstraction.

  7. KarlLeen commented on Aug 18, 2026

    @KarlLeen
    ContributorAuthor

    开始规划 vertical-slice PR 时发现一个问题,PR1 合入的 validator 目前是没法被 loopx/ 生产代码调用的,想在动手前先对齐方向。

    问题

    查了 pyproject.toml:

    • [tool.setuptools.packages.find] include = ["loopx*"] —— 只有 loopx* 会进最终安装包,scripts/ 和 docs/(除了个别 *.md glob)都不会被打包进去。
    • jsonschema 只在 [project.optional-dependencies].test 里,核心 dependencies = []。普通 pip install loopx 拿不到 jsonschema。

    也就是说 scripts/computer_use_runtime_contract_validator.py 和 docs/reference/protocols/schemas/computer_use_runtime_v0/*.schema.json 现在都是仓库 checkout + [test] extras 才能用的东西,对 PR1(协议文档 + 自带 smoke)没问题,但意味着如果 content-ops 里新加的 provider/reducer 想在 module 级别 import 这个 validator,普通用户装出来的 loopx 要么因为 scripts/ 没打包直接 import 失败,要么因为 jsonschema 缺失而报错。这跟"PR1 的 validator/schema 必须被真实调用"的要求直接冲突。

    考虑过三条路径

    A —— 把契约打包成正式安装内容。 挪到 loopx/contracts/computer_use_runtime_v0/,jsonschema 加进核心依赖,scripts/ 里原来那个文件改成薄的 re-export 保持现有 smoke/canary 命令不用改,schema JSON 走 package-data 打包,路径解析从相对文件路径改成 importlib.resources(装进 site-packages 后目录结构会变)。这是唯一能做到"只有一份实现、不存在漂移"的方案,但代价有两个:

    1. 会给每个安装加一个新依赖,直接冲突 README 里明确写的"The Python package has no runtime dependencies outside the standard library"
    2. 现在项目里所有验证都是 pip install -e .(editable install),这测不出打包配置对不对;真要落地这条路径,需要额外做一次 python -m build 打 wheel、装进干净 venv 里的验证,这是目前从没做过的一步,不难但要补上。

    B —— 生产代码保持零依赖,validator 只在测试里真正被调用。 content-ops 自己写一套最小的、纯标准库的结构检查(必填字段存在、stop_reason 是那四个值之一、external_write/credential_use 要求 gate_binding、拒绝 writeback 形状的字段),不 import jsonschema。测试套件里额外 import PR1 真正的 validator,断言生产代码构造/接受的每个 request/receipt 都能通过 PR1 的完整校验。工作量是三条里最小的(大概一两个小时),不碰依赖、不碰 PR1 已合入的文件布局。代价是生产路径上跑的是精简子集,不是完整契约,但这个弱点是诚实的、写在代码里能看出来的,不会假装自己是完整校验。

    第三条路径(原生 Python 重写、不依赖 jsonschema 库但真正跑在生产代码里)思考后感觉不太好。 三个 schema 加起来有 30-40 条字段级约束(required/type/enum/const/嵌套对象的 additionalProperties/一个 if-then 条件),手写忠实复刻要么写成大量逐字段判断(任何一处漏改都是无声漂移),要么等于自己重新发明一个简配版 jsonschema;而且"用现有 fixture 交叉验证两边一致"这个保证,只能证明"这些 fixture 上两边判断一样",不能证明"所有可能输入下都一样",以后 schema 加个新字段,只要没有 fixture 恰好测到,两边悄悄跑偏也不会被发现。评估了一下这是三条里工作量最大、保证却最弱的一条,先排除。

    想问的

    你倾向哪个方向,还是有别的思路?

  8. huangruiteng commented on Aug 18, 2026

    @huangruiteng
    Collaborator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions