Spec-agnostic task validation: openenv validate-task for external task.md packages (PostTrain Arena / BenchFlow)

Status

Design/alignment; implementation branch pending the RFC 008 base decision.

PostTrain/BenchFlow is ready to test an early branch. The draft RFC 008 stack (#942–#958) now proposes the shared adapter/planner/executor/report foundation, so this issue should implement the missing one-shot task-package lane rather than a second validation core. Reviews are underway on the author-facing contracts in #947 and #952.

Problem

PostTrain Arena uses containerized task packages rather than served environments. Its starting-kit package contains a task.md, an environment/, a verifier, and an oracle/solve.sh. The agent runs to completion, then the verifier reads final artifacts and produces a reward.

OpenEnv environments use a different lifecycle: a long-lived server is driven through reset / step / state, while agents interact through MCP and reward is returned by the environment.

flowchart LR
    subgraph Served["OpenEnv served lifecycle"]
      O["Trainer / orchestrator"] -->|"reset · step · state"| E["Long-lived environment"]
      A["Agent"] -->|"MCP tools"| E
      E -->|"Observation + reward"| O
    end

    subgraph OneShot["PostTrain one-shot lifecycle"]
      P["Task package"] --> R["Agent runs to completion"]
      R --> V["Verifier reads artifacts"]
      V --> W["Reward / evidence"]
    end

    Served -. "shared planning/reporting, different runner" .-> OneShot

Neither lifecycle should pretend to be the other. The missing capability is an authoring and validation path for the one-shot package.

Current direction

RFC 008 proposes the shared, spec-neutral pieces:

flowchart LR
    S["Trusted spec adapter"] --> N["Normalized subject + requirements"]
    N --> P["Policy + planner"]
    P --> X["Execution-model-specific runner"]
    X --> R["Shared validation report"]

    X -->|"served"| SV["OpenEnv runtime checks"]
    X -->|"one_shot"| OS["This issue"]

If that architecture is accepted, this issue owns the PostTrain one-shot adapter, policy, runner, CLI surface, and conformance fixtures. It does not own the generic validation planner/executor/report core.

Proposed v1 scope

Pin an explicit PostTrain starting-kit profile/version rather than claiming broad BenchFlow compatibility.
Add a trusted task.md adapter and structural checks for frontmatter, prompt, environment, verifier, oracle, and required paths.
Emit RFC 008 shared reports with stable diagnostics and certified: false for local author validation.
Add a separate task-oriented CLI (openenv validate-task or openenv task validate; spelling still to be decided).
Add representative valid, structurally invalid, and execution-invalid fixtures supplied with PostTrain/BenchFlow.
Add a one-shot Docker runner only after the lifecycle, reward-boundary, and isolation policy is agreed.
Validate oracle and empty-trial behavior with policy-defined thresholds/margins, not brittle exact reward == 1.0 assertions.

Suggested implementation sequence:

flowchart TD
    D["Decide base + one-shot policy ownership"] --> A["PR 1: task.md adapter + structural checks"]
    A --> F["PR 2: shared diagnostics + conformance fixtures"]
    F --> R["PR 3: isolated one-shot runner"]
    R --> C["PR 4: task CLI + docs"]

Explicit non-goals for v1

Reimplementing or wrapping served OpenEnv validation; openenv validate remains the served-environment command.
Public plugin discovery in the first slice.
Official certification of external task packages.
Claiming full generic BenchFlow support from the smaller PostTrain starting-kit checker.
Running hosted-verifier, GPU/TPU-only, multi-agent, or simulated-user packages locally without a defined runner and policy.

Decisions still needed

Base: branch from CLI: validation profiles and shared reports #947 after its CLI semantics are unified, or from a restacked surface that includes Validation: add structured author diagnostics and remediation #952 diagnostics without pulling unrelated remote-provider work.
Policy ownership: add the one-shot lifecycle/reward/isolation contract to RFC 008, or create a focused follow-up RFC.
Profile/version: which PostTrain starting-kit commit and which generic BenchFlow release, if any, are initial conformance targets.
Dependency strategy: pin/import supported BenchFlow APIs versus maintaining a small profile-specific checker with provenance tests.
Fixtures: representative good, structurally bad, and runtime-bad packages from PostTrain Arena.
CLI spelling: validate-task versus task validate.

Acceptance criteria for the first branch

A pinned valid PostTrain fixture passes structural validation.
Invalid fixtures produce stable, repository-relative diagnostics and actionable remediation.
The adapter performs safe structural inspection only; it does not import or execute submitted code.
Reports use the shared schema and never claim certification.
No task package is launched through the served Uvicorn path.
The implementation and docs state the exact profile/version being validated.

Related work

RFC 008: Spec-neutral local and remote validation architecture #942: draft RFC 008 validation architecture and stacked implementation.
[feature] Import external envs (ors, verifiers) #726: importer registry precedent.
envs/tbench2_env: nearby one-shot/pytest task shape, not a drop-in validator.
Harbor/ATIF/ACP trajectory and agent interoperability is a separate follow-up boundary, not part of this v1.

@thegovind

Hi @thegovind! This is Amy from the BenchFlow community. I think a separate validate-env command would be better here, to avoid conflicting with what the existing validate command already does.

@AmyTao

Direction update after reviewing RFC 008 (#942) and its implementation stack:

@AmyTao's suggestion is right in substance: task-package validation should have a separate command instead of extending the existing openenv validate flow.
RFC 008 now provides the spec-neutral adapter, planner, executor, and report foundation, and explicitly reserves one-shot task packages for this issue.
This issue should therefore narrow to the missing PostTrain/BenchFlow lane: the task.md adapter and structural checks, a one-shot Docker oracle/empty-trial runner, a separate task CLI, docs, and one end-to-end example.
The task command should not wrap OpenEnv-native validation; served OpenEnv environments remain under openenv validate. The exact task command can be settled as openenv validate-task versus openenv task validate. I would avoid validate-env, because that is too close to the existing environment command.
Public plugins can support local author validation, but official certification must use pinned trusted adapters.

I have proposed to @zkwentz that I own this focused follow-up rather than opening a competing validation-core PR. The likely implementation sequence is:

PostTrain/BenchFlow adapter plus structural policy.
One-shot local runner and shared report integration.
Task CLI, plugin/docs surface, and example package.

I am holding off on a branch from main. If we develop before #942–#947 land, the clean base is #947 (rfc008-06-cli); otherwise this should start from updated main after that foundation merges.

@thegovind

@thegovind Hi Govind! Could you point me to the branch you are working on for the implementation? I'd like to test on the posttrain arena side! thx

Hi Amy - thanks, and sorry for the delay. There is not an implementation branch yet; I deliberately held off because #898 now depends on the draft RFC 008 stack, and I do not want to create a second validation core in parallel.

I have now reviewed the other two isses Ben prioritized (#947 and #952) and left contract issues for Zach. My proposed first WIP slice is still the PostTrain task.md adapter plus structural checks off the smallest stable core base. the Docker oracle/empty-trial runner would follow after we agree on the one-shot reward and isolation policy. I will link the clearly labeled WIP branch here once Zach confirms whether that base is #947 as-is or a restacked version that includes the diagnostics contract.

To make the first slice useful on your side, could you share 2–3 representative fixtures: one valid PostTrain package, one structurally invalid package, and one that passes structure but should fail oracle/empty-trial validation? It would also help to know the starting-kit commit/profile you want us to pin for the initial conformance target.

Add openenv.validation contracts: RFC 008 slice 0 (#778, #898)

Add validation registries, report, and severity policy: RFC 008 slice…

Add openenv.validation manifest contracts: RFC 008 slice 0a (#778, #898)

Add validation registries, report, and severity policy: RFC 008 slice…

Add openenv.validation manifest contracts: RFC 008 slice 0a (#778, #898)

Add validation registries, report, and severity policy: RFC 008 slice…

Add openenv.validation manifest contracts: RFC 008 slice 0a (#778, #898)

Add validation registries, report, and severity policy: RFC 008 slice…

Checking in on this — status update from the PostTrain Arena side, since our half of the equation is now fully shipped and public:

The OpenEnv protocol adapter has been live in benchflow-ai/posttrainarena since July 10 (PR restructure env server and client code for coding env #6), with end-to-end smoke evidence on native task packages (PR move echo server README to top #10) — FastAPI/WebSocket server + typed client delegating to the task/runtime/verifier engine.
As of this week the full measurement stack behind it is also public and runnable: the attribution pipeline (paired arms, bootstrap CIs, per-domain caps), a sandboxed local replay for task packages, and an experiment ledger with our methodology and numbers (docs/sft-attribution-ledger.md, docs/running-the-attribution-pipeline.md).
Our scripts/check_task.py / check_submission.py already implement the one-shot task-package validation lane discussed above and run secrets-free in fork CI — happy to contribute them as a starting point for validate-task, or to test an implementation branch against our task corpus the moment one exists (per the earlier offer in this thread).

Is there an updated timeline for the RFC 008 base (#1041)? We'd like to align the first validate-task slice with our submission-intake window this fall.

@xdotli

@xdotli thanks for the check-in, and glad the PostTrain side is shipped!

Contracts landed a few weeks back. First end-to-end use of them went up in the last 24h: #1091 (openenv validate walking skeleton).

Let's pick this up at committee today so we can line the validate-task slice up with your fall intake.

Add openenv.validation manifest contracts: RFC 008 slice 0a (#778, #898)

Add validation registries, report, and severity policy: RFC 008 slice…

Add openenv.validation manifest contracts: RFC 008 slice 0a (#778, #898)

Add RFC 008: environment auto-validation (openenv#778)

RFC 008 slice 0a: core types, normalized manifest, and fixtures (#1044)

RFC 008 slice 0b: registries, report schema, and severity policy (#1045)

Now the shared validation foundation is on main, what pinned PostTrain profile and one-shot policy should the first slice target?

Navigation Menu

Repository navigation

Spec-agnostic task validation: openenv validate-task for external task.md packages (PostTrain Arena / BenchFlow) #898

Status

Problem

Current direction

Proposed v1 scope

Explicit non-goals for v1

Decisions still needed

Acceptance criteria for the first branch

Related work

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Spec-agnostic task validation: openenv validate-task for external task.md packages (PostTrain Arena / BenchFlow) #898

Description

Status

Problem

Current direction

Proposed v1 scope

Explicit non-goals for v1

Decisions still needed

Acceptance criteria for the first branch

Related work

Activity

AmyTao commented on Jul 6, 2026

thegovind commented on Jul 13, 2026

AmyTao commented on Jul 16, 2026

thegovind commented on Jul 20, 2026

xdotli commented on Aug 26, 2026

zkwentz commented on Aug 26, 2026

burtenshaw commented on Oct 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions