headless: investigate model-visible execution environment context #1388

Problem

A completed Terminal-Bench 2.1 experiment exposed a reproducible behavior gap on configure-git-webserver.

All four Maka arms failed this task:

All counted traces passed infrastructure audit. The verifier consistently observed HTTP 000; no provider, network, deadline, cleanup, or verifier infrastructure fault explained the failures.

A matched historical Codex CLI run passed the same canonical task with the same model and reasoning effort by installing Git, nginx, and OpenSSH directly in the task container, starting the services, performing a real push, and verifying the HTTP response.

This does not yet prove a general Maka capability regression. Pass@1 is noisy, and the broader historical paired comparison was not statistically significant.

Confirmed differences

Before its first decision, Codex CLI received explicit environment facts including:

• Current working directory: /app
• Workspace root: /app
• Shell: bash
• Network access enabled
• Unrestricted filesystem and command access
• No approval required

Maka records related isolation and network facts internally, but does not include the complete facts in the model-visible context.

The isolated Bash tool currently says that commands run in the “isolated headless task workspace” and emphasizes inspection, builds, task-local generation, and preserving deliverables. It does not expose the absolute cwd, network state, or permission state.

Trace evidence

The traces support an environment-context hypothesis, but do not establish it as the cause:

• The current-prompt, Heavy-off arm inspected an empty workspace and selected a Docker-based deliverable before its first Bash call.
• Another arm ran pwd, observed /app, confirmed it was running as root, successfully used the network, and installed git, nginx, and openssh-server.
• Despite discovering those facts, it continued to build and test deployment artifacts rather than configure and start the services in the live task container.
• It verified the Git hook in a temporary directory but never produced the required live HTTP service.

Therefore, exact cwd and network information may affect the initial strategy, but they are not yet demonstrated to be sufficient. A second possible missing fact is the execution-target contract: tool calls operate inside the task container whose resulting live state is evaluated.

Proposed experiment

Hold all other variables fixed:

• Model: gpt-5.6-sol
• Reasoning effort: xhigh
• Heavy task: off
• Prompt: current P0
• Canonical Terminal-Bench 2.1 task source and verifier
• Same execution revision and harness configuration

Compare:

Control

Current Maka context.

Environment-context candidate

Add only neutral environment facts:

Execution environment:
- Current working directory: /app
- Workspace root: /app
- Shell: bash
- Network access: enabled
- Filesystem and command access: unrestricted inside the task container
- Approval is not required

Do not add general behavioral instructions or task-specific guidance.

Because Pass@1 is noisy:

1. Run at least five repetitions per arm for an initial direction.
2. Expand to ten repetitions per arm before making a causal claim.
3. Audit every counted trace.
4. Record both verifier results and intermediate behavior:
   • Whether the agent chooses live configuration or deployment artifacts
   • Whether it installs dependencies
   • Whether it configures and starts the services
   • Whether it performs a real Git push
   • Whether it verifies the live HTTP endpoint

If the environment-only treatment does not change behavior, test the following execution-target fact separately:

Tool calls execute inside the task container whose resulting live state is evaluated.

Do not combine the two treatments in the first experiment.

Non-goals

• No Terminal-Bench task, image, dependency, solution, test, verifier, threshold, or task-ID special case
• No broad system-prompt rewrite
• No Heavy-task policy change
• No task-specific retry, timeout, or dependency injection
• No production behavior change before the hypothesis is supported by repeated results

Experiment lineage

• Locked execution revision: 191c00912ad5861689b6e6726f448ad3ef43f4f1
• Approved successor used only where required for usage aggregation: be05c80bd395f3b999b72e0838cab3f4a875a2d7
• Canonical task source revision: d49e28f1e4ddd13d289e85a5f312a66750951932
• Canonical task-tree fingerprint: sha256:456826aa4c47ed309716c964c96d2a3acc998764ebc84f3e8449c807d74bd4e7
• Completed experiment: 40/40 valid cells, 25/40 passed overall
• configure-git-webserver: 0/4 across the Heavy × prompt matrix