Skip to content

Keep lean -p as the one-shot default (measured; KIV the graph surface) #568

Description

@justrach

KIV. Lean -p (the default one-shot) stays the default. Do not leave --no-lean on for one-shot coding evals. The named DAG (workflow) is still the right tool when work actually splits — use it there, once the split is large enough to amortize a worker prefix.

This is a parking note, not an implementation ticket.

What we measured

Graph here means the tool-call graph: which tools are in the catalog, whether workflow is advertised, and how a turn batches calls. runTools already fans a turn's tool_calls across the Io pool. Lean keeps eight tools and drops workflow; full catalog is ~4.3k extra schema tokens per model turn.

Three arms on graff-evals, grok-4.6, 1 rep:

arm catalog nudge
graff lean (default -p --yolo) none
graff-full --no-lean none
graff-graph --no-lean short stop-rule: if work splits, one workflow (phase 1 = parallel task per piece, phase 2 = one verify); sequential work stays on the root

Sequential suite (12 tasks)

arm pass wall in tokens out tokens
graff (lean) 12/12 152.2s 160813 6300
graff-full 12/12 168.2s 337775 6945

Same pass rate. Full catalog spent 2.10× input and ~10% more wall. exact-reply (no tools): 4086 vs 8338. The stop rule predicted this suite would lose, and it did.

Fan-out slice (2 tasks)

Independent files that can split: split-three-bugs (three one-line module bugs + a verify script) and split-three-csvs (three ledgers → one totals file).

arm pass wall in tokens out tokens
graff (lean) 2/2 30.2s 31958 1152
graff-full 2/2 32.4s 64629 1355
graff-graph 2/2 49.5s 91644 2666
task arm wall in calls used workflow?
split-three-bugs graff 17.46s 18511 4 no
split-three-bugs graff-full 20.51s 38128 4 no
split-three-bugs graff-graph 34.99s 64099 14 yes
split-three-csvs graff 12.75s 13447 3 no
split-three-csvs graff-full 11.86s 26501 3 no
split-three-csvs graff-graph 14.49s 27545 3 no

What the traces show

  • Lean already fans out inside a turn. Bugs: batched reads, then three edit_file, then the verify command. CSVs: batched reads, then one write. No DAG needed.
  • --no-lean without a nudge never called workflow. It swapped native reads for an installed MCP batch-read and paid ~2× input for the same call count. Catalog ≠ graph use.
  • The nudge produced a real diamond on bugs: one workflow, phase 1 = three parallel fixers, phase 2 = one verify. That is the intended shape.
  • The diamond still lost. Each worker opened ~25k context (lean's whole run was 18.5k). 14 API calls vs 4. Wall 35s vs 17s. The fixers overlapped, but spawn + per-worker prefix + a verify agent lost to three one-line edits on the root.
  • CSVs correctly skipped the DAG even with the nudge. Stop rule held; cost tracked the fat-catalog arm.

Decision

Keep lean as the one-shot default. Use workflow when work actually splits and the pieces are large enough that the root would otherwise serialize fat reads — or when workers can inherit the lean catalog instead of the full one.

Follow-up (only if the sign might flip)

  • Independent pieces large enough that the root cannot cheaply batch them.
  • Workers that inherit lean rather than the full catalog.
  • Re-run the fan-out slice after either change.

Do not prepend third-party skill dumps or knowledge-graph prompts. That is not this harness.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationstatus:backlogWork item has not been started

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions