Skip to content

Add an Antigravity scenario driver and record its baseline (refs #3748) - #3778

Merged
malibio merged 4 commits into
mainfrom
issue-3748-antigravity-harness
Oct 13, 2026
Merged

malibio merged 4 commits into
mainfrom
issue-3748-antigravity-harness

Conversation

@malibio

@malibio malibio commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

What

  • scripts/eval/harness/antigravity.ts runs the harness scenario in one Antigravity conversation with what the installer writes today (the skill and the instructions block). run.ts and trace.ts know it.
  • antigravity-approve.ts is the scenario's permission rule: Antigravity in print mode cannot prompt, so a PreToolUse hook in the scratch checkout approves a short list of commands and leaves the rest to be denied. It is not --dangerously-skip-permissions.

Baseline (Antigravity CLI 1.3.4, live run)

  • Probe: named the project from the workspace folder's name, said "unknown" for the skills.
  • A skill edited mid-session: not noticed; asked whether anything changed, it answered "No" from its earlier read.
  • Task finished elsewhere: not reached (the session stopped to ask the user to choose a task, as the procedure says), so not measured.

Hook support

Antigravity has hooks (SessionStart, PreInvocation, PreToolUse, PostToolUse, Stop) and ANTIGRAVITY_CONVERSATION_ID reaches the agent's commands. The recommendation, with the evidence, is on #3748 and filed as #3777.

Not in this PR

Pi (no ready credentials, Node 22.14 crashes it); the results table is posted on the issue once Pi is in.

Tests

bun run test:scripts passes (1490). The Rust tiers could not build in this fresh worktree (the cold llama.cpp C++ build fails against this machine's OpenSSL headers); the diff has no Rust, and the merge gate builds in the warm gate checkout.

🤖 Generated with Claude Code

@malibio

malibio commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

Code Review Summary

Verdict: request changes (posted as a comment review because the PR is self-authored).

The driver itself (antigravity.ts, the run.ts wiring, the baseline) is a net positive and covers the Antigravity acceptance criteria on #3748: a driver exists and the scenario ran with what the installer writes today; hook support was checked and the recommendation with its evidence is in #3777. That evidence lives in #3777, not on #3748 itself (the PR text says "on #3748"), so either post a short pointer comment there or reword. The Sync-boundary and node-type-literal checkers pass, no .md files were added, and no Rust is touched.

Two things need fixing before merge: the approver's allowlist can be bypassed in ways its doc comment implies it cannot, and the new trace.ts branch does not match the events Antigravity actually emits (against the committed run log it prints nothing useful). I ran both against /tmp/nodespace-harness-scenario/logs/antigravity.jsonl, feeding the approver real hook payloads.

Findings

Critical Issues

  • scripts/eval/harness/antigravity-approve.ts:27-34,50 the line splitter is bypassable, so an "approved" line can run anything. Verified:

    • echo a & id is auto_approve: a single & is not a split point, so id is never checked.
    • echo "a'b" && id && echo "c'd" is auto_approve: withoutLiterals only knows single quotes, so the ' inside the double-quoted strings pairs across && id && and erases it.
    • echo \' ; id ; echo \' is auto_approve: an escaped quote is treated as opening a literal.
    • echo hi > ~/.zshrc, cat f >> /etc/hosts, sort -o /etc/x f and sed -i s/a/b/ ~/.zshrc are approved: redirection and output-file flags are not looked at.
    • git -c core.pager="sh -c id" log, git -c alias.x="!id" x, git config alias.x "!id" and git push --receive-pack=... are approved: git is allowed with any arguments.
    • echo hi <(id) is approved (process substitution is not in the $(/backtick check).

    Principle: a security filter that parses a shell with regexes fails open on every construct it does not model (KISS, fail closed). More regexes will not fix it. Either (a) write a real tokenizer that handles both quote kinds, backslash escapes, &, redirection and process substitution, and answers ask on anything it does not fully understand, or (b) drop the pretence of containment (next item) and keep only what the scenario needs.

  • antigravity-approve.ts:1-15,21-24 the doc comment is not honest about what this is. The list contains sh and node, so sh -c "<anything>" and node -e "<anything>" are approved by design (confirmed auto_approve), and sed/awk can write files or call system(). For a scratch fixture on a disposable checkout that is an acceptable trade, but "it is not a bypass ... this approves a short list" overstates it: the effective permission is arbitrary code as the user, so long as the line starts with an allowed word. Say so plainly (a fixture convenience that keeps the model from wandering, not a sandbox), and drop sh, sed, awk if unneeded. The scenario transcript used only nodespace, git, node test.js, ls, head, cat.

  • antigravity-approve.ts:43-45 does not fail closed on bad input, and fails open on an unexpected shape. JSON.parse on garbage or empty stdin throws uncaught (exit 1, stack trace on stderr, nothing on stdout), and Give Antigravity a NodeSpace plugin: hooks for project, skill list and refusal #3777's own evidence says PreToolUse must answer with a decision. Separately, input that parses but lacks toolCall.name === "run_command" ({}, or a renamed field in a later Antigravity) answers {"decision":"allow"}. Wrap the parse in try/catch and answer ask for anything not recognised. The matcher: "run_command" in hooks.json already makes the non-shell allow branch unreachable, so it can go.

  • antigravity-approve.ts:33 splitting on \n inside quotes denied the scenario's own commit (observed in the run). Step 69 of the log, git add slug.js test.js && git commit -m "widget: collapse repeated dashes in slugify\n\nChecked-by: osprey", was split into pieces including Checked-by: osprey", whose first word is not allowed, so it got ask and was denied ("user denied permission to run command"). The commit trailer is exactly what the scenario's standard is about, so the last turn ended with response: "" and denied_actions: [{"action":"command","display_name":"RunCommand"}]. The same splitter also wrongly denies echo "a;b". A quote-aware tokenizer fixes both directions.

  • scripts/eval/harness/trace.ts (new Antigravity branch) does not match the events in the run log. Running bun scripts/eval/harness/trace.ts /tmp/nodespace-harness-scenario/logs/antigravity.jsonl:

    • Agent text streams as many ACTIVE agent_response steps carrying text_delta; only the final DONE step has the last fragment. The branch requires state === "DONE", so SAYS prints only the tail, for example SAYS ace path /private/tmp/.../widget) ... and SAYS (empty). Use the result event's response, or accumulate deltas per step_index.
    • Tool steps are step_type: "tool" with tool_name and tool_info.parameters / tool_info.output / tool_info.error. The branch reads step.tool_call ?? step.command, which do not exist, so every tool prints STEP tool "": no command and no result, which defeats the file's stated purpose.
    • state === "ERROR" tool steps (the denied commit) are not printed at all.

Suggested Improvements

  • antigravity.ts:78-101 ignores result.status and result.denied_actions. Exit 0 with a denied_actions entry and an empty response returned "" silently (the last turn of the committed run). Treat a non-SUCCESS status as an error and surface a non-empty denied_actions: a denial means the measurement is about the approver's list, not the agent.
  • antigravity.ts:48-54 the agent can disable the approver. .agents/hooks.json is inside the checkout and --mode accept-edits lets the agent edit it; file-edit tools also never pass through the run_command hook. Fine for the measurement, but one more reason the comment must not imply a boundary.
  • antigravity-approve.ts has no tests, though it makes the one security-relevant decision here, and scenario.test.ts shows the harness tests its pure logic. Top-level await Bun.stdin.text() makes it untestable as written: export decide(line), guard the stdin read with import.meta.main, and pin the bypasses above (and confirm the tests fail against the current regexes).
  • antigravity.ts:54 appends .agents to .git/info/exclude on every run without checking includes, and assumes .git is a directory. Fine for the fixture; trivially idempotent.
  • antigravity.ts:67 the -p=<prompt> form works for the prompts used (the live run passed multi-line prompts). Whether a prompt containing = or a leading - parses was not exercised; the comment should not claim more than was tried.

Nitpicks

  • Nit: trace.ts comment says a step is "the user input, the agent's text, or a tool call", but the log also has system_message steps, which print STEP system_message "".
  • Nit: if the allowlist shrinks as suggested, a comment per word on why the scenario needs it would document the list.

Checked and fine

  • run.ts wiring and its usage comment.
  • NODESPACE_WRITE_JOURNAL removed from the child env consistently with the Codex driver; real-home use is documented and gated by an install check with a clear error.
  • Conversation continuation: the id is taken from the init event once and passed with --conversation; the committed run shows one conversation_id across all four prompts.
  • scripts/check-sync-boundary.ts and scripts/check-node-type-literals.ts pass on the branch.

@malibio malibio left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES (posted as a comment review because the PR is self-authored). Details are in the summary comment above.

Blockers:

  1. antigravity-approve.ts is bypassable (single &, a quote inside double quotes, an escaped quote, redirection, git -c), and its doc comment overstates it while sh and node are allowed.
  2. It does not fail closed on non-JSON stdin and answers allow on an unexpected payload shape.
  3. Its newline split denied the scenario's own multi-line commit in the committed run.
  4. The trace.ts Antigravity branch prints truncated agent text and empty tool steps against the real event shapes.

The driver, the run.ts wiring and the baseline are otherwise sound.

@malibio

malibio commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

Review addressed (51209f7)

Addressed

  • The approver is a tokenizer that fails closed (words(), decide()), not regexes: a lone &, $, backticks, </> other than discarding output, (, {, !, a backslash, a quote inside a double quote, a path or a VAR=value before the command, git -c/config/--output/--exec-path, and node/sh with -e/-c all ask. Unreadable stdin and a payload with no tool name ask. The tests pin every bypass the review listed, and the quoted multi-line commit message and echo "a;b" that were wrongly denied.
  • The doc comment says it is not a sandbox: sh <file> and node <file> run files the agent writes, so the rule narrows what a fixture run reaches for and is not a boundary. sed, awk, sort and diff are dropped from the list.
  • trace.ts reads the real step shapes (text deltas joined per step, tool parameters, output and error); run on the log, it prints the commands and their results.
  • The driver throws on a turn whose result.status is not SUCCESS and appends a line saying when the permission system denied an action; the .git/info/exclude entry is idempotent.
  • The pointer: the Antigravity column and the recommendation are on Give Codex a NodeSpace plugin, and finish the harness scenario on Codex, Pi and Antigravity #3748 (comment below).

Not changed

  • The agent can edit .agents/hooks.json and file-edit tools do not pass through the hook. Said in the driver and the approver's comment: the rule is a fixture rule.
  • -p=<prompt> with = or a leading dash: the = form is what Antigravity's flag parser needs for a prompt that follows other flags, and it takes the whole remainder as the value; the scenario's four prompts are the only input.

Re-Review Decision

Decision: RE-REVIEW NEEDED

Rationale: The approver is security-relevant code and was rewritten, not patched; the tokenizer should be read once by someone who did not write it.

@malibio

malibio commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

Re-review of 51209f7 (git diff 181cafe31 51209f7e4)

Summary

The rewrite is a real improvement: the regex filter is gone, the tokenizer fails closed on everything it does not model, and every bypass from the first review now answers ask (I re-ran them, plus about 150 more, through decide). The earlier doc-comment, fail-closed-input, trace and driver findings are addressed. One blocker remains that is about behaviour, not escape: replaying the committed run's own commands through the new approver denies three of its 23 distinct run_command lines, and they are the scenario's central writes. The rest is Improvement-level.

Tests: bun test scripts/eval/harness/antigravity-approve.test.ts passes (8 tests, 48 expects). trace.ts on /tmp/nodespace-harness-scenario/logs/antigravity.jsonl prints full agent text per step, TOOL run_command <CommandLine> and -> <output> lines (71 lines, 60 tool steps, 1 ERROR step).

Status of the earlier findings

Finding Status
Regex approver bypassable (&, quote pairing, \', redirection, git -c, <() Fixed. All listed lines answer ask.
Doc comment overstates containment Fixed in substance, see Improvement 3 for what it still omits.
Non-JSON stdin throws; unknown shape answers allow Fixed for stdin (try/catch to ask) and for a missing name. A present non-run_command name still answers allow, unreachable behind matcher: "run_command". Nit.
Newline split denied the scenario's own commit Fixed (quoted multi-line commit message and echo "a;b" are pinned).
trace.ts truncated text, empty tool steps, ERROR not printed Fixed, verified against the real log.
antigravity.ts ignored status / denied_actions Fixed: throws on non-SUCCESS, appends a denial line.
Approver untested Fixed, with the gaps noted below.
.git/info/exclude not idempotent Fixed.
-p=<prompt> with = / leading - Skipped, acceptable: the four scenario prompts are the only input.
Agent can edit .agents/hooks.json Skipped, acceptable, and now stated in code.

Findings

Critical Issues

None that let a hostile line through the tokenizer. Adversarial results are below.

Blocker (functionality)

  • antigravity-approve.ts words() double-quote branch rejects backslash and $ outright, and that denies the scenario's own writes. Replaying the 23 distinct run_command lines of the committed log (the pre-fix run) through decide, three answer ask:

    • nodespace node update <id> --content "- [x] \slugify("a -- b")` returns `a-b`" --version 1`
    • nodespace node create --type text --parent <id> --content "Test: ... \assert...` Failing run:\n$ node test.js ..."(escaped backticks and a literal$ `)
    • nodespace node update <id> --content "- [x] \node test.js` passes ..." --version 1`

    These are the checkbox ticks and the evidence node, exactly the work the "Implementing a Task" skill asks for. In print mode they are denied, so a rerun against the new approver will show the same shape of failure that the first review flagged for the commit message, only on the nodespace writes. The driver now says "permission system denied N action(s)", which is good, but the measurement is then about the approver's list again. The tests pin only simple approvals, so nothing caught it. Fix by modelling the shell's rule instead of rejecting the character: inside double quotes, \ followed by one of $ ` " \ (or newline) is a literal and is consumed with its escape; $ followed by something that cannot start an expansion (space, end of quote, punctuation other than { ( @ * # ? ! $ - and digits/letters/_) is literal in bash and zsh. Anything else stays ask. Add the three lines above to the approve list in the test.

Suggested Improvements

  1. GIT_REFUSED and the git list are narrower than the comment's intent, and mostly moot. Still auto_approve: git commit -F /etc/passwd and --template=<path> (read), git branch -D main / -m (destructive in the fixture), git diff --no-index /etc/passwd x (reads outside), git status --git-dir=/x, git commit --amend with no --no-edit (launches $EDITOR, which hangs print mode), git diff/log --ext-diff and --textconv (run a driver named in .git/config, which the agent can write with its file tools), and git commit runs .git/hooks/* the agent can also write. None are reachable from a hostile line alone, since -c, config and --output are refused, and --out=/--outp= are not accepted abbreviations for git diff (checked), but the list is a pattern-match on flags, not a closed set. Principle (YAGNI, least surface): the scenario's transcript uses status, diff, add <files>, commit -m, log. Drop branch, show, ls-files, rev-parse unless the scenario uses them, or say they are there for convenience.
  2. nodespace is allowed with any arguments. The CLI has session launch (spawns PTY external agents, which then run outside this hook entirely), skill install|uninstall (writes agent config in the real home, which is why NODESPACE_HOME redirection exists), database remove, uninstall, mcp install. uninstall refuses to leave NODESPACE_HOME, so the blast radius is bounded, but session launch is a hole in the whole approach: it is an approved line that starts another agent with its own permissions. Allow nodespace only with the second words the scenario uses (skill guidance|get, node, query, search, schema, relationship, wait), as git already is.
  3. The doc comment is honest about sh <file>/node <file>, but it does not name touch/mkdir writes outside the checkout, reads of any path by cat/head/grep, or nodespace session launch. "Not a boundary" covers it; one sentence listing what an approved line can still do would stop the git-flag list reading as a security feature. The node <file> point is correct: with node ./a.js allowed and file writes unrestricted, the allowlist is moot as containment, and the comment now says so.
  4. Test names overclaim. "asks about a write outside the checkout" is pinned only for >/>>, sort, sed; touch /etc/x and mkdir -p /etc/x answer auto_approve (acceptable per the comment, but then the test title should say "a redirection or a program off the list"). The stdin entry (import.meta.main, JSON.parse failure to ask) is the fail-closed claim in the doc and is not exercised; a Bun.spawn of the script with garbage on stdin is a three-line test.

Nitpicks

  • Nit: decide({toolCall:{name:"anything_else"}}) answers allow. Behind matcher: "run_command" it cannot fire; ask would be the closed default and costs nothing.
  • Nit: tail -f and cat with no file block forever (print mode has no stdin). Same class as git commit --amend.
  • Nit: git log --format=%(trailers) unquoted is rejected by (; quoted is fine. The scenario's own use of --format is quoted in the log, so no action.
  • Nit: trace.ts does not print system_message steps now, which is fine; the old comment nit is resolved.
  • Nit: ls >/dev/null id and echo hi &>/dev/null id are approved with a discarded redirect mid-line; that is correct shell semantics, just worth knowing the tokenizer is not strictly "redirect at the end".

Adversarial run (run through decide, bun)

Answered ask (correct): echo a2>&1b, echo x>/dev/nullid, echo hi & id, echo hi &>/dev/null&id, echo x |& id, echo x |&id, ls ;; id, echo #x\nid (second command seen), echo <<< x, echo {a,b}, -x, =id, VAR=x echo, 2>&1 id, echo hi 1>&2, echo hi >&2, echo hi >/dev/null/../../etc/x, echo a >/dev/null&&id, echo a&>/dev/null, echo 'x' >/dev/null'id', ls >/dev/null"id", echo 2>&1'x', echo "a" followed by a lone ", echo a\<newline>b, echo '"' && id && echo '"', echo "'" ; id ; echo "'", sh -c id, node -e 1, node --eval=1, node -r x a.js, node -, node --, sh -, git -C /tmp status, git log --output=/tmp/x, git diff --output /tmp/x, git log --open-files-in-pager=id.

The HARMLESS_REDIRECTION question: the regex runs before quoting is known, so it can delete a matched token from inside quotes (echo "a 2>&1 b" becomes the word a b). That alters quoted data only. It cannot create or hide a metacharacter: the removed tokens contain no quote, $, backtick or operator, the leading whitespace is preserved by $1, and the lookahead character is not consumed, so a removal can never join two words or move a quote boundary. A 2>&1 is only removed where it follows whitespace and precedes whitespace, ;, &, | or end. I could not construct a joining or quote-shifting input. Unicode whitespace in \s ( ) is removed as a separator by the regex but treated as word content by the tokenizer; the only consequence is a harmless redirect token disappearing next to it. \r, \v, NUL, ESC and 
 stay inside a word and the shell also treats them as word characters, so no differential. =x and ~, *, ?, ^, [1] as plain args are approved; in zsh =cmd expands to a path and ^/* are globs, which only affect arguments of an allowed command.

Approved, by design or by the doc: grep -f /etc/x a, cat -- /etc/passwd (reads), mkdir -p /etc/x, touch /etc/x (create an empty path outside the checkout; permitted by the "not a boundary" framing, but say so), printf -v x y (a bash builtin variable assignment; harmless), test -f, ls, node ./a.js and sh a.js (arbitrary code from a file the agent wrote; the allowlist is not containment, and the comment now says that).

Verdict

REQUEST CHANGES, one fix (the double-quote escape handling): the tokenizer is sound against every hostile input I could build and the earlier findings are closed, but a re-run of the committed scenario would again be denied on its nodespace writes. I would also narrow nodespace to the subcommands the scenario uses (Improvement 2) in the same change, since session launch defeats the rule outright. Everything else is optional.

/address-review re-review needed only for the escape handling in words().

@malibio malibio left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES (posted as a comment review because the PR is self-authored). Details are in the re-review comment above. One blocker: the approver's double-quote branch rejects backslash and $, which denies 3 of the 23 distinct commands in the committed run (the nodespace node update/create writes with escaped backticks). Everything from the first review is otherwise addressed and I could not bypass the tokenizer. Also recommended: narrow nodespace to the subcommands the scenario uses (session launch starts an agent outside the hook).

@malibio

malibio commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

Re-review addressed (12aedce)

Addressed

  • Blocker: inside double quotes an escaped $, backtick, " or \ is a literal, and a $ that cannot start an expansion is one. The checklist ticks and the evidence node the skill asks for are approved; an unescaped backtick or an expansion ("$HOME", "${X}", "$(id)", "$1", "$$") still asks. Replaying the 23 distinct command lines of the live run through decide: all auto-approved (the replay is not in the repo; the three lines the review named are pinned in the tests).
  • nodespace is approved for node, query, diagnostics and skill guidance|get, after an optional --json. session launch, skill install, database and mcp ask.
  • git: status, diff, log, add, commit; -F, --template, --amend, --no-index, --git-dir, --work-tree, --ext-diff and --textconv ask. show, branch, ls-files and rev-parse are dropped.
  • The doc comment says what an approved line can still do: touch and mkdir can create paths outside the checkout, cat, head and grep can read any path.
  • Tests: the accepted double-quoted forms, the refused expansions, the narrowed nodespace and git forms, and the script run as a process on input that is not JSON (answers ask).

Not changed

  • A named tool other than run_command still answers allow: the hook's matcher is run_command, so it is not reached; the unit documents it.
  • tail -f and cat with no file hang in print mode: agent behaviour, not an approval question.

Re-Review Decision

Decision: RE-REVIEW NEEDED

Rationale: The double-quote branch of the tokenizer is the surface that held the earlier bypasses, and it now accepts more. The fix is the one the review suggested and is pinned by tests in both directions, but it is a permission gate and deserves one more read of that branch alone.

@malibio

malibio commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

Re-review of 12aedce (approver double-quote handling, narrowed nodespace and git)

Verdict: APPROVE, with one one-line fix requested before merge ($[). The blocker from the last review is closed, the earlier findings are closed, and all 14 tests pass (bun test scripts/eval/harness/antigravity-approve.test.ts). Re-review is needed only for the one-line change below.

1. The new double-quote branch

Correct, and I checked each shape against real shell behaviour:

  • \ before a char other than $ ` " \: kept as a literal \, and the next char is read normally in the next iteration. This is what sh does. echo "a\b" approves, correctly.
  • \ at end of line, or before the closing quote with nothing after: unterminated, ask.
  • \+newline inside quotes: sh removes both (line continuation); the reader keeps them as word content inside a quoted word. No command split is possible inside quotes in either reading, so no differential that matters.
  • "a\"b" ; id: the escaped quote does not close the word in the reader or in sh; the line is then unterminated, ask. "\\"; id; echo "\\" asks (the pair \\ is consumed, the quote closes, then id is seen as a command and not on the list).
  • "a\\id" asks (escaped backslash, then a live backtick). "\\$(id)" asks. "\$(id)" approves, correctly a literal.
  • "$", "$ x", "$'x'", "$:", "!!": approved; all literal in sh (ANSI-C quoting and $"..." are not active inside double quotes; history expansion is off non-interactively). "$"id" asks (the quote after $ closes, then id" is unterminated).
  • "${x}", "$(id)", "$x", "$[$(id)]", "$[`id`]": all ask.
  • HARMLESS_REDIRECTION is untouched by the diff and still runs before the reader; I found no new interplay.

One differential: $[. [ is not in the regex's list, but inside double quotes bash (and zsh) evaluate $[expr] as arithmetic, and arithmetic evaluation expands a variable whose value is x[$(cmd)]. printf is on the list and printf -v assigns a variable, so this line answers auto_approve and runs a command that is on no list:

printf -v a 'x[$(touch /private/tmp/.../pwn)]'; echo "$[a]"

I ran that line under bash: the touch executed (and echo printed 0). Unquoted $[1+1] is already refused by UNFOLLOWED, so this is only reachable through the new branch. Fix: add [ to the char class in the double-quote branch (/[A-Za-z0-9_{(\[@*#?!$-]/), and add echo "$[1+1]" to the refused-expansions test. Optionally refuse printf -v as well (printf does not need -v for this scenario).

Severity: the header comment already says the approver is not a containment boundary (sh <file> runs whatever the agent wrote), so this is not a new capability. It is a reader/shell mismatch in the exact branch this commit rewrote, and the fix is one character, so I would take it.

2. nodespace and git

nodespace: --json skill install x asks; nodespace node, --json node get x, node get x approve; bare nodespace, nodespace skill, nodespace --json, --json --json node and -v node x ask. Correct, and session launch/database/mcp are out.

git: git commit -m "x", git add -A, git diff --stat, git commit -am x, git log -n 5 --oneline all approve, so no legitimate use is refused. Two things the prefix match misses:

  • Combined short flags and abbreviations. GIT_REFUSED is a prefix match on the written text, and git accepts both. Verified in a scratch repo: git commit -aF <file> and git commit -a --fil=<file> both ran and read the file into the commit message; -sF x, --templ=, --amen, --git-d= and git diff --textc/--ext also answer auto_approve (--no-i is ambiguous in git, so that one fails harmlessly). git commit -Foo, --file-x and -F alone are refused, as intended; -n is not over-matched (it approves). The pattern is a denylist of spellings, not a closed set, as the first review noted.
  • Impact is small and inside the documented "cat/head can read any path" envelope: a read into a commit message in the fixture repo, --amen hangs on $EDITOR in print mode, and --ext/--textc need a driver the agent could already only configure by writing .git/config. If you want it closed rather than listed: allow-list the git args (-m <msg>, -a, -A, ., --stat, --oneline, -n <N>, paths) instead of denylisting, or refuse any arg starting with - that is not in a small known set. Not blocking.

3. Tests

Both directions are pinned: accepted double-quoted forms (including the three evidence-write shapes), refused expansions, narrowed nodespace, git refusals, and the script run as a process on non-JSON stdin (answers ask). 14 pass, 0 fail, 76 expects. Missing only the $[ case above and a -aF case if you tighten git.

Nitpicks

  • Nit: git commit -m "-F" asks (the quoted word -F matches the prefix). Harmless over-refusal, but it shows the match runs on decoded words, so a commit message beginning with -F or --file would be denied.
  • Nit: decide for a non-run_command tool still answers allow (unchanged, documented behind the matcher).

@malibio malibio left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE with one one-line fix requested (posted as a comment review because the PR is self-authored). The double-quote escape handling is correct, the narrowed nodespace rule works, and the 14 tests pass. One reader/shell mismatch remains: inside double quotes $[expr] is arithmetic, so printf -v a 'x[$(cmd)]'; echo "$[a]" auto-approves and runs cmd (verified under bash). Add [ to the char class in the double-quote branch and a test for it. The git flag denylist also misses -aF and abbreviations (non-blocking). Details in the comment above.

@malibio

malibio commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

Re-review addressed (009e93e)

Addressed

  • $[ inside double quotes (arithmetic expansion, which evaluates a variable's x[$(cmd)] value) now asks: [ is in the character class of what can start an expansion. echo "$[1+1]" and the review's printf -v a 'x[$(id)]'; echo "$[a]" are in the refused-expansions test.
  • printf -v asks, as the review suggested; a printf that only prints is approved and tested.

Not changed

  • Allow-listing git arguments instead of denylisting them: -aF <file> and --fil=<file> read a file into a commit message, which is inside the envelope the header already states (cat, head and grep read any path). Revisit if the approver is ever pointed at anything but the scenario's fixture.
  • git commit -m "-F" asks because the match runs on decoded words: a false deny on a message nothing in the scenario writes.

Tests: 15 for the approver, bun run test:scripts 1505 pass, scripts lint and typecheck clean.

Re-Review Decision

Decision: NO RE-REVIEW NEEDED

Rationale: The review itself said re-review was needed only for the $[ one-liner. This is that one-character change to the character class, the printf -v refusal it suggested, and tests that pin both. No other logic moved.

malibio and others added 4 commits October 13, 2026 12:16
…mode (refs #3748)

Antigravity has no NodeSpace plugin, so the driver runs the scenario with
what the installer writes today: the skill and the instructions block.
Print mode cannot prompt, so the checkout carries a PreToolUse hook that
approves a short list of commands and leaves the rest to be denied; it is
not --dangerously-skip-permissions. trace.ts reads Antigravity's
stream-json steps.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ns said (refs #3748)

- antigravity-approve.ts reads the line with a small tokenizer and asks about
  anything it does not understand: a lone &, $, backticks, redirections other
  than discarding output, a quote inside a double quote, a path or a
  VAR=value before the command, git -c/config/--output, node/sh -e/-c.
  Unreadable input and a payload with no tool name ask. The doc comment says
  it is not a sandbox: sh and node run files the agent writes.
- A quoted multi-line commit message is one word, no longer a false deny.
- Tests pin every bypass the review found.
- trace.ts reads the real step shapes: text deltas joined, tool parameters,
  results and errors.
- The driver fails a turn that does not end in SUCCESS and says when the
  permission system denied an action; the exclude entry is idempotent.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… git (refs #3748)

- Inside double quotes an escaped dollar, backtick, double quote or backslash
  is a literal, and a dollar that cannot start an expansion is one, so the
  checklist ticks and evidence nodes the skill asks for are approved. An
  unescaped backtick or an expansion still asks. Replaying the live run's 23
  distinct command lines: all approved.
- nodespace is approved for node, query, diagnostics and skill guidance|get
  (after an optional --json), not session launch, skill install, database
  or mcp.
- git: status, diff, log, add, commit; not -F, --template, --amend,
  --no-index, --git-dir, --ext-diff, --textconv.
- The doc comment says what an approved line can still do (touch and mkdir
  outside the checkout, reads of any path).
- Tests pin the accepted and refused forms and the script's own stdin path.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rintf -v (refs #3748)

- A dollar followed by an open bracket inside double quotes is arithmetic
  expansion, which evaluates a variable's x[$(cmd)] value; it now asks, as
  the other expansions do.
- printf with -v assigns a variable and asks.
- Tests pin both, and that a printf that only prints is approved.

Skipped: allow-listing git arguments. The denylist is bypassed by combined
short flags and abbreviations (-aF, --fil=), whose impact is reading a file
into a commit message, inside the envelope the header already states (cat,
head and grep read any path). Revisit if the approver is ever pointed at
anything but the scenario's fixture.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@malibio
malibio force-pushed the issue-3748-antigravity-harness branch from 009e93e to fda24db Compare October 13, 2026 10:16
@malibio
malibio merged commit ab6ced1 into main Oct 13, 2026
@malibio
malibio deleted the issue-3748-antigravity-harness branch October 13, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant