Repository navigation
Add an Antigravity scenario driver and record its baseline (refs #3748) - #3778
Conversation
Code Review SummaryVerdict: request changes (posted as a comment review because the PR is self-authored). The driver itself ( Two things need fixing before merge: the approver's allowlist can be bypassed in ways its doc comment implies it cannot, and the new FindingsCritical Issues
Suggested Improvements
Nitpicks
Checked and fine
|
malibio
left a comment
There was a problem hiding this comment.
Verdict: REQUEST CHANGES (posted as a comment review because the PR is self-authored). Details are in the summary comment above.
Blockers:
antigravity-approve.tsis bypassable (single&, a quote inside double quotes, an escaped quote, redirection,git -c), and its doc comment overstates it whileshandnodeare allowed.- It does not fail closed on non-JSON stdin and answers
allowon an unexpected payload shape. - Its newline split denied the scenario's own multi-line commit in the committed run.
- The
trace.tsAntigravity branch prints truncated agent text and empty tool steps against the real event shapes.
The driver, the run.ts wiring and the baseline are otherwise sound.
Review addressed (51209f7)Addressed
Not changed
Re-Review DecisionDecision: RE-REVIEW NEEDED Rationale: The approver is security-relevant code and was rewritten, not patched; the tokenizer should be read once by someone who did not write it. |
Re-review of 51209f7 (
|
| Finding | Status |
|---|---|
Regex approver bypassable (&, quote pairing, \', redirection, git -c, <() |
Fixed. All listed lines answer ask. |
| Doc comment overstates containment | Fixed in substance, see Improvement 3 for what it still omits. |
Non-JSON stdin throws; unknown shape answers allow |
Fixed for stdin (try/catch to ask) and for a missing name. A present non-run_command name still answers allow, unreachable behind matcher: "run_command". Nit. |
| Newline split denied the scenario's own commit | Fixed (quoted multi-line commit message and echo "a;b" are pinned). |
trace.ts truncated text, empty tool steps, ERROR not printed |
Fixed, verified against the real log. |
antigravity.ts ignored status / denied_actions |
Fixed: throws on non-SUCCESS, appends a denial line. |
| Approver untested | Fixed, with the gaps noted below. |
.git/info/exclude not idempotent |
Fixed. |
-p=<prompt> with = / leading - |
Skipped, acceptable: the four scenario prompts are the only input. |
Agent can edit .agents/hooks.json |
Skipped, acceptable, and now stated in code. |
Findings
Critical Issues
None that let a hostile line through the tokenizer. Adversarial results are below.
Blocker (functionality)
-
antigravity-approve.tswords()double-quote branch rejects backslash and$outright, and that denies the scenario's own writes. Replaying the 23 distinctrun_commandlines of the committed log (the pre-fix run) throughdecide, three answerask:nodespace node update <id> --content "- [x] \slugify("a -- b")` returns `a-b`" --version 1`nodespace node create --type text --parent <id> --content "Test: ... \assert...` Failing run:\n$ node test.js ..."(escaped backticks and a literal$ `)nodespace node update <id> --content "- [x] \node test.js` passes ..." --version 1`
These are the checkbox ticks and the evidence node, exactly the work the "Implementing a Task" skill asks for. In print mode they are denied, so a rerun against the new approver will show the same shape of failure that the first review flagged for the commit message, only on the nodespace writes. The driver now says "permission system denied N action(s)", which is good, but the measurement is then about the approver's list again. The tests pin only simple approvals, so nothing caught it. Fix by modelling the shell's rule instead of rejecting the character: inside double quotes,
\followed by one of$ ` " \(or newline) is a literal and is consumed with its escape;$followed by something that cannot start an expansion (space, end of quote, punctuation other than{ ( @ * # ? ! $ -and digits/letters/_) is literal in bash and zsh. Anything else staysask. Add the three lines above to the approve list in the test.
Suggested Improvements
GIT_REFUSEDand the git list are narrower than the comment's intent, and mostly moot. Stillauto_approve:git commit -F /etc/passwdand--template=<path>(read),git branch -D main/-m(destructive in the fixture),git diff --no-index /etc/passwd x(reads outside),git status --git-dir=/x,git commit --amendwith no--no-edit(launches$EDITOR, which hangs print mode),git diff/log --ext-diffand--textconv(run a driver named in.git/config, which the agent can write with its file tools), andgit commitruns.git/hooks/*the agent can also write. None are reachable from a hostile line alone, since-c,configand--outputare refused, and--out=/--outp=are not accepted abbreviations forgit diff(checked), but the list is a pattern-match on flags, not a closed set. Principle (YAGNI, least surface): the scenario's transcript usesstatus,diff,add <files>,commit -m,log. Dropbranch,show,ls-files,rev-parseunless the scenario uses them, or say they are there for convenience.nodespaceis allowed with any arguments. The CLI hassession launch(spawns PTY external agents, which then run outside this hook entirely),skill install|uninstall(writes agent config in the real home, which is whyNODESPACE_HOMEredirection exists),database remove,uninstall,mcp install.uninstallrefuses to leaveNODESPACE_HOME, so the blast radius is bounded, butsession launchis a hole in the whole approach: it is an approved line that starts another agent with its own permissions. Allownodespaceonly with the second words the scenario uses (skill guidance|get,node,query,search,schema,relationship,wait), as git already is.- The doc comment is honest about
sh <file>/node <file>, but it does not nametouch/mkdirwrites outside the checkout, reads of any path bycat/head/grep, ornodespace session launch. "Not a boundary" covers it; one sentence listing what an approved line can still do would stop the git-flag list reading as a security feature. Thenode <file>point is correct: withnode ./a.jsallowed and file writes unrestricted, the allowlist is moot as containment, and the comment now says so. - Test names overclaim. "asks about a write outside the checkout" is pinned only for
>/>>,sort,sed;touch /etc/xandmkdir -p /etc/xanswerauto_approve(acceptable per the comment, but then the test title should say "a redirection or a program off the list"). The stdin entry (import.meta.main,JSON.parsefailure toask) is the fail-closed claim in the doc and is not exercised; aBun.spawnof the script with garbage on stdin is a three-line test.
Nitpicks
- Nit:
decide({toolCall:{name:"anything_else"}})answersallow. Behindmatcher: "run_command"it cannot fire;askwould be the closed default and costs nothing. - Nit:
tail -fandcatwith no file block forever (print mode has no stdin). Same class asgit commit --amend. - Nit:
git log --format=%(trailers)unquoted is rejected by(; quoted is fine. The scenario's own use of--formatis quoted in the log, so no action. - Nit:
trace.tsdoes not printsystem_messagesteps now, which is fine; the old comment nit is resolved. - Nit:
ls >/dev/null idandecho hi &>/dev/null idare approved with a discarded redirect mid-line; that is correct shell semantics, just worth knowing the tokenizer is not strictly "redirect at the end".
Adversarial run (run through decide, bun)
Answered ask (correct): echo a2>&1b, echo x>/dev/nullid, echo hi & id, echo hi &>/dev/null&id, echo x |& id, echo x |&id, ls ;; id, echo #x\nid (second command seen), echo <<< x, echo {a,b}, -x, =id, VAR=x echo, 2>&1 id, echo hi 1>&2, echo hi >&2, echo hi >/dev/null/../../etc/x, echo a >/dev/null&&id, echo a&>/dev/null, echo 'x' >/dev/null'id', ls >/dev/null"id", echo 2>&1'x', echo "a" followed by a lone ", echo a\<newline>b, echo '"' && id && echo '"', echo "'" ; id ; echo "'", sh -c id, node -e 1, node --eval=1, node -r x a.js, node -, node --, sh -, git -C /tmp status, git log --output=/tmp/x, git diff --output /tmp/x, git log --open-files-in-pager=id.
The HARMLESS_REDIRECTION question: the regex runs before quoting is known, so it can delete a matched token from inside quotes (echo "a 2>&1 b" becomes the word a b). That alters quoted data only. It cannot create or hide a metacharacter: the removed tokens contain no quote, $, backtick or operator, the leading whitespace is preserved by $1, and the lookahead character is not consumed, so a removal can never join two words or move a quote boundary. A 2>&1 is only removed where it follows whitespace and precedes whitespace, ;, &, | or end. I could not construct a joining or quote-shifting input. Unicode whitespace in \s ( ) is removed as a separator by the regex but treated as word content by the tokenizer; the only consequence is a harmless redirect token disappearing next to it. \r, \v, NUL, ESC and
stay inside a word and the shell also treats them as word characters, so no differential. =x and ~, *, ?, ^, [1] as plain args are approved; in zsh =cmd expands to a path and ^/* are globs, which only affect arguments of an allowed command.
Approved, by design or by the doc: grep -f /etc/x a, cat -- /etc/passwd (reads), mkdir -p /etc/x, touch /etc/x (create an empty path outside the checkout; permitted by the "not a boundary" framing, but say so), printf -v x y (a bash builtin variable assignment; harmless), test -f, ls, node ./a.js and sh a.js (arbitrary code from a file the agent wrote; the allowlist is not containment, and the comment now says that).
Verdict
REQUEST CHANGES, one fix (the double-quote escape handling): the tokenizer is sound against every hostile input I could build and the earlier findings are closed, but a re-run of the committed scenario would again be denied on its nodespace writes. I would also narrow nodespace to the subcommands the scenario uses (Improvement 2) in the same change, since session launch defeats the rule outright. Everything else is optional.
/address-review re-review needed only for the escape handling in words().
malibio
left a comment
There was a problem hiding this comment.
Verdict: REQUEST CHANGES (posted as a comment review because the PR is self-authored). Details are in the re-review comment above. One blocker: the approver's double-quote branch rejects backslash and $, which denies 3 of the 23 distinct commands in the committed run (the nodespace node update/create writes with escaped backticks). Everything from the first review is otherwise addressed and I could not bypass the tokenizer. Also recommended: narrow nodespace to the subcommands the scenario uses (session launch starts an agent outside the hook).
Re-review addressed (12aedce)Addressed
Not changed
Re-Review DecisionDecision: RE-REVIEW NEEDED Rationale: The double-quote branch of the tokenizer is the surface that held the earlier bypasses, and it now accepts more. The fix is the one the review suggested and is pinned by tests in both directions, but it is a permission gate and deserves one more read of that branch alone. |
Re-review of 12aedce (approver double-quote handling, narrowed
|
malibio
left a comment
There was a problem hiding this comment.
Verdict: APPROVE with one one-line fix requested (posted as a comment review because the PR is self-authored). The double-quote escape handling is correct, the narrowed nodespace rule works, and the 14 tests pass. One reader/shell mismatch remains: inside double quotes $[expr] is arithmetic, so printf -v a 'x[$(cmd)]'; echo "$[a]" auto-approves and runs cmd (verified under bash). Add [ to the char class in the double-quote branch and a test for it. The git flag denylist also misses -aF and abbreviations (non-blocking). Details in the comment above.
Re-review addressed (009e93e)Addressed
Not changed
Tests: 15 for the approver, Re-Review DecisionDecision: NO RE-REVIEW NEEDED Rationale: The review itself said re-review was needed only for the |
…mode (refs #3748) Antigravity has no NodeSpace plugin, so the driver runs the scenario with what the installer writes today: the skill and the instructions block. Print mode cannot prompt, so the checkout carries a PreToolUse hook that approves a short list of commands and leaves the rest to be denied; it is not --dangerously-skip-permissions. trace.ts reads Antigravity's stream-json steps. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ns said (refs #3748) - antigravity-approve.ts reads the line with a small tokenizer and asks about anything it does not understand: a lone &, $, backticks, redirections other than discarding output, a quote inside a double quote, a path or a VAR=value before the command, git -c/config/--output, node/sh -e/-c. Unreadable input and a payload with no tool name ask. The doc comment says it is not a sandbox: sh and node run files the agent writes. - A quoted multi-line commit message is one word, no longer a false deny. - Tests pin every bypass the review found. - trace.ts reads the real step shapes: text deltas joined, tool parameters, results and errors. - The driver fails a turn that does not end in SUCCESS and says when the permission system denied an action; the exclude entry is idempotent. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… git (refs #3748) - Inside double quotes an escaped dollar, backtick, double quote or backslash is a literal, and a dollar that cannot start an expansion is one, so the checklist ticks and evidence nodes the skill asks for are approved. An unescaped backtick or an expansion still asks. Replaying the live run's 23 distinct command lines: all approved. - nodespace is approved for node, query, diagnostics and skill guidance|get (after an optional --json), not session launch, skill install, database or mcp. - git: status, diff, log, add, commit; not -F, --template, --amend, --no-index, --git-dir, --ext-diff, --textconv. - The doc comment says what an approved line can still do (touch and mkdir outside the checkout, reads of any path). - Tests pin the accepted and refused forms and the script's own stdin path. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rintf -v (refs #3748) - A dollar followed by an open bracket inside double quotes is arithmetic expansion, which evaluates a variable's x[$(cmd)] value; it now asks, as the other expansions do. - printf with -v assigns a variable and asks. - Tests pin both, and that a printf that only prints is approved. Skipped: allow-listing git arguments. The denylist is bypassed by combined short flags and abbreviations (-aF, --fil=), whose impact is reading a file into a commit message, inside the envelope the header already states (cat, head and grep read any path). Revisit if the approver is ever pointed at anything but the scenario's fixture. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
009e93e to
fda24db
Compare
What
scripts/eval/harness/antigravity.tsruns the harness scenario in one Antigravity conversation with what the installer writes today (the skill and the instructions block).run.tsandtrace.tsknow it.antigravity-approve.tsis the scenario's permission rule: Antigravity in print mode cannot prompt, so aPreToolUsehook in the scratch checkout approves a short list of commands and leaves the rest to be denied. It is not--dangerously-skip-permissions.Baseline (Antigravity CLI 1.3.4, live run)
Hook support
Antigravity has hooks (
SessionStart,PreInvocation,PreToolUse,PostToolUse,Stop) andANTIGRAVITY_CONVERSATION_IDreaches the agent's commands. The recommendation, with the evidence, is on #3748 and filed as #3777.Not in this PR
Pi (no ready credentials, Node 22.14 crashes it); the results table is posted on the issue once Pi is in.
Tests
bun run test:scriptspasses (1490). The Rust tiers could not build in this fresh worktree (the cold llama.cpp C++ build fails against this machine's OpenSSL headers); the diff has no Rust, and the merge gate builds in the warm gate checkout.🤖 Generated with Claude Code