From a603f247dff017887cbcff8db331461bdc561c66 Mon Sep 17 00:00:00 2001 From: "Jiaxiao (mossaka) Zhou" Date: Wed, 25 Feb 2026 18:54:12 +0000 Subject: [PATCH 1/2] docs: add integration test coverage guide with gap analysis Comprehensive analysis of all integration tests, CI workflows, and smoke tests to improve visibility into what's covered and what's missing. Key findings: - Most integration tests (20+ files) don't run in CI - --block-domains and --env-all are completely untested - DNS restriction enforcement is unverified - Package manager tests only query registries, never install Includes 6 detailed analysis documents covering domain/network, chroot, protocol/security, container/ops, CI/smoke, and test infrastructure. Co-Authored-By: Claude Opus 4.6 (1M context) --- CLAUDE.md | 1 + docs/INTEGRATION-TESTS.md | 306 ++++++++++++++ docs/test-analysis/chroot.md | 437 ++++++++++++++++++++ docs/test-analysis/ci-smoke.md | 506 ++++++++++++++++++++++++ docs/test-analysis/container-ops.md | 369 +++++++++++++++++ docs/test-analysis/domain-network.md | 330 ++++++++++++++++ docs/test-analysis/protocol-security.md | 486 +++++++++++++++++++++++ docs/test-analysis/test-infra.md | 497 +++++++++++++++++++++++ 8 files changed, 2932 insertions(+) create mode 100644 docs/INTEGRATION-TESTS.md create mode 100644 docs/test-analysis/chroot.md create mode 100644 docs/test-analysis/ci-smoke.md create mode 100644 docs/test-analysis/container-ops.md create mode 100644 docs/test-analysis/domain-network.md create mode 100644 docs/test-analysis/protocol-security.md create mode 100644 docs/test-analysis/test-infra.md diff --git a/CLAUDE.md b/CLAUDE.md index b0a51718b..977527ce1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,6 +13,7 @@ This is a firewall for GitHub Copilot CLI (package name: `@github/awf`) that pro - **[LOGGING.md](LOGGING.md)** - Comprehensive logging documentation - **[docs/logging_quickref.md](docs/logging_quickref.md)** - Quick reference for log queries and monitoring - **[docs/releasing.md](docs/releasing.md)** - Release process and versioning instructions +- **[docs/INTEGRATION-TESTS.md](docs/INTEGRATION-TESTS.md)** - Integration test coverage guide with gap analysis ## Development Workflow diff --git a/docs/INTEGRATION-TESTS.md b/docs/INTEGRATION-TESTS.md new file mode 100644 index 000000000..90a26525c --- /dev/null +++ b/docs/INTEGRATION-TESTS.md @@ -0,0 +1,306 @@ +# Integration Tests Coverage Guide + +A comprehensive guide to what the gh-aw-firewall integration tests cover, what they don't cover, and how they relate to real-world usage in GitHub Agentic Workflows. + +**Last updated:** February 2026 + +--- + +## Quick Navigation + +| Area | Tests | Doc | +|------|-------|-----| +| Domain filtering, DNS, network security | 6 files, ~50 tests | [domain-network.md](test-analysis/domain-network.md) | +| Chroot sandbox, languages, package managers | 5 files, ~70 tests | [chroot.md](test-analysis/chroot.md) | +| Protocol support, credentials, tokens | 8 files, ~100 tests | [protocol-security.md](test-analysis/protocol-security.md) | +| Containers, volumes, git, env vars | 7 files, ~45 tests | [container-ops.md](test-analysis/container-ops.md) | +| CI workflows, smoke tests, build-test | 27 workflows | [ci-smoke.md](test-analysis/ci-smoke.md) | +| Test fixtures and infrastructure | 6 helper files | [test-infra.md](test-analysis/test-infra.md) | + +--- + +## Overview + +The test suite is organized in three tiers: + +``` +┌─────────────────────────────────────────────────────┐ +│ Smoke Tests (5 workflows) │ +│ Real AI agents (Claude, Copilot, Codex, Gemini) │ +│ running inside AWF sandbox │ +├─────────────────────────────────────────────────────┤ +│ Build-Test Workflows (8 workflows) │ +│ Real projects (Go, Rust, Java, Node, etc.) │ +│ built and tested through the firewall proxy │ +├─────────────────────────────────────────────────────┤ +│ Integration Tests (26 files, ~265 tests) │ +│ End-to-end AWF container execution with │ +│ domain filtering, chroot, security assertions │ +├─────────────────────────────────────────────────────┤ +│ Unit Tests (19 files) │ +│ Individual module testing (parser, config, logger) │ +└─────────────────────────────────────────────────────┘ +``` + +### Test Counts by Category + +| Category | Files | Approx Tests | CI Workflow | +|----------|-------|-------------|-------------| +| Domain/Network | 6 | 50 | **None** (not run in CI!) | +| Chroot | 5 | 70 | `test-chroot.yml` (4 jobs) | +| Protocol/Security | 8 | 100 | **None** (not run in CI!) | +| Container/Ops | 7 | 45 | **None** (not run in CI!) | +| Unit Tests | 19 | ~200 | `test-coverage.yml` | +| Smoke Tests | 5 | N/A | Per-workflow (scheduled + PR) | +| Build-Test | 8 | N/A | Per-workflow (PR + dispatch) | + +--- + +## What's Well Covered + +### 1. Chroot Filesystem Isolation (Strong) + +The chroot tests are the most mature, run in CI, and cover critical scenarios: + +- **Language runtimes**: Python, Node.js, Go, Java, .NET, Ruby, Rust all verified accessible through chroot +- **Package managers**: pip, npm, cargo, maven, dotnet, gem, go modules — all tested for registry connectivity +- **Security properties**: NET_ADMIN/SYS_CHROOT capability drop, Docker socket hidden, non-root execution +- **/proc filesystem**: Dynamic mount verified for JVM and .NET CLR compatibility +- **Shell features**: Pipes, redirects, command substitution, compound commands all work in chroot + +**CI coverage**: 4 parallel jobs in `test-chroot.yml` exercise these tests on every PR. + +### 2. Credential Isolation (Strong) + +Multi-layered defense tested at each level: + +- **Credential file hiding**: Docker config, GitHub CLI tokens, npmrc auth tokens all verified hidden via `/dev/null` overlays +- **Exfiltration resistance**: base64 encoding, xxd pipelines, grep patterns all tested — return empty +- **Chroot bypass prevention**: Specific regression test for the vulnerability where credentials were accessible at `$HOME` but not `/host$HOME` +- **API proxy sidecar**: Agent gets placeholder tokens; real keys held by proxy. Healthchecks for OpenAI, Anthropic, Copilot +- **One-shot token library**: LD_PRELOAD intercepts `getenv()`, caches value, clears from environment. Tested in both container and chroot modes +- **Token unsetting from /proc/1/environ**: GITHUB_TOKEN, OPENAI_API_KEY, ANTHROPIC_API_KEY all verified cleared + +### 3. Multi-Engine Smoke Tests (Strong) + +Real AI agents running through the full AWF pipeline: + +- **Claude**: GitHub MCP, Playwright browser automation, file I/O, bash tools +- **Copilot**: Same + web-fetch, agentic-workflows tools +- **Codex**: GH CLI safe inputs, Tavily web search, discussion interactions +- **Gemini**: Same feature set as Codex, different engine path + +### 4. Multi-Language Build-Test (Strong) + +8 language ecosystems tested with real open-source projects: + +- Bun, C++, Deno, .NET, Go, Java, Node.js, Rust +- Each clones a test repo, installs dependencies, builds, and runs tests through AWF + +### 5. Exit Code Propagation (Good) + +15 tests covering exit codes 0-255, command exit codes, pipeline behavior. Critical for CI/CD integration where non-zero = failure. + +--- + +## Critical Gaps + +### Gap 1: Most Integration Tests Don't Run in CI + +**Severity: Critical** + +The `test-integration.yml` workflow is actually just a TypeScript type-check. It does **not** run the integration test suite. This means 20+ integration test files covering domains, DNS, environment variables, exit codes, error handling, network security, credentials, and protocols have **no CI pipeline**. + +Only the 5 chroot test files run in CI (via `test-chroot.yml`). + +**Impact**: Regressions in domain filtering, credential hiding, exit code propagation, and network security would go undetected until someone runs tests locally. + +### Gap 2: `--block-domains` Completely Untested + +**Severity: Critical** + +The `--block-domains` CLI flag (deny-list on top of allow-list) has zero integration tests. The `blocked-domains.test.ts` file is a misnomer — it only tests allow-list behavior. The `AwfRunner` test fixture doesn't even expose a `blockDomains` option. + +**Impact**: If `--block-domains` breaks, no test catches it. + +### Gap 3: `--env-all` Never Tested + +**Severity: Critical** + +The `--env-all` flag is the **primary production mode** — used by `gh-aw`'s `BuildAWFArgs()` to pass all GitHub Actions environment variables into the container. The `environment-variables.test.ts` file header mentions it, but no test uses `envAll: true`. + +**Impact**: The most common production invocation pattern has zero test coverage. + +### Gap 4: DNS Restriction Enforcement Untested + +**Severity: High** + +The `--dns-servers` flag restricts DNS traffic to whitelisted resolvers (preventing DNS-based exfiltration). While `dns-servers.test.ts` exists, no test verifies that queries to **non-whitelisted** DNS servers are actually blocked. All tests use the default or explicitly whitelisted servers. + +**Impact**: DNS exfiltration prevention — a key security feature — is unverified. + +### Gap 5: No Package Installation Tests + +**Severity: High** + +Package manager tests (in `chroot-package-managers.test.ts`) only **query** registries (`pip index versions`, `npm view`, `cargo search`, `gem search`) but never **install** packages (`pip install`, `npm install`, `cargo add`). This is the most common real-world operation. + +The build-test workflows partially compensate (they run `npm install`, `cargo build`, etc.) but these are AI-agent-driven and non-deterministic. + +**Impact**: A regression in package installation through the proxy would not be caught by deterministic tests. + +### Gap 6: `git push` and Authenticated Git Untested + +**Severity: High** + +`git-operations.test.ts` tests clone, ls-remote, and config but **not** `git push` (the most important write operation in agentic workflows) or any authenticated git operation. Every production workflow uses authenticated git to push branches and create PRs. + +**Impact**: If authenticated git push breaks through the proxy, no test catches it. + +### Gap 7: No Localhost HTTP Request Tests + +**Severity: High** + +`localhost-access.test.ts` verifies configuration (iptables rules, Squid ACLs) but never makes an actual HTTP request to localhost. MCP servers run on localhost inside the container — a real request test is needed. + +### Gap 8: Docker Warning Tests Entirely Skipped + +**Severity: Medium** + +`docker-warning.test.ts` is wrapped in `describe.skip` with a stale TODO about a build issue. These 5 tests provide zero coverage. + +### Gap 9: Fragile Timing Dependencies + +**Severity: Medium** + +Token-unset tests rely on `sleep 7` to wait for the 5-second unsetting delay. Log-command tests use `if (existsSync())` guards that silently pass when logs aren't created. These patterns mask real failures and cause flakiness. + +### Gap 10: No SSL Bump Integration Tests + +**Severity: Medium** + +The `--ssl-bump` feature is configured in the CLI but has zero integration tests. This feature enables HTTPS content inspection and is important for some security-sensitive deployments. + +--- + +## Coverage Heat Map + +A visual overview of what's tested vs. not: + +``` +Feature Unit Integration CI Smoke Build-Test +───────────────────────────────────────────────────────────────────────── +Domain allow-list ✅ ✅ ❌ ✅ ✅ +Domain deny-list (--block-domains) ❌ ❌ ❌ ❌ ❌ +Wildcard patterns ✅ ✅ ❌ ❌ ❌ +Empty domains (air-gapped) ❌ ✅ ❌ ❌ ❌ +DNS server restriction ✅ ⚠️ * ❌ ❌ ❌ +Network security (SSRF, bypass) ❌ ✅ ❌ ❌ ❌ +Chroot languages ❌ ✅ ✅ ✅ ✅ +Chroot package managers ❌ ✅ ✅ ❌ ✅ +Chroot /proc filesystem ❌ ✅ ✅ ❌ ❌ +Chroot edge cases ❌ ✅ ✅ ❌ ❌ +Credential hiding ❌ ✅ ❌ ❌ ❌ +Token unsetting ❌ ✅ ❌ ❌ ❌ +One-shot tokens (LD_PRELOAD) ❌ ✅ ❌ ❌ ❌ +API proxy sidecar ❌ ✅ ❌ ❌ ❌ +Protocol support (HTTP/HTTPS) ❌ ✅ ❌ ❌ ❌ +IPv6 ❌ ✅ ❌ ❌ ❌ +Exit code propagation ❌ ✅ ❌ ❌ ❌ +Error handling ❌ ✅ ❌ ❌ ❌ +Volume mounts ❌ ✅ ❌ ❌ ❌ +Container workdir ❌ ✅ ❌ ❌ ❌ +Git operations ❌ ✅ ❌ ❌ ❌ +Environment variables ❌ ✅ ❌ ❌ ❌ +--env-all ❌ ❌ ❌ ❌ ❌ +SSL Bump ✅ ❌ ❌ ❌ ❌ +Log commands ✅ ⚠️ * ❌ ❌ ❌ +Docker unavailability ❌ ✅ ❌ ❌ ❌ +Docker warning stub ❌ ❌ ** ❌ ❌ ❌ +Setup action (action.yml) ❌ ❌ ✅ ❌ ❌ +Container security scan ❌ ❌ ✅ ❌ ❌ +Dependency audit ❌ ❌ ✅ ❌ ❌ + +* ⚠️ = Tests exist but have significant gaps (see detailed docs) +** = Tests exist but are skip'd +``` + +--- + +## Test Infrastructure Summary + +### How Tests Run + +- **Serial execution** (`maxWorkers: 1`) — Docker network/container conflicts prevent parallelism +- **120-second timeout** per test — container lifecycle takes 15-25 seconds +- **Batch runner** groups commands sharing the same config into single containers — reduces ~73 startups to ~27 for chroot tests +- **Custom Jest matchers**: `toSucceed()`, `toFail()`, `toExitWithCode()`, `toTimeout()`, `toAllowDomain()`, `toBlockDomain()` +- **4-stage cleanup**: pre-test TypeScript cleanup → AWF normal exit → AWF signal handlers → CI always-cleanup + +### Infrastructure Limitations + +1. Docker + sudo required — no lightweight local testing +2. Batch runner loses individual stderr (merged via `2>&1`) +3. Log-based matchers require `keepContainers: true` +4. Aggressive `docker prune` in cleanup can affect non-AWF containers +5. No retry logic for flaky network tests + +See [test-infra.md](test-analysis/test-infra.md) for full infrastructure analysis. + +--- + +## Recommended Priority Actions + +### P0: Run Integration Tests in CI + +Create a CI workflow that actually runs the non-chroot integration tests. Currently, ~150 tests across 20+ files have no CI pipeline. + +### P1: Add `--block-domains` Tests + +Add `blockDomains` support to `AwfRunner` and write tests for the deny-list feature: +- Block specific subdomain while allowing parent +- Block takes precedence over allow +- Wildcard blocking patterns + +### P1: Add `--env-all` Tests + +Add tests using `envAll: true` to verify: +- All host env vars pass through +- Sensitive tokens are properly filtered +- Proxy env vars (HTTP_PROXY, HTTPS_PROXY) are set correctly + +### P1: Add DNS Restriction Enforcement Test + +Test that DNS queries to non-whitelisted servers are actually blocked: +``` +dig @1.2.3.4 example.com # Should fail with non-whitelisted DNS +``` + +### P2: Add Package Installation Tests + +Test actual `pip install`, `npm install`, `cargo add` through the proxy instead of just registry queries. + +### P2: Add `git push` with Authentication Test + +Test authenticated git push through the proxy — the most critical write operation in agentic workflows. + +### P2: Enable or Remove Skipped Docker Warning Tests + +Either fix the build issue or remove the dead `describe.skip` block. + +### P3: Fix Fragile Timing Dependencies + +Replace `sleep 7` in token-unset tests with signal-file synchronization. Replace `if (existsSync())` guards with hard assertions. + +--- + +## Detailed Analysis Documents + +Each document provides per-test-case analysis with plain-language descriptions, real-world mappings, and gap identification: + +- **[Domain & Network Tests](test-analysis/domain-network.md)** — Domain filtering, DNS, network security, localhost +- **[Chroot Tests](test-analysis/chroot.md)** — Sandbox isolation, languages, package managers, /proc, edge cases +- **[Protocol & Security Tests](test-analysis/protocol-security.md)** — HTTP/HTTPS, IPv6, API proxy, credentials, tokens, exit codes +- **[Container & Operations Tests](test-analysis/container-ops.md)** — Workdir, volumes, git, env vars, logging, Docker availability +- **[CI & Smoke Tests](test-analysis/ci-smoke.md)** — All 27 CI/smoke/build-test workflows analyzed +- **[Test Infrastructure](test-analysis/test-infra.md)** — Runner architecture, batch pattern, cleanup strategy, limitations diff --git a/docs/test-analysis/chroot.md b/docs/test-analysis/chroot.md new file mode 100644 index 000000000..6c81e3779 --- /dev/null +++ b/docs/test-analysis/chroot.md @@ -0,0 +1,437 @@ +# Chroot Integration Tests Analysis + +This document provides a detailed analysis of all chroot integration test files in the `gh-aw-firewall` project, covering what each test validates, how it maps to real-world usage, and identifying gaps in coverage. + +## Table of Contents + +- [Test Infrastructure Overview](#test-infrastructure-overview) +- [1. chroot-languages.test.ts](#1-chroot-languagestestts) +- [2. chroot-package-managers.test.ts](#2-chroot-package-managerstestts) +- [3. chroot-edge-cases.test.ts](#3-chroot-edge-casestestts) +- [4. chroot-copilot-home.test.ts](#4-chroot-copilot-hometestts) +- [5. chroot-procfs.test.ts](#5-chroot-procfstestts) +- [Cross-File Gap Analysis](#cross-file-gap-analysis) + +--- + +## Test Infrastructure Overview + +### Execution Model +All chroot tests use `AwfRunner.runWithSudo()` which invokes `sudo -E node dist/cli.js` with preserved environment variables (`PATH`, `HOME`, `GOROOT`, `CARGO_HOME`, `JAVA_HOME`, `DOTNET_ROOT`). Each invocation spins up a full Docker Compose stack (Squid proxy + agent container). + +### Batch Runner Optimization +Tests that share the same `allowDomains` config are batched into a single AWF container invocation using `runBatch()`. This concatenates commands into a single bash script with delimiter tokens, parsing per-command results from the combined output. This reduces ~73 container startups to ~27 across the suite. + +### Custom Matchers +- `toSucceed()` - exit code 0 +- `toFail()` - non-zero exit code +- `toExitWithCode(n)` - specific exit code +- `toAllowDomain(domain)` / `toBlockDomain(domain)` - Squid log inspection + +### Chroot Architecture Under Test +The agent container mounts the host filesystem at `/host`, then calls `chroot /host` so all paths resolve naturally. Key features: +- Selective path mounting (not full `/` mount by default) +- Empty writable `$HOME` with specific subdirectory overlays +- Dynamic `/proc` mount via `mount -t proc` (not static bind mount) +- Capability drop (`NET_ADMIN`, `SYS_CHROOT`, `SYS_ADMIN`) before user code runs +- UID/GID remapping to match host user + +--- + +## 1. chroot-languages.test.ts + +**Purpose:** Verifies that host-installed language runtimes are accessible through the chroot filesystem. Critical for GitHub Actions runners where tools are pre-installed at the host level. + +### Test Cases + +#### Batched Quick Checks (single container invocation) + +| Test | Command | What It Validates | +|------|---------|-------------------| +| Python version | `python3 --version` | Python3 binary accessible via chroot PATH | +| Python inline | `python3 -c "print(2 + 2)"` | Python interpreter executes inline scripts | +| Python stdlib | `python3 -c "import json, os, sys; ..."` | Python standard library modules load correctly | +| pip version | `pip3 --version` | pip package manager accessible | +| Node.js version | `node --version` | Node.js binary accessible | +| Node.js inline | `node -e "console.log(2 + 2)"` | Node.js evaluates inline JS | +| Node.js modules | `node -e "require('os').platform()"` | Node.js built-in modules resolve | +| npm version | `npm --version` | npm binary accessible | +| npx version | `npx --version` | npx binary accessible | +| Go version | `go version` | Go binary accessible | +| Go env | `go env GOVERSION` | Go environment properly configured | +| Java version | `java --version` | JDK accessible (fallback: `java -version`) | +| .NET version | `dotnet --version` | .NET SDK accessible | +| .NET info | `dotnet --info` | .NET runtime information available | +| Unix utils | `which bash && which ls && which cat` | Core Unix utilities accessible | +| Git version | `git --version` | Git binary accessible | +| curl version | `curl --version` | curl binary accessible | + +#### Individual Tests (separate containers) + +| Test | What It Validates | +|------|-------------------| +| Java compile + run | Creates Hello.java, compiles with `javac`, runs with `java` - validates full JDK toolchain | +| Java stdlib (java.util) | Compiles and runs code using `java.util.Arrays` and `java.util.List` | +| .NET create + run | `dotnet new console` + `dotnet restore` + `dotnet run` - validates full SDK workflow (requires NuGet domains) | + +### Real-World Mapping + +| Test Area | Real-World Scenario | +|-----------|-------------------| +| Python | Claude/Copilot agents installing Python packages, running Python scripts in AI-generated code | +| Node.js/npm | Copilot CLI itself is a Node.js tool; agents run `npm install`, build JS projects | +| Go | Agents building Go projects (common in GitHub Actions context) | +| Java | Agents compiling Java projects with Maven/Gradle (enterprise workflows) | +| .NET | Agents building .NET projects, NuGet restore for dependencies | +| Git | Every agent workflow uses git (clone, commit, push) | +| curl | Agents fetching APIs, downloading artifacts | + +### Gaps and Missing Coverage + +1. **No Rust compile test** - Rust is tested in package-managers but only for `cargo --version` and `rustc --version`. No `cargo build` or `rustc` compile test exists here, despite Rust being a primary language for AWF users. + +2. **No Python virtual environment test** - Real agents frequently create venvs (`python3 -m venv`). The chroot filesystem might not handle venv creation correctly (symlinks, activation scripts). + +3. **No TypeScript compilation test** - `tsc` or `tsx` are common in agent workflows but never tested. + +4. **No Bun runtime test** - Bun is explicitly supported in `entrypoint.sh` (AWF_BUN_INSTALL) but has no corresponding test. + +5. **No multi-language interaction test** - Real agents often chain languages (e.g., Python script calling a Node.js tool), which could fail if PATH ordering is wrong. + +6. **No dynamic library loading test** - Tests only check binary execution. Shared library loading (`ld.so.cache`, `/lib64/`) is implicitly tested but not explicitly verified. + +7. **Java version check uses fallback pattern** - `java --version 2>&1 || java -version 2>&1` catches both formats, but doesn't verify which Java version is found (could pick up wrong JDK). + +8. **Soft failures on network tests** - `.NET` test uses `if (result.success)` guard, meaning the test passes even if .NET can't reach NuGet. This hides real failures. + +--- + +## 2. chroot-package-managers.test.ts + +**Purpose:** Validates that package managers can perform network operations through the firewall with proper domain whitelisting. Tests both online (with allowed domains) and offline behaviors. + +### Test Cases + +#### pip (Python) + +| Test | Domains Allowed | What It Validates | +|------|----------------|-------------------| +| pip list | pypi.org, files.pythonhosted.org | Lists installed packages (verifies pip can read local package DB) | +| pip index versions | pypi.org, files.pythonhosted.org | Queries PyPI registry through firewall | +| pip show pip | localhost only | Shows package info without network (offline capability) | + +#### npm (Node.js) + +| Test | Domains Allowed | What It Validates | +|------|----------------|-------------------| +| npm config list | registry.npmjs.org | npm configuration accessible | +| npm view chalk version | registry.npmjs.org | npm queries registry through firewall | +| npm view (blocked) | localhost only | npm registry access is blocked without domain whitelisting | + +#### Rust (cargo) + +| Test | Domains Allowed | What It Validates | +|------|----------------|-------------------| +| cargo version | crates.io, static.crates.io, index.crates.io | Cargo binary accessible via chroot | +| cargo search serde | crates.io, static.crates.io, index.crates.io | Cargo can search crates.io through firewall | +| rustc version | localhost only | rustc binary accessible (offline) | + +#### Java (maven) + +| Test | Domains Allowed | What It Validates | +|------|----------------|-------------------| +| java version | localhost only | Java runtime accessible | +| javac version | localhost only | Java compiler accessible | +| mvn version | repo.maven.apache.org, repo1.maven.org | Maven binary accessible with repository domains | + +#### .NET (dotnet/nuget) + +| Test | Domains Allowed | What It Validates | +|------|----------------|-------------------| +| dotnet list-sdks | localhost only | SDK listing works offline | +| dotnet list-runtimes | localhost only | Runtime listing works offline | +| dotnet create + build | api.nuget.org, nuget.org, dotnetcli.azureedge.net | Full project lifecycle with NuGet restore | +| dotnet restore (blocked) | localhost only | NuGet restore fails without domain whitelisting | + +#### Ruby (gem/bundler) + +| Test | Domains Allowed | What It Validates | +|------|----------------|-------------------| +| ruby version | localhost only | Ruby binary accessible | +| gem list (local) | localhost only | Lists locally installed gems | +| gem version | rubygems.org, index.rubygems.org | gem binary accessible with registry domains | +| bundler version | rubygems.org, index.rubygems.org | Bundler binary accessible | +| gem search rails | rubygems.org, index.rubygems.org | gem can search rubygems.org through firewall | + +#### Go modules + +| Test | Domains Allowed | What It Validates | +|------|----------------|-------------------| +| go env GOPATH GOPROXY | proxy.golang.org, sum.golang.org | Go module proxy configuration correct | +| go mod init + tidy | localhost only | Go module initialization works offline | + +### Real-World Mapping + +| Test Area | Real-World Scenario | +|-----------|-------------------| +| pip + PyPI | Copilot/Claude agents running `pip install` for Python dependencies in AI-generated code | +| npm + registry | Agents running `npm install` for JS/TS projects; Copilot CLI itself needs npm | +| cargo + crates.io | Agents building Rust projects, adding dependencies with `cargo add` | +| maven | Agents building Java enterprise projects with Maven | +| dotnet + NuGet | Agents building .NET projects, adding NuGet packages | +| gem + rubygems | Agents working with Ruby projects, installing gems | +| go modules | Agents working with Go projects, fetching module dependencies | +| Blocking tests | Ensures firewall actually blocks unauthorized network access - critical security property | + +### Gaps and Missing Coverage + +1. **No pip install test** - Tests query PyPI index but never actually install a package. `pip install requests` through the firewall would be a more realistic test. + +2. **No npm install test** - Tests `npm view` but never `npm install`. Real agents always install packages. + +3. **No cargo build/add test** - Tests `cargo search` but never `cargo add` or `cargo build` with dependencies. + +4. **No Gradle test** - Maven is tested but Gradle (also very common in Java) is completely absent. `entrypoint.sh` even pre-seeds `~/.gradle/gradle.properties` for proxy config but this is never tested. + +5. **No sbt/Scala test** - JVM proxy flags are set via `JAVA_TOOL_OPTIONS` for sbt but never tested. + +6. **No pip blocking test** - npm and .NET have explicit "blocked without domain" tests, but pip does not. There's no test verifying that `pip install` fails when PyPI is not whitelisted. + +7. **No cargo blocking test** - Same gap as pip - no test verifying cargo is blocked without crates.io domains. + +8. **No gem install test** - Tests `gem search` but never `gem install`. Real-world Ruby workflows install gems. + +9. **Soft failure pattern** - Multiple tests use `if (result.exitCode === 0)` or `if (result.success)` guards, meaning the test passes even on failure. This is appropriate for CI flakiness tolerance but masks real regressions. + +10. **No proxy configuration verification** - Tests verify tools can reach registries but don't verify proxy env vars are correctly set. A test checking `echo $HTTP_PROXY` would confirm proxy configuration. + +--- + +## 3. chroot-edge-cases.test.ts + +**Purpose:** Validates edge cases, security features, error handling, and shell compatibility within the chroot environment. + +### Test Cases + +#### General Checks (batched) + +| Test | Command | What It Validates | +|------|---------|-------------------| +| PATH preserved | `echo $PATH` | PATH includes `/usr/bin` and `/bin` | +| HOME set | `echo $HOME` | HOME env var points to a valid path | +| /usr readable | `ls /usr/bin` | Host `/usr/bin` accessible through chroot | +| /etc readable | `cat /etc/passwd` | Host `/etc/passwd` accessible (contains "root") | +| /tmp writable | Write + read + delete in /tmp | Temp directory is writable | +| Docker socket hidden | Check `/var/run/docker.sock` | Docker socket is NOT accessible (security) | +| NET_ADMIN dropped | `iptables -L` | Cannot list iptables rules (permission denied) | +| chroot prevented | `chroot / /bin/true` | Cannot use chroot command (capability dropped) | +| Shell pipes | `echo "hello" \| grep hello` | Pipe operator works in chroot | +| Shell redirect | Write via `>` and read back | Redirection works in chroot | +| Command substitution | `echo "Today is $(date +%Y)"` | `$()` substitution works | +| Compound commands | `echo "first" && echo "second" && echo "third"` | `&&` chaining works | +| Non-root user | `id -u` | UID is not 0 (running as non-root) | +| Username set | `whoami` | Username is not "root" | + +#### Working Directory Handling (individual tests) + +| Test | What It Validates | +|------|-------------------| +| Respect container-workdir | `pwd` with `containerWorkDir: '/tmp'` returns `/tmp` | +| Fallback for nonexistent dir | `pwd` with nonexistent `containerWorkDir` falls back to home | + +#### Exit Code Propagation (individual tests) + +| Test | What It Validates | +|------|-------------------| +| Exit code 0 | `exit 0` propagates correctly | +| Exit code 1 | `exit 1` propagates correctly | +| Failed command | `false` returns exit code 1 | +| Command not found | `nonexistent_command_xyz123` returns exit code 127 | + +#### Network Firewall Enforcement (individual tests) + +| Test | What It Validates | +|------|-------------------| +| Allow HTTPS | `curl -s -o /dev/null -w "%{http_code}" https://api.github.com` succeeds with whitelisted domain | +| Block HTTPS | `curl -s --connect-timeout 5 https://example.com` fails when example.com not whitelisted | +| Block HTTP | `curl -f --connect-timeout 5 http://example.com` fails when example.com not whitelisted | + +### Real-World Mapping + +| Test Area | Real-World Scenario | +|-----------|-------------------| +| PATH/HOME | Every agent command depends on correct environment variables | +| /usr, /etc access | Agents need host binaries and system configs | +| /tmp writable | Build tools, compilers, and agents use temp files extensively | +| Docker socket hidden | Prevents agents from escaping the firewall by spawning unrestricted containers | +| Capability drop | Prevents agents from modifying iptables to bypass firewall | +| Shell features | Agents execute complex shell commands with pipes, redirects, and substitution | +| Non-root execution | Security requirement - agents must not run as root | +| Working directory | `--container-workdir` sets where agent commands execute (typically the repo checkout) | +| Exit codes | AWF must faithfully propagate agent exit codes for CI/CD pass/fail determination | +| Network enforcement | Core firewall functionality - allow whitelisted, block everything else | + +### Gaps and Missing Coverage + +1. **No `--env` passthrough test** - Test for custom environment variables is explicitly skipped (`test.skip`). This is a significant gap since `--env` is a real CLI feature. + +2. **No SYS_ADMIN capability drop test** - Tests verify NET_ADMIN and SYS_CHROOT are dropped but don't test SYS_ADMIN (which is dropped in chroot mode per `entrypoint.sh`). + +3. **No signal handling test** - No test for SIGTERM/SIGINT propagation. The entrypoint has explicit signal handling (`trap cleanup_and_exit TERM INT`) but this is never tested. + +4. **No symlink resolution test** - Chroot mode relies on symlinks (e.g., `/lib` -> `/lib/x86_64-linux-gnu`). No test verifies symlinks work correctly. + +5. **No large output test** - No test for commands producing large stdout/stderr, which could test buffer handling. + +6. **No `--allow-full-filesystem-access` test** - The escape hatch flag is never tested in chroot integration tests. + +7. **No credential hiding test** - The selective mounting hides credential files via `/dev/null` overlays, but no test verifies that `cat ~/.docker/config.json` or `cat ~/.ssh/id_rsa` returns empty/fails. + +8. **No DNS resolution test** - DNS configuration is complex in chroot mode (resolv.conf backup/restore, Docker embedded DNS + external DNS). No test verifies DNS queries resolve correctly. + +9. **No concurrent process test** - No test running multiple processes simultaneously in the chroot, which could reveal issues with /proc, temp files, or resource sharing. + +10. **No exit code for signals** - Tests check exit codes 0, 1, and 127, but not 128+N signal exit codes (e.g., 143 for SIGTERM). + +11. **No timeout propagation test** - No test verifying that AWF's timeout mechanism works and propagates correctly. + +--- + +## 4. chroot-copilot-home.test.ts + +**Purpose:** Verifies that the GitHub Copilot CLI can access and write to `~/.copilot` directory in chroot mode. Essential for package extraction, configuration storage, and log management. + +### Test Cases (all batched, single container) + +| Test | Command | What It Validates | +|------|---------|-------------------| +| Write to ~/.copilot | Create dir + write file + read back | Basic write access to ~/.copilot | +| Nested directories | Create `~/.copilot/pkg/linux-x64/0.0.405/marker.txt` | Deep directory creation (mimics Copilot package extraction) | +| Permissions | `touch` + `rm` in ~/.copilot | File creation and deletion work (correct ownership) | + +### Real-World Mapping + +| Test | Real-World Scenario | +|------|-------------------| +| Write file | Copilot CLI writes configuration files on first run | +| Nested directories | Copilot CLI extracts bundled packages to `~/.copilot/pkg///` | +| Permissions | Copilot CLI needs to manage its own files (create, update, delete) | + +### Gaps and Missing Coverage + +1. **No file persistence test** - Tests write and read within the same invocation. No test verifies files persist between AWF invocations (which they should, as ~/.copilot is bind-mounted from host). + +2. **No ~/.copilot/logs test** - Copilot CLI writes logs to `~/.copilot/logs/` which is separately mounted (`${config.workDir}/agent-logs:${effectiveHome}/.copilot/logs:rw`). No test verifies log writing works. + +3. **No ownership/UID test** - Files should be owned by the AWF user (not root). No test checks `ls -la ~/.copilot/test/file.txt` for correct ownership. + +4. **No concurrent write test** - No test for atomic file writes (important for config files). + +5. **No symlink within ~/.copilot test** - Copilot may create symlinks; no test verifies this works. + +6. **No `.claude.json` creation test** - `entrypoint.sh` creates `~/.claude.json` when `CLAUDE_CODE_API_KEY_HELPER` is set. This is never tested. + +7. **No other home subdirectory tests** - `~/.cache`, `~/.config`, `~/.local`, `~/.anthropic`, `~/.claude` are all mounted but only `~/.copilot` is tested for write access. + +--- + +## 5. chroot-procfs.test.ts + +**Purpose:** Validates the dynamic `/proc` filesystem mount in chroot mode. This is a regression test for commit `dda7c67` which replaced a static `/proc/self` bind mount with `mount -t proc`. + +### Background + +Without the dynamic proc mount: +- .NET CLR fails: "Cannot execute dotnet when renamed to bash" +- JVM misreads `/proc/self/exe` and `/proc/cpuinfo` +- Rustup proxy binaries appear as bash instead of the actual binary + +### Test Cases + +#### Batch 1: Quick /proc checks (single container) + +| Test | Command | What It Validates | +|------|---------|-------------------| +| /proc/self/exe resolves | `readlink /proc/self/exe` | Returns a real path (not "bash") | +| Different binaries differ | `bash -c "readlink ..."` vs `python3 -c "readlink ..."` | Different binaries see different /proc/self/exe | +| /proc/cpuinfo | `cat /proc/cpuinfo \| head -10` | CPU info accessible (needed by JVM, .NET GC) | +| /proc/meminfo | `cat /proc/meminfo \| head -5` | Memory info accessible (needed by JVM, .NET GC) | +| /proc/self/status | `cat /proc/self/status \| head -5` | Process status accessible | + +#### Batch 2: Java /proc tests (single container) + +| Test | Command | What It Validates | +|------|---------|-------------------| +| Java reads /proc/self/exe | Java program reads `/proc/self/exe` via `Files.readSymbolicLink` | JVM sees itself as "java", not "bash" | +| Java availableProcessors | Java program reads `Runtime.availableProcessors()` | JVM correctly reads /proc/cpuinfo for CPU count | + +### Real-World Mapping + +| Test | Real-World Scenario | +|------|-------------------| +| /proc/self/exe resolution | .NET CLR reads /proc/self/exe to find itself (required for startup). JVM reads it for identity. Rustup proxy reads it to determine which tool to invoke. | +| /proc/cpuinfo | JVM uses CPU count for thread pool sizing. .NET GC uses it for heap sizing. | +| /proc/meminfo | JVM and .NET use memory info for heap/GC configuration. | +| Different binary resolution | Ensures the procfs mount is truly dynamic (not cached from parent shell) | +| Java /proc/self/exe | Specific regression test - JVM was misidentifying itself as bash, causing startup issues | + +### Gaps and Missing Coverage + +1. **No .NET /proc/self/exe test** - .NET was the original motivation for the fix, but only Java has a /proc/self/exe verification test. A `dotnet` program reading /proc/self/exe would be valuable. + +2. **No Rust/rustup /proc/self/exe test** - Rustup proxies use /proc/self/exe to determine which tool to invoke. No test verifies this. + +3. **No /proc/self/environ test** - The one-shot-token security feature unsets sensitive tokens from `/proc/1/environ`. No test verifies tokens are actually cleared. + +4. **No /proc/self/maps test** - Some runtimes read memory maps; not tested. + +5. **No /proc isolation test** - The dynamic proc mount should be container-scoped (only container processes visible). No test verifies that host PIDs are NOT visible. + +6. **No /proc/self/fd test** - File descriptor access via /proc is used by some tools; not tested. + +7. **No Node.js /proc test** - Node.js uses /proc for certain operations (e.g., `process.memoryUsage()`, `os.cpus()`). No test verifies Node's /proc access. + +8. **Soft failure pattern on Java tests** - Both Java /proc tests use `if (r.exitCode === 0)` guard, meaning they pass even if Java compilation fails. + +--- + +## Cross-File Gap Analysis + +### High-Priority Missing Tests + +| Gap | Severity | Affected Scenarios | +|-----|----------|--------------------| +| **Credential hiding verification** | Critical | No test verifies `/dev/null` overlays on `~/.docker/config.json`, `~/.ssh/id_rsa`, etc. Prompt injection defense is untested. | +| **Signal handling (SIGTERM/SIGINT)** | High | No test for graceful shutdown and cleanup. Real AWF runs in CI with `timeout` which sends SIGTERM. | +| **DNS resolution in chroot** | High | Complex DNS setup (resolv.conf backup/restore, Docker embedded DNS) is completely untested. | +| **Package installation (pip/npm/cargo)** | High | Tests only query registries but never install packages. Real agents install packages constantly. | +| **`--env` passthrough** | Medium | Skipped test. Custom env vars are a core feature for passing API keys to agents. | +| **One-shot token protection** | Medium | `/proc/1/environ` token clearing is never tested. Security feature with no regression test. | +| **Bun runtime** | Medium | Explicitly supported in entrypoint.sh but never tested. | +| **Gradle build tool** | Medium | Proxy config pre-seeded by entrypoint.sh but never tested. | +| **`~/.claude.json` creation** | Medium | Created by entrypoint.sh for Claude Code API auth but never tested. | + +### Test Pattern Issues + +1. **Soft failure masking** - Many tests use `if (result.success)` or `if (r.exitCode === 0)` guards that silently pass on failure. While appropriate for CI flakiness, these should at minimum log a warning when the underlying check is skipped. + +2. **No negative security tests** - Security features (capability drop, Docker socket hiding, credential hiding) lack comprehensive negative testing. Only NET_ADMIN and SYS_CHROOT drops are verified. + +3. **No cleanup verification** - `entrypoint.sh` has extensive cleanup logic (resolv.conf restoration, hosts file cleanup, script file deletion). None of this is tested. + +4. **Missing `--allow-full-filesystem-access` tests** - The escape hatch that mounts `/:/host:rw` instead of selective mounts is never tested. + +5. **No `--mount` custom volume test** - Custom volume mounts passed via `--mount` flag are never tested in chroot context. + +### Recommended Test Additions (Priority Order) + +1. **Credential exfiltration test** - Verify `cat ~/.docker/config.json`, `cat ~/.ssh/id_rsa`, `cat ~/.config/gh/hosts.yml` all return empty or fail. +2. **Package install test** - `pip install requests`, `npm install chalk`, `cargo add serde` through the firewall. +3. **DNS resolution test** - `nslookup github.com` or `dig github.com` inside the chroot. +4. **Signal propagation test** - Send SIGTERM to AWF process, verify cleanup runs. +5. **`--env` passthrough test** - Pass custom env var, verify it's accessible in chroot. +6. **Token clearing test** - Verify `/proc/1/environ` doesn't contain sensitive tokens after agent starts. +7. **Bun runtime test** - `bun --version` and `bun run` inside chroot. +8. **Gradle proxy test** - Verify `~/.gradle/gradle.properties` contains proxy settings. +9. **`.claude.json` test** - Set `CLAUDE_CODE_API_KEY_HELPER`, verify file is created correctly. +10. **Home subdirectory write tests** - Verify `~/.cache`, `~/.config`, `~/.local` are writable. diff --git a/docs/test-analysis/ci-smoke.md b/docs/test-analysis/ci-smoke.md new file mode 100644 index 000000000..c1a230f8c --- /dev/null +++ b/docs/test-analysis/ci-smoke.md @@ -0,0 +1,506 @@ +# CI Workflows and Smoke Tests Analysis + +This document catalogs all CI workflows and smoke/build-test agentic workflows in gh-aw-firewall, describing what each tests, when it runs, what real-world scenario it validates, coverage gaps, and how it relates to the Node.js integration test suite. + +--- + +## Table of Contents + +1. [CI Workflow Overview](#ci-workflow-overview) +2. [Core CI Workflows](#core-ci-workflows) +3. [Smoke Test Workflows (Agentic)](#smoke-test-workflows-agentic) +4. [Build-Test Workflows (Agentic)](#build-test-workflows-agentic) +5. [Security & Compliance Workflows](#security--compliance-workflows) +6. [Infrastructure Workflows](#infrastructure-workflows) +7. [Relationship Map: CI vs Integration Tests](#relationship-map-ci-vs-integration-tests) +8. [Coverage Gap Analysis](#coverage-gap-analysis) + +--- + +## CI Workflow Overview + +The repo has three tiers of testing: + +| Tier | Type | Count | Purpose | +|------|------|-------|---------| +| **Unit** | Jest (src/*.test.ts) | 19 files | Fast feedback on individual modules | +| **Integration** | Jest (tests/integration/*.test.ts) | 26 files | End-to-end AWF container execution | +| **Smoke/Build-Test** | gh-aw compiled workflows (.lock.yml) | 13 workflows | Real AI agent execution inside AWF sandbox | +| **CI** | Hand-written GitHub Actions (.yml) | 12 workflows | Build, lint, type-check, security, coverage | + +--- + +## Core CI Workflows + +### 1. `test-integration.yml` — TypeScript Type Check + +**File**: `.github/workflows/test-integration.yml` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | TypeScript type-checking via `npm run type-check`. Despite the filename, this does NOT run integration tests. | +| **Triggers** | Push to main, PR to main, manual dispatch | +| **Timeout** | 5 minutes | +| **Real-world mapping** | Validates that TypeScript code compiles without type errors before merge | +| **Gaps** | Only checks types, not runtime behavior. Name is misleading (suggests integration tests). | +| **Integration test relationship** | Complementary — type checking catches compile-time errors; integration tests catch runtime issues. | + +### 2. `test-chroot.yml` — Chroot Integration Tests + +**File**: `.github/workflows/test-chroot.yml` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Runs the chroot integration test suite across 4 parallel jobs: Language Support, Package Managers, /proc Filesystem, Edge Cases | +| **Triggers** | Push to main, PR to main, manual dispatch | +| **Timeout** | 30-45 minutes per job | +| **Real-world mapping** | Validates that the chroot-based filesystem isolation works correctly with multiple languages (Node, Python, Go, Java, .NET, Ruby, Rust) and package managers | +| **Gaps** | Sequential dependency: package-managers waits for languages job. No macOS testing. | +| **Integration test relationship** | **Direct 1:1 mapping** — this workflow runs `tests/integration/chroot-languages.test.ts`, `chroot-package-managers.test.ts`, `chroot-procfs.test.ts`, and `chroot-edge-cases.test.ts` | + +**Jobs breakdown:** +- **test-chroot-languages** — Sets up Node.js, Python 3.12, Go 1.22, Java 21, .NET 8.0. Builds containers locally. Runs `chroot-languages` integration tests. +- **test-chroot-package-managers** (needs: languages) — Adds Ruby 3.2, Rust stable. Runs `chroot-package-managers` integration tests. 45-minute timeout. +- **test-chroot-procfs** (parallel) — Tests /proc filesystem access within chroot. Python, Java only. +- **test-chroot-edge-cases** (parallel) — Tests edge cases. Node.js only. + +Key detail: Containers are built locally (`docker build`), so source changes to `entrypoint.sh` and `docker-manager.ts` ARE reflected in tests. + +### 3. `test-coverage.yml` — Test Coverage + +**File**: `.github/workflows/test-coverage.yml` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Runs unit tests with coverage collection. On PRs: compares coverage against base branch and fails on regression. On push: generates coverage summary. | +| **Triggers** | Push to main, PR to main (ignoring .md files) | +| **Timeout** | 15 minutes | +| **Real-world mapping** | Ensures PRs don't reduce test coverage — guards against "add feature, skip tests" PRs | +| **Gaps** | Only covers unit tests (src/*.test.ts), not integration tests. Node.js 20 only (build.yml tests 20+22). | +| **Integration test relationship** | Only measures coverage of unit tests. Integration test coverage is not tracked. | + +**Notable features:** +- Checks out base branch to compute coverage diff +- Posts coverage comparison as PR comment +- Uploads coverage artifacts (30-day retention) +- Fails PR if coverage regresses + +### 4. `test-action.yml` — Setup Action Tests + +**File**: `.github/workflows/test-action.yml` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Tests the `action.yml` composite action that installs AWF from GitHub releases | +| **Triggers** | Push to main, PR to main (ignoring .md files), manual dispatch | +| **Timeout** | 5-10 minutes per job | +| **Real-world mapping** | Validates that users can install AWF via `uses: github/gh-aw-firewall@v1` in their workflows | +| **Gaps** | Only tests installation, not actual firewall functionality. Tests version v0.7.0 specifically (may go stale). | +| **Integration test relationship** | No overlap — tests the GitHub Action packaging, not the firewall itself | + +**Jobs:** +- **test-action-latest** — Install latest version, verify `awf --version` and `awf --help` work +- **test-action-specific-version** — Install v0.7.0, verify exact version/image-tag outputs match +- **test-action-with-images** — Install v0.7.0 with `pull-images: true`, verify Docker images are pulled +- **test-action-invalid-version** — Install `invalid-version`, verify action fails gracefully + +### 5. `test-examples.yml` — Examples Test + +**File**: `.github/workflows/test-examples.yml` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Runs example shell scripts from `examples/` directory as smoke tests | +| **Triggers** | Push to main, PR to main (ignoring .md files), manual dispatch | +| **Timeout** | 15 minutes | +| **Real-world mapping** | Validates that documentation examples actually work — prevents stale README instructions | +| **Gaps** | Skips `github-copilot.sh` (requires GITHUB_TOKEN). Only 4 of 5 examples tested. | +| **Integration test relationship** | Complementary — examples test real AWF invocations from shell scripts, while integration tests use the Jest/TypeScript test runner | + +**Examples tested:** +1. `basic-curl.sh` — Basic domain allow/block with curl +2. `using-domains-file.sh` — Domain list from file +3. `debugging.sh` — Debug mode with `--keep-containers` +4. `blocked-domains.sh` — Verify blocked domains return errors + +### 6. `build.yml` — Build Verification + +**File**: `.github/workflows/build.yml` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Builds TypeScript project and runs linter across Node.js 20 and 22 matrix | +| **Triggers** | Push to main, PR to main, manual dispatch | +| **Timeout** | 10 minutes | +| **Real-world mapping** | Ensures the project builds successfully on supported Node.js versions | +| **Gaps** | No Node.js 18 testing (though `pkg` in release uses node18 targets). No test execution. | +| **Integration test relationship** | Prerequisite — if build fails, nothing else runs. No direct test overlap. | + +### 7. `lint.yml` — ESLint + +**File**: `.github/workflows/lint.yml` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Runs ESLint on TypeScript source | +| **Triggers** | Push to main, PR to main (ignoring .md files) | +| **Timeout** | 5 minutes | +| **Real-world mapping** | Code quality enforcement | +| **Gaps** | Duplicated with `build.yml` which also runs `npm run lint`. | +| **Integration test relationship** | None — code quality only | + +--- + +## Smoke Test Workflows (Agentic) + +These are gh-aw agentic workflows compiled from `.md` source files into `.lock.yml` GitHub Actions workflows. They run **actual AI agents** (Claude, Copilot, Codex, Gemini) inside the AWF sandbox. + +**Post-processing**: All `.lock.yml` files are post-processed by `scripts/ci/postprocess-smoke-workflows.ts` which replaces GHCR image references with local builds (`--build-local`), removes sparse-checkout, and installs AWF from source. + +### 8. `smoke-claude.lock.yml` — Smoke Claude + +**Source**: `smoke-claude.md` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Claude Code engine running inside AWF sandbox with MCP servers, Playwright, file I/O, and bash tools | +| **Engine** | `claude` (max 15 turns) | +| **Triggers** | Every 12h (schedule), PR (opened/synchronize/reopened), manual dispatch | +| **Timeout** | 10 minutes | +| **Network allowed** | defaults, github, playwright | +| **Tools** | cache-memory, github (repos, pull_requests), playwright, edit, bash | +| **Safe outputs** | add-comment (hide older), add-labels (smoke-claude) | +| **MCP** | ghcr.io/github/gh-aw-mcpg container | +| **Real-world mapping** | Validates that Claude Code can operate within AWF's network sandbox: GitHub API access via MCP, browser automation via Playwright, local file operations — the core use case for agentic workflows | +| **Gaps** | Non-deterministic (AI agent may behave differently). No HTTPS blocking verification. | +| **Integration test relationship** | High-level end-to-end complement. Integration tests verify AWF mechanics (iptables, proxy); this verifies an actual AI agent works through the firewall. | + +**Test requirements:** +1. GitHub MCP: Review last 2 merged PRs +2. Playwright: Navigate to github.com, verify page title +3. File writing: Create test file, verify with cat +4. Bash: Execute commands to verify file creation +5. Post-step: Validate safe outputs were invoked (add_comment for PR triggers) + +### 9. `smoke-copilot.lock.yml` — Smoke Copilot + +**Source**: `smoke-copilot.md` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Copilot engine running inside AWF sandbox with MCP, Playwright, web-fetch, and agentic-workflows tools | +| **Engine** | `copilot` | +| **Triggers** | Every 12h, PR, manual dispatch | +| **Timeout** | 5 minutes | +| **Network allowed** | defaults, node, github, playwright | +| **Tools** | agentic-workflows, cache-memory, edit, bash, github, playwright, web-fetch | +| **Real-world mapping** | Validates Copilot CLI agent works through AWF with broader network access (node registries) and additional tools | +| **Gaps** | Shorter timeout (5min) may cause flaky failures. No blocked-domain verification. | +| **Integration test relationship** | Similar to smoke-claude but for Copilot engine. Tests a different engine implementation path. | + +### 10. `smoke-codex.lock.yml` — Smoke Codex + +**Source**: `smoke-codex.md` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Codex engine with extended tool suite: GH CLI safe inputs, Tavily web search, discussion interactions, and AWF project build | +| **Engine** | `codex` | +| **Triggers** | Every 12h, PR, manual dispatch | +| **Timeout** | 15 minutes | +| **Network allowed** | defaults, github, playwright | +| **Tools** | cache-memory, github, playwright, edit, bash | +| **Safe outputs** | add-comment, create-issue, add-labels, hide-comment | +| **Imports** | shared/gh.md, shared/mcp/tavily.md, shared/reporting.md, shared/github-queries-safe-input.md | +| **Real-world mapping** | Most comprehensive smoke test — validates safe-inputs (gh CLI), Tavily MCP, discussion API, and build capability | +| **Gaps** | Complex prompt may cause non-deterministic failures. Build step (`npm ci && npm run build`) adds latency. | +| **Integration test relationship** | Tests discussion interactions and create-issue safe outputs that integration tests don't cover | + +**Additional test requirements beyond Claude/Copilot:** +- Safe Inputs GH CLI: Query PRs via `safeinputs-gh` +- Tavily web search: Search for "GitHub Agentic Workflows Firewall" +- Discussion interaction: Comment on latest discussion +- Build AWF: Run `npm ci && npm run build` inside sandbox + +### 11. `smoke-gemini.lock.yml` — Smoke Gemini + +**Source**: `smoke-gemini.md` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Gemini engine with same extended tool suite as Codex smoke test | +| **Engine** | `gemini` | +| **Triggers** | Every 12h, PR, manual dispatch | +| **Timeout** | 15 minutes | +| **Real-world mapping** | Validates Gemini (Google) engine works through AWF — important for multi-engine support | +| **Gaps** | Same as Codex. Identical test requirements — could share test definition via imports. | +| **Integration test relationship** | Same as Codex — tests a different engine path through the same infrastructure | + +### 12. `smoke-chroot.lock.yml` — Smoke Chroot + +**Source**: `smoke-chroot.md` + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Chroot filesystem isolation by comparing host vs chroot runtime versions (Python, Node.js, Go) | +| **Engine** | `copilot` | +| **Triggers** | PR (with path filter: src/**, containers/**, package.json, smoke-chroot.md), manual dispatch | +| **Timeout** | 20 minutes | +| **Network allowed** | defaults, github | +| **Tools** | github (repos, pull_requests), bash | +| **Real-world mapping** | Validates the core chroot feature: host binaries must be accessible inside the container with matching versions | +| **Gaps** | Only tests 3 runtimes (Python, Node, Go). No Java/.NET/Ruby/Rust version comparison. Path-filtered — won't run on non-code PRs. | +| **Integration test relationship** | **Overlaps with** `chroot-languages.test.ts` but approaches differently: smoke test runs `awf` → agent compares versions; integration test runs `awf` → Jest assertions compare versions | + +**Unique architecture:** +- Pre-steps capture host versions, run `awf --skip-pull` for each runtime, compare +- Agent only reads result files and creates PR comment with comparison table +- Uses `--skip-pull` (locally-built containers from pre-steps) + +--- + +## Build-Test Workflows (Agentic) + +These are agentic workflows that clone external test repositories and run real build/test commands through the AWF sandbox. They validate that AWF's network filtering allows language-specific package managers to function correctly. + +All build-test workflows share a common pattern: +- **Engine**: `copilot` +- **Triggers**: PR (opened/synchronize/reopened), manual dispatch +- **Tools**: bash, github (with GH_AW_GITHUB_MCP_SERVER_TOKEN) +- **MCP**: ghcr.io/github/gh-aw-mcpg container +- **Safe outputs**: add-comment, add-labels +- **Error handling**: `safeoutputs-missing_tool` on clone failure, table-based reporting +- **Test repos**: `Mossaka/gh-aw-firewall-test-{language}` + +### 13. `build-test-bun.lock.yml` — Build Test Bun + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Install Bun from bun.sh, clone test-bun repo, run `bun install && bun test` on elysia and hono projects | +| **Network** | defaults, github, node, bun.sh | +| **Timeout** | 15 minutes | +| **Real-world mapping** | Validates AWF allows Bun runtime installation and package fetching | +| **Gaps** | Bun installed via curl (no version pinning). Only 2 projects tested. | +| **Integration test relationship** | Extends beyond integration tests — no integration test covers Bun specifically | + +### 14. `build-test-cpp.lock.yml` — Build Test C++ + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Clone test-cpp repo, run CMake + Make on fmt and json (nlohmann) libraries | +| **Network** | defaults, github | +| **Timeout** | 30 minutes | +| **Real-world mapping** | Validates C++ projects with no external network dependencies build inside AWF | +| **Gaps** | No package manager (vcpkg/conan) testing. Build-only, no test execution. | +| **Integration test relationship** | No integration test equivalent | + +### 15. `build-test-deno.lock.yml` — Build Test Deno + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Install Deno from deno.land, clone test-deno repo, run `deno test` on oak and std | +| **Network** | defaults, github, node, deno.land, jsr.io, dl.deno.land | +| **Timeout** | 15 minutes | +| **Real-world mapping** | Validates AWF allows Deno installation and JSR/deno.land package fetching | +| **Gaps** | No version pinning for Deno install. Only 2 projects. | +| **Integration test relationship** | No integration test equivalent | + +### 16. `build-test-dotnet.lock.yml` — Build Test .NET + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Clone test-dotnet repo, `dotnet restore && dotnet build && dotnet run` on hello-world and json-parse | +| **Runtime** | dotnet 8.0 | +| **Network** | defaults, github, dotnet | +| **Timeout** | 15 minutes | +| **Real-world mapping** | Validates NuGet package restoration and .NET build through AWF proxy | +| **Gaps** | Very simple projects. No `dotnet test` execution. | +| **Integration test relationship** | Complements `chroot-package-managers.test.ts` which tests `dotnet --version` but not actual NuGet restore | + +### 17. `build-test-go.lock.yml` — Build Test Go + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Clone test-go repo, `go mod download && go test ./...` on color, env, uuid projects | +| **Runtime** | go 1.22 | +| **Network** | defaults, github, go | +| **Timeout** | 15 minutes | +| **Real-world mapping** | Validates Go module proxy access and test execution through AWF | +| **Gaps** | Small projects with few dependencies. No CGO testing. | +| **Integration test relationship** | Complements `chroot-package-managers.test.ts` which tests `go mod download` but on a minimal project | + +### 18. `build-test-java.lock.yml` — Build Test Java + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Clone test-java repo, configure Maven proxy via `~/.m2/settings.xml`, `mvn compile && mvn test` on gson and caffeine | +| **Runtime** | java 21 | +| **Network** | defaults, github, java | +| **Timeout** | 15 minutes | +| **Real-world mapping** | Validates Maven proxy configuration workaround (Maven ignores JAVA_TOOL_OPTIONS proxy settings) | +| **Gaps** | Only Maven tested. No Gradle testing. Maven proxy config requires manual `settings.xml` — error-prone for users. | +| **Integration test relationship** | Extends `chroot-package-managers.test.ts` which doesn't test Maven dependency download | + +### 19. `build-test-node.lock.yml` — Build Test Node.js + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Clone test-node repo, `npm install && npm test` on clsx, execa, p-limit | +| **Runtime** | node 20 | +| **Network** | defaults, github, node | +| **Timeout** | 15 minutes | +| **Real-world mapping** | Validates npm registry access through AWF proxy | +| **Gaps** | Only npm tested, no yarn/pnpm. Small packages only. | +| **Integration test relationship** | Complements `chroot-package-managers.test.ts` which tests `npm install` on a minimal package | + +### 20. `build-test-rust.lock.yml` — Build Test Rust + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Clone test-rust repo, `cargo build && cargo test` on fd and zoxide | +| **Runtime** | rust stable | +| **Network** | defaults, github, rust, crates.io | +| **Timeout** | 30 minutes | +| **Real-world mapping** | Validates crates.io access and Cargo build/test through AWF proxy. Longer timeout for compilation. | +| **Gaps** | Only 2 projects. No nightly Rust testing. | +| **Integration test relationship** | Complements `chroot-package-managers.test.ts` which tests `cargo build` on a minimal project | + +--- + +## Security & Compliance Workflows + +### 21. `container-scan.yml` — Container Security Scan + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Trivy vulnerability scanning of Agent and Squid container images | +| **Triggers** | Push/PR to main (containers/** paths), weekly Sunday schedule, manual dispatch | +| **Timeout** | 15 minutes per job | +| **Real-world mapping** | Ensures container images don't ship with CRITICAL/HIGH CVEs | +| **Gaps** | Only scans Agent and Squid, not api-proxy or agent-act. | +| **Integration test relationship** | None — security scanning, not functional testing | + +### 22. `dependency-audit.yml` — Dependency Vulnerability Audit + +| Attribute | Value | +|-----------|-------| +| **What it tests** | `npm audit` on main package and docs-site package. Uploads SARIF to GitHub Security tab. | +| **Triggers** | Push/PR to main (ignoring .md), weekly Monday schedule, manual dispatch | +| **Timeout** | 5 minutes per job | +| **Real-world mapping** | Catches vulnerable npm dependencies before they ship | +| **Gaps** | Only npm, not container base image packages. | +| **Integration test relationship** | None | + +### 23. `codeql.yml` — CodeQL Analysis + +| Attribute | Value | +|-----------|-------| +| **What it tests** | CodeQL static analysis for JavaScript/TypeScript and GitHub Actions code | +| **Triggers** | Push/PR to main, weekly Monday schedule, manual dispatch | +| **Timeout** | 360 minutes (6 hours) | +| **Real-world mapping** | Catches security vulnerabilities (XSS, injection) and code quality issues | +| **Gaps** | No Shell/Bash analysis (container scripts, iptables rules not analyzed) | +| **Integration test relationship** | None — static analysis | + +### 24. `pr-title.yml` — PR Title Check + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Semantic PR title format (e.g., `feat:`, `fix:`, `docs:`) using `amannn/action-semantic-pull-request` | +| **Triggers** | PR to main (opened, edited, synchronize, reopened) | +| **Real-world mapping** | Enforces conventional commit format for automated changelog generation | +| **Gaps** | N/A | +| **Integration test relationship** | None | + +--- + +## Infrastructure Workflows + +### 25. `release.yml` — Release Pipeline + +| Attribute | Value | +|-----------|-------| +| **What it tests** | End-to-end release: version bump, build 4 container images (squid, agent, api-proxy, agent-act), create binaries (linux-x64/arm64, darwin-x64/arm64), generate changelog, create GitHub release | +| **Triggers** | Manual dispatch only (patch/minor/major choice) | +| **Real-world mapping** | Production release pipeline | +| **Gaps** | Binary smoke test only for linux-x64 (arm64 and macOS verified as valid ELF/Mach-O but not executed) | +| **Integration test relationship** | None — infrastructure | + +**Container images built:** +- `squid:VERSION` (linux/amd64 + arm64) — with cosign signing and SBOM +- `agent:VERSION` (linux/amd64 + arm64) — with cosign signing and SBOM, no-cache +- `api-proxy:VERSION` (linux/amd64 + arm64) — with cosign signing and SBOM +- `agent-act:VERSION` (linux/amd64 only) — with cosign signing and SBOM (retry logic) + +### 26. `deploy-docs.yml` — Deploy Documentation + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Builds and deploys docs-site to GitHub Pages | +| **Triggers** | Push to main (docs-site/** paths), manual dispatch | +| **Real-world mapping** | Documentation deployment | +| **Integration test relationship** | None | + +### 27. `copilot-setup-steps.yml` — Copilot Setup Steps + +| Attribute | Value | +|-----------|-------| +| **What it tests** | Installs gh-aw extension for GitHub Copilot Agent | +| **Triggers** | Manual dispatch, push to workflow file | +| **Real-world mapping** | Configures Copilot Agent environment with gh-aw | +| **Integration test relationship** | None | + +--- + +## Relationship Map: CI vs Integration Tests + +| CI Workflow | Related Integration Tests | Overlap Level | +|-------------|--------------------------|---------------| +| `test-chroot.yml` | `chroot-languages.test.ts`, `chroot-package-managers.test.ts`, `chroot-procfs.test.ts`, `chroot-edge-cases.test.ts` | **Direct** — CI runs these exact test files | +| `test-examples.yml` | `blocked-domains.test.ts`, `wildcard-patterns.test.ts` | **Indirect** — examples test similar scenarios (domain allow/block) | +| `test-coverage.yml` | All `src/*.test.ts` unit tests | **Direct** — runs unit test suite with coverage | +| `smoke-chroot.lock.yml` | `chroot-languages.test.ts` | **Overlapping** — both test runtime version matching, different approaches | +| `build-test-*.lock.yml` | `chroot-package-managers.test.ts` | **Complementary** — build-tests use real projects; integration uses minimal packages | +| `smoke-{claude,copilot,codex,gemini}.lock.yml` | None | **Unique** — only place that tests actual AI agents through AWF | +| `test-action.yml` | None | **Unique** — only place that tests the setup action | +| `build.yml` | None | **Prerequisite** — validates build on Node 20+22 | + +--- + +## Coverage Gap Analysis + +### What's Well Covered + +1. **Chroot functionality** — Tested at 3 levels: unit tests, integration tests, CI workflow, and smoke test +2. **Domain filtering** — Unit tests (domain-patterns), integration tests (blocked-domains, wildcard-patterns), examples +3. **Multi-engine support** — Smoke tests cover Claude, Copilot, Codex, Gemini +4. **Multi-language support** — Build-tests cover 8 languages (Bun, C++, Deno, .NET, Go, Java, Node, Rust) +5. **Container security** — Trivy scans, cosign signing, SBOM attestation + +### Gaps Identified + +1. **No integration tests run in CI** — The `test-integration.yml` is actually just a type-check. The non-chroot integration tests (blocked-domains, dns-servers, environment-variables, exit-code-propagation, etc.) have no dedicated CI workflow. + +2. **No macOS CI testing** — All CI runs on `ubuntu-latest`. AWF produces darwin binaries but never tests them in CI. + +3. **No arm64 CI testing** — Containers are built for arm64 in release but never tested on arm64 runners. + +4. **Duplicate lint execution** — Both `build.yml` and `lint.yml` run `npm run lint` on PRs. + +5. **Missing Python build-test** — Python pip/conda package installation through AWF proxy has no build-test workflow (despite Python being tested in chroot-languages). + +6. **Missing Ruby build-test** — Ruby gem installation through AWF proxy has no build-test workflow. + +7. **Maven proxy workaround not tested in integration** — The `~/.m2/settings.xml` workaround is only documented in build-test-java.md, not validated by integration tests. + +8. **No load/performance testing** — No tests for concurrent connections, large file transfers, or many-domain allowlists. + +9. **Smoke test non-determinism** — AI agent behavior varies between runs. A passing smoke test doesn't guarantee the next run passes. + +10. **No negative security testing in CI** — Integration tests cover `network-security.test.ts` (iptables bypass attempts), but this isn't run by any CI workflow. + +11. **Stale version in test-action.yml** — Tests hardcode `v0.7.0` which may diverge from current release. + +12. **No integration test coverage tracking** — `test-coverage.yml` only tracks unit test coverage. + +13. **api-proxy container not scanned** — `container-scan.yml` only scans agent and squid images, not the api-proxy image added later. diff --git a/docs/test-analysis/container-ops.md b/docs/test-analysis/container-ops.md new file mode 100644 index 000000000..7183d9764 --- /dev/null +++ b/docs/test-analysis/container-ops.md @@ -0,0 +1,369 @@ +# Container, Volume, and Operational Integration Tests Analysis + +This document provides a detailed analysis of integration tests covering container configuration, volume mounts, environment variables, git operations, log commands, and Docker-in-Docker removal verification. + +--- + +## Table of Contents + +1. [Container Working Directory Tests](#1-container-working-directory-tests) +2. [Volume Mount Tests](#2-volume-mount-tests) +3. [Git Operations Tests](#3-git-operations-tests) +4. [Environment Variable Tests](#4-environment-variable-tests) +5. [Log Commands Tests](#5-log-commands-tests) +6. [Docker Warning Tests](#6-docker-warning-tests) +7. [No Docker Tests](#7-no-docker-tests) +8. [Cross-Cutting Gaps](#8-cross-cutting-gaps) + +--- + +## 1. Container Working Directory Tests + +**File:** `tests/integration/container-workdir.test.ts` + +### What It Tests + +| Test | Description | +|------|-------------| +| Default working directory | Verifies that when `--container-workdir` is not specified, the container starts with `/workspace` as the working directory (Dockerfile default) | +| Custom working directory | Verifies `--container-workdir /tmp` changes the working directory | +| Command execution in workdir | Creates a file and lists it from `/tmp`, confirming commands execute relative to the custom workdir | +| Home directory as workdir | Sets workdir to `$HOME` (from host `process.env.HOME`), verifies it resolves correctly | +| Relative path navigation | Runs `cd .. && pwd` from `/tmp` to verify relative paths work from within the workdir | + +### Real-World Mapping + +This maps directly to how gh-aw invokes AWF via `BuildAWFArgs()` in `pkg/workflow/awf_helpers.go`. The `--container-workdir` flag is used to set the agent's working directory to the cloned repository directory (typically `/home/runner/work/repo/repo` on Actions runners). Getting this wrong means the AI agent can't find the code it needs to work on. + +### Gaps and Missing Coverage + +| Gap | Priority | Rationale | +|-----|----------|-----------| +| **Non-existent directory** | High | What happens if `--container-workdir /nonexistent` is specified? Should it fail or create the dir? | +| **Directory with spaces** | Medium | Paths like `--container-workdir "/my project"` could break shell quoting | +| **Deeply nested path** | Low | A path like `/a/b/c/d/e/f/g` that doesn't exist | +| **Permissions on workdir** | High | Verify the agent user can write to the custom workdir (not just navigate to it) | +| **Interaction with volume mounts** | High | When `--container-workdir /data` is set AND `--mount /host/dir:/data:ro` is used, can the agent navigate correctly? | +| **Workdir inside chroot** | Medium | In chroot mode, `/workspace` maps differently — no test verifies chroot + workdir interaction | + +### Edge Cases + +- Setting workdir to `/` (root filesystem) +- Setting workdir to a symlinked directory +- Setting workdir to a directory owned by a different user + +--- + +## 2. Volume Mount Tests + +**File:** `tests/integration/volume-mounts.test.ts` + +### What It Tests + +| Test # | Test Name | Description | +|--------|-----------|-------------| +| 1 | Read-only custom mount | Host file at `testDir/test.txt` → mounted at `/data/test.txt:ro` → agent can read it | +| 2 | Read-write custom mount | Container writes to `/data/output.txt` → file appears on host filesystem | +| 3 | Multiple custom mounts | Two separate directories mounted at `/mount1` and `/mount2` — both accessible | +| 4 | Blanket mount removed | When custom mounts are provided, host paths outside mounts are NOT accessible (security isolation) | +| 5 | No /host mount | With custom mounts, `/host` (the full host filesystem) is not mounted (verified by `ls /host` failing) | +| 6 | Essential mounts (HOME) | Even with custom mounts, `$HOME` is still set and its directory exists | +| 7 | Backward compatibility | Without custom mounts, `/host` blanket mount is present (legacy default behavior) | +| 8 | Default mode is rw | Mount without `:ro` or `:rw` suffix defaults to read-write | +| 9 | Debug logging | Debug logs contain a message about custom volume mount configuration | +| 10 | Current working directory | Mounts a project directory at `/workspace` and reads a file from it | +| 11 | Mixed ro/rw mounts | One directory mounted as `:ro` (reads config), another as `:rw` (writes log) | + +### Real-World Mapping + +Volume mounts are central to AWF's value proposition. In production gh-aw workflows: +- **Test 4-5 (security isolation):** When custom mounts are specified, the agent should ONLY see what it's given. This prevents a compromised agent from reading SSH keys, cloud credentials, or other sensitive host files. +- **Test 7 (backward compatibility):** The default "blanket mount" mode (`/host`) is how chroot mode works — the entire host FS is mounted read-only with specific writable overlays. +- **Test 10 (workspace mount):** Maps to the typical gh-aw pattern of mounting the cloned repo directory as `/workspace`. +- **Test 11 (mixed):** Maps to mounting source code as read-only but a build output directory as read-write. + +### Gaps and Missing Coverage + +| Gap | Priority | Rationale | +|-----|----------|-----------| +| **Read-only enforcement** | Critical | Test 1 verifies reading works on `:ro` mounts, but does NOT verify that WRITING to a `:ro` mount FAILS. This is a security property. | +| **Invalid mount paths** | High | No test for host paths that don't exist, or container paths that conflict with system dirs | +| **Mount path traversal** | High | No test for attempts to escape mount boundaries (e.g., `cat /data/../../../etc/passwd`) | +| **Symlink mounts** | Medium | Mounting a host directory that contains symlinks pointing outside the mount | +| **Large file mounts** | Low | Performance with large files or many files in mounted directory | +| **Mount over system directories** | High | What happens if you mount to `/usr`, `/etc`, or `/bin`? | +| **Empty directory mount** | Low | Mounting an empty directory | +| **File (not directory) mount** | High | CLAUDE.md explicitly warns against file bind mounts due to atomic write issues. No test validates this limitation or documents the failure mode. | +| **Nested mount paths** | Medium | E.g., mounting `/data` and `/data/subdir` separately | +| **Mount with special chars in path** | Medium | Paths containing spaces, unicode, or shell metacharacters | + +### Edge Cases + +- Mount the same host directory to two different container paths +- Mount a FIFO or device file +- Mount a directory on a different filesystem (e.g., NFS, tmpfs) +- Concurrent writes from host and container to the same rw mount + +--- + +## 3. Git Operations Tests + +**File:** `tests/integration/git-operations.test.ts` + +### What It Tests + +| Test | Description | +|------|-------------| +| git ls-remote to allowed domain | `git ls-remote` to `github.com` succeeds, returns a commit hash | +| git ls-remote to subdomain | Same as above (tests subdomain matching, though the URL is identical — likely a test naming issue) | +| git ls-remote blocked | `git ls-remote` to `gitlab.com` fails when only `github.com` is allowed | +| git clone allowed | `git clone --depth 1` of a public repo succeeds, contains README | +| git clone blocked | `git clone` from `gitlab.com` fails | +| git config --global list | Git global config can be read/listed | +| git config set | Can set `user.email` via `git config --global` and read it back | +| Sequential git operations | Two `git ls-remote` commands back-to-back both succeed | + +### Real-World Mapping + +Git operations are the most common network activity in agentic workflows. Every AI agent that works on code needs to: +1. Clone the repository (or it's pre-cloned on the runner) +2. Fetch remote refs to check for updates +3. Push commits (if the workflow creates PRs) + +The firewall must transparently proxy HTTPS git operations through Squid. Git uses HTTPS CONNECT tunnels, which map to Squid's `TCP_TUNNEL` decision. + +### Gaps and Missing Coverage + +| Gap | Priority | Rationale | +|-----|----------|-----------| +| **git push** | Critical | No test for `git push` — the most important write operation in agentic workflows. Agents create branches and push PRs. | +| **git with authentication** | Critical | No test with `GITHUB_TOKEN` or PAT. Production agents always use authenticated git to push and access private repos. The test header mentions "Git with authentication" but no such test exists. | +| **git fetch** | High | Listed in the file header but not actually tested. `git fetch` inside an existing clone is a common operation. | +| **git submodule operations** | Medium | Submodules require fetching from potentially different domains | +| **git over SSH** | Low | AWF blocks non-HTTP(S), but no test confirms git-over-SSH fails gracefully | +| **git LFS** | Medium | Large File Storage uses different endpoints that may need separate domain allowlisting | +| **Concurrent git operations** | Low | Multiple parallel clones/fetches | +| **git with custom proxy config** | Medium | Does `git config http.proxy` interact with AWF's transparent proxy? | +| **Subdomain test is a duplicate** | Bug | "should allow git ls-remote to subdomain" uses exactly the same URL as the first test (`github.com`). Should test actual subdomain like `api.github.com` or `gist.github.com`. | + +### Edge Cases + +- Git operation that exceeds Squid's idle timeout +- Repository with very large history (timeout during clone) +- `git push --force` (should be blocked by workflow permissions, not firewall, but worth noting) +- Git credential helper interactions + +--- + +## 4. Environment Variable Tests + +**File:** `tests/integration/environment-variables.test.ts` + +### What It Tests + +| Test | Description | +|------|-------------| +| Single env var | `-e TEST_VAR=hello_world` is passed to container, `echo $TEST_VAR` outputs it | +| Multiple env vars | Three variables (`VAR1`, `VAR2`, `VAR3`) all reach the container | +| Special characters | Value with spaces (`"value with spaces"`) is correctly preserved | +| Empty value | An empty string value is detected as empty inside the container | +| PATH preservation | Default `PATH` inside the container includes `/usr/bin` or `/bin` | +| HOME set | `$HOME` is set to `/root` or `/home/*` | +| No sensitive leakage | `printenv | grep TOKEN|SECRET|PASSWORD|KEY` shows nothing by default | +| Numeric values | String `"12345"` passed as env var arrives correctly | + +### Real-World Mapping + +Environment variables are how secrets and configuration flow into agentic workflows: +- `GITHUB_TOKEN` — for Git operations and API calls +- `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` — for AI engine API calls +- `HOME` / `PATH` — for tool discovery and configuration +- Custom env vars from workflow `env:` blocks + +The `--env-all` flag is used in production by gh-aw's `BuildAWFArgs()` to pass all GitHub Actions environment variables (including secrets) into the container. + +### Gaps and Missing Coverage + +| Gap | Priority | Rationale | +|-----|----------|-----------| +| **`--env-all` flag** | Critical | The file header mentions it but NO test actually uses `envAll: true`. This is the primary mode used in production. | +| **Env var with equals sign in value** | High | `KEY=value=with=equals` — the parsing uses first `=` as delimiter, but no test verifies | +| **Env var with newlines** | High | Multi-line values (common in PEM certificates, SSH keys) | +| **Env var with shell metacharacters** | Medium | Values containing `$`, backticks, `$(...)` that could be expanded | +| **Very long env var values** | Low | Docker has limits on environment variable size | +| **Env var overriding system vars** | High | What happens if `-e PATH=/empty` or `-e HOME=/nonexistent` is passed? | +| **Proxy env vars** | High | AWF sets `HTTP_PROXY`, `HTTPS_PROXY` internally. No test verifies these exist inside the container or that user-provided proxy vars don't conflict. | +| **JAVA_TOOL_OPTIONS** | Medium | AWF sets JVM proxy properties. No test verifies this. | +| **Env var ordering/precedence** | Medium | If the same key is specified twice, which value wins? | +| **Sensitive var leakage test is weak** | High | The "no sensitive leakage" test just greps for keywords in printenv. It doesn't set actual secrets on the host and verify they DON'T appear. The comment even acknowledges "This depends on what's in the host environment." | + +### Edge Cases + +- Unicode characters in env var names or values +- Env var names with dots (e.g., `npm_config_registry`) +- Boolean-like values (`true`, `false`, `0`, `1`) +- Env vars that Docker treats specially (e.g., `DOCKER_HOST`) + +--- + +## 5. Log Commands Tests + +**File:** `tests/integration/log-commands.test.ts` + +### What It Tests + +**Live Integration Tests (3 tests):** + +| Test | Description | +|------|-------------| +| Log generation | Runs `curl` through AWF with `--keep-containers`, verifies `squid-logs/access.log` is created and non-empty | +| Log parsing | Runs two curls (one allowed, one blocked), reads the log file, and verifies parsed entries have required fields (`timestamp`, `host`, `statusCode`, `decision`) | +| Allowed vs blocked distinction | Same two-curl setup, filters entries by decision type (`allowed` vs `blocked`), verifies at least one entry exists | + +**Unit Tests for LogParser (4 tests):** + +| Test | Description | +|------|-------------| +| Squid log format parsing | Parses a synthetic log line, verifies `host`, `statusCode`, `decision` fields | +| Blocked entry identification | Parses a `TCP_DENIED` / 403 log line, confirms it's classified as blocked | +| Unique domains | Parses 3 entries (2 unique domains), verifies deduplication | +| Domain filtering | Filters by `github.com` subdomain matching | + +### Real-World Mapping + +Log analysis is critical for: +1. **GitHub Actions step summaries** — `awf logs summary >> $GITHUB_STEP_SUMMARY` shows what domains were accessed/blocked +2. **Debugging blocked requests** — when an agent fails because a required domain isn't in the allowlist +3. **Security auditing** — reviewing what external services an agent contacted +4. **Compliance** — proving that sensitive internal services weren't accessed + +### Gaps and Missing Coverage + +| Gap | Priority | Rationale | +|-----|----------|-----------| +| **`awf logs stats` command** | High | The stats subcommand is never actually invoked end-to-end. Only the parser is unit-tested. | +| **`awf logs summary` command** | High | Similarly never invoked as a real CLI command | +| **`awf logs` command (view)** | High | The base `logs` command with `--follow`, `--format`, etc. is never tested | +| **JSON output format** | Medium | Stats/summary can output JSON — not tested | +| **Markdown output format** | Medium | Summary defaults to markdown — not tested | +| **Pretty output format** | Medium | Stats defaults to pretty — not tested | +| **`--source` flag** | Medium | Specifying custom log source path | +| **`--list` flag** | Medium | Listing available log sources | +| **Empty logs** | Medium | What do commands output when no logs exist? | +| **Integration tests are fragile** | High | All 3 live tests have `if (fs.existsSync(...))` guards that silently pass when logs aren't created. A timing issue could make the test pass vacuously. | +| **Log rotation / large logs** | Low | Behavior with very large access logs | +| **iptables log parsing** | Medium | The `LogParser` has `parseIptablesLog()` and `readIptablesLog()` methods that are never tested in integration | + +### Edge Cases + +- Concurrent requests producing interleaved log entries +- Log entries with unusual User-Agent strings containing quotes +- Very long URLs in log entries +- Squid log buffering delays (tests already work around this with `setTimeout(1000)`) + +--- + +## 6. Docker Warning Tests + +**File:** `tests/integration/docker-warning.test.ts` + +### What It Tests + +**NOTE: This entire test suite is `describe.skip`'d due to a Node.js build issue in local container images.** + +| Test | Description | +|------|-------------| +| docker run warning | Running `docker run alpine echo hello` shows a helpful error about DinD removal (v0.9.1) | +| docker-compose warning | `docker-compose up` fails (docker-compose not installed) | +| which docker | `which docker` shows `/usr/bin/docker` exists (stub script) | +| docker --help | Shows the DinD removal warning and link to breaking changes | +| docker version | Fails with helpful error message | + +### Real-World Mapping + +Docker-in-Docker was removed in v0.9.1 (PR #205) because it was a security risk and unnecessary for agentic workflows. These tests verify that when agents try to use Docker (which some MCP servers or build tools might attempt), they get a clear error message rather than a confusing failure. + +### Gaps and Missing Coverage + +| Gap | Priority | Rationale | +|-----|----------|-----------| +| **Tests are entirely skipped** | Critical | The entire suite is `describe.skip`. These tests provide zero coverage. The comment says "tests will be enabled once the build issue is fixed" — this is a stale TODO. | +| **Requires `buildLocal: true`** | Context | These tests only work with locally-built images that include the Docker stub script. The GHCR images may or may not have the stub. | +| **No equivalent in `no-docker.test.ts`** | — | The `no-docker.test.ts` file covers the same scenario from a different angle (see below). | + +### Edge Cases + +- N/A (tests are skipped) + +--- + +## 7. No Docker Tests + +**File:** `tests/integration/no-docker.test.ts` + +### What It Tests + +| Test | Description | +|------|-------------| +| docker not available | `which docker` fails (docker-cli not installed in container) | +| docker run fails gracefully | `docker run alpine echo hello` fails with stderr containing "docker" or "not found" | +| docker-compose not available | `which docker-compose` fails | +| docker socket not mounted | `/var/run/docker.sock` is not present in the container | + +### Real-World Mapping + +This is the complement to the docker-warning tests. While docker-warning tests verify the stub script provides helpful messages (when building locally), these tests verify the baseline: Docker is simply not available in GHCR images. This is a key security property — the agent cannot escape the firewall by starting new containers. + +### Gaps and Missing Coverage + +| Gap | Priority | Rationale | +|-----|----------|-----------| +| **docker buildx / docker compose (plugin)** | Medium | The new `docker compose` (without hyphen) is a plugin. Not tested. | +| **containerd / nerdctl** | Low | Alternative container runtimes that could be present | +| **podman** | Low | Another alternative runtime | +| **Interplay with buildLocal** | Medium | When `buildLocal: true` is used, the docker-warning stub IS installed. These tests don't use `buildLocal`, so they test a different code path. No test covers both paths in the same suite. | +| **Socket at alternative paths** | Low | Docker socket can be at non-default paths | + +### Edge Cases + +- Agent installing Docker via `apt-get` (should fail due to network restrictions unless docker.io is in allowlist) +- Agent downloading a static Docker binary via curl (should fail unless the download domain is allowed) + +--- + +## 8. Cross-Cutting Gaps + +### Architectural Gaps + +| Gap | Description | Affected Tests | +|-----|-------------|----------------| +| **Chroot mode interaction** | None of these operational tests verify behavior in chroot mode. All use the default container mode. Chroot changes path semantics significantly. | All files | +| **`--env-all` never tested** | The most commonly used env mode in production is completely untested | environment-variables | +| **Cleanup verification** | Tests call `cleanup()` in beforeAll/afterAll but never verify cleanup succeeded. If cleanup fails silently, tests may interfere with each other. | All files | +| **Signal handling** | No test sends SIGINT/SIGTERM to AWF during operation and verifies cleanup | All files | +| **Timeout behavior** | No test verifies what happens when the agent command exceeds `--timeout` | All files | +| **`--keep-containers` interaction** | Only log-commands tests use `keepContainers: true`. No test verifies the flag preserves containers AND that subsequent cleanup removes them. | log-commands | +| **Error messages** | No test verifies user-facing error messages for invalid inputs (bad mount format, invalid workdir, etc.) | volume-mounts, container-workdir | + +### Test Infrastructure Observations + +1. **Fragile log assertions:** The log-commands tests use `if (fs.existsSync(...))` guards that allow tests to pass even when no logs are generated. This should be `expect(fs.existsSync(...)).toBe(true)` to avoid false positives. + +2. **Duplicate subdomain test:** In `git-operations.test.ts`, "should allow git ls-remote to subdomain" uses the exact same URL as the first test. It should use an actual subdomain like `gist.github.com`. + +3. **Test isolation:** Each test spins up a full Docker environment (Squid + Agent containers), which takes 20-60 seconds. This makes the test suite slow (~10-15 minutes total) but provides high-fidelity end-to-end coverage. + +4. **Environment dependency:** Several tests depend on the host having Docker, sudo access, and network connectivity to github.com. This makes them unsuitable for offline or restricted CI environments. + +5. **Timeout margins:** Most individual test timeouts are 120s while the inner command timeouts are 30-60s. This provides a reasonable margin, but some tests (git clone) could be tight on slow networks. + +### Missing Test Categories + +| Category | Description | +|----------|-------------| +| **Concurrent operations** | No test runs multiple AWF instances simultaneously to verify network isolation | +| **Resource limits** | No test for container memory/CPU limits | +| **Filesystem permissions** | Limited testing of file ownership (UID/GID) inside the container | +| **Binary execution** | No test compiles or runs a binary inside the container | +| **Network partition** | No test for behavior when Docker network is unavailable or degraded | +| **Upgrade/migration** | No test for behavior differences between image versions | diff --git a/docs/test-analysis/domain-network.md b/docs/test-analysis/domain-network.md new file mode 100644 index 000000000..59a91a58f --- /dev/null +++ b/docs/test-analysis/domain-network.md @@ -0,0 +1,330 @@ +# Domain & Network Integration Test Analysis + +> Generated 2026-02-25. Covers 6 test files in `tests/integration/` related to domain filtering, wildcard patterns, network security, DNS, and localhost access. + +--- + +## Table of Contents + +1. [blocked-domains.test.ts](#1-blocked-domainstestts) +2. [wildcard-patterns.test.ts](#2-wildcard-patternstestts) +3. [empty-domains.test.ts](#3-empty-domainstestts) +4. [network-security.test.ts](#4-network-securitytestts) +5. [dns-servers.test.ts](#5-dns-serverstestts) +6. [localhost-access.test.ts](#6-localhost-accesstestts) +7. [Cross-Cutting Gaps](#7-cross-cutting-gaps) + +--- + +## 1. blocked-domains.test.ts + +**File:** `tests/integration/blocked-domains.test.ts` (185 lines, 9 tests) + +### What It Tests + +| # | Test | Description | +|---|------|-------------| +| 1 | `should block specific domain even when parent is allowed` | Allows `github.com`, accesses `api.github.com` — verifies subdomain auto-inclusion (expects success, not actually testing block) | +| 2 | `should allow requests to allowed domains` | Simple allow-list test: `github.com` allows `api.github.com/zen` | +| 3 | `should block requests to non-allowed domains` | With only `github.com` allowed, `example.com` is blocked (expects `toFail`) | +| 4 | `should handle multiple blocked domains` | Two domains in allow-list (`github.com`, `npmjs.org`), verifies `api.github.com` succeeds | +| 5 | `should show allowed domains in debug output` | Verifies `[INFO] Allowed domains:` appears in stderr with `--log-level debug` | +| 6 | `should handle case-insensitive domain matching` | Accesses `https://API.GITHUB.COM/zen` with `github.com` in allow-list — DNS/Squid are case-insensitive | +| 7 | `should handle domains with trailing dots` | Allows `github.com.` (FQDN trailing dot) — verifies normalization handles it | +| 8 | `should handle domains with leading/trailing whitespace` | Allows ` github.com ` — verifies `parseDomains()` trims whitespace | +| 9 | `should block IP address access when only domain is allowed` | Resolves `api.github.com` to IP, curls the raw IP — expects block | + +### Real-World Mapping + +| Test | Real-World Scenario | +|------|---------------------| +| #1-2 | When Claude/Copilot agents access `api.github.com` with `github.com` whitelisted — the most common AWF configuration | +| #3 | When a malicious prompt causes an agent to access a non-whitelisted domain (e.g., exfiltrating code to attacker domain) | +| #4 | Multi-service workflow: agent needs GitHub API + npm registry | +| #6-8 | Robustness against variant domain representations in agent HTTP clients (Node.js fetch, Python requests, curl) | +| #9 | SSRF mitigation: agent resolves domain to IP and curls IP directly to bypass domain filtering | + +### Gaps and Missing Coverage + +1. **No actual `--block-domains` flag testing**: Despite the file name, no test uses the `--block-domains` CLI option. All tests use `--allow-domains` only. The `blockedDomains` field in `AwfRunner` options isn't even defined. +2. **No block-overrides-allow precedence test**: The Squid config places blocked domain ACLs before allowed domain ACLs. No test verifies that `--block-domains internal.corp.com --allow-domains corp.com` actually blocks `internal.corp.com`. +3. **No `--block-domains-file` test**: File-based blocklist parsing is untested in integration. +4. **No `--allow-domains-file` test**: File-based allowlist parsing is untested in integration (only unit-tested in `cli.test.ts`). +5. **Test #1 is misleading**: Named "should block specific domain even when parent is allowed" but doesn't actually test blocking — it just tests subdomain inclusion. The test name suggests `--block-domains` functionality. +6. **No test for blocking a subdomain while allowing parent**: e.g., `--allow-domains github.com --block-domains api.github.com` then verify `api.github.com` is blocked but `github.com` works. + +### Edge Cases Missing + +- Domain with port number (e.g., `example.com:8080`) +- Punycode / internationalized domain names (e.g., `xn--nxasmq6b.example.com`) +- Domain that is a prefix of an allowed domain (e.g., allow `github.com`, test `github.com.evil.com` — should be blocked) +- Multiple consecutive requests to different domains in the same container session +- Very long domain names (close to 253 char RFC limit) + +--- + +## 2. wildcard-patterns.test.ts + +**File:** `tests/integration/wildcard-patterns.test.ts` (185 lines, 11 tests) + +### What It Tests + +| # | Test | Describe Block | Description | +|---|------|----------------|-------------| +| 1 | `should allow subdomain with *.github.com pattern` | Leading Wildcard | `*.github.com` allows `api.github.com` | +| 2 | `should allow raw.githubusercontent.com` | Leading Wildcard | `*.githubusercontent.com` + `github.com` allows `raw.githubusercontent.com` | +| 3 | `should allow nested subdomains` | Leading Wildcard | `*.github.com` allows `api.github.com` (duplicate of #1) | +| 4 | `should match domain case-insensitively` | Case Insensitivity | `github.com` allows `API.GITHUB.COM` | +| 5 | `should match wildcard pattern case-insensitively` | Case Insensitivity | `*.GitHub.COM` allows `API.GITHUB.COM` | +| 6 | `should allow exact domain match` | Plain Domain | `github.com` allows `github.com/robots.txt` | +| 7 | `should allow subdomains of plain domain` | Plain Domain | `github.com` implicitly allows `api.github.com` | +| 8 | `should allow domains matching any of multiple patterns` | Multiple Patterns | Three wildcard patterns, one domain tested | +| 9 | `should combine wildcard and plain domain patterns` | Multiple Patterns | Mixed wildcard + plain domain list | +| 10 | `should block domain not matching any pattern` | Non-Matching | `*.github.com` blocks `example.com` | +| 11 | `should block similar-looking domain` | Non-Matching | `*.github.com` blocks `notgithub.com` (suffix attack) | + +### Real-World Mapping + +| Test | Real-World Scenario | +|------|---------------------| +| #1-3 | Agent accessing GitHub API, raw file downloads — extremely common in agentic workflows | +| #2 | When Claude downloads raw files from GitHub repos for code review | +| #4-5 | HTTP client libraries may normalize case differently — ensures consistent behavior | +| #6-7 | Core domain filtering: `github.com` in frontmatter allows all GitHub subdomains | +| #8-9 | Complex workflows needing multiple code hosting platforms (GitHub + GitLab + Bitbucket) | +| #10-11 | Security: preventing domain confusion attacks where `notgithub.com` could be attacker-controlled | + +### Gaps and Missing Coverage + +1. **No mid-domain wildcard test**: `api-*.example.com` pattern is mentioned in `domain-patterns.ts` docs but never tested in integration. This pattern is supported by the `wildcardToRegex()` function. +2. **No deeply nested subdomain test**: e.g., `*.github.com` should match `a.b.c.github.com`. The regex `[a-zA-Z0-9.-]*` allows dots, so this should work, but it's untested. +3. **No bare wildcard exclusion test**: `*.github.com` should NOT match `github.com` itself (only subdomains). This is a critical semantic difference that's untested. +4. **Test #3 is a duplicate of #1**: Both test `*.github.com` allowing `api.github.com`. +5. **No protocol-specific wildcard test**: e.g., `https://*.github.com` — the domain-patterns module supports this but it's not integration-tested. +6. **No test for multiple wildcards matching the same domain**: e.g., both `*.github.com` and `*.com` — verifies no regex conflict. + +### Edge Cases Missing + +- Wildcard with only TLD: `*.com` (should this be rejected as too broad? Currently `validateDomainOrPattern` doesn't explicitly reject it) +- Trailing dot with wildcard: `*.github.com.` +- Empty subdomain match: does `*.github.com` match `.github.com`? +- Wildcard pattern that overlaps with a plain domain in the allow-list + +--- + +## 3. empty-domains.test.ts + +**File:** `tests/integration/empty-domains.test.ts` (149 lines, 8 tests) + +### What It Tests + +| # | Test | Describe Block | Description | +|---|------|----------------|-------------| +| 1 | `should block all network access when no domains are specified` | Network Blocking | `allowDomains: []` blocks `https://example.com` | +| 2 | `should block HTTPS traffic` | Network Blocking | Empty allow-list blocks `https://api.github.com` | +| 3 | `should block HTTP traffic` | Network Blocking | Empty allow-list blocks `http://httpbin.org` | +| 4 | `should allow commands that do not require network` | Offline Commands | `echo "Hello"` works with empty allow-list | +| 5 | `should allow file system operations` | Offline Commands | File create/read/delete works without network | +| 6 | `should allow local computations` | Offline Commands | `expr 2 + 2` works without network | +| 7 | `should indicate no domains in debug output` | Debug Output | Stderr matches `No allowed domains specified` or `all network access will be blocked` | +| 8 | `should block network even when DNS resolution succeeds` | DNS Behavior | DNS resolves but HTTP connection still blocked | + +### Real-World Mapping + +| Test | Real-World Scenario | +|------|---------------------| +| #1-3 | "Air-gapped" agent mode: running code analysis or refactoring that needs zero network access. This is the safest mode for untrusted code. | +| #4-6 | Agent performs local-only tasks (file editing, computation) inside the sandbox — must work even with all network blocked | +| #7 | Operator debugging: seeing confirmation that no domains are configured | +| #8 | Defense-in-depth: even if DNS leaks (which it does — DNS is allowed through), the HTTP/HTTPS layer still blocks connections | + +### Gaps and Missing Coverage + +1. **No test for DNS exfiltration**: DNS queries are allowed even with empty domains. An attacker could encode data in DNS queries to exfiltrate secrets. There's no test for DNS-based data exfiltration (though `--dns-servers` limits which DNS servers are reachable). +2. **No test for non-HTTP protocols**: What about SSH (port 22), SMTP (port 25), or raw TCP? These should be blocked by iptables rules, but aren't tested in this file. +3. **No test for localhost access with empty domains**: Can the agent connect to `127.0.0.1` services with empty domains? +4. **No test for the `--allow-domains` flag being completely omitted** (vs. passed as empty): The CLI behavior may differ. + +### Edge Cases Missing + +- UDP traffic (e.g., DNS-over-UDP to a custom server on a non-standard port) +- ICMP/ping with empty domains +- Agent trying to curl localhost or 127.0.0.1 with empty domains + +--- + +## 4. network-security.test.ts + +**File:** `tests/integration/network-security.test.ts` (232 lines, 14 tests) + +### What It Tests + +| # | Test | Describe Block | Description | +|---|------|----------------|-------------| +| 1 | `should drop NET_ADMIN after iptables setup` | Capability Restrictions | `iptables -t nat -L OUTPUT` fails inside container | +| 2 | `should block iptables flush attempt` | Capability Restrictions | `iptables -t nat -F OUTPUT` fails (can't flush NAT rules) | +| 3 | `should block iptables delete attempt` | Capability Restrictions | `iptables -t nat -D OUTPUT 1` fails (can't delete rules) | +| 4 | `should block iptables insert attempt` | Capability Restrictions | `iptables -t nat -I OUTPUT -j ACCEPT` fails (can't add ACCEPT rule) | +| 5 | `should block curl --connect-to bypass` | Firewall Bypass | `curl --connect-to ::github.com: https://example.com` — tries to resolve `example.com` via `github.com` IP | +| 6 | `should block NO_PROXY bypass` | Firewall Bypass | `NO_PROXY='*'` doesn't bypass the firewall (because firewall uses iptables, not env vars) | +| 7 | `should block ALL_PROXY bypass` | Firewall Bypass | `ALL_PROXY=''` doesn't bypass the firewall | +| 8 | `should block AWS metadata endpoint` | SSRF Protection | `169.254.169.254` is blocked | +| 9 | `should block AWS metadata with path` | SSRF Protection | `169.254.169.254/latest/meta-data/` is blocked | +| 10 | `should block GCP metadata endpoint` | SSRF Protection | `metadata.google.internal` with GCP header is blocked | +| 11 | `should block Azure metadata endpoint` | SSRF Protection | `169.254.169.254/metadata/instance` with Azure header is blocked | +| 12 | `should block DNS over HTTPS (DoH)` | DNS Security | `cloudflare-dns.com/dns-query` is blocked | +| 13 | `should block Google DoH endpoint` | DNS Security | `dns.google/dns-query` is blocked | +| 14 | `should maintain firewall after bypass attempt` | Effectiveness | Attempts `iptables flush` then `curl example.com` — both fail | + +### Real-World Mapping + +| Test | Real-World Scenario | +|------|---------------------| +| #1-4 | **Critical security**: Malicious prompt injection causes agent to try modifying iptables rules to disable the firewall. The `capsh --drop=cap_net_admin` in `entrypoint.sh` prevents this. | +| #5 | Sophisticated bypass: agent uses `curl --connect-to` to route blocked domain through allowed domain's IP | +| #6-7 | Agent sets proxy-bypass env vars hoping to avoid Squid. AWF uses iptables DNAT (L3), not proxy env vars, so this doesn't work. | +| #8-11 | **SSRF protection**: Agent on cloud infrastructure (AWS/GCP/Azure) tries to access instance metadata to steal credentials. This is a top OWASP risk. | +| #12-13 | Agent tries DNS-over-HTTPS to exfiltrate data or resolve blocked domains via an encrypted channel | +| #14 | Compound attack: first try to disable firewall, then access blocked domain | + +### Gaps and Missing Coverage + +1. **No test for `http_proxy`/`https_proxy` override**: Agent could set `http_proxy=direct://` or `https_proxy=` to try bypassing. While AWF uses iptables DNAT, this should still be tested. +2. **No test for iptables -A (append) bypass**: Tests cover `-F` (flush), `-D` (delete), `-I` (insert), but not `-A` (append). +3. **No test for `ip route` manipulation**: Agent might try to add a route to bypass Squid. Should be blocked by dropped NET_ADMIN. +4. **No test for `nsenter` or container escape**: Agent in container tries to escape to host namespace. +5. **No test for SSRF via DNS rebinding**: Agent resolves allowed domain that DNS-rebinds to internal IP (e.g., `169.254.169.254`). +6. **No test for SSRF via redirect**: Agent accesses allowed domain that returns HTTP 302 redirect to `http://169.254.169.254`. +7. **No `169.254.0.0/16` link-local range test**: Only `169.254.169.254` (metadata) is tested — but other link-local IPs could be dangerous. +8. **No test for IMDSv2 (token-based)**: AWS IMDSv2 uses PUT to get a token first, then GET metadata. Only IMDSv1-style requests are tested. +9. **No test for internal Docker network access**: Can agent reach `172.30.0.10` (Squid) or `172.30.0.1` (gateway) directly? + +### Edge Cases Missing + +- `curl --resolve` bypass (similar to `--connect-to`) +- Agent using `socat` or `nc` for raw TCP connections +- Agent trying to use container's Docker socket if mounted +- Agent trying to use IPv6 link-local addresses +- Agent modifying `/etc/resolv.conf` to point to a malicious DNS server + +--- + +## 5. dns-servers.test.ts + +**File:** `tests/integration/dns-servers.test.ts` (115 lines, 6 tests) + +### What It Tests + +| # | Test | Description | +|---|------|-------------| +| 1 | `should resolve DNS with default servers` | `nslookup github.com` works with default DNS (8.8.8.8, 8.8.4.4) | +| 2 | `should resolve DNS with custom Google DNS` | `nslookup github.com 8.8.8.8` works explicitly | +| 3 | `should resolve DNS with Cloudflare DNS` | `nslookup github.com 1.1.1.1` works | +| 4 | `should show DNS servers in debug output` | stderr contains `DNS` or `dns` | +| 5 | `should resolve multiple domains sequentially` | `nslookup github.com && nslookup api.github.com` both succeed | +| 6 | `should resolve DNS for allowed domains` | `dig github.com +short` returns IP addresses | + +### Real-World Mapping + +| Test | Real-World Scenario | +|------|---------------------| +| #1 | Default configuration: agent resolves GitHub API before making requests | +| #2-3 | Organization mandates specific DNS servers (corporate DNS policy, Cloudflare for privacy) | +| #5 | Agent resolves multiple APIs during a workflow (GitHub + npm + PyPI) | +| #6 | Verifying DNS resolution actually returns IPs (not NXDOMAIN or empty) | + +### Gaps and Missing Coverage + +1. **No `--dns-servers` CLI option test**: None of these tests actually use the `--dns-servers` flag. They all use default DNS. The `dnsServers` option exists in `AwfRunner` but is never passed. +2. **No DNS restriction test**: With `--dns-servers 8.8.8.8`, can the agent still query `1.1.1.1`? The iptables rules should block non-whitelisted DNS servers, but this is untested. +3. **No IPv6 DNS server test**: `isValidIPv6()` exists in the CLI, but no integration test uses IPv6 DNS. +4. **No invalid DNS server test**: What happens with `--dns-servers 999.999.999.999`? The CLI validates, but no integration test covers this path. +5. **No DNS resolution of blocked domains test**: Can the agent resolve a domain that's NOT in the allow-list? (DNS resolution should succeed per design, but HTTP access should be blocked.) +6. **No DNS timeout/failure test**: What happens when DNS servers are unreachable? + +### Edge Cases Missing + +- DNS resolution for non-existent domains (NXDOMAIN) +- DNS resolution for domains with CNAME chains +- DNS resolution with `--dns-servers 127.0.0.1` (using container-local DNS) +- Agent trying to use `dig @attacker-dns.com` with a non-whitelisted DNS server + +--- + +## 6. localhost-access.test.ts + +**File:** `tests/integration/localhost-access.test.ts` (138 lines, 8 tests) + +### What It Tests + +| # | Test | Description | +|---|------|-------------| +| 1 | `should automatically enable host access when localhost is in allowed domains` | `allowDomains: ['localhost']` triggers security warning, auto-enables host access, auto-allows common dev ports | +| 2 | `should map localhost to host.docker.internal` | `localhost` in allow-list becomes `host.docker.internal` in config | +| 3 | `should preserve http:// protocol prefix` | `http://localhost` becomes `http://host.docker.internal` | +| 4 | `should preserve https:// protocol prefix` | `https://localhost` becomes `https://host.docker.internal` | +| 5 | `should work with localhost combined with other domains` | `['localhost', 'github.com', 'example.com']` — all domains present, localhost mapped | +| 6 | `should allow custom port range to override default` | `allowHostPorts: '8080'` overrides default port list (no "allowing common development ports" message) | +| 7 | `should resolve host.docker.internal from inside container` | `getent hosts host.docker.internal` returns an IP | +| 8 | `should work for Playwright-style testing scenario` | Simulates Playwright test context (just echo commands, no actual server) | + +### Real-World Mapping + +| Test | Real-World Scenario | +|------|---------------------| +| #1-2 | **Playwright testing**: Agent runs Playwright tests against a local dev server (e.g., `localhost:3000`). AWF maps `localhost` to `host.docker.internal` so container can reach host services. | +| #3-4 | Protocol-specific access: agent needs HTTP-only or HTTPS-only to localhost | +| #5 | Agent needs both localhost (for dev server) and external domains (for API calls) | +| #6 | Custom dev server on non-standard port (e.g., `8080` for Spring Boot) | +| #7 | Container DNS resolution: `host.docker.internal` must resolve to the Docker host IP | +| #8 | End-to-end Playwright-style usage (though mocked — no actual HTTP server) | + +### Gaps and Missing Coverage + +1. **No actual HTTP request to localhost**: No test starts a server on the host and verifies the agent can reach it. Test #8 is just echo commands. +2. **No test for port filtering**: Do the auto-allowed ports (3000, 4200, 5173, 8080, etc.) actually work? Is port 22 (SSH) blocked? +3. **No test for dangerous port blocking**: `squid-config.ts` defines `DANGEROUS_PORTS` (22, 3306, 5432, 6379, etc.) that should be blocked even with `--allow-host-ports`. No integration test verifies this. +4. **No test for `--enable-host-access` without `localhost` keyword**: The flag can be set directly without using the `localhost` keyword. +5. **No test for multiple localhost entries**: e.g., `['localhost', 'http://localhost']` — how is this handled? +6. **No test for `localhost:PORT` format**: Users might try `localhost:3000` as a domain. + +### Edge Cases Missing + +- Agent trying to access host Docker socket via `host.docker.internal` +- Agent accessing host-bound services on non-standard interfaces (e.g., `0.0.0.0` vs `127.0.0.1`) +- Port range specification (e.g., `3000-4000`) +- Localhost with IPv6 (`::1`) + +--- + +## 7. Cross-Cutting Gaps + +### Infrastructure Issues + +1. **No `--block-domains` support in `AwfRunner`**: The test runner (`awf-runner.ts`) doesn't have a `blockDomains` option, making it impossible to test the blocklist feature in integration tests without modifying the runner. + +2. **No protocol-specific integration tests**: The `domain-patterns.ts` module supports `http://`, `https://` prefixes on domains, and `squid-config.ts` generates protocol-specific ACLs (`allowed_http_only`, `allowed_https_only`). None of this is integration-tested. + +3. **No SSL Bump integration tests**: The `--ssl-bump` feature generates entirely different Squid config (with CA certs, `ssl_bump peek/stare/bump`). No integration test covers this. + +4. **No `--allow-urls` integration tests**: URL-level filtering (requires `--ssl-bump`) is untested in integration. + +### Security Scenarios Not Covered + +| Scenario | Risk | Current Coverage | +|----------|------|-----------------| +| DNS rebinding attack | Agent resolves allowed domain -> attacker DNS returns internal IP | None | +| HTTP redirect to internal | Allowed domain returns 302 to `169.254.169.254` | None | +| Domain fronting | Agent uses TLS SNI of allowed domain but HTTP Host header of blocked domain | None | +| Slowloris / connection exhaustion | Agent opens many connections to exhaust Squid resources | None | +| Data exfiltration via DNS | Encode secrets in DNS query labels | None | +| Agent modifying proxy config | Agent writes to `/etc/environment` or `/etc/profile.d/` | None | +| Time-of-check/time-of-use | Domain resolves to allowed IP during check, then changes | None | + +### Recommended New Test Files + +1. **`block-domains.test.ts`** — Dedicated tests for `--block-domains` and `--block-domains-file` functionality +2. **`protocol-filtering.test.ts`** — Tests for `http://` and `https://` protocol-specific domain filtering +3. **`host-access.test.ts`** — Tests for `--enable-host-access` with an actual HTTP server on the host, port filtering, dangerous port blocking +4. **`ssl-bump.test.ts`** — Tests for SSL Bump mode with URL-level filtering +5. **`dns-restriction.test.ts`** — Tests for `--dns-servers` actually restricting which DNS servers the agent can query diff --git a/docs/test-analysis/protocol-security.md b/docs/test-analysis/protocol-security.md new file mode 100644 index 000000000..5a929b443 --- /dev/null +++ b/docs/test-analysis/protocol-security.md @@ -0,0 +1,486 @@ +# Protocol, Proxy, and Security Integration Tests Analysis + +This document provides a detailed analysis of the integration tests covering protocol support, IPv6, API proxy, credential hiding, token unsetting, one-shot tokens, exit code propagation, and error handling in the AWF (Agentic Workflow Firewall). + +--- + +## Table of Contents + +1. [Protocol Support Tests](#1-protocol-support-tests) +2. [IPv6 Integration Tests](#2-ipv6-integration-tests) +3. [API Proxy Sidecar Tests](#3-api-proxy-sidecar-tests) +4. [Credential Hiding Security Tests](#4-credential-hiding-security-tests) +5. [Token Unsetting Tests](#5-token-unsetting-tests) +6. [One-Shot Token Protection Tests](#6-one-shot-token-protection-tests) +7. [Exit Code Propagation Tests](#7-exit-code-propagation-tests) +8. [Error Handling Tests](#8-error-handling-tests) +9. [Cross-Cutting Observations](#9-cross-cutting-observations) + +--- + +## 1. Protocol Support Tests + +**File:** `tests/integration/protocol-support.test.ts` + +### What It Tests + +| Test Case | Description | +|-----------|-------------| +| **HTTPS to allowed domain** | Sends `curl -fsS https://api.github.com/zen` with `github.com` allowed. Verifies the request succeeds (exit code 0). | +| **HTTPS to non-allowed domain** | Sends `curl -f https://example.com` with only `github.com` allowed. Verifies the CONNECT is denied by Squid and curl fails. | +| **HTTPS verbose output** | Runs `curl -v` and greps for SSL/TLS connection info, confirming TLS handshakes happen through the proxy. | +| **HTTP/2 connections** | Uses `curl --http2` to verify HTTP/2 negotiation works through the Squid CONNECT tunnel. | +| **HTTP/1.1 fallback** | Uses `curl --http1.1` to verify HTTP/1.1 also works, confirming protocol negotiation flexibility. | +| **HTTP requests** | Attempts plain HTTP to `github.com`. Expects failure because HTTP requests hit Squid's intercept port where HTTP-to-HTTPS redirects fail. Documents a known limitation. | +| **Custom headers** | Passes `-H "Accept: application/json"` to verify headers traverse the proxy. | +| **User-Agent header** | Passes `-A "Test-Agent/1.0"` to verify custom User-Agent traverses the proxy. | +| **IPv4 connections** | Uses `curl -4` to force IPv4, verifying IPv4-only connectivity works. | +| **IPv6 (may not be available)** | Uses `curl -6` with `|| exit 0` fallback. Tests IPv6 connectivity but gracefully handles environments without IPv6. | +| **curl max-time** | Uses `--max-time 5` to verify the timeout option works through the proxy. | +| **curl connect-timeout** | Uses `--connect-timeout 10` to verify connection timeout works through the proxy. | + +### Real-World Mapping + +These tests map to the core firewall functionality: when an AI agent (Claude, Copilot, Codex) makes HTTP/HTTPS requests, the proxy must: +- Allow HTTPS to whitelisted domains (API calls to `api.github.com`, `api.anthropic.com`, etc.) +- Block HTTPS to non-whitelisted domains (preventing data exfiltration) +- Support various HTTP versions and headers that real tools use +- Handle connection timeouts gracefully when domains are blocked + +### Gaps and Missing Coverage + +| Gap | Impact | Recommendation | +|-----|--------|----------------| +| **No test for multiple allowed domains simultaneously** | Real workflows allow 5-10+ domains. No test verifies correct behavior with many domains. | Add test with 3-5 allowed domains, verify each works and a non-listed one is blocked. | +| **No test for wildcard subdomain matching** | Allowing `github.com` should also allow `api.github.com`, `raw.githubusercontent.com`, etc. | Add test verifying subdomain matching behavior. | +| **No test for large request/response bodies** | AI agents may download large files (repos, models). | Add test downloading a file >1MB through the proxy. | +| **No test for concurrent requests** | AI agents make many parallel requests. | Add test making 5-10 concurrent curl requests. | +| **No test for WebSocket upgrade** | Some MCP servers use WebSocket. | Add test attempting WebSocket connection through proxy (expected to fail or document limitation). | +| **No test for POST/PUT/PATCH/DELETE methods** | Only GET-like requests are tested. AI agents use POST for API calls. | Add test for POST with JSON body through the proxy. | +| **HTTP intercept mode behavior not thoroughly validated** | The HTTP test documents a "known limitation" but doesn't verify the exact failure mode. | Add test that checks the specific error (403 page vs connection refused). | +| **No test for redirect following** | `curl -L` following redirects across domains. | Test that redirects from allowed domain to non-allowed domain are blocked. | + +### Edge Cases + +- What happens when Squid is slow to start and the first request arrives before it's ready? +- What happens with very long domain names (>255 chars)? +- What about requests to IP addresses directly (bypassing DNS)? + +--- + +## 2. IPv6 Integration Tests + +**File:** `tests/integration/ipv6.test.ts` + +### What It Tests + +| Test Case | Description | +|-----------|-------------| +| **Accept IPv6 DNS servers** | Configures `dnsServers: ['2001:4860:4860::8888', '2001:4860:4860::8844']` and verifies they appear in debug logs. | +| **Accept mixed IPv4/IPv6 DNS** | Configures both `8.8.8.8` and `2001:4860:4860::8888`, verifies both appear in logs. | +| **DNS resolution with IPv4-only DNS** | Runs `nslookup github.com` with IPv4 DNS only, verifies resolution works. | +| **IPv6 traffic blocked for non-whitelisted** | When IPv6 is available, attempts `curl -6 https://example.com` (not in allowlist). Expects failure. | +| **IPv6 curl graceful failure** | Tests `curl -6` with fallback, ensuring graceful handling when IPv6 is unavailable. | +| **ip6tables status logging** | Verifies that starting with IPv6 DNS servers logs ip6tables availability status. | +| **IPv6 chain cleanup** | Verifies that ip6tables chains are cleaned up on exit. | +| **IPv4 curl with IPv4 DNS** | Baseline test confirming IPv4 curl works with IPv4 DNS servers. | +| **Dual-stack DNS configuration** | Configures 4 DNS servers (2 IPv4 + 2 IPv6), verifies all appear in logs. | +| **Valid IPv6 loopback** | Tests that `::1` (loopback) is accepted as a DNS server. | +| **Link-local addresses** | Tests graceful handling by using IPv4 DNS to avoid link-local issues. | +| **Empty IPv6 DNS server list** | Tests that IPv4-only DNS servers work and DNS resolution succeeds. | + +### Real-World Mapping + +GitHub Actions runners may have IPv6 enabled. The firewall must handle IPv6 DNS configuration correctly to prevent DNS-based data exfiltration while allowing legitimate IPv6 DNS resolution. The ip6tables rules must match the iptables rules for IPv4. + +### Gaps and Missing Coverage + +| Gap | Impact | Recommendation | +|-----|--------|----------------| +| **No test for actual IPv6 traffic through proxy** | Only tests DNS config and curl -6 blocking. Doesn't verify IPv6 traffic flows correctly through Squid when allowed. | Add conditional test (when IPv6 available) that makes successful IPv6 request to allowed domain. | +| **Invalid IPv6 address rejection not tested** | The test for "Invalid IPv6 address rejected at CLI level" actually uses a valid IPv6 address (`::1`). | Add test with genuinely invalid IPv6 like `not:an:ipv6` and verify CLI error. | +| **No test for IPv6-only environment** | All tests either have IPv4 or mixed. What if IPv4 is unavailable? | Add conditional test for IPv6-only DNS configuration. | +| **Link-local test doesn't actually test link-local** | Uses IPv4 DNS "to avoid link-local issues" - doesn't actually test `fe80::` handling. | Add test with `fe80::1%eth0` to verify proper rejection or handling. | +| **ip6tables cleanup not strongly verified** | Checks for presence of log keywords but doesn't verify rules are actually removed. | After cleanup, run `ip6tables -L` and verify no AWF rules remain. | +| **Many tests only check log output** | Several tests just verify strings appear in stderr rather than testing actual network behavior. | Where feasible, add actual connectivity tests. | + +### Edge Cases + +- What happens when IPv6 is available but ip6tables is not installed? +- Race condition: ip6tables rules being applied while IPv6 traffic is already in flight. +- DNS64/NAT64 environments where IPv6 traffic is translated to IPv4. + +--- + +## 3. API Proxy Sidecar Tests + +**File:** `tests/integration/api-proxy.test.ts` + +### What It Tests + +| Test Case | Description | +|-----------|-------------| +| **Anthropic healthcheck** | Starts API proxy with `ANTHROPIC_API_KEY`, checks `/health` on port 10001. Expects `"status":"healthy"` and `"anthropic-proxy"`. | +| **OpenAI healthcheck** | Same pattern with `OPENAI_API_KEY` on port 10000. | +| **ANTHROPIC_BASE_URL set** | Verifies the agent container has `ANTHROPIC_BASE_URL=http://172.30.0.30:10001`. | +| **ANTHROPIC_AUTH_TOKEN placeholder** | Verifies the agent gets `ANTHROPIC_AUTH_TOKEN=placeholder-token-for-credential-isolation` (real key held by sidecar). | +| **OPENAI_BASE_URL set** | Verifies `OPENAI_BASE_URL=http://172.30.0.30:10000`. | +| **Anthropic API routing through Squid** | Makes actual POST to `/v1/messages` via proxy. Expects authentication error (proving the request reached Anthropic's servers through Squid). | +| **Health endpoint with Anthropic-only** | Verifies port 10000 health shows `openai:false, anthropic:true` when only Anthropic key is provided. | +| **Copilot healthcheck** | Starts with `COPILOT_GITHUB_TOKEN` on port 10002. | +| **COPILOT_API_URL set** | Verifies `COPILOT_API_URL=http://172.30.0.30:10002`. | +| **COPILOT_TOKEN placeholder** | Verifies `COPILOT_TOKEN=placeholder-token-for-credential-isolation`. | +| **Copilot in health providers** | Verifies health endpoint reports `copilot:true`. | + +### Real-World Mapping + +The API proxy sidecar is the core credential isolation mechanism. In production agentic workflows: +- The AI agent never sees the real API key (only a placeholder) +- The sidecar injects the real key when proxying requests to the API +- This prevents prompt injection attacks from exfiltrating API keys +- All API traffic still routes through Squid for domain filtering + +### Gaps and Missing Coverage + +| Gap | Impact | Recommendation | +|-----|--------|----------------| +| **No test for OpenAI API routing** | Anthropic routing is tested end-to-end, but OpenAI routing is not. | Add test making POST to OpenAI endpoint and verifying auth error response. | +| **No test for Copilot API routing** | Only healthcheck and env vars tested for Copilot. | Add end-to-end routing test for Copilot. | +| **No test for all three keys simultaneously** | Real workflows may have multiple API keys. | Test with all three keys and verify all proxies are healthy. | +| **No test for API proxy with blocked domain** | What happens when Squid blocks the API domain? | Test with `enableApiProxy: true` but without the API domain in allowDomains. | +| **No test for API proxy failure/crash** | What if the sidecar crashes mid-request? | Test resilience to sidecar failures. | +| **No negative test for key leakage** | Should verify the real API key is NOT accessible in agent container. | Add test that greps for the fake API key in `/proc/*/environ` and other locations. | +| **No test for streaming responses** | Claude and OpenAI APIs use SSE streaming. | Test that streaming responses work through the proxy chain. | +| **No test for concurrent API requests** | AI agents make many parallel API calls. | Test multiple concurrent requests through the proxy. | + +### Edge Cases + +- What happens if `ANTHROPIC_API_KEY` is set but `api.anthropic.com` is not in allowed domains? +- What if the API proxy sidecar starts but Squid hasn't passed healthcheck yet? +- What about API key rotation during a long-running agent session? + +--- + +## 4. Credential Hiding Security Tests + +**File:** `tests/integration/credential-hiding.test.ts` + +### What It Tests + +| Test Case | Description | +|-----------|-------------| +| **Docker config.json hidden (normal mode)** | Reads `~/.docker/config.json` inside container. Expects empty output (file mounted from `/dev/null`). | +| **GitHub CLI hosts.yml hidden** | Reads `~/.config/gh/hosts.yml`. Expects no `oauth_token` or `gho_` strings. | +| **NPM .npmrc hidden** | Reads `~/.npmrc`. Expects no `_authToken` or `npm_` strings. | +| **Credential files are 0 bytes** | Uses `wc -c` on multiple credential files. All should be 0 bytes. | +| **Debug logs show credential hiding** | Verifies debug output contains "Using selective mounting" or "Hidden.*credential". | +| **Chroot mode hides at /host paths** | Reads `/host$HOME/.docker/config.json`. Expects empty or "No such file". | +| **Chroot mode debug logs** | Verifies chroot-specific credential hiding log messages. | +| **Chroot bypass prevention** | Critical security test: verifies credentials hidden at direct `$HOME` path (not just `/host` path). Previously a bypass vulnerability. | +| **Chroot GitHub CLI tokens hidden at direct path** | Same bypass prevention for `hosts.yml`. | +| **Full filesystem access security warnings** | With `--allow-full-filesystem-access`, verifies security warnings are displayed. | +| **Full access: Docker config NOT hidden** | With full access flag, verifies the real Docker config is accessible (not hidden). | +| **Simulated exfiltration: base64 encoding** | Runs `cat ... | base64` on hidden credential file. Gets empty output. | +| **Multiple encoding attempts** | Runs `cat ... | base64 | xxd -p` pipeline. Still gets empty output. | +| **Grep for tokens finds nothing** | Greps for `oauth_token`, `_authToken`, `auth:` patterns. Finds nothing. | +| **MCP logs /tmp/gh-aw/mcp-logs/ hidden (normal)** | Verifies the MCP logs directory is hidden (tmpfs overlay). | +| **MCP logs hidden (chroot)** | Same at `/host/tmp/gh-aw/mcp-logs/`. | +| **MCP log files unreadable** | Tries to read a specific log file path. Expects "No such file". | + +### Real-World Mapping + +This is a critical security layer. In real agentic workflows: +- The AI agent has filesystem access to write code, run tests, etc. +- A prompt injection attack could instruct the agent to read credential files +- Credential hiding prevents exfiltration of Docker Hub tokens, GitHub CLI tokens, NPM auth tokens +- The MCP logs hiding prevents reading log files that might contain sensitive operation details +- The `--allow-full-filesystem-access` flag is a conscious security trade-off for trusted workloads + +### Gaps and Missing Coverage + +| Gap | Impact | Recommendation | +|-----|--------|----------------| +| **No test for Cargo credentials** | `~/.cargo/credentials.toml` may contain crates.io tokens. | Add test verifying Cargo credentials are hidden. | +| **No test for Composer auth** | `~/.composer/auth.json` may contain Packagist tokens. | Add test for Composer credentials. | +| **No test for pip/PyPI credentials** | `~/.pip/pip.conf` or `~/.config/pip/pip.conf` may contain index tokens. | Add test for Python package manager credentials. | +| **No test for SSH keys** | `~/.ssh/id_rsa`, `~/.ssh/id_ed25519` could be exfiltrated. | Add test verifying SSH keys are not accessible (or document if they're needed). | +| **No test for `/proc/self/environ` credential reading** | Agent could read its own environment via `/proc/self/environ`. | Add test reading `/proc/self/environ` and verifying sensitive tokens are cleared. (Covered partially by token-unset tests.) | +| **No test for symlink bypass** | Agent could create symlinks from accessible paths to hidden paths. | Test that symlinks to credential files resolve to `/dev/null` content. | +| **No test for bind mount information leakage** | `mount` or `cat /proc/mounts` could reveal the `/dev/null` pattern. | Test that mount info doesn't reveal the hiding mechanism. | +| **No test for credential files that don't exist on host** | Tests assume credential files exist on the host machine. | Add conditional logic or mock files for test portability. | +| **No test for cloud provider credentials** | `~/.aws/credentials`, `~/.config/gcloud/`, `~/.azure/` could contain cloud tokens. | Add tests for cloud provider credential files. | + +### Edge Cases + +- What if a credential file is a symlink on the host? +- What if the agent creates a new credential file (e.g., `docker login` creates `~/.docker/config.json`)? +- What about credential discovery via `find / -name "*.json" -exec grep -l auth`? +- What if a tool writes credentials to a non-standard location? + +--- + +## 5. Token Unsetting Tests + +**File:** `tests/integration/token-unset.test.ts` + +### What It Tests + +| Test Case | Description | +|-----------|-------------| +| **GITHUB_TOKEN unset from /proc/1/environ** | After 7s delay, reads `/proc/1/environ` and verifies `GITHUB_TOKEN` is gone. Also verifies agent can still read via `getenv`. | +| **OPENAI_API_KEY unset** | Same pattern for OpenAI key. | +| **ANTHROPIC_API_KEY unset** | Same pattern for Anthropic key. | +| **Multiple tokens simultaneously** | Sets all three tokens, verifies all cleared from `/proc/1/environ` after 7s. | +| **Works in non-chroot mode** | Sets `AWF_CHROOT_ENABLED=false`, verifies token clearing works in container-only mode. | + +### Real-World Mapping + +The entrypoint.sh runs the agent in the background, waits 5 seconds for initialization, then unsets sensitive tokens from the parent shell's environment. This prevents a secondary attack vector: even if credential files are hidden, an attacker could read `/proc/1/environ` to find API keys passed as environment variables. The one-shot token library caches values so the agent process can still use them. + +### Gaps and Missing Coverage + +| Gap | Impact | Recommendation | +|-----|--------|----------------| +| **Fixed 7s sleep is fragile** | If agent startup takes longer (or shorter), tests may be flaky. | Use a more robust synchronization mechanism (e.g., signal file). | +| **No test for COPILOT_GITHUB_TOKEN** | Listed in `unset_sensitive_tokens()` but not tested here. | Add test for `COPILOT_GITHUB_TOKEN`. | +| **No test for all tokens in sensitive list** | `unset_sensitive_tokens()` lists 12+ tokens. Only 3 are tested. | Add tests for `GH_TOKEN`, `GITHUB_API_TOKEN`, `GITHUB_PAT`, `CLAUDE_API_KEY`, etc. | +| **No test for /proc/self/environ** | Tests check `/proc/1/environ` (PID 1) but not `/proc/self/environ` of child processes. | Add test reading `/proc/self/environ` from a child process. | +| **No test for timing race** | What if agent reads `/proc/1/environ` before the 5s unsetting delay? | Add test that reads immediately (before sleep) and after, showing the transition. | +| **No test for token values in /proc/*/cmdline** | Tokens passed via `--env` might appear in cmdline. | Verify token values don't appear in `/proc/*/cmdline`. | +| **No negative test: token not set** | Should verify that unsetting a non-existent token doesn't cause errors. | Add test with no tokens set. | + +### Edge Cases + +- What if the agent process forks and the child inherits the pre-unset environment? +- What if `/proc/1/environ` is not readable (restricted `/proc` mount)? +- What about tokens set via `--env` flag vs inherited from host environment? + +--- + +## 6. One-Shot Token Protection Tests + +**File:** `tests/integration/one-shot-tokens.test.ts` + +### What It Tests + +#### Container Mode (7 tests) + +| Test Case | Description | +|-----------|-------------| +| **Cache GITHUB_TOKEN** | Uses `printenv` (forks new process each time). Both reads succeed. Verifies debug log shows token access with value preview. | +| **Cache COPILOT_GITHUB_TOKEN** | Same pattern for Copilot token. | +| **Cache OPENAI_API_KEY** | Same pattern for OpenAI key. | +| **Multiple tokens independently** | Sets GITHUB_TOKEN and OPENAI_API_KEY. Verifies both cached independently. | +| **Non-sensitive vars unaffected** | Sets `NORMAL_VAR`. Verifies it's readable multiple times without one-shot-token log messages. | +| **Python same-process caching** | Uses Python `os.getenv()` to call getenv() directly (same process). Both reads succeed via cache. | +| **Clear from /proc/self/environ** | Python test: verifies first getenv() caches, checks `os.environ` dict, second getenv() returns cached value. | + +#### Chroot Mode (5 tests) + +| Test Case | Description | +|-----------|-------------| +| **Cache GITHUB_TOKEN in chroot** | Same as container mode but in chroot. Verifies library copied to chroot. | +| **Cache COPILOT_GITHUB_TOKEN in chroot** | Same for Copilot token in chroot. | +| **Python caching in chroot** | Python `os.getenv()` test in chroot mode. | +| **Non-sensitive vars in chroot** | Verifies non-sensitive vars unaffected in chroot. | +| **Multiple tokens in chroot** | Multiple independent tokens in chroot mode. | + +#### Edge Cases (3 tests) + +| Test Case | Description | +|-----------|-------------| +| **Empty token value** | Sets `GITHUB_TOKEN=''`. Both reads return empty. | +| **Token not set** | `NONEXISTENT_TOKEN` not in environment. Both reads return empty. | +| **Special characters** | Sets token with `@#$%` characters. Both reads preserve special chars. | + +### Real-World Mapping + +The one-shot token library (`LD_PRELOAD`) is the core mechanism preventing token theft within the agent process. When an AI agent like Claude Code calls `getenv("ANTHROPIC_API_KEY")`: +1. First call: library caches value in memory, unsets from environment +2. Subsequent calls: library returns cached value +3. `/proc/self/environ` no longer contains the token +4. A prompt injection attack that reads `/proc/self/environ` or spawns a subprocess to `printenv` gets nothing + +### Gaps and Missing Coverage + +| Gap | Impact | Recommendation | +|-----|--------|----------------| +| **No test for ANTHROPIC_API_KEY caching** | Tested in container mode but not explicitly with debug log verification like GITHUB_TOKEN. | Already covered (Test 3 in container mode). | +| **No test for Node.js `process.env` access** | Python is tested but Node.js (used by Claude Code, Copilot CLI) is not. | Add Node.js test with `process.env.GITHUB_TOKEN`. | +| **No test for Go `os.Getenv()` access** | Go is common in GitHub Actions. | Add Go test verifying LD_PRELOAD works with Go binaries. | +| **No test for compiled C program** | The library intercepts C's `getenv()`. Testing with a compiled C program would be most direct. | Add test with simple C program calling getenv(). | +| **No test for `environ` global variable** | Some programs access the `environ` array directly instead of getenv(). | Test that `extern char **environ` iteration doesn't show tokens. | +| **No test for token very long values** | What if a token is 10KB+? | Test with very long token value. | +| **No test for concurrent getenv() calls** | Multi-threaded programs calling getenv() simultaneously. | Test thread safety of the LD_PRELOAD library. | +| **No test for LD_PRELOAD being stripped** | If agent can modify LD_PRELOAD before forking. | Verify LD_PRELOAD is preserved in child processes. | +| **No test for `CLAUDE_API_KEY` or `CODEX_API_KEY`** | Listed in sensitive tokens but not tested. | Add tests for all listed sensitive tokens. | + +### Edge Cases + +- What if LD_PRELOAD is disabled by the runtime (e.g., setuid binaries)? +- What about statically linked binaries that don't use dynamic getenv()? +- What if a program calls `secure_getenv()` instead of `getenv()`? +- Race condition: what if two threads call getenv() for the same token simultaneously? + +--- + +## 7. Exit Code Propagation Tests + +**File:** `tests/integration/exit-code-propagation.test.ts` + +### What It Tests + +#### Basic Exit Codes (6 tests) + +| Test Case | Description | +|-----------|-------------| +| **Exit code 0** | `exit 0` propagates as 0. Verifies "Process exiting with code: 0" in stderr. | +| **Exit code 1** | `exit 1` propagates as 1. | +| **Exit code 2** | `exit 2` propagates as 2. | +| **Exit code 42** | Custom exit code propagates correctly. | +| **Exit code 127** | Running `nonexistent_command_xyz` produces 127 (command not found). | +| **Exit code 255** | Maximum standard exit code propagates correctly. | + +#### Command Exit Codes (6 tests) + +| Test Case | Description | +|-----------|-------------| +| **true** | `true` command returns 0. | +| **false** | `false` command returns 1. | +| **test success** | `test 1 -eq 1` returns 0. | +| **test failure** | `test 1 -eq 2` returns 1. | +| **grep found** | `echo "hello world" \| grep hello` returns 0. | +| **grep not found** | `echo "hello world" \| grep xyz` returns 1. | + +#### Pipeline Exit Codes (3 tests) + +| Test Case | Description | +|-----------|-------------| +| **Last command in pipeline** | `echo "test" \| cat \| exit 5` returns 5. | +| **Compound command success** | `echo "a" && echo "b" && exit 0` returns 0. | +| **Compound command failure** | `echo "a" && false && echo "c"` returns 1 (short-circuits at `false`). | + +### Real-World Mapping + +Exit code propagation is essential for CI/CD integration. When AWF runs in a GitHub Actions step: +- Exit code 0 means the agent succeeded → workflow continues +- Non-zero exit code means the agent failed → workflow marks step as failed +- The correct exit code helps users diagnose what went wrong + +### Gaps and Missing Coverage + +| Gap | Impact | Recommendation | +|-----|--------|----------------| +| **No test for signal-induced exit codes** | SIGTERM → 143, SIGKILL → 137. Not tested. | Add tests for signal exit codes. | +| **No test for exit codes > 255** | Shell wraps codes modulo 256. | Test `exit 256` returns 0, `exit 257` returns 1. | +| **No test for exit code from killed process** | What exit code does a `kill -9`'d process produce? | Test that AWF propagates signal-killed process codes. | +| **No test for exit code from timeout** | When `--max-time` causes curl to timeout. | Test exit code from timed-out commands. | +| **No test for exit code from OOM-killed process** | When a command is killed by OOM. | Document behavior (may not be testable in CI). | +| **No test for pipefail behavior** | `set -o pipefail` changes which exit code is returned from pipelines. | Test with `set -o pipefail` to verify behavior. | +| **No test for nested AWF invocations** | AWF running AWF (if that's ever a pattern). | May not be relevant but worth documenting. | + +### Edge Cases + +- What if the Docker container itself crashes (not the command)? +- What if `docker wait` returns a different code than the process's actual exit code? +- What about exit codes from `exec`'d processes within the command? + +--- + +## 8. Error Handling Tests + +**File:** `tests/integration/error-handling.test.ts` + +### What It Tests + +#### Network Errors (3 tests) + +| Test Case | Description | +|-----------|-------------| +| **Blocked domain** | `curl -f https://example.com` with only `github.com` allowed. Expects non-zero exit. | +| **Connection refused** | `curl http://localhost:12345` where no server listens. Expects "connection failed". | +| **DNS resolution failure** | curl to non-existent domain (in allowlist). Expects "dns failed". | + +#### Command Errors (3 tests) + +| Test Case | Description | +|-----------|-------------| +| **Command not found** | `nonexistent_command_xyz123` returns exit code 127. | +| **Permission denied** | `cat /etc/shadow` fails with permission denied. | +| **File not found** | `cat /nonexistent/file/path` fails with "No such file". | + +#### Script Errors (2 tests) + +| Test Case | Description | +|-----------|-------------| +| **Bash syntax error** | `bash -c "if then fi"` caught with `\|\| echo`. | +| **Division by zero** | `bash -c "echo $((1/0))"` caught with `\|\| echo`. | + +#### Process Signals (1 test) + +| Test Case | Description | +|-----------|-------------| +| **SIGTERM from command** | `kill -TERM $$` self-terminates. Verifies firewall handles it gracefully (result is defined). | + +#### Recovery After Errors (1 test) + +| Test Case | Description | +|-----------|-------------| +| **Continue after failure** | Runs `false` then `echo "recovery test"`. Verifies second command works (they're separate AWF invocations). | + +### Real-World Mapping + +Error handling is critical for production reliability: +- Network errors occur when agents try to access domains not in the allowlist +- Command errors occur when agent-generated commands have bugs +- Script errors happen when AI-generated code has syntax issues +- Signal handling ensures the firewall cleans up properly when interrupted +- Recovery after errors ensures the system doesn't enter a broken state + +### Gaps and Missing Coverage + +| Gap | Impact | Recommendation | +|-----|--------|----------------| +| **No test for Docker daemon unavailable** | What if Docker is not running when AWF starts? | Test CLI behavior when Docker socket is missing. | +| **No test for disk full** | What if `/tmp` is full and work directory can't be created? | Test behavior with limited disk space. | +| **No test for Squid proxy crash** | What if Squid crashes mid-session? | Test agent behavior when Squid becomes unreachable. | +| **No test for network partition** | What if the Docker network goes down? | Test behavior when `awf-net` becomes unavailable. | +| **No test for SIGKILL handling** | SIGKILL cannot be caught. What state is left behind? | Document and test cleanup behavior after SIGKILL. | +| **No test for SIGINT (Ctrl+C) during command** | The most common user interruption pattern. | Test that SIGINT properly stops containers and cleans up. | +| **No test for concurrent error + cleanup** | What if error occurs during cleanup? | Test nested error handling scenarios. | +| **No test for invalid CLI arguments** | What happens with `--allow-domains ""` or `--dns-servers invalid`? | Already tested at unit level but not integration. | +| **No test for container startup timeout** | What if container image pull takes too long? | Test behavior with unreachable registry. | +| **No test for resource cleanup verification** | After error, verify Docker networks/containers/volumes are cleaned up. | Add post-test assertions checking Docker state. | + +### Edge Cases + +- What if the error occurs in a `trap` handler itself? +- What if multiple signals arrive in rapid succession? +- What about errors in the iptables cleanup phase? +- What if the work directory is deleted by another process during execution? + +--- + +## 9. Cross-Cutting Observations + +### Strengths + +1. **Defense in depth**: The security tests cover multiple layers (credential hiding, token unsetting, one-shot tokens, API proxy isolation). +2. **Both modes tested**: Most security tests cover both normal and chroot mode. +3. **Real-world attack simulation**: The credential hiding tests simulate actual exfiltration attacks (base64, xxd, grep patterns). +4. **Custom matchers**: The `toSucceed()`, `toFail()`, `toExitWithCode()` matchers provide clear, readable assertions. +5. **Bypass prevention**: Tests specifically cover the chroot bypass vulnerability (Test 8) that was previously discovered and fixed. +6. **Comprehensive API proxy coverage**: Tests cover all three API providers (OpenAI, Anthropic, Copilot) and verify credential isolation. + +### Systemic Gaps + +1. **No stress testing**: No tests verify behavior under load, concurrent requests, or resource exhaustion. +2. **No timing/performance tests**: No tests verify that proxy overhead is within acceptable bounds. +3. **Limited negative testing for security**: While credential hiding is well-tested, other security bypasses (iptables manipulation, network namespace escape, container escape) are not tested. +4. **Fragile timing dependencies**: The token-unset tests rely on fixed `sleep 7` delays which could be flaky in slow CI environments. +5. **Test isolation concerns**: Tests share cleanup fixtures. A failed test could leave state that affects subsequent tests. +6. **No test for the full workflow**: No end-to-end test runs an actual AI agent (even a mock one) through the complete AWF pipeline. +7. **Missing SSL Bump tests**: The `--ssl-bump` feature is configured in the CLI but has no integration tests in this set. +8. **Missing blocked-domains tests**: The `--block-domains` feature (deny-list on top of allow-list) has no integration tests. +9. **No test for `--env-all` security implications**: Passing all host env vars could leak credentials. +10. **No test for log output sanitization**: Verifying that real API keys never appear in logs/stderr. diff --git a/docs/test-analysis/test-infra.md b/docs/test-analysis/test-infra.md new file mode 100644 index 000000000..e52411959 --- /dev/null +++ b/docs/test-analysis/test-infra.md @@ -0,0 +1,497 @@ +# Test Infrastructure Analysis + +This document provides a comprehensive analysis of the test fixtures, runners, matchers, and infrastructure used by the gh-aw-firewall integration test suite. + +--- + +## Table of Contents + +- [Overview](#overview) +- [Test Runner Architecture](#test-runner-architecture) +- [Abstractions Provided](#abstractions-provided) +- [Batch Runner Pattern](#batch-runner-pattern) +- [Cleanup Strategy](#cleanup-strategy) +- [CI Workflow Post-Processing](#ci-workflow-post-processing) +- [Limitations](#limitations) +- [Improvement Opportunities](#improvement-opportunities) + +--- + +## Overview + +The test infrastructure lives in two primary locations: + +| Path | Purpose | +|------|---------| +| `tests/fixtures/` | Reusable test helpers: runners, matchers, cleanup, log parsing | +| `tests/setup/` | Jest configuration and setup files | +| `scripts/ci/` | CI-specific cleanup and workflow post-processing scripts | + +The suite contains **26 integration test files** across `tests/integration/`, all executed serially via Jest with a 120-second per-test timeout. + +### Key Files + +| File | Lines | Role | +|------|-------|------| +| `tests/fixtures/awf-runner.ts` | 331 | Core test runner — wraps AWF CLI invocations | +| `tests/fixtures/batch-runner.ts` | 118 | Batches multiple commands into a single container | +| `tests/fixtures/assertions.ts` | 179 | Custom Jest matchers (`toSucceed`, `toFail`, etc.) | +| `tests/fixtures/docker-helper.ts` | 297 | Low-level Docker operations helper | +| `tests/fixtures/cleanup.ts` | 209 | TypeScript port of `cleanup.sh` | +| `tests/fixtures/log-parser.ts` | 224 | Squid and iptables log parsing | +| `tests/setup/jest.integration.config.js` | 24 | Jest config for integration tests | +| `tests/setup/jest.setup.ts` | 9 | Registers custom matchers globally | +| `scripts/ci/cleanup.sh` | 56 | Bash cleanup script for CI | +| `scripts/ci/postprocess-smoke-workflows.ts` | 150 | Post-processes compiled workflow YAML for CI | + +--- + +## Test Runner Architecture + +### AwfRunner (`tests/fixtures/awf-runner.ts`) + +The `AwfRunner` class is the central abstraction for integration tests. It wraps the AWF CLI binary and provides two execution modes: + +#### `run(command, options)` — Direct execution +- Runs `node dist/cli.js -- ` directly +- Suitable for tests that don't require iptables (no sudo) +- Rarely used in practice (most tests need sudo for iptables) + +#### `runWithSudo(command, options)` — Privileged execution +- Runs `sudo -E --preserve-env=PATH,HOME,... node dist/cli.js -- ` +- Preserves critical environment variables (`PATH`, `HOME`, `GOROOT`, `CARGO_HOME`, `JAVA_HOME`, `DOTNET_ROOT`) +- Required for real firewall operation (iptables NAT rules) +- Used by ~95% of integration tests + +#### AwfOptions Interface + +All CLI flags are exposed as typed options: + +```typescript +interface AwfOptions { + allowDomains?: string[]; + keepContainers?: boolean; + logLevel?: 'debug' | 'info' | 'warn' | 'error'; + buildLocal?: boolean; + imageRegistry?: string; + imageTag?: string; + timeout?: number; // Default: 120000ms + env?: Record; + volumeMounts?: string[]; + containerWorkDir?: string; + tty?: boolean; + dnsServers?: string[]; + allowHostPorts?: string; + allowFullFilesystemAccess?: boolean; + enableApiProxy?: boolean; +} +``` + +#### AwfResult Interface + +Every invocation returns a structured result: + +```typescript +interface AwfResult { + exitCode: number; + stdout: string; + stderr: string; + success: boolean; // exitCode === 0 + timedOut: boolean; + workDir?: string; // Extracted from stderr: /tmp/awf- +} +``` + +The `workDir` is particularly important — it's extracted from AWF's stderr logs and used by log-based assertions (`toAllowDomain`, `toBlockDomain`) to locate Squid access logs. + +#### Typical Test Pattern + +```typescript +describe('Feature X', () => { + let runner: AwfRunner; + + beforeAll(async () => { + await cleanup(false); // Pre-test cleanup + runner = createRunner(); + }); + + afterAll(async () => { + await cleanup(false); // Post-test cleanup + }); + + test('should do something', async () => { + const result = await runner.runWithSudo( + 'curl -f https://api.github.com/zen', + { + allowDomains: ['github.com'], + logLevel: 'debug', + timeout: 60000, + } + ); + expect(result).toSucceed(); + }, 120000); // Jest timeout (must be >= AWF timeout) +}); +``` + +### Jest Configuration (`tests/setup/jest.integration.config.js`) + +Key settings: + +| Setting | Value | Rationale | +|---------|-------|-----------| +| `testTimeout` | 120000 (2 min) | Firewall tests involve Docker container lifecycle | +| `maxWorkers` | 1 | Serial execution — avoids Docker network/container conflicts | +| `verbose` | true | Full test output for CI debugging | +| `preset` | `ts-jest` | TypeScript compilation | +| `setupFilesAfterEnv` | `jest.setup.ts` | Registers custom matchers before tests run | + +Tests are discovered from `tests/integration/**/*.test.ts`. + +--- + +## Abstractions Provided + +### Custom Jest Matchers (`tests/fixtures/assertions.ts`) + +Six custom matchers extend Jest's `expect()`, registered globally via `jest.setup.ts`: + +| Matcher | Asserts | Input | +|---------|---------|-------| +| `toSucceed()` | `result.success === true` (exit code 0) | `AwfResult` | +| `toFail()` | `result.success === false` (non-zero exit) | `AwfResult` | +| `toExitWithCode(code)` | `result.exitCode === code` | `AwfResult` | +| `toTimeout()` | `result.timedOut === true` | `AwfResult` | +| `toAllowDomain(domain)` | Domain appears as `TCP_TUNNEL` in Squid logs | `AwfResult` | +| `toBlockDomain(domain)` | Domain appears as `TCP_DENIED` in Squid logs | `AwfResult` | + +**Type declarations** are in `tests/jest-custom-matchers.d.ts`, included in test files via: +```typescript +/// +``` + +#### Log-Based Matchers + +`toAllowDomain` and `toBlockDomain` are more sophisticated — they: +1. Extract `workDir` from the `AwfResult` +2. Read `${workDir}/squid-logs/access.log` synchronously (Jest matchers must be sync) +3. Parse the Squid log using `LogParser` +4. Check for `TCP_TUNNEL` (allowed) or `TCP_DENIED` (blocked) entries for the domain + +These require `keepContainers: true` to preserve the work directory. + +### Docker Helper (`tests/fixtures/docker-helper.ts`) + +A general-purpose Docker operations wrapper: + +| Method | Purpose | +|--------|---------| +| `pullImage(image)` | Pull a Docker image | +| `run(options)` | Run a container with full option support | +| `stop(name)` | Stop a container | +| `rm(name, force?)` | Remove a container | +| `inspect(name)` | Get container state and network info | +| `logs(name, options?)` | Retrieve container logs | +| `exec(name, command)` | Execute command in running container | +| `networkExists(name)` | Check if a Docker network exists | +| `createNetwork(name, subnet?)` | Create a Docker network | +| `removeNetwork(name)` | Remove a Docker network | +| `listContainers(options?)` | List containers by filter | +| `wait(name)` | Wait for container exit and return exit code | +| `isRunning(name)` | Check if a container is currently running | + +All methods use `execa` with `reject: false` to handle errors gracefully. + +**Note:** This helper is available but less commonly used in practice — most tests go through `AwfRunner.runWithSudo()` which handles the full container lifecycle automatically. + +### Log Parser (`tests/fixtures/log-parser.ts`) + +Parses two log formats: + +#### Squid Log Parser + +Parses the `firewall_detailed` log format: +``` +%ts.%03tu %>a:%>p %{Host}>h %Hs %Ss:%Sh %ru "%{User-Agent}>h" +``` + +Into typed `SquidLogEntry` objects with fields: `timestamp`, `clientIp`, `clientPort`, `host`, `destIp`, `destPort`, `protocol`, `method`, `statusCode`, `decision`, `hierarchy`, `url`, `userAgent`. + +Filtering methods: +- `filterByDecision(entries, 'allowed' | 'blocked')` — Filter by `TCP_TUNNEL`/`TCP_DENIED` +- `filterByDomain(entries, domain)` — Filter by exact or subdomain match +- `getUniqueDomains(entries)` — Deduplicated domain list +- `wasAllowed(entries, domain)` / `wasBlocked(entries, domain)` — Boolean checks + +#### iptables Log Parser + +Parses kernel log entries prefixed with `[FW_BLOCKED_UDP]` or `[FW_BLOCKED_OTHER]` from `dmesg` output. + +--- + +## Batch Runner Pattern + +### Problem + +Each `runner.runWithSudo()` call spawns a full Docker container lifecycle: config generation, Docker Compose up (Squid + Agent), iptables setup, command execution, teardown. This takes **15-25 seconds per invocation**. + +The chroot language test suite originally had ~73 individual test invocations, each with this overhead. + +### Solution: `batch-runner.ts` + +The batch runner groups commands that share the same `AwfOptions` (particularly `allowDomains`) into a **single AWF container invocation**. This reduced the chroot suite from ~73 to ~27 container startups. + +#### How It Works + +1. **Script Generation** — Each command is wrapped in delimiters: + ```bash + echo "===BATCH_START:python_version===" + (python3 --version) 2>&1 + _EC=$? + echo "" + echo "===BATCH_EXIT:python_version:$_EC===" + ``` + +2. **Single Invocation** — The concatenated script runs as one `runWithSudo()` call. + +3. **Result Parsing** — The combined stdout is parsed back into per-command results: + ```typescript + const batch = await runBatch(runner, [ + { name: 'python_version', command: 'python3 --version' }, + { name: 'node_version', command: 'node --version' }, + ], { allowDomains: ['github.com'] }); + + expect(batch.get('python_version').exitCode).toBe(0); + ``` + +#### Interface + +```typescript +interface BatchCommand { + name: string; // Unique identifier for this command + command: string; // Shell command to execute +} + +interface BatchCommandResult { + stdout: string; // Captured output (stdout + stderr merged) + exitCode: number; // Per-command exit code +} + +interface BatchResults { + get(name: string): BatchCommandResult; // Throws if name not found + overall: AwfResult; // Raw AWF result for the whole batch +} +``` + +#### Design Decisions + +- Each command runs in a **subshell** `(cmd) 2>&1` so failures don't abort the batch +- **stdout and stderr are merged** via `2>&1` — individual stderr is not preserved +- Exit code is captured immediately into `_EC=$?` before `echo` resets `$?` +- Delimiter tokens (`===BATCH_START:`, `===BATCH_EXIT:`) chosen to be unlikely in real output +- If the batch is killed early, missing commands get `exitCode: -1` + +#### Usage Pattern in Tests + +The batch runner is used in `beforeAll` to run all commands once, then individual `test()` blocks assert against named results: + +```typescript +describe('Quick checks (batched)', () => { + let batch: BatchResults; + + beforeAll(async () => { + batch = await runBatch(runner, [ + { name: 'python_version', command: 'python3 --version' }, + { name: 'go_version', command: 'go version' }, + ], { allowDomains: ['github.com'], timeout: 120000 }); + }, 180000); + + test('Python available', () => { + expect(batch.get('python_version').exitCode).toBe(0); + }); + + test('Go available', () => { + expect(batch.get('go_version').exitCode).toBe(0); + }); +}); +``` + +This pattern is used extensively in `chroot-languages.test.ts` (17 batched commands) and `chroot-package-managers.test.ts`. + +--- + +## Cleanup Strategy + +The cleanup system uses a **defense-in-depth** approach across four stages, accounting for the fact that Docker container and network resources can leak when processes are killed mid-lifecycle. + +### Stage 1: Pre-Test Cleanup (TypeScript) + +**File:** `tests/fixtures/cleanup.ts` + +The `Cleanup` class is a TypeScript port of the shell script, providing the same operations as programmable methods: + +```typescript +class Cleanup { + removeContainers() // docker rm -f awf-squid awf-agent + stopDockerComposeServices() // docker compose down -v for all /tmp/awf-*/ + cleanupIptables() // Remove FW_WRAPPER chain from DOCKER-USER + removeNetwork() // docker network rm awf-net + pruneContainers() // docker container prune -f + pruneNetworks() // docker network prune -f (fixes "Pool overlaps") + removeWorkDirectories() // rm -rf /tmp/awf-* + cleanAll() // All of the above in sequence +} +``` + +Called in `beforeAll` and `afterAll` of every test `describe` block: +```typescript +beforeAll(async () => { await cleanup(false); }); +afterAll(async () => { await cleanup(false); }); +``` + +### Stage 2: Normal Exit (AWF Built-in) + +AWF's own cleanup in `src/cli.ts`: +- `docker compose down -v` stops containers +- Deletes the work directory `/tmp/awf-` + +### Stage 3: Signal/Error (AWF Built-in) + +SIGINT/SIGTERM handlers in `src/cli.ts` trigger the same cleanup as normal exit. Cannot catch SIGKILL. + +### Stage 4: CI Always Cleanup (Shell Script) + +**File:** `scripts/ci/cleanup.sh` + +A bash script that performs the same operations as the TypeScript `Cleanup` class. Run as a safety net in CI workflows with `if: always()`. + +Operations: +1. `docker rm -f awf-squid awf-agent` +2. `docker compose -f /tmp/awf-*/docker-compose.yml down -v` +3. Remove `FW_WRAPPER` iptables chain from `DOCKER-USER` +4. `docker network rm awf-net` +5. `docker container prune -f` +6. `docker network prune -f` (critical for subnet pool management) +7. `rm -rf /tmp/awf-*` + +### Why Multi-Stage? + +The `timeout` command used in CI can SIGKILL the AWF process after a grace period, bypassing stages 2-3. Without stages 1 and 4, orphaned Docker networks accumulate and eventually exhaust the subnet pool ("Pool overlaps" errors). + +--- + +## CI Workflow Post-Processing + +### `scripts/ci/postprocess-smoke-workflows.ts` + +After `gh-aw compile` generates `.lock.yml` workflow files, this script transforms them for CI use: + +| Transformation | Why | +|----------------|-----| +| Replace "Install awf binary" step with `npm ci && npm run build` | Use locally-built code instead of pre-built GHCR binary | +| Remove `sparse-checkout` blocks | Full repo checkout needed for npm build | +| Remove `depth: 1` shallow clone | Full checkout needed | +| Replace `--image-tag X --skip-pull` with `--build-local` | Use locally-built container images | + +Processes 30+ workflow files across smoke tests, build tests, and agentic workflows. Ensures CI tests use the current source code rather than stale published images. + +--- + +## Limitations + +### 1. Serial Execution Only + +`maxWorkers: 1` means all 26 test files run sequentially. A full integration suite run takes **30-60+ minutes** depending on the number of container startups. + +**Root cause:** All tests share the same Docker network (`172.30.0.0/24`), container names (`awf-squid`, `awf-agent`), and iptables chains. Parallel execution would cause conflicts. + +### 2. No Per-Test Isolation + +Tests within a `describe` block share the same `AwfRunner` instance. While each `runWithSudo()` call creates a fresh container, there's no mechanism to isolate host-level side effects (iptables rules, Docker networks) between individual tests. + +### 3. Batch Runner Loses Individual stderr + +The batch runner merges stdout and stderr (`2>&1`), so per-command stderr is mixed into stdout. Tests can't distinguish between a command's stdout and stderr output. + +### 4. Log-Based Matchers Require `keepContainers` + +`toAllowDomain` and `toBlockDomain` read Squid logs from the work directory, which is deleted during normal cleanup. Tests using these matchers must pass `keepContainers: true` and manually call `cleanup()` after assertions. + +### 5. Timeout Duplication + +Every test has **two timeout values**: the AWF timeout in `AwfOptions` (default 120s) and the Jest test timeout (the second argument to `test()`). These must be kept in sync manually, with the Jest timeout always exceeding the AWF timeout. + +### 6. No Retry Logic + +Flaky tests (network issues, Docker daemon slowness) have no built-in retry mechanism. The test infrastructure treats every failure as final. + +### 7. Docker Dependency + +All integration tests require: +- Docker daemon running +- sudo access for iptables +- Port 3128 available for Squid +- The `172.30.0.0/24` subnet unoccupied + +This makes local development testing impossible without a Linux environment with Docker. + +### 8. Cleanup Is Aggressive + +`docker container prune -f` and `docker network prune -f` in the cleanup routine can affect non-AWF containers and networks on the same host. This is safe in CI but could be problematic in shared development environments. + +--- + +## Improvement Opportunities + +### 1. Dynamic Network Allocation + +Instead of hardcoding `172.30.0.0/24`, assign a unique subnet per test run. This would enable limited parallelism (2-3 workers) and eliminate "Pool overlaps" errors at the source. + +### 2. Test Grouping by Domain Config + +The batch runner already groups commands by `AwfOptions`. This pattern could be extended: tests that share `allowDomains` and other config could be automatically grouped into fewer container invocations, even across test files. + +### 3. Container Caching / Reuse + +For tests that share the same `allowDomains` config, the Squid proxy container could be kept running between tests, avoiding the ~10s container startup per invocation. Only the agent container would need to restart. + +### 4. Parallel-Safe Container Names + +Using unique prefixes (e.g., `awf--squid` instead of `awf-squid`) would allow multiple test runs or workers simultaneously. + +### 5. Flaky Test Retry + +A Jest `retryTimes` configuration or custom retry wrapper for network-dependent tests would improve CI reliability without masking real failures. + +### 6. Separate stderr in Batch Runner + +The batch runner could be enhanced to capture stderr separately by writing it to a temp file: +```bash +(cmd) 2>/tmp/batch_stderr_name +``` +This would preserve per-command stderr for better failure diagnostics. + +### 7. Automatic Timeout Synchronization + +A helper that sets both the AWF timeout and the Jest timeout from a single value would eliminate the timeout duplication issue: +```typescript +function testWithTimeout(name, fn, timeoutMs) { + const awfTimeout = timeoutMs - 30000; // Buffer for container lifecycle + test(name, () => fn(awfTimeout), timeoutMs); +} +``` + +### 8. Targeted Cleanup + +Replace `docker container/network prune -f` with targeted removal of AWF-specific resources only (e.g., by label or name pattern). This would make the cleanup safe for shared environments. + +### 9. Test Fixture for Common Assertions + +Many tests repeat the same pattern: run a curl command, check `toSucceed()` or `toFail()`. A higher-level fixture could encapsulate this: +```typescript +await expectDomainAllowed(runner, 'github.com'); +await expectDomainBlocked(runner, 'example.com', { allowDomains: ['github.com'] }); +``` + +### 10. Pre-built Docker Image Caching + +Integration tests that use `--build-local` rebuild containers from scratch every time. A CI-level image cache (built once, reused across tests) would save significant time. From a447275891099e690bfd89df6bb521c10100069a Mon Sep 17 00:00:00 2001 From: "Jiaxiao (mossaka) Zhou" Date: Wed, 25 Feb 2026 19:50:21 +0000 Subject: [PATCH 2/2] docs: address review comments and move gaps to issue #1039 - Fix workflow counts (28 lock.yml, 15 hand-written) in ci-smoke.md - Fix postprocess script count (29 files) and note missing smoke-gemini - Clarify API proxy coverage (routing tested only for Anthropic) - Fix smoke test count from 5 to 4 (Gemini workflow was removed) - Fix "skip'd" to "skipped" - Remove gaps/action items from INTEGRATION-TESTS.md (moved to #1039) - Keep INTEGRATION-TESTS.md as pure coverage reference Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/INTEGRATION-TESTS.md | 144 ++---------------------- docs/test-analysis/ci-smoke.md | 4 +- docs/test-analysis/protocol-security.md | 2 +- docs/test-analysis/test-infra.md | 2 +- 4 files changed, 13 insertions(+), 139 deletions(-) diff --git a/docs/INTEGRATION-TESTS.md b/docs/INTEGRATION-TESTS.md index 90a26525c..a3e5f6e86 100644 --- a/docs/INTEGRATION-TESTS.md +++ b/docs/INTEGRATION-TESTS.md @@ -1,6 +1,6 @@ # Integration Tests Coverage Guide -A comprehensive guide to what the gh-aw-firewall integration tests cover, what they don't cover, and how they relate to real-world usage in GitHub Agentic Workflows. +A reference guide to what the gh-aw-firewall integration tests cover and how they relate to real-world usage in GitHub Agentic Workflows. **Last updated:** February 2026 @@ -25,8 +25,8 @@ The test suite is organized in three tiers: ``` ┌─────────────────────────────────────────────────────┐ -│ Smoke Tests (5 workflows) │ -│ Real AI agents (Claude, Copilot, Codex, Gemini) │ +│ Smoke Tests (4 workflows) │ +│ Smoke workflows (Claude, Copilot, Codex, Chroot) │ │ running inside AWF sandbox │ ├─────────────────────────────────────────────────────┤ │ Build-Test Workflows (8 workflows) │ @@ -46,17 +46,17 @@ The test suite is organized in three tiers: | Category | Files | Approx Tests | CI Workflow | |----------|-------|-------------|-------------| -| Domain/Network | 6 | 50 | **None** (not run in CI!) | +| Domain/Network | 6 | 50 | None | | Chroot | 5 | 70 | `test-chroot.yml` (4 jobs) | -| Protocol/Security | 8 | 100 | **None** (not run in CI!) | -| Container/Ops | 7 | 45 | **None** (not run in CI!) | +| Protocol/Security | 8 | 100 | None | +| Container/Ops | 7 | 45 | None | | Unit Tests | 19 | ~200 | `test-coverage.yml` | -| Smoke Tests | 5 | N/A | Per-workflow (scheduled + PR) | +| Smoke Tests | 4 | N/A | Per-workflow (scheduled + PR) | | Build-Test | 8 | N/A | Per-workflow (PR + dispatch) | --- -## What's Well Covered +## What's Covered ### 1. Chroot Filesystem Isolation (Strong) @@ -88,7 +88,6 @@ Real AI agents running through the full AWF pipeline: - **Claude**: GitHub MCP, Playwright browser automation, file I/O, bash tools - **Copilot**: Same + web-fetch, agentic-workflows tools - **Codex**: GH CLI safe inputs, Tavily web search, discussion interactions -- **Gemini**: Same feature set as Codex, different engine path ### 4. Multi-Language Build-Test (Strong) @@ -103,86 +102,6 @@ Real AI agents running through the full AWF pipeline: --- -## Critical Gaps - -### Gap 1: Most Integration Tests Don't Run in CI - -**Severity: Critical** - -The `test-integration.yml` workflow is actually just a TypeScript type-check. It does **not** run the integration test suite. This means 20+ integration test files covering domains, DNS, environment variables, exit codes, error handling, network security, credentials, and protocols have **no CI pipeline**. - -Only the 5 chroot test files run in CI (via `test-chroot.yml`). - -**Impact**: Regressions in domain filtering, credential hiding, exit code propagation, and network security would go undetected until someone runs tests locally. - -### Gap 2: `--block-domains` Completely Untested - -**Severity: Critical** - -The `--block-domains` CLI flag (deny-list on top of allow-list) has zero integration tests. The `blocked-domains.test.ts` file is a misnomer — it only tests allow-list behavior. The `AwfRunner` test fixture doesn't even expose a `blockDomains` option. - -**Impact**: If `--block-domains` breaks, no test catches it. - -### Gap 3: `--env-all` Never Tested - -**Severity: Critical** - -The `--env-all` flag is the **primary production mode** — used by `gh-aw`'s `BuildAWFArgs()` to pass all GitHub Actions environment variables into the container. The `environment-variables.test.ts` file header mentions it, but no test uses `envAll: true`. - -**Impact**: The most common production invocation pattern has zero test coverage. - -### Gap 4: DNS Restriction Enforcement Untested - -**Severity: High** - -The `--dns-servers` flag restricts DNS traffic to whitelisted resolvers (preventing DNS-based exfiltration). While `dns-servers.test.ts` exists, no test verifies that queries to **non-whitelisted** DNS servers are actually blocked. All tests use the default or explicitly whitelisted servers. - -**Impact**: DNS exfiltration prevention — a key security feature — is unverified. - -### Gap 5: No Package Installation Tests - -**Severity: High** - -Package manager tests (in `chroot-package-managers.test.ts`) only **query** registries (`pip index versions`, `npm view`, `cargo search`, `gem search`) but never **install** packages (`pip install`, `npm install`, `cargo add`). This is the most common real-world operation. - -The build-test workflows partially compensate (they run `npm install`, `cargo build`, etc.) but these are AI-agent-driven and non-deterministic. - -**Impact**: A regression in package installation through the proxy would not be caught by deterministic tests. - -### Gap 6: `git push` and Authenticated Git Untested - -**Severity: High** - -`git-operations.test.ts` tests clone, ls-remote, and config but **not** `git push` (the most important write operation in agentic workflows) or any authenticated git operation. Every production workflow uses authenticated git to push branches and create PRs. - -**Impact**: If authenticated git push breaks through the proxy, no test catches it. - -### Gap 7: No Localhost HTTP Request Tests - -**Severity: High** - -`localhost-access.test.ts` verifies configuration (iptables rules, Squid ACLs) but never makes an actual HTTP request to localhost. MCP servers run on localhost inside the container — a real request test is needed. - -### Gap 8: Docker Warning Tests Entirely Skipped - -**Severity: Medium** - -`docker-warning.test.ts` is wrapped in `describe.skip` with a stale TODO about a build issue. These 5 tests provide zero coverage. - -### Gap 9: Fragile Timing Dependencies - -**Severity: Medium** - -Token-unset tests rely on `sleep 7` to wait for the 5-second unsetting delay. Log-command tests use `if (existsSync())` guards that silently pass when logs aren't created. These patterns mask real failures and cause flakiness. - -### Gap 10: No SSL Bump Integration Tests - -**Severity: Medium** - -The `--ssl-bump` feature is configured in the CLI but has zero integration tests. This feature enables HTTPS content inspection and is important for some security-sensitive deployments. - ---- - ## Coverage Heat Map A visual overview of what's tested vs. not: @@ -222,7 +141,7 @@ Container security scan ❌ ❌ ✅ ❌ ❌ Dependency audit ❌ ❌ ✅ ❌ ❌ * ⚠️ = Tests exist but have significant gaps (see detailed docs) -** = Tests exist but are skip'd +** = Tests exist but are skipped ``` --- @@ -249,51 +168,6 @@ See [test-infra.md](test-analysis/test-infra.md) for full infrastructure analysi --- -## Recommended Priority Actions - -### P0: Run Integration Tests in CI - -Create a CI workflow that actually runs the non-chroot integration tests. Currently, ~150 tests across 20+ files have no CI pipeline. - -### P1: Add `--block-domains` Tests - -Add `blockDomains` support to `AwfRunner` and write tests for the deny-list feature: -- Block specific subdomain while allowing parent -- Block takes precedence over allow -- Wildcard blocking patterns - -### P1: Add `--env-all` Tests - -Add tests using `envAll: true` to verify: -- All host env vars pass through -- Sensitive tokens are properly filtered -- Proxy env vars (HTTP_PROXY, HTTPS_PROXY) are set correctly - -### P1: Add DNS Restriction Enforcement Test - -Test that DNS queries to non-whitelisted servers are actually blocked: -``` -dig @1.2.3.4 example.com # Should fail with non-whitelisted DNS -``` - -### P2: Add Package Installation Tests - -Test actual `pip install`, `npm install`, `cargo add` through the proxy instead of just registry queries. - -### P2: Add `git push` with Authentication Test - -Test authenticated git push through the proxy — the most critical write operation in agentic workflows. - -### P2: Enable or Remove Skipped Docker Warning Tests - -Either fix the build issue or remove the dead `describe.skip` block. - -### P3: Fix Fragile Timing Dependencies - -Replace `sleep 7` in token-unset tests with signal-file synchronization. Replace `if (existsSync())` guards with hard assertions. - ---- - ## Detailed Analysis Documents Each document provides per-test-case analysis with plain-language descriptions, real-world mappings, and gap identification: diff --git a/docs/test-analysis/ci-smoke.md b/docs/test-analysis/ci-smoke.md index c1a230f8c..42189b680 100644 --- a/docs/test-analysis/ci-smoke.md +++ b/docs/test-analysis/ci-smoke.md @@ -25,8 +25,8 @@ The repo has three tiers of testing: |------|------|-------|---------| | **Unit** | Jest (src/*.test.ts) | 19 files | Fast feedback on individual modules | | **Integration** | Jest (tests/integration/*.test.ts) | 26 files | End-to-end AWF container execution | -| **Smoke/Build-Test** | gh-aw compiled workflows (.lock.yml) | 13 workflows | Real AI agent execution inside AWF sandbox | -| **CI** | Hand-written GitHub Actions (.yml) | 12 workflows | Build, lint, type-check, security, coverage | +| **Smoke/Build-Test** | gh-aw compiled workflows (.lock.yml) | 28 workflows | Real AI agent execution inside AWF sandbox | +| **CI** | Hand-written GitHub Actions (.yml) | 15 workflows | Build, lint, type-check, security, coverage | --- diff --git a/docs/test-analysis/protocol-security.md b/docs/test-analysis/protocol-security.md index 5a929b443..e8ab84078 100644 --- a/docs/test-analysis/protocol-security.md +++ b/docs/test-analysis/protocol-security.md @@ -470,7 +470,7 @@ Error handling is critical for production reliability: 3. **Real-world attack simulation**: The credential hiding tests simulate actual exfiltration attacks (base64, xxd, grep patterns). 4. **Custom matchers**: The `toSucceed()`, `toFail()`, `toExitWithCode()` matchers provide clear, readable assertions. 5. **Bypass prevention**: Tests specifically cover the chroot bypass vulnerability (Test 8) that was previously discovered and fixed. -6. **Comprehensive API proxy coverage**: Tests cover all three API providers (OpenAI, Anthropic, Copilot) and verify credential isolation. +6. **Broad API proxy coverage**: Tests cover all three API providers (OpenAI, Anthropic, Copilot) for healthchecks and env wiring; end-to-end request routing and credential isolation are currently verified in depth only for Anthropic. ### Systemic Gaps diff --git a/docs/test-analysis/test-infra.md b/docs/test-analysis/test-infra.md index e52411959..6631e774c 100644 --- a/docs/test-analysis/test-infra.md +++ b/docs/test-analysis/test-infra.md @@ -392,7 +392,7 @@ After `gh-aw compile` generates `.lock.yml` workflow files, this script transfor | Remove `depth: 1` shallow clone | Full checkout needed | | Replace `--image-tag X --skip-pull` with `--build-local` | Use locally-built container images | -Processes 30+ workflow files across smoke tests, build tests, and agentic workflows. Ensures CI tests use the current source code rather than stale published images. +Processes 29 workflow files (5 smoke, 8 build-test, 13 agentic, 3 secret-digger) across the suite. Note: the script currently expects `.github/workflows/smoke-gemini.lock.yml`; if that workflow does not exist, either add it or update the script to keep the expected workflow list in sync. Ensures CI tests use the current source code rather than stale published images. ---