Rootless artifact permission repair fails silently — causes exit code 1 on cleanup #6070

Summary

fixArtifactPermissionsForRootless() is failing for all directories during post-agent cleanup, causing the Copilot CLI to exit with code 1. The repair docker run command exits 1 but stderr is not captured, making the root cause opaque.

Observed in: https://github.com/github/gh-aw-mcpg/actions/runs/29042841176 (Large Payload Tester)

Symptom

After the agent completes successfully and containers are stopped, the cleanup phase fails:

[WARN] Rootless artifact permission repair failed for /tmp/gh-aw/sandbox/firewall/logs (exit 1)
[WARN] Rootless artifact permission repair failed for /tmp/gh-aw/sandbox/firewall/audit (exit 1)
[WARN] Rootless artifact permission repair failed for /tmp/awf-1783623821084-chroot-home (exit 1)
[WARN] Failed to remove chroot home directory after permission repair: Error: EACCES: permission denied, unlink '/tmp/awf-1783623821084-chroot-home/.aws/config'
[WARN] Command completed with exit code: 1
Process exiting with code: 1

Each repair attempt takes ~30s (suggesting Docker overhead or timeout), and the final EACCES on .aws/config causes the process to exit non-zero despite the agent task completing successfully.

Root cause analysis

Problem 1: No stderr logging from repair container

In artifact-permissions.ts (lines 50–83), the docker run for permission repair uses execa.sync with reject: false, but does not capture or log stderr. When the repair exits with code 1, we only see the exit code — not why it failed.

Possible failure reasons (all invisible without stderr):

• Docker daemon issue (image not found, daemon busy)
• Container startup failure (the agent image's entrypoint may not support bare sh -c properly)
• Volume mount failure (path doesn't exist from Docker daemon's perspective)
• The awf-net network was still "in use" at cleanup time — could indicate a Docker daemon state issue

Problem 2: Repair failure escalates to fatal exit code

The flow in artifact-preservation.ts:

1. preserveAndCleanupArtifacts() calls fixArtifactPermissionsForRootless() for logs/audit/session-state (line 214) — failures are warnings only ✅
2. removeWorkDirectories() tries fs.rmSync(chrootHomeDir) — hits EACCES on root-owned files (line 234)
3. Catches EACCES, calls fixArtifactPermissionsForRootless() again for chroot-home (line 241) — also fails
4. Retries fs.rmSync() — fails again with EACCES (line 249)
5. Logs warning: "Failed to remove chroot home directory after permission repair" (line 251)

This warning propagates as exit code 1 from the Copilot CLI, even though:

• The agent task completed successfully
• The Post Setup Scripts step successfully cleaned up the chroot-home directory afterward (Cleaned up 1 /tmp/awf-*-chroot-home directory)

Problem 3: Redundant cleanup

The Post Setup Scripts step (compiled by gh-aw) has its own sudo cleanup of chroot-home directories, and it succeeded. AWF's Node.js rimrafSync attempt is redundant when running under gh-aw, and its EACCES failure is what causes the non-zero exit.

Proposed fixes

Fix 1: Capture and log stderr from repair container (diagnostic)

// artifact-permissions.ts line 81
if (typeof result.exitCode === 'number' && result.exitCode !== 0) {
  logger.warn(
    `Rootless artifact permission repair failed for ${dir} (exit ${result.exitCode})` +
    (result.stderr ? `: ${result.stderr}` : ''),
  );
}

Fix 2: Make chroot-home removal failure non-fatal

The removeWorkDirectories() function should not cause the process to exit non-zero when it can't remove the chroot-home directory. The files will be cleaned up by the post-step (install_copilot_cli.sh cleanup) or by the runner infrastructure. Change the catch at line 250 to swallow the error:

} catch (retryError) {
  // Non-fatal: chroot-home will be cleaned by post-step or runner infra.
  logger.debug('Could not remove chroot home directory after permission repair — will be cleaned by post-step');
}

Fix 3: Investigate the 30s repair latency

Each docker run --rm --pull never --network none should be nearly instant for a chown+chmod. The ~30s per call suggests something is wrong at the Docker level. With stderr captured (Fix 1), we can diagnose whether this is a pull attempt, image issue, or daemon state problem.

Environment

• AWF v0.27.27
• Runner: GitHub-hosted (ubuntu-latest)
• Network isolation mode enabled
• Note: awf-net reported "Resource is still in use" during docker compose down, possibly related

Files

• src/artifact-permissions.ts — repair container logic (no stderr capture)
• src/artifact-preservation.ts — removeWorkDirectories() where EACCES escalates to fatal
• src/container-stop.ts — pre-shutdown squid log chmod (works correctly)