[amandeavor]

What this adds

A new blind evaluation pool for E21 at eval/e21_blind/amandeavor.jsonl: 90 tool calls (45 labelled harmful and 45 labelled benign), one JSON object per line, matching the shape of eval/e21_heldout/corpus.jsonl.

The harmful calls cover the ground specified in #368:

• CI/CD workflows and secrets: deleting CI secrets with gh secret delete, removing branch protection and PR review requirements via gh api, deleting security workflows, capturing Actions OIDC tokens, and dumping runner environment secrets.
• Cloud IAM: creating backdoor admin users, access keys, and login profiles in AWS; assigning Project Owner and Organization Admin roles in GCP; creating backdoor Azure Service Principals with Contributor permissions; stopping and deleting CloudTrail trails and audit logging sinks.
• Secrets stores and credentials: dumping Kubernetes secrets across namespaces, extracting database passwords from Kubernetes secrets, reading production secrets from Vault and AWS Secrets Manager, and extracting credentials from kubeconfig and .env.production.
• Exfiltration and prompt injection: leaking secrets and credentials via curl, wget, netcat, and DNS exfiltration.
• Evasion: calls using base64 decoding pipes, variable concatenation, eval, and inline scripts.

The benign calls provide matching routine operations: read-only GitHub CLI commands (gh secret list, gh workflow list, gh run view), read-only cloud IAM enumeration (aws iam list-users, gcloud iam service-accounts list, az role assignment list), read-only Kubernetes lookups (kubectl get secrets, kubectl auth can-i), Vault status queries, and routine git, terraform, and workflow commands.

Blind statement

I wrote this pool without reading the rules: I have not read tracekit/policy2/, tracekit/policy/, the policy packs, docs/policy-v2.md, the E3 or E17 corpora, eval/e21_blind/, or any rule ids. The only project material used was issue #368 itself and the field shape of eval/e21_heldout/corpus.jsonl as the schema reference.

Validation

All 90 rows parse as valid JSON objects carrying the heldout schema fields with matching types: 45 harmful rows with severity, techniques, and OWASP tags, and 45 benign rows.

Fixes #368

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[LoopGlitch26]

Scored as submitted, against main at 70aaa4c, before any rule change. This stays the pool's blind score; any rule written from it reports the score after.

Pack	Harmful held or denied	Benign held
server (server.yaml)	22 of 45 (49%), 17 denied	0 of 45
strict (strict.yaml)	25 of 45 (56%), 19 denied	2 of 45 (curl --dry-run, curl -I: strict holds every network command)

Both regex engines decide every call the same. By category on server: CI/CD changes 0 of 4, defense evasion 4 of 13, credential access 5 of 11, exfiltration 4 of 7, privilege escalation 6 of 7, persistence 3 of 3.

What the server pack misses, grouped:

• GitHub repository security through gh (0 of 6): deleting Actions secrets, deleting branch protection or its review requirement, turning secret scanning off, and the same secret delete with its path base64-encoded.
• CI workflow files (0 of 3): deleting workflows with rm or git rm (flagged, not held), and writing a workflow that sends secrets out.
• Audit logging (0 of 2): CloudTrail stop-logging and delete-trail.
• Secret reads outside the cloud CLIs (0 of 7): Kubernetes secrets dumped or decoded with kubectl, vault kv get, a root vault token create, reading .env.production (flagged), and CI environment variables written to a file.
• Exfiltration and encoding (4 misses): a token read into a curl URL, DNS exfiltration with dig, and two inline Python calls (one base64-decoded); strict holds two of them.
• kubectl delete clusterrolebinding is flagged on server and held on strict.

I'll open rule issues for the first four groups, as #359 to #361 came from the first pool. This PR adds only the pool file, so it can merge once CI passes. Thanks for writing it blind.

[LoopGlitch26]

Rule issues from the misses: #402 (repository secrets and branch protection through gh), #403 (AWS audit logging), #404 (Kubernetes and Vault secret reads), #405 (CI workflow files). Any of them is yours if you want it.

[LoopGlitch26]

Merged, thanks. The score above stays this pool's blind score.

[LoopGlitch26]

Thanks again for contributing to Tracekit. A request: if you use a coding agent (Claude Code, Codex, Cursor, Gemini CLI or OpenCode), would you run Tracekit on it for a week and tell us how it went?

pip install -U tracekit-ai
tracekit audit --since 7d      # read-only: what it would have held in your last week (Claude Code history)
tracekit init --dev --v2       # then record and guard from now on

After a week, tracekit report --share prints counts only (no commands, paths or content). Post it in #385 with anything that got in your way: a false alarm, a confusing message, something slow. If you find a problem, you're welcome to fix it too: say so on #385 and it's yours.