You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(scan): detect credentials exposed in source - #1134
Credentials embedded in current source can expose access even when their containing code is unused or lives in a test. Teach the scan workflow to assess this source-reader disclosure path offline, retain the original evidence, and distinguish supported exposures from placeholders and public material.
Changes
Apply the credential-discovery guidance to the core audit, baseline auditor, and focused investigators, with source access and uncertain live validity reflected in the finding.
Add a generated-source eval with six credential cases and negative controls. Grade retained findings, applicable classifications, exact source evidence, and complete coverage without giving the model gold labels.
Reuse the native account and permission preflight, isolate eval state and model tools, and preserve saved-login refreshes and authentication precedence. Use the shared permission-profile identifier after merging main so the eval accepts its configured profile and detects runtime fallback.
Share the bundled SDK environment helper while preserving Unix PATH behavior and Windows path aliases. Add deterministic eval checks to CI.
Testing
Passed all 135 deterministic secret-discovery eval checks, including native child-process authentication precedence, exact evidence preservation, profile rejection, fallback cancellation, and cleanup.
Passed all 54 focused SDK checks for custom validation, CLI diagnostics, and errors.
Passed SDK build:ci, types, and format.
Passed Ruff 0.16.9 check and format checks, plugin source compatibility, and all 9 source-checker tests.
Three independent native reviews and fresh verification found no actionable regressions on the final integrated commit.
Full seeded SDK suite: 3,438 passed, 53 skipped, and 11 failures caused by the host working-directory Git marker. All 73 authentication tests passed unchanged from an isolated working directory.
Full randomized SDK suite: 3,448 passed, 53 skipped, and one TUI subprocess import failure caused by the isolated working directory. That test passed unchanged from the package directory. No unexplained full-suite failures remain.
The optional model-backed eval was not run during this update. Deterministic checks verify the harness and grading behavior; they do not measure model recall.
Risk and rollout
The production change is scan guidance. It retains source evidence and existing credential-access controls, and it does not use discovered credentials or validate them against services. The optional eval runs in isolated temporary state with offline, read-only model tools. No public CLI command, flag, environment variable, or default is added or changed.
Public disclosure review
No customer, partner, prospect, or user identities, data, or identifying details are included.
No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.
Adds source-level credential detection to security scans and a synthetic evaluation suite to verify discovery, redaction, coverage, and runtime isolation.
Changes:
Extends audit prompts to treat exposed credentials as standalone findings.
Adds deterministic secret-discovery fixtures, grading, runtime safeguards, and tests.
Runs deterministic checks in CI and refreshes the workflow compatibility hash.
The reason will be displayed to describe this comment to others. Learn more.
Reviewed the prompt changes, isolated evaluation harness, fixtures, grader, authentication/fallback checks and cancellation cleanup. The existing Windows-path issue remains applicable at this head: grade.mjs strips './' but does not normalize backslashes, so a correct src\client.py location is missed and counted as a false positive. Please address #1134 (comment) before approval. No additional finding; hosted node-ci is green, but this missing grading case is not covered. No local or live-model tests run.
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🔵 Needs a closer look
Whitespace-only CODEX_HOME values can bypass the default saved login and break evaluation authentication.
Review effort: Balanced Findings: None
Previously missed (1)
In code that hasn't changed since last review
Treat whitespace-only CODEX_HOME as unset
evals/secret-discovery/run.mjs:27
Treat whitespace-only CODEX_HOME as unset, as the SDK does in sdk/typescript/src/auth.ts:33-38. Currently a value such as CODEX_HOME=" " resolves to that literal relative directory, so the eval misses the default saved login and can fail authentication instead of using ~/.codex.
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🔵 Needs a closer look
The new masking contract conflicts with the repository’s explicit policy against wrapper-owned credential redaction.
Review effort: Balanced Findings: None
Previously missed (2)
In code that hasn't changed since last review
Stop grader from requiring credential masking
evals/secret-discovery/grade.mjs:208
The grader makes credential masking a pass condition, thereby enforcing the wrapper-owned redaction policy prohibited by AGENTS.md:49-62; that policy specifically requires preserving diagnostic/report text and not replacing credential-shaped content solely because it may be sensitive. Remove this leakage failure condition and align the prompt and eval documentation accordingly.
Remove credential redaction from saved scan drafts
This new requirement redacts credential text before saving scan drafts, which conflicts with the repository policy in AGENTS.md:49-62 prohibiting wrapper-owned credential redaction in saved diagnostics, reports, and summaries. Remove this masking requirement and preserve the source-backed diagnostic payload; the corresponding eval/docs that enforce masking must be updated as part of the same change.
This issue also appears in the following locations of the same file:
Corrected in b4a5f65. Both items in Copilot’s review identify a policy conflict: the pre-persistence instruction transforms returned worker evidence, and the eval enforces that masking contract.
The update removes the added masking requirements and grading checks. Offline credential discovery, original finding evidence, saved-login support, and existing authentication, access, storage, and publication controls remain intact. All 125 deterministic eval tests passed, including raw-result preservation through the SDK child-process boundary, along with focused SDK tests, all five portable plugin checks, types, formatting, and installed-package checks.
The reason will be displayed to describe this comment to others. Learn more.
Reviewed the current-source credential guidance and eval. All 135 deterministic eval tests, SDK types, and source checks pass locally; full CI passed on this head. These checks do not establish model recall.
One integration requirement before combining with #1193: the eval uses discovery_eval, while that cleanup hardcodes codex_security_deep_scan_worker in the shared preflight/fallback helpers. The existing native-preflight eval test passes here and fails with the cleanup helper. Preserve configurable profile IDs or update the eval's profile configuration and callers together.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
soyeon-oai
left a comment
kmbroai
left a comment
Summary
Credentials embedded in current source can expose access even when their containing code is unused or lives in a test. Teach the scan workflow to assess this source-reader disclosure path offline, retain the original evidence, and distinguish supported exposures from placeholders and public material.
Changes
Testing
build:ci,types, andformat.Risk and rollout
The production change is scan guidance. It retains source evidence and existing credential-access controls, and it does not use discovered credentials or validate them against services. The optional eval runs in isolated temporary state with offline, read-only model tools. No public CLI command, flag, environment variable, or default is added or changed.
Public disclosure review