Skip to content

fix: resolve idna CVE-2024-3651 false negative via Python-scoped reachability gate - #406

Open
heatherzh01 wants to merge 4 commits into
mainfrom
fix/tc-6585-false-negatives
Open

heatherzh01 wants to merge 4 commits into
mainfrom
fix/tc-6585-false-negatives

Conversation

@heatherzh01

Copy link
Copy Markdown

Summary

Fixes a false negative where idna CVE-2024-3651 (quadratic-complexity DoS in
idna.encode) was scored not exploitable even though the sample application
calls idna.encode() directly on user-provided input in its Flask handler.

The fix is scoped entirely to the Python ecosystem — Go, C, and Java verdicts
are byte-for-byte unchanged from baseline.

Root cause

  1. CCA query name collapse. For a top-level package function (idna.encode),
    the reachability agent sent the package name as the function name
    ("idna,idna"). Call Chain Analyzer looked up idna() instead of encode(),
    returned "function not found", and that failure was recorded as a genuine
    "not reachable" verdict.
  2. No recovery on CCA failure. A failed/blocked CCA result was treated as a
    real negative verdict, tripping the hard reachability gate — so even with a
    direct idna.encode() call site in application source, the verdict was forced
    to not-exploitable.

Changes

File Change
react_internals.py Reachability Rule 8 rewritten: a top-level module.function symbol uses the function's own name (encode), never the package name; never function_name == package_name.
transitive_code_search.py _validate_query_format rejects the degenerate package==function query with a retry hint; new cca_output_indicates_no_analysis() / CCA_NO_ANALYSIS_MARKERS distinguish a failed/blocked CCA result from a real verdict.
reachability_agent.py Python-only: a failed-analysis CCA False is kept out of cca_results; app_source_callsite is recorded on a main-application-source hit. Other ecosystems keep baseline behaviour.
full_text_search.py keyword_result_has_app_source_hit() + APP_SOURCE_SECTION_LABEL.
cve_agent.py Plumbs app_source_callsite through to checklist items.
cve_summarize.py New Python-only REACHABILITY-LIKELY gate; message hardened against hypothetical "mitigations" and uncertain hedging.

Why the gate is Python-scoped

The inference "direct application-source call site ⇒ reachable ⇒ lean
exploitable"
is sound for Python top-level library functions
(idna.encode(user_input)), but it over-reports Go DoS CVEs whose vulnerable
path is library-internal. It is confined to Python via
_app_callsite_gate_applies(items, ecosystem).

Scope / explicitly out of scope

  • Not included: cache-schema bump, test-harness configs, Text4Shell/Java
    reachability work, and the Go-DoS over-report problem — separate efforts.
  • Non-Python ecosystems are unchanged by design.

Risk

Low. All behavioural changes are gated on ecosystem == "python"; the single
global change only blocks a malformed package==function query and returns a
retry hint.

@heatherzh01
heatherzh01 requested a review from tmihalac October 6, 2026 01:05
@heatherzh01

Copy link
Copy Markdown
Author

/test-heavy

@heatherzh01

Copy link
Copy Markdown
Author

/test vulnerability-analysis-on-pr

@heatherzh01

Copy link
Copy Markdown
Author

/test-heavy

1 similar comment
@tmihalac

tmihalac commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

/test-heavy

@tmihalac

tmihalac commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Automated multi-agent review — idna CVE-2024-3651 Python-scoped gate

Reviewed across five dimensions (correctness, tests, silent failures, comments, general quality). The two most severe findings were verified by hand against the branch. Headline: as written, the fix does not fire — the new gate is currently dead code.


🔴 CRITICAL

1. The fix never fires — app_source_callsite is not a declared AgentState channel. [verified]
src/vuln_analysis/functions/react_internals.py:1119-1134 — AgentState declares cca_results, package_validated, is_reachability, ecosystem, etc., but not app_source_callsite. LangGraph silently drops keys returned from a node that are not declared channels, so reachability_agent.py:399 (extra["app_source_callsite"]) never persists into state. Consequently cve_agent.py:229 answer.get("app_source_callsite") is always None → _reachability_app_callsite_without_cca never returns True → _app_callsite_gate_applies never fires → the Python REACHABILITY-LIKELY gate is inert. The per-iteration "sticky" read at reachability_agent.py:394 is broken by the same cause.
Fix: add app_source_callsite: bool | None = None to AgentState (next to package_validated, ~line 1133).

2. Marker substring-match runs over source-contaminated tool output → false-positive exploitable.
src/vuln_analysis/tools/transitive_code_search.py:95-106 — cca_output_indicates_no_analysis() does str(tool_output) and substring-scans the entire stringified (False, path_summary) tuple. path_summary carries summarized application/dependency source lines. The two generic markers "Invalid input format." and "Invalid query:" are exactly the kind of text that appears in real source (string literals, raised errors, comments). A genuine "not reachable" verdict whose summarized code line contains a marker is then dropped from cca_results → for a Python CVE with an app-source hit, _app_callsite_gate_applies fires and REACHABILITY_LIKELY_GATE_MSG forces exploitable. A real negative becomes a confident false positive.
Fix: inspect only the known status slot (the "no analysis" marker is always the sole element of path_summary), or return a structured status sentinel instead of substring-scanning free-form output.

3. app_source_callsite is a bare keyword hit, not a call site — then used to force exploitable.
src/vuln_analysis/utils/full_text_search.py:92-108 only proves the keyword appears ≥1× in main-application source. A comment, string literal, import line, or unrelated same-named symbol all count as a hit. REACHABILITY_LIKELY_GATE_MSG then instructs the summary LLM that the verdict "MUST be 'exploitable'", "MUST NOT be 'uncertain'", "do NOT hedge" — amplifying a weak signal into an unhedgeable verdict. (Live once 1 is fixed.)


🟠 IMPORTANT

4. The package == function guard is global and shared with the mandatory Function Locator. [verified]
src/vuln_analysis/tools/transitive_code_search.py:146-154 — _validate_query_format is shared by transitive_search (Call Chain Analyzer, :360) and package_and_function_locator (Function Locator, :470), across all ecosystems. The rest of the PR is carefully Python-scoped; this guard is not. (a) It rejects legitimate Go/Java/C fuzzy Function Locator queries where package and function names coincide. (b) The error text is idna/CCA-specific ("For a top-level package function such as idna.encode … never repeat the package name") and is misleading when returned from the Function Locator path.
Fix: scope the guard to Python and/or the CCA tool, make the message tool-neutral — or confirm no legitimate package == function case exists for Function Locator across Go/Java/C.

5. "Keep markers in sync" is a manual coupling that fails silently toward false negative.
src/vuln_analysis/tools/transitive_code_search.py:88-95 — CCA_NO_ANALYSIS_MARKERS duplicates literal strings emitted at :145, :148-153, :383-386. If any producer message is reworded without updating the markers, cca_output_indicates_no_analysis() returns False, the failed-analysis False is recorded as a genuine not-reachable verdict, the hard reachability gate fires, and a reachable Python CVE becomes a false negative — no error, no log, no failing test.
Fix: define the producer messages as named constants and build the markers from them.

6. Keyword-header format coupling fails silently toward false negative.
src/vuln_analysis/utils/full_text_search.py:105 — the regex shares only APP_SOURCE_SECTION_LABEL with the producer (:253); the " (N of M results)" shape is not covered by a shared constant. If search_index()'s header format changes, keyword_result_has_app_source_hit() silently returns False and reintroduces the idna-class false negative this PR exists to fix.

7. Inaccurate comment about the non-Python path.
src/vuln_analysis/functions/cve_summarize.py:227-228 claims "Every other ecosystem falls through to the softer REACHABILITY-UNVERIFIED handling below." That is only true when cca_results is empty. For the Go/C DoS case the same block discusses, the failed verdict is recorded as False (baseline), so the earlier _all_cca_not_reachable (gate 2) fires the hard gate and control never reaches REACHABILITY-UNVERIFIED. This contradicts the accurate comment at reachability_agent.py:348-353.

8. CVE/ticket references in source comments (project convention: describe behavior, not task origin).
Strip the CVE IDs (keep the behavioral description and the generic idna.encode() illustration) from: full_text_search.py:103, cve_summarize.py:118,209,215-216, transitive_code_search.py:88, reachability_agent.py:346,388. The references in tests/test_cve_summarize_gates.py are acceptable — tests may document the bug they reproduce.

9. Test coverage: four bug-fix functions ship with no tests; no plumbing test.
Untested: cca_output_indicates_no_analysis() + each marker, post_observation Python-only CCA-False drop (including the non-Python keeps-False baseline guarantee), keyword_result_has_app_source_hit() regex/boundary, and _validate_query_format equality guard. There is also no integration test of summarize_cve's elif routing or the graph → answer → summarize plumbing — which is precisely why finding 1 passes CI green while the feature is inert. These belong in test_reachability_agent.py, test_transitive_code_search.py, and a new full_text_search test. TestGateMessage is brittle phrase-by-phrase matching on a prompt constant — it passes even if the gate stops firing; keep one smoke check and drop the rest.

10. CCA tool exceptions are swallowed and indistinguishable from "no analysis".
With handle_tool_errors=True, an actual CCA crash becomes a plain ToolMessage with no error-level log; post_observation sees neither "true" nor "false" → empty cca_results, identical to "function not found" and "CCA never called". A systemic CCA failure is laundered into a reachability verdict with no alert.
Fix: log CCA tool exceptions at error level and carry a distinct "errored" signal separate from "no analysis".


🟡 Suggestions

  • Ecosystem normalized three ways — reachability_agent.py:354 uses the literal "python", cve_summarize.py uses Ecosystem.PYTHON.value. They agree today (both from image.ecosystem.value) but the two halves of one mechanism are written in different idioms; a future change to either source decouples them. Unify via a shared helper.
  • Duplicated rationale — the "Go DoS over-reports, so scope to Python" argument is stated in full in both cve_summarize.py:214-230 and reachability_agent.py:348-353, with partial restatements elsewhere. Keep one canonical rationale and reduce the others to a one-line note + pointer.
  • Gate ordering — _all_cca_not_reachable is checked before _app_callsite_gate_applies ; for a Python CVE with ≥2 reachability questions where one yields a genuine False, preempts the app-callsite rescue. Defensible, but undocumented at the ordering site. (Moot until 1 is fixed.)

@exploit-iq-pac

exploit-iq-pac Bot commented Oct 6, 2026

Copy link
Copy Markdown

Caution

There are some errors in your PipelineRun template.

PipelineRun Error
vulnerability-analysis-on-pr CEL expression evaluation error: expression "event == \"pull_request\" &&\n!body.pull_request.draft &&\n(target_branch == \"main\" || target_branch == \"rh-aiq-main\") &&\n(\"src/**\".pathChanged() || \"metrics_lib/**\".pathChanged() || \"pyproject.toml\".pathChanged() || \"uv.lock\".pathChanged() || \"Dockerfile\".pathChanged() || \".dockerignore\".pathChanged())\n" failed to evaluate: no such key: pull_request

@heatherzh01

This comment was marked as duplicate.

@heatherzh01

Copy link
Copy Markdown
Author

/test-heavy

@heatherzh01

Copy link
Copy Markdown
Author

/test vulnerability-analysis-on-pr

1 similar comment
@heatherzh01

Copy link
Copy Markdown
Author

/test vulnerability-analysis-on-pr

@heatherzh01

Copy link
Copy Markdown
Author

/test-heavy

@TamarW0

TamarW0 commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Question on the reject_equal_pkg_fn guard and the original root cause

The Rule 8 prompt rewrite should prevent the model from generating idna,idna going forward. That makes the reject_equal_pkg_fn guard in _validate_query_format a safety net rather than the primary fix.

However, the guard currently does a hard reject on all package == function queries, which blocks legitimate Python cases where the function name genuinely matches the package name — e.g. socket.socket(), decorator.decorator(), copy.copy(). In those cases the agent cannot comply with the retry hint ("provide the function's own name, do not repeat the package name") because the real name IS the package name. The agent would exhaust retries, record no CCA result, and lose reachability evidence.

Two questions:

  1. What caused the model to produce idna,idna originally? Was it purely the old Rule 8 wording ("use the class name, NOT the method name") that led the model to treat idna as a class and drop encode? Or did something in the agent's observation/state (e.g. Function Locator output format) also contribute? Understanding the trigger helps gauge how likely it is to recur despite the new Rule 8.

  2. Could the guard be softened? For example: emit a warning and let the agent confirm the name is correct on retry, rather than a hard reject. That way the safety net still catches the name-collapse bug without blocking legitimate pkg == fn queries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants