Repository navigation
perf(indexer): compute interface fingerprints in Rust on the tree bridge - #3495
Conversation
 Tested 1 flow, found no issues. What we tested
Took 2 attempts. TREX adjusted its test setup between them. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Fingerprint rules and language configuration codebase_rag/constants/ast_js.py, codebase_rag/constants/ast_python.py, codebase_rag/constants/interface.py, codebase_rag/parsers/interface_fingerprint.py, codebase_rag/parsers/utils.py, codebase_rag/types_defs.py |
Python defines the language, node-type, and callable-scan rules used for fingerprinting. The parser packages these rules for the bridge and uses shared node-type sets in callable-parameter scans. |
Rust traversal and scope support rust/cgr_graph_rs/src/trees/mod.rs, rust/cgr_graph_rs/src/trees/scopes.rs, rust/cgr_graph_rs/src/trees/tests.rs |
Rust adds preorder tree traversal and scope-binding helpers. Scope walks use the traversal, and their node-type collections use the shared aliases. |
Rust fingerprinting and bridge integration rust/cgr_graph_rs/src/trees/interface.rs, rust/cgr_graph_rs/src/trees/python.rs, codebase_rag/parsers/interface_fingerprint.py, codebase_rag/parsers/native_trees.py, codebase_rag/constants/embedded_graph.py |
The Rust implementation fingerprints syntax trees using configured rules. The tree bridge loads those rules and exposes the fingerprint method. The Python parser selects the bridge when available and falls back to its Python implementation when the bridge reports NotBorrowed. Bridge validation compares the Rust and Python fingerprints. |
Parity tests and release updates codebase_rag/tests/conftest.py, codebase_rag/tests/test_interface_fingerprint.py, codebase_rag/tests/test_native_interface.py, rust/cgr_graph_rs/src/trees/tests.rs, codebase_rag/tests/test_rust_extension_packaging.py, docs/architecture/embedded-graph.md, docs/reports/INDEXER_RUST_PLAN.md, pyproject.toml, rust/cgr_graph_rs/Cargo.toml, rust/cgr_graph_rs/README.md, rust/cgr_graph_rs/pyproject.toml |
Tests compare native and Python fingerprints across language samples and Python package files, and check fallback and bridge-validation cases. Documentation and performance reports describe the landed implementation. Package version references change to 0.1.3. |
Priority:
Estimated code review effort: 4 (Complex) | ~45 minutes
Change: Refactor
Sequence Diagram(s)
sequenceDiagram
participant Fingerprint as interface_fingerprint
participant Bridge as native_trees.interface_bridge
participant TreeBridge
participant Rust as Rust interface_fingerprint
participant Python as python_interface_fingerprint
Fingerprint->>Bridge: select available interface bridge
Bridge-->>Fingerprint: return TreeBridge or None
Fingerprint->>TreeBridge: request fingerprint for supported language
TreeBridge->>Rust: fingerprint tree using source and configured rules
Rust-->>Fingerprint: return fingerprint
Fingerprint->>Python: use Python implementation when bridge reports NotBorrowed
Python-->>Fingerprint: return reference fingerprint
Possibly related PRs
- vitali87/code-graph-rag#3492: Adds the Rust and Python scope walks and scope facts used by this interface-fingerprint implementation.
- vitali87/code-graph-rag#3475: Adds the tree bridge framework that this change extends with interface fingerprinting and parity checks.
Merge Risk: 
0e58b
-
Normal file indexing is unaffected by the identified discrepancy. Nested-definition fingerprints can differ between Rust and Python, warranting bounded follow-up rather than blocking this release.
Pre-merge checks: 
-
Check name Status Explanation Title check The title clearly and concisely describes the primary change: moving interface fingerprint computation to Rust on the tree bridge. It uses Conventional Commits format. Description check The description includes all required sections, explains the implementation, documents performance and correctness results, identifies test limitations, and records related issues and checklist status… Linked Issues check The changes implement the active coding requirements in #3473. Rust now computes interface fingerprints on existing py-tree-sitter trees for Python, Java, JavaScript, TypeScript, and TSX. Python remai…Out of Scope Changes check The changed constants, rule types, bridge validation, traversal helper, Rust scope integration, tests, version updates, and documentation support the #3473implementation. The Java registry scan appea…Full details: Description check
-
Explanation
The description includes all required sections, explains the implementation, documents performance and correctness results, identifies test limitations, and records related issues and checklist status.
Full details: Linked Issues check
-
Explanation
The changes implement the active coding requirements in
#3473. Rust now computes interface fingerprints on existing py-tree-sitter trees for Python, Java, JavaScript, TypeScript, and TSX. Python remains the reference and fallback. The startup probe compares both implementations. The shared scope facts and Rust scope walks support the fingerprint path. The reported tests compare both paths across language samples and package files, and the reported measurements reduce the combined profile share from 21.9% to 5.1%. The reported graph comparisons preserve nodes, edges, and fingerprints for code-graph-rag, rich, and click.
Full details: Out of Scope Changes check
-
Explanation
The changed constants, rule types, bridge validation, traversal helper, Rust scope integration, tests, version updates, and documentation support the
#3473implementation. The Java registry scan appears only as documented future plan work; no unrelated registry implementation was added. The dependency and crate version updates support the new bridge method.
-
Finishing Touches
-
📝 Generate docstrings
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
[Medium impact] The PR appears safe to merge; no blocking defects were found. SummaryThe PR moves file interface fingerprints to Rust while keeping Python as the reference and fallback.
No blocking issues were found. Tests were inspected but not run. Diagram%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[File interface fingerprint] --> B{Language has rules?}
B -->|No| C[Return None]
B -->|Yes| D{Bridge supports fingerprint?}
D -->|No| E[Python reference]
D -->|Yes| F[Rust walk]
F -->|NotBorrowed| E
F -->|Success| G[Return digest]
E --> G
Reviews (1) · Last reviewed commit: "docs(indexer): record stage 3 of the Rus..." · Reviewed by Greptile |
|
Main reached 0.1.3 with #3495, so the pagerank fix needs a version above it for publish.yml to ship the changed library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ChfULvr1gXZn6MQXr4xNFS
Summary
Gson.java: 112 dependents, ~65 s per sync) #2531) to decide whether the files that depend on it must be read again.TreeBridge(perf(indexer): hash definitions in Rust on py-tree-sitter's own trees #3475) now computes it on py-tree-sitter's own trees, in Python, Java, JavaScript, TypeScript and TSX.python_interface_fingerprintstays the reference and the fallback, and the startup probe compares the two before the bridge is used.Demo
The GIF, recorded with VHS, indexes rich from scratch: 213 Python files, no embeddings. The first run uses
main's source (4f497f2) withmain's own build of the extension, asmain's users get it. The second uses this branch, in the same environment. After each run it exports the graph and hashes every module's fingerprint, as "module fingerprint" lines (fingerprints.sh, printed at the start):03f7f80e33988fbb.That is one run of each. The median of five alternating runs is below: 19.8 s against 17.0 s. The GIF is stored on the
claude/cgr-rust-porting-mrwohh-evidencebranch and is not part of this PR's diff.How it works
The walk.
rust/cgr_graph_rs/src/trees/interface.rsportsinterface_fingerprint.pyfunction by function:It hashes the same length-prefixed tokens with the same SHA-256, so it returns the same string.
The rules.
bridge_rules()hands the bridge_RULESand every node type, field and token the walk reads, all fromconstants/, as stages 1 and 2 do. Their shapes areInterfaceRulesand friends intypes_defs.py. The node-type sets thatparsers/utils.pykept privately for the callable scan moved toconstants/ast_python.pyandconstants/ast_js.py, so both sides read one definition. A length prefix or byte order the Rust side does not write, or an unknown flavor, fails the bridge's construction, and the bridge stays off.Scope facts. Where the Python checks ask which names a function binds, they call #3492's scope facts. The Rust walk calls #3492's Rust scope walks directly (
names_bound_withinandadd_localsinscopes.rs) and keeps the facts per def for the file, without calling back into Python.The probe. The startup probe now also fingerprints its sample both ways. A
relayfunction was added to the sample, with the kinds of calls, keyword arguments and returns the fingerprint looks at. Without a guard, the Python twin's scope lookups would go through the very bridge being probed.native_trees._checkingkeeps them in Python while the probe runs, so it compares two separate computations, not Rust with itself.Walking faster. The walk never asks a node for its parent, because tree-sitter answers that by walking down from the root; it keeps the enclosing definitions on its own stack instead. It and #3492's scope walks now share one cursor-based preorder iterator (
Preorderintrees/mod.rs). The node-type sets hash with FNV (NodeTypes), about a third cheaper per lookup than Rust's default hasher. That is safe here because their keys are the grammar's type names, never text from the file; sets of names read from the file keep the default hasher.Fallbacks, as in #3492:
NotBorrowed, and that call runs the Python code.interface_fingerprintleaves every file to Python (native_trees.interface_bridge()).None.Results
Per call, each file fingerprinted both ways:
codebase_rag/, 1,743Most of what is left is tree-sitter's own cursor, 93 ns to 125 ns a step. The walk also visits a function's body more than once: first to check what reads it, then to project it.
End to end, sync time without embeddings, alternating
main(with its own extension build) and this branch in the same environment:mainThis container ran about 1.6 times slower than when #3492 was measured: the same code on the same tree took 214 s to 224 s then. Compare within this table only. The 1.13x is what #3492's profile predicted (12.9% at 8.6x). Under py-spy,
interface_fingerprintfell from 12.9% of the code-graph-rag run to 1.9%. The scope walks fell from 6.6% to 3.2%, since the fingerprint no longer calls them from Python.Correctness:
mainrun beside it, byte for byte, fingerprints included (215,163 and 215,164 nodes and edges). The two pairs differ from each other by oneCALLSedge, on both sides.mainitself (noted in perf(indexer): walk each Python scope once, in Rust when the tree bridge can #3492). Its five runs on each side wrote the same set of lines.Type of Change
Related Issues
Closes #3473: #3492 was its scope-walk half, and this is
interface_fingerprint, the rest. Related to #2531, which introduced the fingerprint.Test Plan
uv run pytest -n auto -m "not integration", with the extras CI installs): 24,130 passed and 5 failed, none of them because of this change.go/typeshelper (two Go oracle tests).test_project_selection.py::TestOutsideAnyIndexedRepo::test_several_projects_are_refused_with_the_indexed_names, fails onmaintoo, about 1 run in 740 (see below). It passed on rerun.codebase_rag/tests/test_native_interface.py(29 tests) compares Rust and Python on a sample in each language, on every definition in those samples, on each sample's elided and kept bodies, and on every file of thecodebase_ragpackage. It also covers:test_interface_fingerprint.pynow runs on both paths through theboth_interface_walksfixture: 158 runs, all passing.src/trees/tests.rspin the Python twin's value on a sample, and that the fingerprint ignores what callers cannot see and changes with what they can.make test-integration, requires Docker). Not run: no Docker here.main-against-branch graph comparisons above;cargo test --release(61 passed);cargo clippy --features python: nothing new; the two existing warnings inlib.rsremain.Checklist
!in the title and its release notes inchanges/breaking/<PR number>.md: not a breaking change.make pre-commit)# type: ignore,cast(),Any, orobjecttype hintsFor the maintainer
cgr-graph-rsgoes from 0.1.2 to 0.1.3, as the "Crate version bumped with its code" check requires, and the requirement's floor moves to 0.1.3. Inuv.lock, only the crate's version line changed, rewritten in place as in perf(indexer): walk each Python scope once, in Rust when the tree bridge can #3492.uv lock --checkpasses.main:test_several_projects_are_refused_with_the_indexed_namesderives a project name from pytest's temporary folder, and that name carries a hash of the path. When the hash happens to look likecobra__125fa53b,close_project_namessuggests it, and the test's expectation (no close match, so every indexed project is listed) fails. The failing run's name wasscratch__25fabe29, at 0.625 against the 0.6 cutoff. About 1 random hash in 740 does that. Fixing the folder name, or the hash, in the test would remove it. That fix is unrelated to this change and is left out of it.java/type_resolver.py:_find_registry_entries_underat 9.5% of the code-graph-rag run, for its 7 Java files. When the registry's trie finds no names under a prefix, it scans every registered name. The plan records it as step 7, a Python fix; it is not part of this PR.Summary by CodeRabbit