Repository navigation
fix(parser): resolve Go and Ruby imports to files, not bare strings - #1027
Conversation
Go and Ruby both reached `_resolve_module_to_file` and fell through every
arm of `_do_resolve_module` to its bare `return None`, so every import
edge kept the raw import string as its target. Measured on cli/cli: 8687
go IMPORTS_FROM edges, 0 matching a File node, 3493 of them naming a
directory that exists on disk; `importers_of("pkg/iostreams/iostreams.go")`
answered 0 for a package imported 435 times. On jekyll: 227 ruby edges, 0
matching a File node.
Go gets a go.mod anchored resolver. The module path declared in the
nearest go.mod is the only thing that tells this repository's packages
from its dependencies, so it is read (cached on path/mtime/size) and an
import under that prefix maps to the directory it names, then to the
non-test .go files in it -- a Go import names a package, which is a
directory of files, not one file. Nested modules win over their
ancestors while the ancestors stay in the chain, local `replace` targets
and `vendor/` are honoured, and an import whose first segment has no dot
is the standard library and resolves to nothing. Resolving those by
walking up from the caller looking for a matching directory name would
bind 149 of cli/cli's 3575 stdlib imports to repository directories
(context 133, embed 11, io 5), which is why the module prefix is the
anchor and not a path search.
Ruby gets load-path resolution. The extractor now dispatches on the
call's method name rather than testing whether the node text contains
"require": that substring test made 34 of jekyll's 227 edges (15%) false
positives, with targets like `Missing --ssl_cert or --ssl_key. Both are
required.` from a `raise`, and it swallowed those call subtrees whole so
their calls were dropped too. It never saw `autoload`, which is how
jekyll declares its entire internal module graph (49 statements). The
form now survives to the resolver, because `require` and
`require_relative` take identical strings and resolve them differently:
`require_relative` against the requiring file, `require`/`load`/
`autoload`/`require_all` against load roots discovered from the
repository -- gemspec `require_paths`, lib/test/spec, the Rails `app`
roots, and `$LOAD_PATH.unshift` in a repository-root script. A path built
by an expression emits no edge rather than a guessed one, and a gem
resolves to nothing.
Both reuse the existing machinery: the case-exact directory-listing
probes (renamed from `_python_*` to neutral names, no behaviour change),
`_path_is_within` as the repository clamp, and the same cache key shape
and eviction cap as `_resolve_module_to_file`. Every Go import specifier
also carries its own line now; 8648 of cli/cli's 8690 import edges were
stamped with the line of the `import (` token.
Measured, before -> after:
cli/cli go edges naming a file 0/8687 -> 17527/22721 (77.1%)
importers_of(iostreams) 0 -> 424
impact radius 0 nodes -> 500 (1764 total), 212 files
cobra go edges naming a file 0/190 -> 159/337 (47.2%)
gin go edges naming a file 0/519 -> 230/718 (32.0%)
jekyll ruby edges naming a file 0/227 -> 159/264 (60.2%)
importers_of(test/helper.rb) 0 -> 51
rubocop ruby edges naming a file 0/458 -> 274/361 (75.9%)
Graphs for gson (java), okhttp (kotlin), ripgrep (rust), guzzle (php),
Newtonsoft.Json (csharp) and Alamofire (swift) are byte-identical before
and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
code-review-graph reviewOverall risk: 0.75 (HIGH) — 163 changed function(s)/class(es), 47 affected flow(s), 68 test gap(s) Risk-scored changes
Affected execution flows
Test gaps
Token savings: this graph-backed report used ~257,881 fewer tokens (~84%) than reading every changed file in full (estimated, chars/4 approximation). Powered by code-review-graph — local-first analysis; no code leaves the CI runner. |
A Go import names a PACKAGE, which is a directory of files, so resolving one by emitting an edge per member file makes the edge count grow with imports times package size. On kubernetes/kubernetes that was 588,972 IMPORTS_FROM edges against 93,624 import statements, a 6.5GB database, and 524s of build time against staging's 149s. 73,507 of those edges named a file under vendor/, which the default ignore patterns exclude, so they were confident-looking paths that matched no node. And an incremental update disagreed with a rebuild, because an edge's target set depended on which files were in the package at the moment the importing file happened to be parsed. So the parser emits ONE edge naming the directory, tagged extra.import_scope, and the read path expands a directory to its members: importers_of matches edges targeting a file's own package, imports_of reports import_target_kind, the impact traversal follows a package target at every hop without spending one, and bare-call resolution expands the members once per post-processing pass instead of storing them. Ruby's require_all is the same shape with tree scope, because it loads every file below the directory rather than the directory's own. No resolver may name a path the build does not index, so an import into vendor/ keeps its bare module string, which is visibly external. get_impact_radius bounds every list by a fixed ceiling -- the same 100 nodes / 150 edges / 200 files get_review_context applies to the same radius -- and reports what it dropped. edges and changed_nodes had no ceiling and max_results was not exposed on the MCP tool, so one changed file of cli/cli cost 138k tokens and one of kubernetes 189k, against a documented 12k budget. Measured on kubernetes/kubernetes, against the same branch before this change: build 524.6s -> 144.4s (staging: 149.4s), database 6.50GB -> 5.07GB, IMPORTS_FROM 588,972 -> 93,650, import edges naming a path with no node 73,512 -> 5, importers_of for apimachinery/pkg/runtime unchanged at 2,682, get_impact_radius 189,163 -> 23,677 tokens and 6.5s -> 1.5s warm. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The directory branch of one relaxation hop must be an index seek on (target_qualified, kind). Left to itself SQLite drove it from the covering index on kind alone and rescanned every IMPORTS_FROM row once per frontier directory: 18 seconds of a 19-second kubernetes traversal, against 1.5 with CROSS JOIN and INDEXED BY. The same shape of mistake filled _impact_frontier_dirs by scanning every File node in the graph instead of the frontier. Neither is visible in a token budget or on a small fixture, so the SQL moves to module level where a test can EXPLAIN the text the traversal actually runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Correction to the scale table before merge. The build-time row claimed kubernetes went from 149.4s on staging to 129.0s here, i.e. faster than staging. That does not reproduce. Measured isolated and alternating on one idle machine:
So the branch is about 24% slower to build and 21.5% larger than staging, never faster. That is the real cost of resolving imports that previously resolved to nothing, and it is worth paying, but the table said the opposite. What does hold, and is the point of the change: against the pre-fix version of this branch, kubernetes build time goes 524.6s to ~146s, IMPORTS_FROM edges 588,972 to 93,650 (one per import statement), edges naming an unindexed path 73,512 to 5, and get_impact_radius 189,163 tokens to 23,669. The PR body and CHANGELOG entry need this row corrected before merge. |
…this branch An independent re-measurement found the kubernetes build-time row wrong in the direction that favoured this change. It read 149.4s on staging against 129.0s here, which says this branch is faster than staging. Measured isolated and alternating on one idle machine it is not: staging 115.1s and 122.8s against 148.7s and 146.5s here, and 4.41 GB of database against 5.36 GB. The branch is about 24% slower to build and 21.5% larger, never faster. The CHANGELOG now says that plainly, alongside what the cost buys: on cli/cli the CALLS edges bound to an indexed node go from 5503 to 22564 of the same 67503, because a bound import gives the call resolver a file to attribute a cross-file call to. The comparisons that do hold are against the pre-fix revision of this branch, which emitted one edge per member file: 524.6s of build against about 146s, 588,972 IMPORTS_FROM edges against 93,650, 73,512 edges naming an unindexed path against 5, and a 189,163-token get_impact_radius against 23,669. Those are kept and labelled as what they are. Also re-measured and corrected, on clean alternating builds of cli/cli, cobra, gin, jekyll and rubocop against origin/staging: - the grouped-block line defect is 8636 of cli/cli's 8687 import edges, not 8648 of 8690 - jekyll's non-require import edges are 31 of 227 (14%), not 34 (15%) - internal/ is 1317 of the 3495 cli/cli go import edges that bind in-repo, not "3493 of 8687" Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Corrected in 2ccc937, and staging is merged in (5209f57, no conflicts). The build-time row is gone and replaced with your measurement: kubernetes staging 115.1s and 122.8s against 148.7s and 146.5s here, database 4.41 GB against 5.36 GB, about 24% slower and 21.5% larger, never faster. The description says so in its own section and the CHANGELOG carries the same sentence. The four numbers that hold against the pre-fix revision (524.6s to about 146s, 588,972 to 93,650 IMPORTS_FROM edges, 73,512 to 5 unindexed-path edges, 189,163 to 23,669 tokens) are kept in a separate table labelled as a comparison with that revision and not with staging. Everything else in the table was re-measured from clean alternating builds against origin/staging on cli/cli, cobra, gin, jekyll and rubocop, because the per-file fan-out had been replaced by directory-scoped edges after those numbers were taken. What changed:
Reproduced unchanged and left alone: all 15 importers_of counts, the impact-radius counts for cobra, gin, jekyll and rubocop, the 149 standard-library collisions (context 133, embed 11, io 5), jekyll's 49 autoload statements, and the six other-language repositories, which are still byte-identical on both sides including database size. Gates on the merged branch: 4033 passed, 772 skipped, 2 xfailed, 2 xpassed; ruff clean; mypy clean on 75 files; bandit 0 issues. |
graph.py was the only overlap. Staging added a read-transaction helper (contextlib.contextmanager, _active_read_txn, _read_transaction, _get_stats_locked) and migration-safety retries; this branch added the package-scoped import resolver and the pinned impact traversal, which needs collections.abc.Iterable. The import block now carries both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One conflict, in GraphStore.__init__ of code_review_graph/graph.py. Staging wrapped the schema init and migration run in a guard that turns an unreadable or wrongly shaped database into CorruptGraphDatabaseError; this branch added the crg_parent_dir SQLite function that the directory-scoped import traversal joins on. Both are kept: the guard keeps its original scope, and the function is registered after it, because a failure to register a function is not a corrupt file and must not reach the code path that deletes a database. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One conflict, in GraphStore.__init__. Staging (via #1027) registers the SQLite function crg_parent_dir after the open/migrate guard; this branch ends that guard with a described GraphStoreError fallback. Kept both: the fallback stays the last arm of the except, the registration stays after it, so a failure registering the helper is still not mistaken for a corrupt database. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What this fixes
_do_resolve_modulehad nogoand norubyarm. Both languages fellthrough every branch to its bare
return None, so every Go and RubyIMPORTS_FROMedge kept the raw import string as its target and matched nonode.
importers_ofandget_impact_radiustherefore answered a confident0for packages with hundreds of real importers.Verified against the diagnosis before touching anything, on a clean build of
each repository:
importers_of("pkg/iostreams/iostreams.go")returned 0 results for apackage imported 435 times.
require line at all, and all 49
autoloadstatements produced no edge.were stamped with the line of the
import (token.Tests were written first and watched fail for exactly those reasons.
Go
The module path in the nearest
go.modis the anchor; there is no other wayto tell
github.com/cli/cli/v2/pkg/cmdutil(this repository) fromhub.lumenfield.work/spf13/cobra(a download).go.modis read once per file, cachedon path/mtime/size like the existing composer.json reader.
An import equal to that module path, or prefixed by
<module>/, names apackage directory. A Go import names a package, which is a directory of
files, so the edge target is that directory, tagged
extra.import_scope: package, and the read path expands the directory to itsmember files. One import statement produces one edge, not one edge per member
file: fanning out makes the edge count grow with imports times package size,
and makes an incremental update disagree with a rebuild, because the target
set then depends on which files were in the package when the importing file
happened to be parsed.
_test.gofiles are not importable from anotherpackage, so they are never members.
go.modwins over its ancestor, and the ancestors stay in the chain,so a file inside cli/cli's nested module under
.github/codeqlstillresolves imports of the outer module.
replace ... => ./dir(single and block form) is honoured.vendor/<import path>is checked before an import is called external.resolves to nothing. Resolving those by walking up from the caller looking
for a matching directory name binds 149 of cli/cli's 3575 standard
library imports to repository directories (
context133,embed11,io5), which is why the module prefix is the anchor and not a path search.That count is reproduced by script, not asserted.
Extraction parity checked spec by spec: all 8689 of cli/cli's (file, module)
pairs are accounted for after the change, 5194 still bare and 3495 bound,
0 dropped.
Ruby
The extractor now dispatches on the call's method name instead of testing
whether the node text contains the letters "require". That substring test is
what produced targets like
Jekyll is a blog-aware, static site generator in Ruby(from a constant assignment) andlocal-require(from aBenchmark.ipslabel), and it also returned "handled" for those whole call nodes, so every
call inside them was dropped from the call graph as well.
The form now survives to the resolver, because
requireandrequire_relativetake identical strings and resolve them by differentalgorithms:
require_relative "x"-> against the requiring file's own directory.require/load/autoload-> against load roots discovered from therepository: gemspec
require_paths(defaulting tolib),lib,test,spec, the Railsapproots when the tree is Rails-shaped, and anydirectory a
$LOAD_PATH.unshiftin a repository-root script names.autoload :Const, "path"resolves its second argument. This is how jekylldeclares its entire internal module graph and it produced no edge at all.
require_all "dir"names a directory tree, so it emits one edge targetingthat directory, tagged
extra.import_scope: tree, expanded at read time toevery
.rbfile below it. jekyll declares six subsystems this way(commands, converters, converters/markdown, drops, generators, tags).
require File.join(__dir__, "app", "helper"),require path, an interpolated string) emits no edge rather than aguessed one. The old regex took the first quoted string and emitted
app.Rails implicit autoloading (Zeitwerk) is explicitly out of scope, and the
code says so where a reader will look. There is no import statement to
resolve: the dependency is a bare constant reference (
Jekyll::Drops::Drop)that a runtime resolver maps to a file by naming convention. Supporting it
needs a constant-reference resolver plus autoload-root discovery from
config/application.rb, which is a different feature from module-to-fileimport resolution.
Reused machinery, not a third resolver
_python_file_exists/_python_dir_exists/_python_dir_entries/_python_repo_boundaryto neutral names. Pure rename, no behaviour change,no call site outside the parser.
_path_is_withinis the repository clamp, applied via one shared_parts_withinhelper: ago.modreplace => ../outsideand arequire_relative "../../../elsewhere"both name real files and neitherbecomes an edge target.
IMPORTS_FROM, source file -> target, one rowper import statement. Only the target spelling gained a directory form, and
only behind an explicit
extra.import_scopetag.Measured, before -> after
Every number below was re-measured for this description. Baseline is
origin/stagingatf0e4eb7; "after" is this branch with staging merged.The two checkouts were built alternately from clean, on the same machine,
against the same clone of each repository: cli/cli
0cf1092(943.go),spf13/cobra
adbc881(36), gin-gonic/gin5c6a15f(99), jekyll541d8b2(162
.rb), rubocop5bb0567(1780).Import edges that bind into the repository
The edge count does not grow with resolution, because one import statement
still produces one edge. The unbound remainder is, top-first,
fmt,net/http,testing,strings,github.com/stretchr/testify/assertfor Goand
pathname,fileutils,yaml,rainbowfor Ruby: standard library andgems, which is the correct answer. cobra and gin look low in absolute terms
because they barely import themselves; cobra has 12 in-repo import statements
and all 12 now bind. rubocop's ruby edge count falls because the false
positives are gone.
importers_of, hand-verified against the sourcepkg/iostreams/iostreams.gogo.modunder.github/codeqland correctly bind to that module's ownpkg/iostreamsinternal/ghrepo/repo.gopkg/cmdutil/factory.gocommand.goargs.godoc/md_docs.gointernal/bytesconv/bytesconv.gobinding/binding.gorender/render.gotest/helper.rbrequire "helper"lineslib/jekyll.rbrequire "jekyll"/require_relative "../lib/jekyll"lines, including extension-less scripts (exe/jekyll,script/console,benchmark/*)lib/jekyll/filters.rblib/jekyll.rblib/rubocop.rblib/rubocop/version.rblib/rubocop/cop/base.rblib/rubocop.rbEvery edge behind these counts sits on a real
require/require_relative/autoload/load line, against 31 of 227 (14%) that did
not before.
get_impact_radiuson one central file per repopkg/iostreams/iostreams.gocommand.gointernal/bytesconv/bytesconv.golib/jekyll.rblib/rubocop.rbCalls that now bind to an indexed node
This is the part worth having. Resolving an import gives the call resolver a
file to attribute a cross-file call to, so the gain is not confined to
IMPORTS_FROM.Share of all edges naming nothing the build indexed
A directory target counts as named when the directory holds indexed files.
rubocop is the one row that gets worse, and it is worth reading rather than
hiding. Its ruby import edges went from 0 bound to 277 of 364; the overall
share rises because CALLS edges went 95886 -> 121103. Those 25217 calls
are the ones the old substring test was swallowing: rubocop's specs are full
of
it 'registers an offense for EnforcedStyle: require_parentheses' do ... end, whose node text contains "require", so_extract_importsclaimed thewhole block and the dispatcher never reached call extraction. More real edges,
a larger unresolved call target population, and a separate problem from the
one this PR fixes.
What it costs
Resolving imports that previously resolved to nothing costs build time and
disk. This branch is slower to build and produces a larger database than
staging on every repository measured. It is never faster.
Build time
Median of 5 alternating clean builds per side once the machine had settled,
with the observed range.
The three sub-2s repositories are noise-dominated and no difference should be
read into them. rubocop's two ranges overlap, so that row does not establish a
direction either. cli/cli is the smallest repository here where the two sets
of samples do not overlap at all.
Database
Identical on every repeat, so these are exact rather than median.
At scale: kubernetes/kubernetes
Measured isolated and alternating on one idle machine, two runs per side.
An earlier revision of this description carried a kubernetes build row of
149.4s on staging against 129.0s here, which read as this branch being faster
than staging. That does not reproduce and it was wrong in the direction that
flattered the change. The table above replaces it.
Where the cost goes
Not the resolver. On cli/cli the extra work is downstream of parsing: the
go.modreads and directory listings are cached, the edge count is unchangedat 121398 against 121396, and 8689 import edges are still 8689. What grows is
what a bound import lets the rest of the pipeline do. 17061 more CALLS edges
now bind to an indexed node, and each resolution attempt that lands writes
target and candidate metadata that an unresolved edge never had:
edges.extragoes from 2.4 MB to 11.1 MB, and a bound target is an absolute file path
rather than a short module string, which grows the row and six indexes on it.
Going from 0 to 3495 bound imports and from 5503 to 22564 bound calls on a
943-file repository costs about 1.7s of build and 45 MB of database. That is
the trade being made, and it is visible here rather than buried.
The comparison that does hold against a pre-fix revision
The first revision of this branch emitted one edge per member file. Against
that revision, on kubernetes, the directory-scoped edge is a straight win:
IMPORTS_FROMedgesget_impact_radiuson one fileThose four are the numbers this design change earned. They are not a
comparison with staging and are not offered as one.
Not regressing the other languages
Shared code was touched (the probe renames,
_extract_imports,_resolve_module_to_file), so the six other-language repositories from thediagnosis were rebuilt on both sides and compared. Every graph is
identical: edge totals, per-kind distribution, per-language import
resolution and database size all match exactly.
One Python behaviour was nearly broken and caught by the existing guard: an
early self-edge filter dropped
from . import runnerinsidecode_review_graph/eval/__init__.py, which legitimately resolves to thepackage's own
__init__.py.test_import_target_integrity.pyfailed, thefilter was removed, and the Python audit is back to 100%.
The same gap in other languages
C# and Swift are the other two zeros the diagnosis found, and both have the
same symptom (no branch in
_do_resolve_module) but not the same shape,so neither is fixed here rather than half-fixed:
using X.Y;names anamespace, not a file, and C# imposes no filename convention. Resolving it
needs a repository-wide namespace -> files index built before edges are
written, which is a cross-file pre-pass, not a manifest read like
go.modor a gemspec. There is already a query-time compensation in
tools/query.pythat matches a file's declared
csharp_namespacesagainst raw namespaceedge targets, which is why
importers_ofon a C# file returns results eventhough no
IMPORTS_FROMedge does;get_impact_radius, which reads theedge table, gets nothing.
there is no
_extract_importarm either, so the stored targets are rawstatement text. Extracting the module name is cheap; resolving it is not,
because a Swift import names a module whose files are listed by a
Package.swifttarget declaration, a Swift DSL to parse rather than a dataformat to read. Extracting the name without resolving it would only make an
unresolvable target look resolvable.
Two further gaps the diagnosis measured are real but are a different defect
(a branch that exists and is too narrow, not a missing branch), so they are
also left alone here:
directory only, so
gson/src/test/java/...never reachesgson/src/main/java/com/google/gson/Gson.java. The fix is siblingsource-root discovery, which is cheap, but it is a separate change with its
own before/after numbers.
Gates
Run on this branch with staging merged.
uv run --python 3.13 python -m pytest tests/ -q-> 4033 passed,772 skipped, 2 xfailed, 2 xpassed
uv run ruff check code_review_graph/-> all checks passeduv run --with mypy mypy code_review_graph/ --ignore-missing-imports --no-strict-optional-> no issues in 75 source filesbandit -r code_review_graph/ -c pyproject.toml-> 0 issues at everyseverity
Known limits
require File.expand_path("../lib/jekyll", __dir__)emits no edge.__dir__is statically known, so this is resolvable in principle; theconservative "no guessed path" rule is applied instead. jekyll has 4 such
lines (
test/helper.rb,benchmark/schwartzian_transform.rb,script/stackprof,script/rubyprof). Nothing is lost against the oldbehaviour, which emitted an unresolvable bare string.
$LOAD_PATH.unshiftis read from repository-root scripts, the gemspecs, theRakefile, the Gemfile,
spec/spec_helper.rbandtest/test_helper.rbonly.A push written elsewhere is not visible. The alternative is reading every
file in the repository before resolving the first import.
require(
Jekyll::External.require_with_graceful_fail("kramdown"), 11 sites injekyll) no longer emit an edge. Every one of them names a gem, so the edge
they used to emit was an unresolvable bare string.
includes files excluded by
//go:buildon the current platform(
pkg/iostreams/console_windows.gois a member ofpkg/iostreams).repository measured. See "What it costs".
🤖 Generated with Claude Code