feat(extract): add dbt model .sql extractor for ref()/source() lineage - #2409
feat(extract): add dbt model .sql extractor for ref()/source() lineage#2409amanhooda98 wants to merge 2 commits into
Conversation
dbt models wrap SQL in Jinja, so the tree-sitter-sql extractor can't see them and ref()/source() dependencies stay invisible in the graph. Parses the Jinja AST (no rendering) to resolve ref() as depends_on edges and source() as reads_from edges, keyed on bare model/table names to match dbt's global-uniqueness guarantee.
There was a problem hiding this comment.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify review — findings
This PR expands the project's CI/CD and repository tooling infrastructure. It adds new GitHub Actions workflows for publishing to PyPI via trusted publishing and for building/releasing a self-graph asset, migrates the existing CI to use uv with frozen lockfiles, and introduces a skillgen-check job plus a pre-commit hook that guard against drift between generated skill files and their source fragments in tools/skillgen/. It also adds various repo config files (.dockerignore, .gitattributes, .pre-commit-config.yaml, FUNDING.yml) and updates .gitignore rules, notably to keep generated skill bundles tracked while ignoring other artifacts. The large set of changed symbols (tests, skillgen expected fixtures, worked examples, docs, changelog) suggests accompanying regeneration of blessed skill artifacts and related test/snapshot updates across the surface area.
No blocking issues surfaced.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 12374 functions depend on the 12355 functions this change touches.
Health — this change adds coupling hotspots:
- worse:
extract()— 361 callers, 29 callees - worse:
_rebuild_code()— 68 callers, 49 callees - worse:
build_from_json()— 142 callers, 14 callees - new:
deduplicate_entities()— 49 callers, 21 callees - worse:
detect()— 82 callers, 11 callees - new:
ingest_scip_json()— 76 callers, 4 callees - new:
extract_files_direct()— 15 callers, 19 callees - new:
_extract_generic()— 18 callers, 15 callees - …and 167 more
Verification — 12374 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 12374 function(s) in the blast radius were not formally verified this run
Summary
dbt models wrap SQL in Jinja (
{{ config(...) }},{{ ref('model') }},{{ source('schema','table') }}), so the existingtree-sitter-sqlextractor can't parse them —
ref()/source()dependencies stayinvisible in the graph. This adds a dedicated extractor that parses
the Jinja AST directly (no rendering, no dbt project context needed)
and resolves
ref()asdepends_onedges andsource()asreads_fromedges.Design
_get_extractor()already content-sniffs ambiguous suffixes (e.g.
.hfor ObjC vsC++). Same precedent here: a
.sqlfile routes toextract_dbt_sqlif it contains any
{{/{%in its first 4KB, otherwise fallsthrough to the generic SQL extractor.
manifest.jsonwith a fully resolved DAG, but consuming it directlywould mean tracking a schema that changes across dbt-core versions
(the
manifest_schema_versionhas bumped repeatedly, and fields likedepends_on/sourcesaren't guaranteed stable), and dbt Fusion hasmoved to a Parquet manifest instead of JSON — a second format to
support. It's also only produced by actually running
dbt parse(orcompile/run), which requires a working dbt environment: aprofiles.yml, valid warehouse credentials, installed packages, anda successful connection. Parsing the
.sqlsource directly needsnone of that — just the file on disk.
path.stem, notpath.name, and stubtarget nodes carry no
origin_file— both are required for a modelreferenced by multiple callers, or one nested in a subdirectory, to
resolve onto a single node instead of a dangling duplicate.
extract.py changes
Two small additions to
_get_extractor()/extract(), both required:.sqlfile toextract_dbt_sql; without it the new extractor is never reachedthrough the normal
extract()path..sqlcan now hard-fail on two different missing deps(
tree-sitter-sqlorjinja2) depending on content._EXTRA_FOR_EXTENSION's existing flat per-extension lookup onlyknows one extra per extension, so without a separate
_EXTRA_FOR_MISSING_PACKAGEcheck, a dbt model missingjinja2gets told to
pip install "graphifyy[sql]"— the wrong extra,verified by reverting the check locally and reproducing the
incorrect hint.
Optional dependency
pip install jinja2(orpip install 'graphifyy[dbt]'). If notinstalled,
extract_dbt_sqlreturns{"nodes": [], "edges": [], "error": ...}and the rest of the pipeline is unaffected.