Skip to content

feat(extract): add dbt model .sql extractor for ref()/source() lineage - #2409

Open
amanhooda98 wants to merge 2 commits into
Graphify-Labs:v8from
amanhooda98:feat/dbt-jinja-lineage-extractor
Open

feat(extract): add dbt model .sql extractor for ref()/source() lineage#2409
amanhooda98 wants to merge 2 commits into
Graphify-Labs:v8from
amanhooda98:feat/dbt-jinja-lineage-extractor

Conversation

@amanhooda98

Copy link
Copy Markdown

Summary

dbt models wrap SQL in Jinja ({{ config(...) }}, {{ ref('model') }},
{{ source('schema','table') }}), so the existing tree-sitter-sql
extractor can't parse them — ref()/source() dependencies stay
invisible in the graph. This adds a dedicated extractor that parses
the Jinja AST directly (no rendering, no dbt project context needed)
and resolves ref() as depends_on edges and source() as
reads_from edges.

Design

  • Detection is content-sniffed, not path-based. _get_extractor()
    already content-sniffs ambiguous suffixes (e.g. .h for ObjC vs
    C++). Same precedent here: a .sql file routes to extract_dbt_sql
    if it contains any {{/{% in its first 4KB, otherwise falls
    through to the generic SQL extractor.
  • No dependency on dbt's manifest.json. dbt already produces a
    manifest.json with a fully resolved DAG, but consuming it directly
    would mean tracking a schema that changes across dbt-core versions
    (the manifest_schema_version has bumped repeatedly, and fields like
    depends_on/sources aren't guaranteed stable), and dbt Fusion has
    moved to a Parquet manifest instead of JSON — a second format to
    support. It's also only produced by actually running dbt parse (or
    compile/run), which requires a working dbt environment: a
    profiles.yml, valid warehouse credentials, installed packages, and
    a successful connection. Parsing the .sql source directly needs
    none of that — just the file on disk.
  • File node label uses path.stem, not path.name, and stub
    target nodes carry no origin_file — both are required for a model
    referenced by multiple callers, or one nested in a subdirectory, to
    resolve onto a single node instead of a dangling duplicate.

extract.py changes

Two small additions to _get_extractor()/extract(), both required:

  • A dispatch check routes a Jinja-templated .sql file to
    extract_dbt_sql; without it the new extractor is never reached
    through the normal extract() path.
  • .sql can now hard-fail on two different missing deps
    (tree-sitter-sql or jinja2) depending on content.
    _EXTRA_FOR_EXTENSION's existing flat per-extension lookup only
    knows one extra per extension, so without a separate
    _EXTRA_FOR_MISSING_PACKAGE check, a dbt model missing jinja2
    gets told to pip install "graphifyy[sql]" — the wrong extra,
    verified by reverting the check locally and reproducing the
    incorrect hint.

Optional dependency

pip install jinja2 (or pip install 'graphifyy[dbt]'). If not
installed, extract_dbt_sql returns {"nodes": [], "edges": [], "error": ...} and the rest of the pipeline is unaffected.

dbt models wrap SQL in Jinja, so the tree-sitter-sql extractor can't
see them and ref()/source() dependencies stay invisible in the graph.
Parses the Jinja AST (no rendering) to resolve ref() as depends_on
edges and source() as reads_from edges, keyed on bare model/table
names to match dbt's global-uniqueness guarantee.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

This PR expands the project's CI/CD and repository tooling infrastructure. It adds new GitHub Actions workflows for publishing to PyPI via trusted publishing and for building/releasing a self-graph asset, migrates the existing CI to use uv with frozen lockfiles, and introduces a skillgen-check job plus a pre-commit hook that guard against drift between generated skill files and their source fragments in tools/skillgen/. It also adds various repo config files (.dockerignore, .gitattributes, .pre-commit-config.yaml, FUNDING.yml) and updates .gitignore rules, notably to keep generated skill bundles tracked while ignoring other artifacts. The large set of changed symbols (tests, skillgen expected fixtures, worked examples, docs, changelog) suggests accompanying regeneration of blessed skill artifacts and related test/snapshot updates across the surface area.

No blocking issues surfaced.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 12374 functions depend on the 12355 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: extract() — 361 callers, 29 callees
  • worse: _rebuild_code() — 68 callers, 49 callees
  • worse: build_from_json() — 142 callers, 14 callees
  • new: deduplicate_entities() — 49 callers, 21 callees
  • worse: detect() — 82 callers, 11 callees
  • new: ingest_scip_json() — 76 callers, 4 callees
  • new: extract_files_direct() — 15 callers, 19 callees
  • new: _extract_generic() — 18 callers, 15 callees
  • …and 167 more

Verification — 12374 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 12374 function(s) in the blast radius were not formally verified this run

@amanhooda98
amanhooda98 changed the base branch from main to v8 August 3, 2026 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant