Skip to content

ingest_repo.py silently ingests **zero** documents from a git-ignored directory #1

Closed

Description

@boris-virto

ingest_repo.py silently ingests zero documents from a git-ignored directory

Summary

list_md() falls back to os.walk only when the git calls raise. For a directory that
sits inside a git repository but is itself fully .gitignored, both git calls succeed and
git ls-files '*.md' returns an empty list — so the fallback never runs, the walk never
happens, and the ingester writes a valid .sql containing no documents and exits 0.

Nothing in the output says the corpus is empty. psql -f out.sql succeeds. search_docs
then answers every question with "no results", which is indistinguishable from "the corpus
does not cover that" — the failure only surfaces later, as bad answers.

This is the same failure class skills/onboard-project/SKILL.md already warns about for
Tier 1:

Empty output means the path matched no component — fix the globs now, because a silent
Tier 1 is worse than none: it looks like "no rules apply".

Reproduction

cd /tmp && rm -rf demo && mkdir -p demo/notes && cd demo
git init -q . && printf 'notes/\n' > .gitignore
printf '# Runbook\n\nHow the thing works.\n' > notes/runbook.md
git add .gitignore && git -c user.email=a@b -c user.name=a commit -qm init

python3 /path/to/hypermnesia/ingest/ingest_repo.py notes demo-notes out.sql
# stderr: demo-notes: 0 docs, 0 chunks -> out.sql
# exit:   0
grep -c INSERT out.sql   # 0

notes/runbook.md exists and is readable. Point the ingester at a directory that is not
inside a git repo and the same file is picked up, because then git rev-parse fails and the
os.walk branch runs.

Root cause

ingest/ingest_repo.py, list_md() (lines 47-61 at f398573):

try:
    commit = subprocess.check_output(["git", "-C", repo_dir, "rev-parse", "HEAD"], ...)
    files  = subprocess.check_output(["git", "-C", repo_dir, "ls-files", "*.md"], ...)
except Exception:
    commit, files = "nogit", []
    for root, dirs, fs in os.walk(repo_dir):
        ...

The fallback is keyed on "is this a git repo", but what the caller actually cares about is
"did we find any files". A git-ignored subdirectory satisfies the first and fails the
second. git -C <dir> also walks up to the enclosing repository, so passing a
subdirectory is enough to take the git branch.

Why it bites in practice

Working notes, research docs and scratch documentation are exactly the material worth putting
in a doc corpus, and exactly the material people keep out of git. In our case the whole
intended corpus (a temp_docs/ directory, git-ignored by design) ingested as zero documents
on the first run, with no error. We only caught it by reading list_md before trusting the
result; the obvious symptom — "search finds nothing" — would have been blamed on the
embedder, the query, or the reranker long before the ingester.

Suggested fixes

Either of these would have made it loud; the first is one line.

1. Refuse to write an empty corpus. A zero-document ingest is nearly always a mistake:

if not files:
    sys.exit(f"{repo}: no *.md found under {repo_dir} — "
             f"if the directory is git-ignored, pass --walk to bypass `git ls-files`")

2. Make the enumeration mode explicit rather than inferred, e.g. --walk /
--git (default: today's auto-detect). The auto-detect is a good default, but it needs an
override, because "this directory is deliberately not tracked" is a legitimate, common
configuration that the heuristic reads as "this directory is empty".

A third, more surgical option: treat an empty git ls-files result as "no answer" and fall
through to the walk. That fixes this case without a new flag, but it also changes behaviour
for a genuinely empty tracked directory, so the explicit flag is probably cleaner.

Environment

  • HyperMnesia at f398573
  • Python 3.13.5, git 2.x, macOS 15

Happy to send a PR for whichever shape you prefer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions