Repository navigation
ingest_repo.py silently ingests **zero** documents from a git-ignored directory #1
Description
ingest_repo.py silently ingests zero documents from a git-ignored directory
Summary
list_md() falls back to os.walk only when the git calls raise. For a directory that
sits inside a git repository but is itself fully .gitignored, both git calls succeed and
git ls-files '*.md' returns an empty list — so the fallback never runs, the walk never
happens, and the ingester writes a valid .sql containing no documents and exits 0.
Nothing in the output says the corpus is empty. psql -f out.sql succeeds. search_docs
then answers every question with "no results", which is indistinguishable from "the corpus
does not cover that" — the failure only surfaces later, as bad answers.
This is the same failure class skills/onboard-project/SKILL.md already warns about for
Tier 1:
Empty output means the path matched no component — fix the globs now, because a silent
Tier 1 is worse than none: it looks like "no rules apply".
Reproduction
cd /tmp && rm -rf demo && mkdir -p demo/notes && cd demo
git init -q . && printf 'notes/\n' > .gitignore
printf '# Runbook\n\nHow the thing works.\n' > notes/runbook.md
git add .gitignore && git -c user.email=a@b -c user.name=a commit -qm init
python3 /path/to/hypermnesia/ingest/ingest_repo.py notes demo-notes out.sql
# stderr: demo-notes: 0 docs, 0 chunks -> out.sql
# exit: 0
grep -c INSERT out.sql # 0notes/runbook.md exists and is readable. Point the ingester at a directory that is not
inside a git repo and the same file is picked up, because then git rev-parse fails and the
os.walk branch runs.
Root cause
ingest/ingest_repo.py, list_md() (lines 47-61 at f398573):
try:
commit = subprocess.check_output(["git", "-C", repo_dir, "rev-parse", "HEAD"], ...)
files = subprocess.check_output(["git", "-C", repo_dir, "ls-files", "*.md"], ...)
except Exception:
commit, files = "nogit", []
for root, dirs, fs in os.walk(repo_dir):
...The fallback is keyed on "is this a git repo", but what the caller actually cares about is
"did we find any files". A git-ignored subdirectory satisfies the first and fails the
second. git -C <dir> also walks up to the enclosing repository, so passing a
subdirectory is enough to take the git branch.
Why it bites in practice
Working notes, research docs and scratch documentation are exactly the material worth putting
in a doc corpus, and exactly the material people keep out of git. In our case the whole
intended corpus (a temp_docs/ directory, git-ignored by design) ingested as zero documents
on the first run, with no error. We only caught it by reading list_md before trusting the
result; the obvious symptom — "search finds nothing" — would have been blamed on the
embedder, the query, or the reranker long before the ingester.
Suggested fixes
Either of these would have made it loud; the first is one line.
1. Refuse to write an empty corpus. A zero-document ingest is nearly always a mistake:
if not files:
sys.exit(f"{repo}: no *.md found under {repo_dir} — "
f"if the directory is git-ignored, pass --walk to bypass `git ls-files`")2. Make the enumeration mode explicit rather than inferred, e.g. --walk /
--git (default: today's auto-detect). The auto-detect is a good default, but it needs an
override, because "this directory is deliberately not tracked" is a legitimate, common
configuration that the heuristic reads as "this directory is empty".
A third, more surgical option: treat an empty git ls-files result as "no answer" and fall
through to the walk. That fixes this case without a new flag, but it also changes behaviour
for a genuinely empty tracked directory, so the explicit flag is probably cleaner.
Environment
- HyperMnesia at
f398573 - Python 3.13.5, git 2.x, macOS 15
Happy to send a PR for whichever shape you prefer.
Activity
fix(ingest): abort an incremental ingest whose snapshot doesn't match…
fix(ingest): abort an incremental ingest whose snapshot doesn't match…
fix(ingest): a git-ignored directory ingested as zero documents, sile…