Repository navigation
Fix partial hash for 64–128 KiB files; nested roots, scan throughput, index self-hashing - #1
Merged
Merged
Conversation
…self Nested roots: `~` and `~/src` may both be registered. Each file has one row, owned by the most specific root containing it regardless of which scan wrote it last, so scans reuse each other's hashes instead of stealing rows and rehashing everything. roots/find/summary count each entry once; `forget` of a nested root hands its rows to the enclosing root, and `forget` of an outer root keeps the nested root's rows. `scan` notes when a new root overlaps. Throughput: the scan summary reports entries/s and hashed bytes and rate. Progress also fires on a timer, so slow filesystems still report, and `--progress [SECS]` logs timestamped rate lines when stderr is not a tty. Index files: the index, -wal and -shm are recorded but never opened for hashing. Closing any descriptor to a file drops every POSIX lock the process holds on it, which let a concurrent reader delete the WAL mid-scan (SIGBUS in walFindFrame). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For WINDOW < size <= 2*WINDOW the partial hash read only the first 64 KiB, so same-sized files differing after that collided and `cdm dupes` reported them as duplicates without --verify. Such files are now hashed whole; bytes_read() matches. The domain tag stays cdm-partial-v1: every other size hashes exactly as before. Bumping it would leave reused v1 rows (scan reuses on unchanged size/mtime) incomparable with new ones and silently hide duplicates. Instead, schema 3 clears only the partial hashes in the affected size range so the next --checksum scan recomputes them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cdmaestas
added a commit
that referenced
this pull request
Oct 1, 2026
Imports have recorded each file's fileset and storage pool since #9; this puts them to use. `cdm storage` totals files and bytes by pool and by fileset (with names: the person at the terminal owns the index); `find --pool` / `--fileset` filter on them, as do the same parameters on MCP `find`; `stat` shows both. Scanned files have neither and are counted apart. The MCP `storage` tool is a shape tool, so its names follow ADR 0002's rule. The plan assumed pool and fileset names were both shape; on a real cluster the fileset in use was named after its user. Pool names name storage tiers and are returned by default; fileset names are often a person's, project's or customer's, so without --expose-names filesets are reported by rank of size ("fileset #1" is the largest) and never named. Recorded as ADR 0007. Closes #11. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two commits:
Hash files up to two windows whole in
partial_hashcdm dupes(without--verify) reported false duplicates. Such files are now hashed whole;bytes_read()matches.cdm-partial-v1: every other size hashes identically. Bumping it would have left reused v1 rows (scan reuses on unchanged size/mtime) incomparable with new ones, silently hiding duplicates.--checksumscan recomputes them. Read-only connections (MCP) will ask forcdm doctoronce to upgrade.Support nested roots, report scan throughput, never hash the index itself
~and~/src): one row per file, owned by the most specific root; hashes shared;forgethands rows to the enclosing root.--progress [SECS]for non-tty logs.-waland-shmare recorded but never opened for hashing (closing a descriptor dropped SQLite's POSIX locks → SIGBUS inwalFindFrame).Test plan
pytest -q --cov=cdm --cov-fail-under=80: 191 passed, 1 skipped (mcp extra not installed)ruff check .cleantests/test_hashing.py: 100 KiB files differing at byte 80,000 get different partial hashes (collided before the fix);bytes_readmatches actual reads; migration clears only affected rows🤖 Generated with Claude Code