Skip to content

Fix partial hash for 64–128 KiB files; nested roots, scan throughput, index self-hashing - #1

Merged
cdmaestas merged 2 commits into
mainfrom
fix/partial-hash-small-files
Sep 26, 2026
Merged

cdmaestas merged 2 commits into
mainfrom
fix/partial-hash-small-files

Conversation

@cdmaestas

Copy link
Copy Markdown
Owner

Summary

Two commits:

Hash files up to two windows whole in partial_hash

  • For 64 KiB < size ≤ 128 KiB the partial hash read only the first 64 KiB, so same-sized files differing past that collided and cdm dupes (without --verify) reported false duplicates. Such files are now hashed whole; bytes_read() matches.
  • Domain tag kept at cdm-partial-v1: every other size hashes identically. Bumping it would have left reused v1 rows (scan reuses on unchanged size/mtime) incomparable with new ones, silently hiding duplicates.
  • Schema 3 migration clears only the partial hashes in the affected size range, so the next --checksum scan recomputes them. Read-only connections (MCP) will ask for cdm doctor once to upgrade.

Support nested roots, report scan throughput, never hash the index itself

  • Overlapping roots (~ and ~/src): one row per file, owned by the most specific root; hashes shared; forget hands rows to the enclosing root.
  • Scan summary and progress report entries/s and hash throughput; --progress [SECS] for non-tty logs.
  • Index, -wal and -shm are recorded but never opened for hashing (closing a descriptor dropped SQLite's POSIX locks → SIGBUS in walFindFrame).

Test plan

  • pytest -q --cov=cdm --cov-fail-under=80: 191 passed, 1 skipped (mcp extra not installed)
  • ruff check . clean
  • First commit alone: 175 passed
  • New tests/test_hashing.py: 100 KiB files differing at byte 80,000 get different partial hashes (collided before the fix); bytes_read matches actual reads; migration clears only affected rows

🤖 Generated with Claude Code

cdmaestas and others added 2 commits September 26, 2026 14:02
…self

Nested roots: `~` and `~/src` may both be registered. Each file has one row,
owned by the most specific root containing it regardless of which scan wrote
it last, so scans reuse each other's hashes instead of stealing rows and
rehashing everything. roots/find/summary count each entry once; `forget` of a
nested root hands its rows to the enclosing root, and `forget` of an outer
root keeps the nested root's rows. `scan` notes when a new root overlaps.

Throughput: the scan summary reports entries/s and hashed bytes and rate.
Progress also fires on a timer, so slow filesystems still report, and
`--progress [SECS]` logs timestamped rate lines when stderr is not a tty.

Index files: the index, -wal and -shm are recorded but never opened for
hashing. Closing any descriptor to a file drops every POSIX lock the process
holds on it, which let a concurrent reader delete the WAL mid-scan (SIGBUS in
walFindFrame).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For WINDOW < size <= 2*WINDOW the partial hash read only the first 64 KiB,
so same-sized files differing after that collided and `cdm dupes` reported
them as duplicates without --verify. Such files are now hashed whole;
bytes_read() matches.

The domain tag stays cdm-partial-v1: every other size hashes exactly as
before. Bumping it would leave reused v1 rows (scan reuses on unchanged
size/mtime) incomparable with new ones and silently hide duplicates.
Instead, schema 3 clears only the partial hashes in the affected size range
so the next --checksum scan recomputes them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cdmaestas
cdmaestas merged commit 0e2e962 into main Sep 26, 2026
@cdmaestas
cdmaestas deleted the fix/partial-hash-small-files branch September 26, 2026 20:06
cdmaestas added a commit that referenced this pull request Oct 1, 2026
Imports have recorded each file's fileset and storage pool since #9; this
puts them to use. `cdm storage` totals files and bytes by pool and by
fileset (with names: the person at the terminal owns the index);
`find --pool` / `--fileset` filter on them, as do the same parameters on
MCP `find`; `stat` shows both. Scanned files have neither and are counted
apart.

The MCP `storage` tool is a shape tool, so its names follow ADR 0002's
rule. The plan assumed pool and fileset names were both shape; on a real
cluster the fileset in use was named after its user. Pool names name
storage tiers and are returned by default; fileset names are often a
person's, project's or customer's, so without --expose-names filesets are
reported by rank of size ("fileset #1" is the largest) and never named.
Recorded as ADR 0007.

Closes #11.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant