Skip to content

feat(padron): I add an optional SQLite lookup index - #124

Open
manduinca wants to merge 1 commit into
crafter-research:mainfrom
manduinca:feat/padron-sqlite-index
Open

manduinca wants to merge 1 commit into
crafter-research:mainfrom
manduinca:feat/padron-sqlite-index

Conversation

@manduinca

Copy link
Copy Markdown

Closes #15

I added padron index to build an optional SQLite index after padron sync. Single and batch lookups use a compatible index automatically, fall back to the TXT scan when it is missing, stale, or corrupt, and retain the existing result shape. padron status now reports index presence and size.

I use the existing row parser, a TEXT PRIMARY KEY schema, a transaction, unique staging files, and an atomic rename for publication. The index records the schema version and TXT identity; sync invalidates the index before extraction. I also made both streaming paths skip malformed candidates so they agree with the index on the first valid duplicate RUC. I documented the option and its limits in LIMITATIONS.md and added a reproducible synthetic benchmark.

Validation

  • bun test packages/cli/tests/unit: 454 passed, 1 skipped.
  • bun test packages/cli/tests/e2e: 88 passed, 2 skipped. I used temporary configuration with inherited SUNAT/CPE variables removed and the mock driver; the opt-in SUNAT beta tests stayed disabled.
  • SQLite fixtures passed under Bun 1.4.3 and Node 24.22.1. I built the published Node bundle and checked its index, status, ruc, and batch commands with a synthetic local fixture.
  • Focused tests cover parity, malformed/duplicate rows, stale/corrupt indexes, sync success/failure, source changes during opening/building, concurrent builds, and temporary-file cleanup.
  • Biome checks for the four changed TypeScript files and git diff --check: passed.

Performance and limits

I measured a synthetic 998,888,890-byte TXT containing 1,000,000 rows with 920 padding characters per name and a warm OS cache. For batches of 1,000 RUCs, five indexed runs took 11.38–16.43 ms on Bun and 11.42–13.47 ms on Node 24; streaming took about 880/1,015 ms and indexing about 6.46/6.03 s. The resulting index occupied 1,024,036,864 bytes.

This meets the issue's batch target on that fixture; it does not validate a real padrón's cardinality/distribution, cold-cache performance, or other machines. I used no credentials, real taxpayer fixtures, or live fiscal operations. A forced process kill can leave an unpublished staging file; TXT replacement during sync keeps the existing streaming limitations described in LIMITATIONS.md.

@vercel

vercel Bot commented Oct 22, 2026

Copy link
Copy Markdown

@manduinca is attempting to deploy a commit to the Crafter Station Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P2] sqlite index for padrón (sub-ms RUC lookups)

1 participant