sql: rewrite scid string literals to use scid() for index support - #8984
sql: rewrite scid string literals to use scid() for index support#8984vincenzopalazzo wants to merge 2 commits into
Conversation
06ba1c8 to
922b1e1
Compare
922b1e1 to
8627c69
Compare
Short channel IDs were stored as TEXT strings (e.g., "735095x480x1") in
SQLite, which prevented efficient use of indexes on SCID columns. Change
storage to INTEGER (the u64 encoding), using a custom "SCID" column type
so the result-reading code can detect these columns and format them back
as "NNNxNNNxNNN" strings for backward-compatible JSON output.
Add two new SQL functions:
- scid('NNNxNNNxNNN') -> integer: for efficient WHERE clause filtering
- fmt_scid(integer) -> 'NNNxNNNxNNN': for formatting in SQL expressions
Fixes #8941
When users query with WHERE in_channel='735095x480x1', the string
literal is compared directly against the integer SCID column, forcing
SQLite to perform a full table scan even when an index exists.
Automatically rewrite scid string literals (matching NNNxNNNxNNN format)
to use the scid() function before passing the query to SQLite, so
'735095x480x1' becomes scid('735095x480x1'). This allows SQLite to
use indexes on SCID columns transparently.
Queries already using scid() explicitly are detected and left unchanged.
Changelog-Fixed: sql plugin now automatically translates short_channel_id string literals to integers for efficient index usage.
Fixes: #8941
8627c69 to
84562df
Compare
| /* Look for single-quoted string literals */ | ||
| if (*p == '\'') { | ||
| const char *start = p + 1; | ||
| const char *end = strchr(start, '\''); |
There was a problem hiding this comment.
This func probably doesnt understand SQLs - '' escape for a literal quote inside a string ('O''Brien' haha). Any query containing such a string - anywhere, even unrelated to scids -will be mis-split, so the scanner will think the string ends at the first ' of the escape, then treat the rest of the text as a new token. This can corrupt a perfectly valid, unrelated query, not just ones touching scid columns. Maybe when we scan for the closing quote we should treat '' as an escaped quote (skip past it and keep scanning) the way SQLite itself does?
|
|
||
| /* Rewrite scid string literals to use scid() function so | ||
| * SQLite can use indexes on integer SCID columns. */ | ||
| query = rewrite_scid_literals(tmpctx, query); |
There was a problem hiding this comment.
Isnt the rewrite is applied to every single-quoted literal in the query, regardless of what its being compared against? Any literal that happens to match digits-x-digits-x-digits gets silently wrapped in scid(...), even if its compared against an unrelated TEXT column (a node alias, a label, a bolt11)
3 participants
Summary
WHERE in_channel='735095x480x1', the string literal was compared directly against the integer SCID column, forcing SQLite into a full table scan even with an index presentjson_sql()passes user queries directly tosqlite3_prepare_v2()without translating scid string literals to integersNNNxNNNxNNNformat to use thescid()function before query execution, so'735095x480x1'becomesscid('735095x480x1')transparentlyFixes #8941
Changelog-Fixed: sql plugin now automatically translates short_channel_id string literals to integers for efficient index usage.
Test plan
SELECT * FROM forwards WHERE in_channel='735095x480x1'now use indexesscid('...')explicitly are left unchanged