[feature-wip](inverted index) Introduce SPIMI V4 inverted index storage format - #63633
Open
airborne12 wants to merge 78 commits into
Open
[feature-wip](inverted index) Introduce SPIMI V4 inverted index storage format#63633airborne12 wants to merge 78 commits into
airborne12 wants to merge 78 commits into
Conversation
airborne12
requested review from
gavinchou,
liaoxin01 and
yiguolei
as code owners
May 25, 2026 14:25
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
airborne12
force-pushed
the
inverted-index-spimi
branch
2 times, most recently
from
May 26, 2026 01:16
d47cb9d to
ab92f46
Compare
4 tasks
airborne12
force-pushed
the
inverted-index-spimi
branch
from
May 26, 2026 02:33
ab92f46 to
9281771
Compare
5 tasks
Member
Author
|
run buildall |
airborne12
force-pushed
the
inverted-index-spimi
branch
from
May 26, 2026 03:22
9281771 to
0f96cfa
Compare
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
airborne12
changed the title
Member
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
2 similar comments
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Member
Author
|
run buildall |
Member
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
I found blocking correctness issues in the V4 SPIMI spill path and validation gaps that can expose unsupported V4 tables.
Critical checkpoint conclusions:
- Goal/test proof: the PR wires V4 SPIMI write/read and adds broad tests, but the tests do not cover the production multi-spill case with absolute row ids and include a timing assertion that is likely flaky in CI.
- Scope/focus: mostly focused on V4 SPIMI, but the new p0 latency benchmark is not a stable functional test.
- Concurrency/lifecycle: no new shared concurrent state issue identified in the reviewed path; cache-reader lifetime changes appear to use existing searcher cache patterns.
- Configuration/compatibility: V4 is exposed through table properties/config and thrift/proto, but FE validation still permits unsupported V4 index shapes that BE later rejects during load.
- Parallel paths: cloud/non-cloud schema propagation has V4 branches, but unsupported parser/array paths need consistent FE rejection.
- Data correctness: multi-spill merging can corrupt posting doc ids, and flush can persist a stale doc_count for the triggering row. These affect query correctness for larger V4 fulltext segments.
- Test coverage/results: there are many unit/regression tests, but the missing absolute-doc-id multi-spill coverage lets the main spill corruption escape; the latency regression should not be p0 threshold-based.
- Observability/performance: no additional observability blocker found beyond the functional issues above.
User focus: no additional user-provided focus points were present.
| // Apply doc_id offset and append. | ||
| for (auto& d : docs) { | ||
| d.doc_id += offset; | ||
| } |
Contributor
There was a problem hiding this comment.
This offset corrupts production V4 spills. InvertedIndexColumnWriter::add_values() appends the segment-level _rid into SpimiPostingBuffer, and SpillManager::FlushBuffer() emits those doc ids as-is, so spill inputs already contain absolute row ids. When a large segment crosses the SPIMI memory budget more than once, the second and later spills get running added again here, shifting postings beyond their real rows (often beyond total_doc_count) and causing MATCH queries to return wrong/missing rows. The current merger tests build artificial inputs with local doc ids, so they do not cover the actual writer->spill contract. Please either make spill buffers localize doc ids before emitting, or remove this offset for V4 spill inputs and add an end-to-end test with multiple spills using absolute _rid values.
| // and continue accepting tokens. | ||
| if (_spimi_writer->ShouldFlush()) { | ||
| _spimi_writer->FlushPending(_spimi_doc_count); | ||
| } |
Contributor
There was a problem hiding this comment.
The doc count passed to the spill is stale for the row that triggered the flush. At this point the current row's tokens have already been appended with doc id _rid, but _spimi_doc_count is only advanced below this block. If this is the first row past the budget, the spill manifest can advertise a doc_count that excludes a doc whose postings are present. That stale count is then used by SegmentMerger for offsets/metadata. Update _spimi_doc_count before FlushPending() (or pass static_cast<int32_t>(_rid) + 1) so each spill's metadata covers all postings it contains.
| @@ -1275,11 +1277,15 @@ public static TInvertedIndexFileStorageFormat analyzeInvertedIndexFileStorageFor | |||
| return TInvertedIndexFileStorageFormat.V2; | |||
| } else if (invertedIndexFileStorageFormat.equalsIgnoreCase("v3")) { | |||
| return TInvertedIndexFileStorageFormat.V3; | |||
| } else if (invertedIndexFileStorageFormat.equalsIgnoreCase("v4")) { | |||
| return TInvertedIndexFileStorageFormat.V4; | |||
| } else if (invertedIndexFileStorageFormat.equalsIgnoreCase("default")) { | |||
Contributor
There was a problem hiding this comment.
Exposing explicit V4 here needs matching FE validation for the supported index shapes. The BE writer now rejects V4 when should_analyzer is false and also rejects array string indexes, but InvertedIndexUtil.checkInvertedIndexParser() still accepts parser none and array-with-parser-none for the same table property. That lets CREATE TABLE ... PROPERTIES("inverted_index_storage_format"="V4") succeed for keyword/array inverted indexes, and the first load then fails in BE writer init. Please reject unsupported V4 parser/type combinations during analysis, or implement those paths end-to-end.
| // on top of the actual reader latency. A real reader regression | ||
| // would shift the median far past this cap. | ||
| assertTrue(ratio < 2.0, | ||
| "${tag}: V4 median ${v4.median} us / V2 median ${v2.median} us = ${ratio} " + |
Contributor
There was a problem hiding this comment.
A p0 regression suite should not fail on wall-clock query timing. This median ratio includes planner/executor startup, cache state, BE scheduling, network/runner noise, and unrelated concurrent load; on shared CI it can exceed 2x without a functional regression, especially with only 9 retained samples. This will create flaky failures for unrelated changes. Please keep this as logging/manual benchmark coverage, move it out of p0/per-PR execution, or replace the assertion with deterministic functional checks.
Contributor
FE UT Coverage ReportIncrement line coverage |
Member
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 31767 ms |
Contributor
TPC-DS: Total hot run time: 172092 ms |
Contributor
FE Regression Coverage ReportIncrement line coverage |
airborne12
force-pushed
the
inverted-index-spimi
branch
from
June 2, 2026 15:38
3eae47c to
b334f74
Compare
…ms chain-copy
The DOCS_ONLY posting chain stored phrase-shaped docCode VInts (delta<<1|flag [+freq]) that the emit only decoded to throw the freq away and re-encode bare deltas. Store the bare doc-delta directly — written EAGERLY at doc open (no deferred close; FinalizeBlocks skips omit buffers), one entry per doc, which is byte-for-byte the on-disk DOCS_ONLY slim block. Slim omit terms (df < skip_interval) then take the same chain-copy fast path as phrase-on terms (EmitSlimTermPreEncoded grows omit support: no prox bytes, prox_pointer=0); PFOR omit terms replay doc_count bare deltas. The chain also shrinks in memory (no freq codes), trimming the DOCS_ONLY buffer.
Format ownership: the chain format follows the BUFFER's omit flag, the on-disk format follows the WRITER's — chain-copy and the bare-delta replay are gated on the flags AGREEING. The one legal mixed combination (omit writer over a phrase buffer, which tests use to emit DOCS_ONLY from a generic buffer) keeps the docCode decode+re-encode replay; the OmitTfapByteNeutral A/B pair caught the first cut keying this off the writer flag alone.
Omit records semantics: DecodeCompactTerm/DecodeTermToRecords now yield one record per DOC for omit buffers (per-occurrence multiplicity is not recoverable from a bare-delta chain, and nothing downstream needs it — the omit emit ignores freq). Norms are the only per-occurrence consumer, and omit fields never write norms in V2/CLucene either; EmitSegment grows a DCHECK so a future norms-over-omit caller fails loudly.
Validated: SPIMI-wide ASAN suite 486 pass / 0 fail, including the OmitTfapByteNeutral{VInt,Pfor} byte-equality oracles, OmitTfapDirectEmitMatchesRecordsPath, and the DOCS_ONLY roundtrip.
…, last-term swap Three byte-identical term-dictionary wins, one per term across the whole vocabulary: Utf8ToWideInto fills a reused member wstring instead of heap-allocating one per Add/AddInline; the front-coded suffix is staged once (AppendSCharsFromWide, sharing one EncodeSChar core with WriteSCharsFromWide so the encodings cannot drift) and emitted with a single bulk WriteBytes instead of a virtual WriteByte per encoded byte; and the .tis last-term update swaps the scratch instead of copying the wstring (the .tii boundary entries, 1 in 128, still copy and never reach the swap branch). SPIMI-wide ASAN suite 486 pass / 0 fail.
…d emit) The windowed-term replay decoded every position from the prox chain (prefix-summing within-doc deltas to absolutes) only for AddPosition to recompute the SAME deltas and re-encode the SAME LEB128 bytes into the windowed position buffer. Since the chain bytes and the rebuilt buffer are byte-identical, EmitFromCompactDirect now copies the whole prox chain once (one memcpy per slice), decodes only the per-doc docCode entries (df values — cheap relative to occurrences), recovers each doc's position byte offset with a continuation-bit scan (no value decode), and hands everything to FreqProxEncoder::EmitWindowedTermPreDecoded — which produces exactly FinishTermWindowed's output via the same WindowFrameEncoder::Encode call. High-frequency terms (wiki-class head words, millions of occurrences) skip the per-occurrence decode + re-encode entirely. Gated on V4 windowed + phrase-on with a phrase BUFFER (the chain must carry positions); the omit-writer-over-phrase-buffer combination keeps the replay. SPIMI-wide ASAN suite 486 pass / 0 fail, including the V4Inline direct-vs-records A/B's windowed (df=600) case.
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29465 ms |
Contributor
TPC-DS: Total hot run time: 168328 ms |
…wrote -4) 根因：SpimiIndexWriter::EmitDirect 调用 SpimiFulltextWriter::EmitSegment 时 漏传最后一个实参 inline_small_terms（默认 false）。直写路径（单次 flush、 无 spill —— 生产最常见形态）产出的 V4 段因此从不内联小 term：.tis 头 FORMAT=-4，小 term 的 frq/prx 留在外置文件，查询每个小 term 多付一次 读 GET。这与设计意图（fulltext_writer.h 注释：最终段与 V4 spill 段 lockstep 开启内联）相悖；spill-merge 路径一直正常（SpillManager 显式传 true、SegmentMerger 内部按 use_windowed 派生），导致同一字段 「有 spill 则内联、无 spill 则不内联」的不一致。 修复：EmitDirect 补传 inline_small_terms=true。EmitSegment 内部已有 use_windowed（index_version >= kIndexVersionV4）门控，V0/V2 兼容段不受 影响（仍 FORMAT=-4、无内联）。其余三个发射点（SpimiFulltextWriter:: Finish、SpillManager::FlushBuffer、SegmentMerger::Merge）复核无同类 漏传，未改动。 读侧安全性：持久读侧门控为 .fnm 的 index_version >= V4 + .tis 头 FORMAT=-5 分发；生产读路径（query_term_docs 等）已长期解码 spill-merge 产出的 -5 内联段，本修复只是让直写段进入同一已验证格式。 验证（TDD，ASAN UT 构建）： - 新增 SpimiIndexWriterTest.DirectEmitV4InlinesSmallTerms：修复前失败 （FORMAT=-4、TermInfo.inlined=false），修复后通过（FORMAT=-5、内联 frq 字节非空）。 - 新增 DirectEmitNonV4StaysExternalFormat 反向断言：非 V4 直写段保持 FORMAT=-4、无内联（修复前后均通过）。 - ASAN 宽 SPIMI 套件全绿：50 个 suite 共 441 用例，436 通过、5 个既有 env 门控/前置 PR 跳过、0 失败；含端到端 InvertedIndexReaderTest.SpimiV4SingleQueryProbe（生产写路径直写内联段 + 生产读路径查询）。
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
…n dense terms 设计要点（解剖结论：/tmp/idx_anatomy 36 组受控 .idx，.frq 是 V4 相对 V2 唯一 普遍反向的组件；根源是 V2 TurboPFor p4nd1enc 的 delta-1 域 + 0-bit 常数块对 delta≡1/freq≡1 的稠密 term 近乎免费，而 V4 PFOR 位宽下限 1 bit/值 + 每 128 值块 1B 宽度头 ⇒ 全等值块仍要 17B）： - 取块级方案 (a)：PFOR 子块新增「常数块」形态 [0x00][VInt(c)]。0x00 在旧格式 中是非法宽度字节 ⇒ 无歧义；块内 N 个值全等且 1+VIntLen(c) 严格小于普通块时 才发射，平手保持普通块（确定性、非全等块逐字节不变）。相比窗口级 kWinConst： 粒度更细（混合 term 的局部全等块同样受益）、skip 表语义/窗口边界/range-GET framing 与 kWinRaw/kWinZstd 字节语义完全不动、merger 字节拷贝快路径天然兼容。 - 仅 windowed V4 写路径开启（EncodePforPart → allow_const=true）；legacy kCodeModeSpimiPfor 路径保持旧字节。解码三处同步识别 0x00：DecodeBlockFromBytes 与两份 DecodePforRun 副本（query 路径 frq_window_decode_internal.h、merge 路径 posting_decoder.cpp）；0x80（patch 标志+宽度 0）仍按损坏字节硬失败，新增截断 /超长 VInt 损坏用例。 - W 选型与 zstd 交互自动正确：常数块长度进入 dd_len/fq_len ⇒ AnalyticRawFrqSize 与 MeasureAndCacheFrq 的代价评估、skip 偏移与 1.10x 预算逐字节一致（analytic vs measured 的 DCHECK 交叉验证保留）；常数窗 payload 极小，zstd 候选自然落选。 - omit/tfap 核查结论：DOCS_ONLY 无 freq 域（omit 路径已覆盖）；phrase-on 的全 1 freq 块此前仍付 17B/块，现收敛为 2B/块——phrase-on 收益的另一半来源。 效果（RELEASE benchmark 18-cell 矩阵，对照同 HEAD 未打补丁基线，V2 逐字节不变， 节省全部落在 .frq，.prx/.tis 不动）： - httplogs p0 30K：V4/V2 1.138→1.043（idx -8.4%，.frq -12%） - httplogs p1 30K/200K：0.658→0.580 / 0.610→0.523（.frq -23%） - textbench p1 20K/200K/1M：0.779→0.760 / 0.718→0.690 / 0.702→0.672 - agentlogs p1 in/out：0.851→0.836 / 0.837→0.824 - 其余 cell（textbench p0、weibo、wikipedia、agentlogs p0）0.00%~-0.01%，无一变大 - 写 CPU A/B（交替 4 轮、V2 同机对照）：V4 httplogs -0.5%/-4.3%、textbench -0.9%/+1.7%，均不超过 V2 对照漂移 ⇒ 无回归（全等块还省去 pack_bits/nth_element） 测试：TDD（RED 4 例修复前失败→GREEN）；ByteIdentityGolden 按机制重基线 6/7 （唯一无全等值块的 prox_varfreq700 digest 不变，证明非全等块字节稳定）；宽 SPIMI ASAN 套件 442 用例 440 过 / 2 env-skip / 0 失败。
V4 SPIMI posting buffer 的内存预算从编译期常量 kDefaultMemoryBudget
（128 MiB）改为跟随 config::inverted_index_ram_buffer_size（MB，mDouble
可热改）——与 V2/CLucene setRAMBufferSizeMB 同源同值，默认 512 MB。
动机：固定 128 MiB 预算下 >2M 行的大段进入病态 spill churn——httplogs 9M
phrase 写入 CPU 相对 V2 达 3.81x；跟随 512 MB config 后回到 0.99x，峰值
内存与 V2 同量级。
实现：
- SpimiPostingBuffer::ConfiguredMemoryBudgetBytes()：在 MB 域（double）
先夹紧到 [16 MiB, 8 GiB] 再乘 2^20，超大配置不会在 size_t 转换处溢出；
非有限值（NaN/inf）或 <= 0 回退 kDefaultMemoryBudget，config 未初始化
（全零）时安全。
- SpimiIndexWriter 构造时读一次（新增生产 ctor 注入 Limits，保留随机
hash seed），热改 config 对新段生效；Append 热路径零 config 读，每行
spill gate 仍只读 ShouldFlush() latch。
- kDefaultMemoryBudget 降级为测试兜底 + 非法 config 的回退值。
- benchmark [BENCH-CONFIG] 凭证行 buffer_budget_mb 改打实际生效值
（ConfiguredMemoryBudgetBytes() >> 20），不再打编译期常量。
本提交显式取代 a863652（256->128 MiB）的固定低预算决策（用户拍板）：
wiki 类大文档小段场景放弃 546->259 MB 的写峰值收益，512 MB 默认下其
写峰值将回升到 ~546 MB 量级（高于 V2 的 288 MB）；需要旧行为的部署可
调低 inverted_index_ram_buffer_size（热改生效）。预算只移动 spill 切分
点，不改变最终索引字节：新增 UT SpillCadenceDoesNotChangeIndexBytes
（同语料 0/1/3 次 spill 三种节奏，V4 段全部落盘文件逐字节一致）作为
回归护栏；httplogs 9M 实测 128/512 两档 idx_bytes=58928778 逐字节一致。
a863652 当年 +0.75% .idx 随 spill 数变化的现象，在 spill-merge 预解码
再编码重做之后不再复现。
测试：SpimiIndexWriterTest 新增 5 个 UT（config 跟随 / 构造时读一次 /
夹紧上下限 / 非法值回退 / spill 节奏 byte-identity）；宽 SPIMI ASAN
套件 447 ran / 445 PASSED / 2 SKIPPED（均为既有 env-gated 测量用例）/
0 FAILED。顺带清扫 inverted_index_writer.{h,cpp} 中 5 处陈旧的
"256MiB ShouldFlush latch" 注释（预算已 config 驱动）。
…rride
Batch 0 of the spill/merge streaming-concatenation work: evaluation
infrastructure that pins the byte gold standard BEFORE any merge
implementation changes.
1. SPIMI_RAM_BUFFER_MB env override in benchmark_spimi.hpp
ensure_env_inited (same style as SPIMI_FRQ_ZSTD): forces the
spill+merge path (e.g. 128) on corpora that would not spill under
the 512 default now that the buffer budget follows
inverted_index_ram_buffer_size. Verified live: [BENCH-CONFIG]
prints buffer_budget_mb=128 with the env set, 512 without.
2. SpimiMergeByteIdentityGolden (G1): fixed term matrix (40 inline
tiny terms, slim 511/512/513 boundary trio, windowed df=5000
multi-window, PFOR const-block df=2048, heavy-positions df=600)
built through the production spill contract (absolute doc_ids,
k contiguous doc ranges, SpillManager::FlushBuffer) for
k in {direct,1,2,7} x {phrase, DOCS_ONLY, frq-ZSTD}; chained
FNV-1a digests over the merged .tis/.tii/.frq/.prx pinned as
literals captured at this HEAD. MergedEqualsDirectWrite
additionally asserts the live invariant merge == direct write
byte-for-byte (it already holds today); the upcoming streaming
merge batches must keep every one of these bytes identical.
3. ScopedHeapHighWater test utility: heap high-water sampling for
the later peak-memory RED cases (ASAN
__sanitizer_get_current_allocated_bytes 200us sampler; jemalloc
thread.peak fallback; GTEST_SKIP-able when neither exists) with
self-tests.
Wide SPIMI ASAN suite: 451 tests, 449 pass, 2 pre-existing
env-driven skips, 0 sanitizer reports.
…nput Batch 1 (S2) of the spill/merge streaming work. At Finish() time the residual posting buffer used to be FlushBuffer()ed into one more spill (7 tmp files written) and immediately LoadSpill()ed back — a full disk round-trip of the (often largest) final segment for nothing. New SpillManager::EmitBufferToInput emits the residual buffer through EmitSegment into MemoryByteOutput sinks and hands back a SegmentMerger::Input shaped exactly like LoadSpill's, appended as the LAST merge input (highest doc range, absolute-doc-id order contract unchanged). The merger is untouched. Encoding parameters are lockstep with FlushBuffer (index_version, omit_tfap, omit_norms, inline_small_ terms) and MemoryByteOutput/FileByteOutput report identical FilePointer streams, so the merged output stays byte-identical — verified by the batch-0 golden digests (all 10 unchanged) and a direct-write baseline comparison in the new tests. Edge cases preserved: k=0 pure direct write never enters EmitMerged; empty residual (ShouldFlush latched, zero records) contributes no input, mirroring FlushBuffer's empty-skip (NULL/empty-segment behavior intact); zero spills + non-empty residual keeps the MergeSingleInput byte-copy fast path reachable via the single in-memory input. RED -> GREEN: new SpillManager::TotalSpillsCreated() (monotonic, not reset by cleanup) lets tests observe spills created DURING Finish; FinishResidualBufferDoesNotCreateNewSpill failed on the old path (3 spills vs 2) and passes now, with the final segment byte-equal to the direct-write baseline. Wide SPIMI ASAN suite: 452 passed / 2 pre-existing env-skips / 0 sanitizer reports.
…df tier selection Kill the per-term vector<DecodedDoc> materialization in the k-way spill merge: a df=N term used to allocate N structs plus N per-doc positions heap blocks (plus a defensive stable_sort of already-ordered runs), dominating merge peak memory on high-df terms. - PostingDecoder::DecodeFlat appends a term run into flat arrays (doc_deltas/freqs + verbatim within-doc position VInt bytes + per-doc byte offsets), re-basing each subsequent input's first delta against the previous run's last absolute doc id. Decode() is now a thin wrapper over the same single envelope/codec core (DecodeFrqFlat + ResolvePrxInner), so both shapes share one dispatch implementation. - SegmentMerger drains the term heap in two phases: collect every input's (input, TermInfo) first — df precedes the posting pointers in the .tis entry — so the merged sigma-df drives the slim/windowed/ inline tier exactly like a direct write of the same data (sigma-df >= 512 upgrades slim runs straddling spills to windowed; never the inputs' per-run is_slim). - Emission reuses the existing encoders byte-for-byte: EmitSlimTermPreEncoded (rebuilt docCode VInts + spliced raw .prx payload through the same FlushProxRaw ZSTD policy), EmitWindowedTermPreDecoded (extended to DOCS_ONLY: dd-only windowed terms, mirroring FinishTermWindowed's omit-mode call), and a flat replay for legacy (pre-V4) PFOR re-encode. Byte identity: merged output stays byte-for-byte equal to a direct write of the same data — batch-0 golden digests unchanged (phrase/docsonly/frqzstd x direct/k1/k2/k7) and every new RED case also asserts 4-stream equality against direct write. Peak (ASAN net-allocation high-water across LoadSpill+Merge, new RED suite spill_merge_peak_red_test.cpp, red-before/green-after): R1 windowed df~800K k=2: 60MB -> 26MB (line 32MB) R3 const-block df=1M k=2: 68MB -> 28MB (line 32MB) R4 DOCS_ONLY df~800K k=2: 49MB -> 7MB (line 16MB) R5 heavy-pos df~600K k=3: 77MB -> 34MB (line 48MB) R2/R6/R7 guards (sigma-df 511/512/513 straddle, k=3 absolute ids, staged-inline lockstep) green before and after. Full SPIMI ASAN suite: 459 passed / 2 pre-existing env-skips / 0 sanitizer reports.
EmitMerged previously loaded every spill's full .tis/.tii/.frq/.prx back
into memory (LoadSpill), so the k-way merge peak carried the SUM of all
spill bytes for the whole merge. Replace the full load with k x 4
forward sliding-window cursors (ForwardByteSource): the spill layout
already guarantees sequential consumption (.tis in term order, posting
pointers monotonically non-decreasing), and both the .frq and .prx
decodes are strictly forward and self-delimiting given doc_freq, so no
block end needs to be known up front.
- byte_source.{h,cpp} (new): ForwardByteSource (inline hot path, virtual
Refill only on window exhaustion) + MemoryByteSource (borrowing /
owning) + SpillFileByteSource (pread window of min(1MiB, stream len)).
- posting_decoder: DecodeFlat core now consumes ForwardByteSource; the
(ptr,len) overload wraps a memory source. kProxRaw scans VInts straight
off the source; ZSTD/windowed envelopes inflate exactly as before.
- term_enum: parse over ForwardByteSource; the legacy vector ctor keeps
the borrowed inline-span contract; the streaming ctor copies inline
spans into a per-entry scratch (bounded by the inline cap).
- segment_merger: StreamInput (four cursors) is the core Merge input;
the vector<Input> overload is a thin wrapper, so output bytes are
identical on both entries. take() copies inline spans into the
TermSource; MergeSingleInput chunk-copies streams and reads the .tis
footer via ReadAt, keeping the k=1 byte-copy fast path.
- spill_manager: OpenSpillCursor(i, out, buffer_bytes=1MiB); LoadSpill
stays for tests/diagnostics.
- spimi_index_writer: EmitMerged opens cursors; the residual buffer's
in-memory Input is wrapped as an owning memory cursor.
RED->GREEN (ASAN heap high-water, spill_merge_peak_red_test):
- new R8 (httplogs-shaped 500K-doc term sea, k=7, file-backed sinks):
91,163,420B peak with the full load -> 21,071,702B with cursors
(inputs 77,545,850B; line 64MB).
- R1/R3/R4/R5 lines tightened (input bytes no longer a legitimate
residency allowance); R9 stresses 4KB cursor windows (phrase +
DOCS_ONLY) for refill/seek/inline-span lifetimes under ASAN.
- byte golden unchanged: all 10 digests (phrase/docsonly/frqzstd x
direct/k1/k2/k7) hold literally; merged == direct write in every cell.
- spill_segment_merger_test: fix a latent past-the-end TermEntry read in
CompactModeFlushBufferDoesNotSkipData (UB exposed by the new heap
layout).
Wide ASAN SPIMI suite: 463 ran / 461 passed / 2 pre-existing env-skips /
0 sanitizer reports.
Batch 4 of the spill-merge overhaul: when a merged term's sum-df stays
below the skip interval (512), every input run is necessarily SLIM, so
the k-way merge now splices the posting bytes directly instead of
flat-decoding and re-encoding every value:
- .frq chain: only each subsequent run's FIRST docCode is re-based
(delta vs the previous run's last doc, freq low-bit and trailing freq
VInt preserved); all remaining bytes are copied verbatim while a
read-only scan tracks the re-base anchor and sum-freq. Canonical
LEB128 keeps the first run's chain fully verbatim.
- .prx: the whole-term envelope is resolved to its raw payload and
spliced verbatim (kProxRaw scans sum-freq VInts, kProxZstd inflates
first); the merged block's mode-byte/ZSTD policy is re-run on the
concatenated payload by EmitSlimTermPreEncoded, so output bytes stay
identical to a direct write (and to the flattened re-encode path).
Dispatch priority: k=1 whole-segment byte copy > k>1 slim concat >
flattened re-encode (sum-df >= 512 upgrades unchanged). New
PostingDecoder::ConcatSlimRun hosts the splice core; SegmentMerger
gains thread-local MergeStats (verbatim hit-rate observability) and a
test-only force-reencode hook for the cross-path byte assertion.
Tests (RED->GREEN): spill_merge_slim_concat_test pins concat == forced
re-encode == direct write byte-for-byte across omit x k in {2,3,7},
the 511/512/513 sum-df upgrade boundary, the slim .prx 512B ZSTD gate
in both directions (raw inputs -> compressed merged block, compressed
inputs -> inflate then concat), and 4KB streaming-cursor refill
boundaries. Hit rate on the matrix: 124/126 slim terms take the
verbatim path. Byte-identity golden digests unchanged; full SPIMI ASAN
suite green (465 tests).
…truncated-spill error, frqzstd/V1-envelope goldens
Close the four non-blocker gaps from the adversarial review of the
streaming spill-merge batches (0 production code changes, all existing
golden literals unchanged):
- G3-b: FinishSingleSpillEmptyResidualByteIdentity{,DocsOnly} pin the
production-reachable single-spill + empty-residual Finish, hitting
MergeSingleInput x file cursor x CopyWholeStream chunked branch
(BorrowStable==nullptr) byte-identical to a direct write; MergeStats
asserts the path (single_input_segments==1, zero concat/reencode).
DOCS_ONLY side checks the zero-length .prx no-op via filesystem size
(openInput rejects empty files).
- G3-a: FinishTruncatedSpillFileThrowsCorrupt truncates _spill_0.frq to
3 bytes (spill path pinned via inverted_index_spimi_spill_path) and
asserts Finish rethrows doris::Exception cleanly through
FINALLY_CLOSE + CleanupSpillFiles — first direct test of the
SpillFileByteSource short-read corrupt branch.
- G1-a: golden matrix adds frqzstd_k7 (== frqzstd_direct digest) and
frqzstd_docsonly_direct/k2 (equal pair) literals.
- G1-b: LegacyV1WholeTermZstdEnvelopeGolden feeds a V1 whole-term
kCodeModeZstd envelope (V4-production-unreachable, shadow/debug only)
into the merge: k=2 V1 spills (win_5000 df_i=2500 >= 512 forces the
envelope, asserted on the input bytes), merged output byte-identical
to a V0 direct write + digest pinned.
Wide ASAN SPIMI suite: 471 ran / 469 passed / 2 env-gated skipped,
zero ASAN errors.
…comparison V2/CLucene retains per-doc norms (.nrm) for analyzed phrase-on fields, but Doris fulltext MATCH is a filter and never consumes BM25 norms — the .nrm stream is dead weight that inflates V2 idx sizes versus V4 (which omits norms unconditionally). Add a runtime mBool config (default false, zero behavior change) that drops norms on the shared CLucene write path (storage format V2 and V3) when enabled; V4/SPIMI already omits norms on its own path. Wire the SPIMI_RAM_DIR and SPIMI_OMIT_NORMS benchmark envs to the matching configs (inverted_index_ram_dir_enable / inverted_index_v2_omit_norms) so UT microbenchmarks and the E2E cluster measure under the same apples-to-apples conditions, and print v2_omit_norms in the [BENCH-CONFIG] provenance line. Add UT NormsOmittedWhenV2OmitNormsConfigOn covering the new config=true branch (analyzed phrase-on field -> no .nrm), with RAII restore of the process-global config so it cannot pollute other cases. Verified: 2/2 InvertedIndexWriterTest norm cases pass under ASAN.
Member
Author
|
run buildall |
The CI Clang Formatter uses clang-format 16, but this file was committed under clang-format 20 (local ldb_toolchain), which wraps a designated struct-init and an array literal differently. Reformat the two hunks to v16 to green the check. No behavior change.
Member
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29523 ms |
Contributor
TPC-H: Total hot run time: 29578 ms |
Contributor
TPC-DS: Total hot run time: 168480 ms |
Contributor
TPC-DS: Total hot run time: 169091 ms |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Introduces a new inverted index storage format V4 powered by SPIMI
(Single-Pass In-Memory Indexing), replacing the CLucene
IndexWriteron the write path for analyzed (fulltext) string columns.
Why
CLucene's
IndexWriteraccumulates per-tokenPostinglinked-listnodes plus a term hash table plus a char[] interning pool. On Doris
fulltext columns this dominates BE memory during write and shows up
in OOM kills on large segments. The encoding is byte-equivalent to
Lucene 2.x, but the in-memory representation is the cost. SPIMI
keeps a flat
(term_id, doc_id, position)record array plus asingle intern arena, then sorts + emits the same Lucene 2.x sibling
files (
.tis/.tii/.frq/.prx/.fnm/segments_N) onfinish(). Theon-disk format is unchanged; only the writer's working memory shape
changes.
Measured impact (SPIMI_BENCH=1, ~614 K occurrences/segment)
.idxon-disk sizeRepetitive vocab is the architectural trade-off region: V4's
compact-mode VInt-delta stream scales per-occurrence while CLucene's
Posting struct scales per-unique-term. Absolute memory in this
regime is sub-MB on both sides, so the percentage swing has no
production impact. Storage-size delta on repetitive is the
documented PFOR header cost.
What's in this PR
be/src/storage/index/inverted/spimi/):SpimiPostingBuffer(flat record + arena + intern map withhybrid compact-mode VInt-delta migration),
SegmentWriter,TermDictWriter,FieldInfosWriter,SegmentInfosWriter,PFOR encoder for high-doc-freq postings,
ByteOutputfamilyabstracting CLucene's
IndexOutput.SpimiQueryIndexReader,SpimiTermDocsReader,SpimiProxReader,SpimiTermEnum,SpimiSearcherBuilder;SpimiFulltextIndexReaderis theDoris-side adapter (overrides
type() -> SPIMI_FULLTEXTsothe searcher cache routes correctly).
column_reader.cppdispatch: V4 storage format → SPIMIreader; V1/V2/V3 unchanged.
EmitSegmentpost-flush self-validation:ValidateClosedSegmentByteCountsre-queries on-disk filelengths after close, throws
INVERTED_INDEX_FILE_CORRUPTEDonmismatch — guards against the async-S3 partial-flush class of
bugs that single-node tests can't see.
be/test/storage/index/inverted/spimi/plus extended tests under
be/test/storage/segment/:SPIMI_THROW_CORRUPTsite (segments_N / .frq / .prx / PFOR / .tis-.tii readers)
fault-injection case
.idxbyte parity)randomized V2/V4 alternation + full distribution report
repetitive workloads
(
InvertedIndexReaderTest.SpimiV2V4QueryLatencyBenchmark)using the corrected
SpimiFulltextIndexReader::create_shareddispatch
SPIMI_BENCHenv-var tier: default UT runs use 12 Koccurrences (fast regression guard);
SPIMI_BENCH=1scales to~614 K,
SPIMI_BENCH=largescales to ~6 M for full-segmentstress. Keeps headline benchmark numbers reproducible without
ballooning every UT pass.
inverted_index_p0/storage_format/test_storage_format_v4— V2 vs V4 black-box parity across MATCH_ANY / MATCH_ALL /
MATCH_PHRASE / MATCH_PHRASE_PREFIX / MATCH_REGEXP, NULL/empty
handling, and the
support_phrase=false(omit_tfap) no-proxwrite+read path.
test_storage_format_v4_cloud— same coverage gated byisCloudMode()so the async-S3 upload path gets exercised.test_storage_format_v4_query_latency— cluster-levelV2 vs V4 query timing distribution.
PropertyAnalyzer,TabletIndex,OlapTable): acceptinverted_index_storage_format=V4inCREATE TABLE PROPERTIES; propagate through the protocol to BE.
What's NOT in this PR (known gaps)
currently emits a single
_0segment per column; compaction isdocumented as a follow-up in
SPIMI_DESIGN.md.omit_norms=true; the readside synthesizes a default-norm array. Score-using paths
(
MATCH_ALLwith relevance ordering) fall back to V2 behavioron V4 columns. Listed in design doc.
(
should_analyzer=false) and numeric (BKD) paths remain on theexisting writers.
Release note
Add inverted index storage format V4, an in-house SPIMI-based writer
that reduces BE write-side memory by ~55 % and CPU by ~68 % on
diverse-vocab fulltext workloads while keeping segment on-disk
format Lucene 2.x compatible. Enable by setting
inverted_index_storage_format = "V4"in CREATE TABLE PROPERTIES.Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)