Improve documentation search ranking with match position and proximity

[BorisTyshkevich]

Parent design: #420
Depends on: #421

Purpose

Improve documentation search quality by incorporating where the query appears inside each searchable field, whether the full phrase matches, how close multiple tokens occur, and whether a match respects identifier boundaries or camel-case components.

This issue refines ranking and result snippets. It must not change the public search interaction or duplicate the Reference drawer UI work in #422.

Goals

• Use first-match position as a secondary relevance signal.
• Reward full-query phrase matches over separated token matches.
• Reward close token proximity within the same field.
• Distinguish identifier-boundary and camel-case component matches from arbitrary substrings.
• Produce bounded snippets centered around the first useful documentation match.
• Preserve physical source provenance and merge source-specific evidence correctly.
• Keep name/alias/syntax field importance above prose position bonuses.
• Keep ranking deterministic and bounded.

Position signals

For each available searchable field, compute the first case-insensitive match position of the full normalized query:

positionCaseInsensitiveUTF8(name, q) AS name_phrase_pos
positionCaseInsensitiveUTF8(alias_to, q) AS alias_phrase_pos
positionCaseInsensitiveUTF8(syntax, q) AS syntax_phrase_pos
positionCaseInsensitiveUTF8(categories, q) AS category_phrase_pos
positionCaseInsensitiveUTF8(description, q) AS description_phrase_pos

0 means no match; positive values are 1-based character positions.

Every UNION ALL branch must project compatible position columns. A branch lacking a field projects typed zero values.

Multi-token positions

For a normalized query with several tokens, return first positions per token for fields where proximity matters, at least:

• name;
• alias;
• syntax;
• description.

Conceptually:

description_token_positions = [18, 73, 0]
syntax_token_positions      = [4, 12, 0]

The exact SQL representation may be arrays or fixed generated scalar columns. It must remain type-compatible across all union branches and bounded by the maximum normalized query length/token count.

Do not calculate textual proximity across different fields. A token in syntax and another in description receives field-coverage credit, not proximity credit.

Earlier-occurrence bonus

Earlier matches are generally more relevant, especially within full Markdown documentation bodies.

Suggested position bonus:

position 1–80       +80
position 81–240     +50
position 241–800    +25
position above 800  +10

Position bonuses are secondary. They must not allow an early prose occurrence to outrank exact, prefix, alias, or strong syntax matches.

The total prose-derived contribution must remain capped.

Phrase bonus

Reward the full normalized query appearing as a phrase.

Relative priority:

1. phrase in canonical name;
2. phrase in alias;
3. phrase in syntax;
4. phrase in category/type;
5. phrase in description.

A phrase match must outrank the same tokens scattered through the same field, while still respecting the baseline field weights from #421.

Token proximity

For matched token positions within one field, calculate the span:

const matched = positions.filter((position) => position > 0);
const span = Math.max(...matched) - Math.min(...matched);

Suggested proximity bonus:

span ≤ 40 characters    +50
span ≤ 120              +30
span ≤ 300              +15

Requirements:

• Require at least two matched tokens for a proximity bonus.
• Never combine positions from different fields.
• Cap proximity contribution.
• Missing tokens do not receive a proximity bonus; the foundational search contract still requires every token to match somewhere in the row.

Identifier-boundary and camel-case scoring

Substring retrieval should remain permissive, but JavaScript ranking should distinguish stronger lexical matches.

Identifier boundaries are characters outside [A-Za-z0-9_] or the beginning/end of the string.

Examples:

• search sum matching sum as a complete identifier word: strongest boundary bonus;
• search array matching the Array component in groupArray: useful camel-case component bonus;
• search sum matching inside checksum: no boundary/camel bonus;
• arbitrary substring: baseline substring score only.

Implement locale-stable, deterministic matching suitable for ClickHouse identifiers. Do not depend on browser-specific segmentation APIs.

Suggested ordering:

whole identifier word > camel-case component > arbitrary substring

These bonuses refine the field’s existing match score rather than replacing it.

Match-centered snippets

Search must continue to avoid returning complete documentation bodies for every candidate.

Calculate match positions against the full field, then return a bounded snippet around the first useful description match:

if(
    description_pos > 0,
    substringUTF8(description, greatest(1, description_pos - 100), 400),
    substringUTF8(description, 1, 400)
) AS description

Exact SQL may vary with supported ClickHouse functions, but behavior must be:

• maximum bounded snippet size;
• centered with modest leading context when description matched;
• beginning-of-description fallback when only another field matched;
• no full Markdown payload during search;
• plain text result rendering remains UI-owned.

The snippet model should indicate when leading/trailing content was omitted so the UI may render ellipses safely without guessing.

Suggested result fields:

interface DocSearchSnippet {
  text: string;
  clippedStart: boolean;
  clippedEnd: boolean;
}

Shared union projection

Extend the normalized search row with compatible evidence fields, for example:

name_phrase_pos
alias_phrase_pos
syntax_phrase_pos
category_phrase_pos
description_phrase_pos
name_token_positions
alias_token_positions
syntax_token_positions
description_token_positions
description_clipped_start
description_clipped_end

Every included source branch must emit the same ordered types. Missing source fields use explicit typed zero/empty values.

The foundational physical source_table column remains required.

JavaScript scoring

Add pure helpers such as:

function positionBonus(position: number): number {
  if (position <= 0) return 0;
  if (position <= 80) return 80;
  if (position <= 240) return 50;
  if (position <= 800) return 25;
  return 10;
}

function proximityBonus(positions: readonly number[]): number {
  const matched = positions.filter((position) => position > 0);
  if (matched.length < 2) return 0;
  const span = Math.max(...matched) - Math.min(...matched);
  if (span <= 40) return 50;
  if (span <= 120) return 30;
  if (span <= 300) return 15;
  return 0;
}

The exact weights may be tuned with fixtures, subject to these invariants:

• exact canonical name remains highest;
• name/alias remains above syntax;
• syntax remains above pure prose;
• phrase > separated tokens within the same field;
• earlier occurrence > later occurrence within the same field;
• close tokens > distant tokens;
• whole-word/camel component > arbitrary substring;
• prose-derived score is capped;
• deterministic tie-breakers from Build capability-aware documentation search service #421 remain stable.

Source-aware evidence merge

Duplicate logical entities may carry different match evidence from different physical tables.

Do not collapse evidence prematurely.

For each contributing source retain enough data to determine:

• which field matched;
• phrase position;
• token positions/proximity;
• snippet quality;
• source table.

Merged result policy:

• retain all source provenance;
• score each source row with the shared policy;
• use the strongest source-specific evidence as the merged relevance score;
• prefer structured snippets for concise descriptions when similarly relevant;
• use broad system.documentation text when it contains the only or clearly stronger match;
• smallest positive position may be used only within comparable fields; do not treat position 20 in a concise structured description as automatically equivalent to position 20 in a full Markdown document;
• preserve deterministic selection when scores tie.

The full documentation entry source remains controlled by existing docEntry(target) policy.

Query and payload bounds

• Preserve Build capability-aware documentation search service #421’s per-branch and outer row limits.
• Preserve the maximum displayed result count.
• Keep snippet text bounded, recommended 400–500 UTF-8 characters.
• Cap token count used for generated position expressions to a documented safe maximum derived from the 200-character query bound.
• Generated SQL length must remain bounded.
• Position arrays/scalars must not cause unbounded payload growth.

Suggested implementation boundary

src/core/doc-search.ts

Own pure logic for:

• lexical boundary/camel-case classification;
• phrase, position, and proximity scoring;
• caps and tie behavior;
• source-aware evidence merge;
• snippet metadata normalization.

SQL branch builders

Extend capability-generated projections with match positions, token evidence, and bounded context snippets. Keep optional columns capability-safe.

SchemaCatalogService

No public API redesign should be necessary. It continues to execute one union, normalize, merge, rank, and cache the refined result model.

Reference drawer

No required interaction changes. #422 consumes the improved summaries/order through the existing docSearch() response.

Quality fixtures

Add realistic fixtures covering queries such as:

• array aggregation
• array join
• merge tree ttl
• json input
• date truncate
• sum
• group array

Fixtures should include competing rows where:

• one has an exact/prefix name match;
• one has an early description phrase;
• one has tokens far apart in Markdown;
• one matches only in examples-like prose;
• one matches a camel-case component;
• one contains an accidental substring.

Expected ordering must be explicit and stable.

Tests

Position extraction/projection

• phrase positions are projected for every available field;
• missing fields emit compatible zero values;
• multi-token evidence has compatible union types;
• positions are calculated against full fields, not truncated snippets;
• generated expressions are bounded by token limits.

Position scoring

• earlier description match outranks a later description match when all else is equal;
• position bonuses never outrank stronger field classes;
• no-match position 0 contributes nothing;
• thresholds are deterministic at boundaries.

Phrase and proximity

• full phrase outranks separated tokens in the same field;
• close tokens outrank distant tokens;
• one token receives no proximity bonus;
• cross-field positions receive no proximity bonus;
• prose contribution remains capped.

Lexical matching

• whole identifier word outranks arbitrary substring;
• camel-case component receives the intended bonus;
• sum inside checksum does not receive boundary credit;
• underscores and digits behave consistently;
• matching is case-insensitive but tie behavior remains deterministic.

Snippets

• description match returns context around the first useful match;
• leading and trailing clipping flags are correct;
• matches near the beginning do not request negative offsets;
• matches near the end remain bounded;
• non-description matches fall back to the beginning summary;
• snippets never exceed the configured limit;
• multi-byte UTF-8 text remains valid.

Source merge

• structured and broad evidence are both retained;
• strongest source-specific score determines result relevance;
• structured concise summary wins when equivalently relevant;
• broad summary wins when it contains the only match;
• source provenance remains complete;
• same logical result remains deterministic across input row order.

Regression

• baseline exact/prefix/alias ordering from Build capability-aware documentation search service #421 remains intact;
• result count and cache contracts remain unchanged;
• generated query remains one UNION ALL request;
• all branches still report actual physical source table;
• Reference drawer requires no ranking-specific UI logic.

Acceptance criteria

• Search ranking incorporates first-match position without weakening name/alias/syntax priority.
• Full phrases rank above separated tokens within the same field.
• Close same-field token matches rank above distant matches.
• Whole-word and camel-case component matches receive deterministic lexical bonuses.
• Search summaries are bounded and centered around useful description matches.
• Position and token evidence remains type-compatible across the dynamic union.
• Duplicate logical results merge source-aware evidence while retaining physical provenance.
• Prose-derived score is capped and cannot overpower exact or strong identifier matches.
• Ranking quality fixtures demonstrate stable, improved ordering.

Non-goals

• Fuzzy edit-distance search.
• Semantic/vector search.
• Regular expressions.
• Search-as-you-type.
• UI redesign or kind filters.
• Full Markdown transfer in result rows.
• Changing the existing full-entry source-selection policy.