feat(connectors): add Meilisearch source connector

[countradooku]

Summary

• add the Meilisearch source connector using the official meilisearch-sdk
• document source configuration, retry behavior, and primary-key cursor requirements
• add Docker-backed integration coverage for producing indexed Meilisearch documents into Iggy

Motivation

This splits the source portion out of #3404 so the Meilisearch source can be reviewed and merged independently from the sink connector.

Closes #3496.

Validation

• cargo sort --no-format --workspace
• cargo test -p iggy_connector_meilisearch_source
• cargo clippy -p iggy_connector_meilisearch_source --all-targets -- -D warnings
• env CARGO_BIN_EXE_iggy-server=... CARGO_BIN_EXE_iggy-connectors=... cargo test -p integration --test mod -- connectors::meilisearch::meilisearch_source
• git diff --check

[codecov]

Codecov Report

❌ Patch coverage is 77.74968% with 176 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.23%. Comparing base (60b8789) to head (cd089c6).
⚠️ Report is 29 commits behind head on master.

Files with missing lines	Patch %	Lines
...e/connectors/sources/meilisearch_source/src/lib.rs	77.74%	148 Missing and 28 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3498      +/-   ##
============================================
- Coverage     76.27%   76.23%   -0.04%     
  Complexity      969      969              
============================================
  Files          1325     1326       +1     
  Lines        163541   164332     +791     
  Branches     136421   137290     +869     
============================================
+ Hits         124742   125286     +544     
- Misses        35097    35234     +137     
- Partials       3702     3812     +110     

Components	Coverage Δ
Rust Core	76.26% <77.74%> (-0.02%)	⬇️
Java SDK	62.71% <ø> (ø)
C# SDK	71.16% <ø> (-1.13%)	⬇️
Python SDK	89.45% <ø> (ø)
PHP SDK	84.52% <ø> (ø)
Node SDK	96.34% <ø> (+0.26%)	⬆️
Go SDK	43.08% <ø> (ø)

Files with missing lines	Coverage Δ
...e/connectors/sources/meilisearch_source/src/lib.rs	77.74% <77.74%> (ø)

... and 56 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ryerraguntla]

overall correct direction - cursor-by-primary-key is the right shape for Meilisearch, test coverage is present but coverage gap — only happy-path covered. No test for: (a) state restored across restart → cursor starts from last_primary_key, not from zero; (b) second poll produces zero duplicates. refer The elasticsearch_source.rs has state_persists_across_connector_restart as a pattern. For a cursor-based source, duplicate-freedom is the core invariant; it should be integration-tested.

[ryerraguntla]

/author

[countradooku]

/ready

[ryerraguntla]

/author

[countradooku]

/ready

[ryerraguntla]

/author

[countradooku]

/ready

[hubcio]

two items that don't anchor to changed lines:

• setting a nested-array filter through env-var overrides gets silently mangled before it reaches the connector: the env array parser in core/configs/src/configs_impl/parsing.rs strips both leading brackets (trim_start_matches('[') trims repeatedly) and its splitter tracks quotes but not bracket depth, so [["a","b"],"c"] becomes three broken string fragments. they all pass the connector's filter validation and get ANDed into a syntactically invalid meilisearch filter: open() succeeds (it never runs a search), every poll gets a 400 with no retry, and the connector sits at Running forever. pre-existing parser bug, but this is the first connector whose docs point at nested arrays - deserves a separate issue on core/configs, and until then the README shouldn't recommend the env route for nested filters.

• coverage note: the integration test loads the shipped config.toml and overrides via env, so the riskiest configurations are exactly the untested ones - non-empty query (the cursor-ordering problem), include_metadata = true (fixture forces "false"), any filter at all, and the settings-object filterable branch (needs meilisearch v1.14, container pins v1.13).

[hubcio]

cursor pagination breaks whenever query is non-empty. meilisearch bucket-sorts by ranking rules and sort sits 5th, behind 4 relevancy rules, so hits come back relevance-bucketed and pk-ordered only within a bucket - but the cursor takes the last hit's pk as if it were the batch max. two failure modes: with any non-empty query the last hit can carry a lower pk than earlier hits, so already-sent documents get re-emitted on the next poll; and once the match set exceeds batch_size, the truncated low-relevance tail can hold pks below the new cursor - those documents are never fetched again, silently. empty query is safe (placeholder search skips the relevancy rules).

the gate is basically free: check_primary_key_cursor_settings already fetches the full Settings, and Settings.ranking_rules is on that struct - open() can reject a non-empty query unless sort precedes the relevancy rules, with zero extra requests. the README should steer people to filter instead (telling them to reorder rankingRules would wreck relevance for every other consumer of the index). nothing exercises this path today: the integration setup runs query = "" and the fixture never sets a query.

[hubcio]

meilisearch stores all numeric values as f64 (their datatypes docs: ~15 significant digits, integers beyond 2^53 lose precision, strings recommended for large ids). the cursor literal here renders full u64 precision but the server compares rounded values - around 2^63 roughly 2048 consecutive ids collapse onto one f64, so id > X can skip a whole block of documents after emitting just one of them. the unit tests with 9007199254740993 (2^53+1, the first unrepresentable integer) only round-trip this crate's own string rendering and never touch meilisearch, so they don't cover this. README says "must be an integer" with no range bound - either document the 2^53 limit or reject larger values.

[hubcio]

when a non-empty batch has zero valid pks this returns Err, and poll errors never reach the runtime - the SDK loop logs and continues, and connector status stays Running (set once before the loop; set_error is only reachable from send/state-save failures, and no error metric moves either). so a string-pk index passes open() - which checks sortable/filterable but never value types, and the README's own settings example is enough to get there - then errors every poll forever while reporting healthy. it also contradicts README lines 54-56, which say invalid documents "are skipped with a warning". valid_documents_should_reject_all_invalid_primary_keys pins the current behavior, so this is deliberate - but aligning with the README (skip + warn) plus a pk value-type probe in open() would avoid the permanent quiet stall. probe alone isn't sufficient (empty index has nothing to sample, and mixed pk types can appear later), so surfacing poll errors to connector status is the real long-term fix.

[hubcio]

timeout here bounds a single attempt and the backoff sleeps sit outside any budget - the sink applies the same config key as a total budget across all retries (its retry loop tracks elapsed time and clamps each sleep to the remainder). same knob name, opposite meaning within one connector pair. worst case open() chains three of these loops (health, primary key, settings) to nearly 10 minutes against a peer that accepts TCP and never responds - meilisearch-sdk sets no reqwest timeout, so this wrapper is the only bound. and since sources initialize sequentially through a blocking FFI open, one unreachable meilisearch stalls the entire connectors-runtime boot: other sources, sinks, and the control API. adopting the sink's total-budget shape (or renaming the key) fixes both.

[hubcio]

a restored cursor that fails validation gets wiped to None and the next poll runs unfiltered - the entire index is re-emitted. that's fail-open in the one spot where everything else in open() fails closed (missing pk, not sortable, not filterable all abort). influxdb_source handles the same case by refusing to start ("Refusing to start to prevent silent cursor reset"), and failing open() is not a dead end for the operator - POST /sources/{key}/restart re-runs it once the state is fixed. also the warn below prints the primary key field name, not the rejected value - the one thing needed to recover by hand.

[hubcio]

"the next in-process poll can skip the unsent batch; a restart reloads the last persisted cursor" undersells it. the in-memory cursor advances before the send, and the next successful send persists a cursor past the failed batch - so one failed send followed by any success makes the skip permanent. restart only helps if the process dies before another batch goes through. worth saying plainly: once a later batch commits, the unsent one is gone, and the skip is certain rather than "can".

[hubcio]

a few gaps in this list: batch_size is silently clamped to >= 1 and capped server-side by maxTotalHits (default 1000); retry_delay/max_retry_delay get swapped (with a warn) when inverted; polling_interval sleeps before the first search, so the first batch lands one interval late; and the pk cursor only ever sees inserts - updates to already-emitted documents are never re-emitted, which makes "avoids offset pagination skips" read broader than the actual guarantee. the update blind spot is the one most worth a sentence.

[hubcio]

ships include_metadata = true while the code default is false and the README says false too - and the sink plus influxdb source both default true. three-way disagreement; whichever way it lands, code, example and README should agree (family parity says flip the code default to true).

[hubcio]

#[allow(dead_code)] isn't needed - error is read in the failed/canceled arm's format string below. the allow would hide a real dead-code warning later.

[hubcio]

re-declares "iggy_messages" while fixtures::TEST_INDEX is exported and is what the fixture actually feeds the connector - the sink test imports it. if TEST_INDEX ever changes, this test indexes into a different index than the connector reads.