fix(connectors): bring quickwit_sink up to convention

[mfyuce]

Summary

• Defer index_id YAML parse to open() — return InvalidConfigValue instead of panicking via expect()
• Use ClientWithMiddleware (build_retry_client) for ingest retries with exponential backoff on 5xx / connection errors
• check_connectivity_with_retry in open() against /health/livez
• Map 4xx responses (except 429) to PermanentHttpError so the circuit breaker is not tripped by bad payloads
• Add verbose_logging / max_retries / retry_delay / max_retry_delay / max_open_retries / open_retry_max_delay to QuickwitSinkConfig (all optional, forward-compatible)
• Downgrade per-batch consume() log to debug! (upgraded to info! when verbose_logging = true)
• Set Content-Type: application/x-ndjson on ingest requests
• Drop unused dashmap / once_cell deps; add reqwest-middleware
• 5 unit tests covering verbose flag, client init state, and index_id init state

Test plan

• cargo clippy -p iggy_connector_quickwit_sink --all-targets -- -D warnings — clean
• cargo test -p iggy_connector_quickwit_sink — 5 tests pass
• Integration tests: cargo test -p integration -- connectors::quickwit

🤖 Generated with Claude Code

[github-actions]

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

• /ready - back to S-waiting-on-review after addressing feedback
• /author - flip to S-waiting-on-author while you finish changes
• /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

[ryerraguntla]

lib.rs:ingest() — create_index() treats 409 Conflict as InitError; concurrent open() calls (multi-instance, restart race) both see has_index()=false, both POST, second gets 4xx → InitError → connector
never opens. Fix: absorb 409 (and 400 "already exists") as Ok(()) in create_index(). Retry middleware also retries a 5xx create that succeeded server-side; on retry, server returns 409 → same path. Same fix
covers both.

[mfyuce]

Sorry for the late response -- I was running a local benchmark to make sure the setup is working end-to-end.

Fixed in 512b71f04: create_index now handles 409 CONFLICT as a success case -- another instance beat us to it, but the index exists, which is what we wanted. Only other 4xx errors propagate as InitError.

[ryerraguntla]

reqwest::Client::new() has no timeout; health probe and has_index()/create_index() can hang indefinitely under network partition or slow-starting Quickwit. Fix: reqwest::Client::builder().timeout(...) with configurable or sensible default (e.g. 30s)

[mfyuce]

Fixed in 512b71f04: added request_timeout: Option<String> to QuickwitSinkConfig (default "30s") and wired it into reqwest::Client::builder().timeout(request_timeout).build() in open(). The example config TOML has the field commented in as a reference.

[ryerraguntla]

@mfyuce - all the above are minors. I am not sure of quickwit s production data set distribution to mention about the need for circuit breakers (if there are huge number of records/documents for the same cursor key for a given batch size) . Please make a judgement call about the need for circuit breaker implementation . otherwise it is all set for second reviewer's comments before merging.

[ryerraguntla]

/author

[mfyuce]

Thanks for the thorough review @ryerraguntla! Both issues addressed in the latest push:

409 / "already exists" on create_index() — create_index() now absorbs 409 CONFLICT and 400 BAD_REQUEST whose body contains "already exists" as Ok(()) with an info! log. Covers the concurrent-open race and the retry-after-succeeded-5xx path.

No timeout on reqwest::Client — Added request_timeout: Option<String> to QuickwitSinkConfig (default 30s, configurable via TOML). The raw client is now built with Client::builder().timeout(request_timeout), bounding health probes, has_index(), and create_index() under network partition.

/ready

[mfyuce]

@ryerraguntla — judgment call on the circuit breaker:

The existing build_retry_client wraps a HttpRetryMiddleware that already retries 429 and 5xx with exponential backoff, and the PermanentHttpError mapping cuts retries for permanent 4xx errors (bad data won't hammer the backend). Ingest throughput is also naturally rate-bounded by batch_length and poll_interval.

QuickWit is typically an internal service in the same cluster, so prolonged partition is rare, and the runtime already isolates failures per connector. A full half-open / trip-threshold circuit breaker on top of this would add complexity without clear benefit for this specific sink. Happy to revisit if the second reviewer sees a concrete failure mode that the existing retry policy doesn't cover.

[codecov]

Codecov Report

❌ Patch coverage is 67.47573% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.65%. Comparing base (d6059a1) to head (329a751).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
core/connectors/sinks/quickwit_sink/src/lib.rs	67.47%	67 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #3523       +/-   ##
=============================================
- Coverage     74.13%   17.65%   -56.48%     
  Complexity      937      937               
=============================================
  Files          1258     1255        -3     
  Lines        131452   114439    -17013     
  Branches     107320    90352    -16968     
=============================================
- Hits          97455    20208    -77247     
- Misses        30909    93739    +62830     
+ Partials       3088      492     -2596     

Components	Coverage Δ
Rust Core	2.57% <67.47%> (-72.22%)	⬇️
Java SDK	62.44% <ø> (ø)
C# SDK	71.40% <ø> (-0.71%)	⬇️
Python SDK	88.88% <ø> (ø)
PHP SDK	84.29% <ø> (ø)
Node SDK	91.35% <ø> (+0.22%)	⬆️
Go SDK	40.14% <ø> (ø)

Files with missing lines	Coverage Δ
core/connectors/sinks/quickwit_sink/src/lib.rs	58.23% <67.47%> (-9.59%)	⬇️

... and 746 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kriti-sc]

Can you please add a brief rationale/problem statement/why behind the change? I am finding the PR hard to review without clarity on the goal being achieved.
@mfyuce

[mfyuce]

Sorry for the late response -- was doing a local benchmark to make sure the setup is working.

@kriti-sc good call, here is the rationale:

The original quickwit_sink was missing several patterns that exist in every other production-grade connector in this repo. Without them, it would fail silently or hang in real deployments:

1. No request timeout -- reqwest::Client::new() has no timeout. Under network partition, has_index(), create_index(), and ingest() block forever. This PR adds request_timeout (default 30 s).

2. No connectivity check on open() -- the connector would report Running in the runtime's /stats endpoint even if QuickWit was unreachable, then fail silently on first ingest. This PR adds the same check_connectivity_with_retry probe that postgres_sink, http_sink, and elasticsearch_sink all use.

3. No retry middleware -- transient 5xx or 429 responses from QuickWit caused immediate batch failure and offset advancement. This PR wires HttpRetryMiddleware with exponential backoff, matching the behavior of the other HTTP-based sinks.

4. 409 Conflict on create_index() crashed open() -- in a multi-instance or restart race, two connectors can call create_index() simultaneously. The second one got a 409 and propagated it as an InitError, killing the connector. This PR absorbs 409 (and "already exists" 400) as Ok(()).

The goal is to bring quickwit_sink to the same robustness level as postgres_sink before it sees production traffic.

[kriti-sc]

instead of 30s, define a const like the DEFAULT_* defined at the top of this file, and use that here

[mfyuce]

Fixed in 3807e63. Added const DEFAULT_REQUEST_TIMEOUT: &str = "30s"; at the top of the file and updated the parse_duration call in open() to use it as the fallback. Also added explicit logs for server (5xx) versus client (4xx) errors on index creation.

[kriti-sc]

this and the else arm below are the same. is this intentioanl? what kind of errors is the else arm expected to catch?

also, would be good to add an error! to the else arm.

[mfyuce]

Good catch. The two arms are intentional -- 4xx (client errors) are permanent failures such as invalid index YAML or a misconfigured URL; 5xx (server errors) are transient and normally handled by the retry middleware, but they surface here when retries are exhausted.

The error! on the else arm was missing by mistake. Fixed in the latest push: both arms now log at error! level with distinct messages (Permanent client error vs Server error) so they are easy to tell apart in operator logs.

[kriti-sc]

these errors are the same pattern as in create_index above. here PermanentError and HttpRequestFailed error is used, while above InitError. What is the difference and is this intentional?

[mfyuce]

Intentional -- the two functions are called from different lifecycle phases:

• create_index is called from open() (the initialization phase). Failure there means the connector cannot start at all, so InitError is correct -- the runtime treats it as a fatal setup failure.
• ingest is called from consume() (the operation phase). Failure there is an HTTP-level error during normal operation, so HttpRequestFailed is correct.

The distinction matters for how the runtime handles each: InitError aborts the connector on first open; HttpRequestFailed during consume is subject to the retry/backoff policy.

[kriti-sc]

PR largely ok. A few minor changes + justification of some changes requested.

[mfyuce]

@kriti-sc The requested DEFAULT_REQUEST_TIMEOUT constant and explicit HTTP 4xx/5xx logging changes have been implemented and pushed. Ready for review!

[mfyuce]

/ready

[ryerraguntla]

/author please check the prechecks issues and resolve the conflicts

[mfyuce]

/ready

Hi @ryerraguntla,

I have resolved the conflicts and cleaned up the branch:

1. Rebase: Rebased the branch on origin/master (upstream master) and cleanly isolated the quickwit connector commits, purging any unrelated commits.
2. Prechecks: Verified that formatting (cargo fmt), clippy, whitespaces, and newlines checks pass perfectly.
3. Payload Handling: Brought the connector further up to project conventions by adding support for Payload::Raw (tries parsing JSON, falls back to Base64 wrapping if binary/non-JSON) and Payload::Text (wrapped inside a JSON object) to match the elasticsearch connector behavior.
4. Performance: Optimized the NDJSON generation logic during ingestion to build the payload using a single string buffer and avoid intermediate vector allocations.
5. Testing: Added comprehensive unit tests for the conversion logic. All 11 unit tests pass cleanly.

[ryerraguntla]

@mfyuce - while I review, please add the issue it is closing and the motivation for implementing. Follow the published Iggy PR template . Try to add code coverage to the maximum possible with out significantly increasing the test time.

[hubcio]

this is a clear improvement overall (drops the old panic, adds retry + a startup probe, handles raw/text payloads). inline notes are the should-fix items.

PR description nits (no line to anchor to): says "5 unit tests" but there are 11; the "map 4xx so the circuit breaker is not tripped" line describes a mechanism this sink doesn't have (no circuit breaker here); and "drop unused dashmap / once_cell" - once_cell was never a dependency here, only dashmap was.

one runtime-wide caveat, already tracked so not this pr's job: a permanently-failing consume() batch is silently dropped because the offset is already committed at poll and the FFI return code is ignored. that's #2927 (consume return discarded) + #2928 (commit-before-processing ordering) - together they make at-least-once unachievable for any sink today, which is worth keeping in mind next to the at-least-once comment inline. one small thing not yet noted on #2927: the dropped batch is still counted by the messages-processed metric, so dashboards read clean while data is lost.

[hubcio]

QuickwitSinkConfig is missing #[serde(deny_unknown_fields)]. with every knob optional, a typo'd key like max_retires is silently ignored and the connector just runs on defaults with no error. influxdb_sink (the sink this mirrors) sets it.

two smaller things on this struct: the Serialize derive is unused (the config is only ever deserialized; the runtime serializes an untyped value, not this type) so it can be dropped, and the retry/timeout knobs are undocumented while http_sink documents each field with its default.

[hubcio]

max_retry_delay reads backwards from the rest. influxdb, where this startup-probe path is copied from, calls the same knob retry_max_delay and pairs it with open_retry_max_delay. worth renaming the field + the DEFAULT_MAX_RETRY_DELAY const to match, since config keys turn into a compat contract once this ships.

[hubcio]

every other http sink names this timeout (influxdb, http_sink, doris) - request_timeout is the lone outlier. rename the field + DEFAULT_REQUEST_TIMEOUT const before release, config keys are hard to change later.

[hubcio]

this comment only covers the duplicate-write window. it's silent on the loss window: on a permanent 4xx, or once retries are exhausted, the offset was already committed at poll, so the batch is dropped and never redelivered. worth stating both so the real guarantee (at-most-once on final failure) is clear.

[hubcio]

these urls (here, plus has_index and create_index) are rebuilt with format! on every batch, and none trim a trailing slash - a url ending in / produces //api/v1/.... build them once in open() and trim_end_matches('/') the base, like influxdb's base_url().

[hubcio]

ndjson starts from String::new() and reallocs as it grows over the batch (up to batch_length records). pre-size it with String::with_capacity(...) - the commit says "optimize allocations" but this is the main alloc site.

separately: records where to_string fails get skipped silently while messages_count still counts them, so the success log can overcount.

[hubcio]

this permanent-vs-transient split has no runtime effect: the error is collapsed to a return code at the ffi boundary and the runtime discards it, and this sink owns no circuit breaker - so the description's "so the circuit breaker is not tripped" doesn't apply here. the two distinct log lines are still handy for triage, keep those; it's just the error variant that doesn't gate anything.

[hubcio]

these two arms build an identical Err(InitError(...)) and differ only in the log wording, and the status is already in the message. collapse them into one else. keep the success and 409 arms separate - the 409-absorb is load-bearing for the create race.

[hubcio]

String::from_utf8(bytes.clone()) clones a second time. from_utf8(bytes) can consume the buffer, and on the error path e.into_bytes() hands it back for the base64 branch. the earlier clone at the parse call is load-bearing (simd parses destructively) so keep that one.

[hubcio]

/health/livez is liveness - the node can be live but not yet ready to serve. /health/readyz is the readiness gate you actually want before create_index/ingest. minor, since the retry client papers over an early 5xx, but readyz is the more correct probe.