[alongubkin]

Problem

The OTLP HTTP exporter made one attempt per batch. A transient connection failure or retryable collector response could discard logs during a rollout.

Solution

• Upgrade the OpenTelemetry crates from 0.30 to 0.32.
• Enable the upstream experimental HTTP retry policy instead of maintaining a custom HTTP client.
• Retry at most three times with bounded exponential delay and jitter.
• Bound each export attempt to 750 ms and retain the existing explicit shutdown flush.

Retries are in memory. Process termination or retry exhaustion can still lose records that were not acknowledged by the collector.

Verification

• cargo check -p alien-worker-runtime --all-features
• The upstream retry implementation was traced before selecting the upgrade.
• No external telemetry service or production system was changed.

ALIEN-850

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the previously reported response-body retry gap is fixed and no new actionable issue was found.

Summary

The PR adds bounded, in-memory retry handling for transient OTLP log delivery failures.

• Reuses identical serialized batches across retry attempts.
• Retries transient transport failures and selected retryable HTTP statuses within a 60-second delivery budget.
• Honors Retry-After, exponential backoff, jitter, and per-attempt timeouts.
• Extends explicit logger shutdown to accommodate the retry budget.
• The latest revision correctly routes truncated successful response bodies back through the retry loop and adds targeted wire-level coverage.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Serialized OTLP log batch] --> B[Send HTTP attempt]
    B --> C{Result}
    C -->|Success with readable body| D[Return response]
    C -->|Permanent status or permanent error| E[Return without retry]
    C -->|429, 502, 503, 504| F[Calculate Retry-After or backoff with jitter]
    C -->|Transient transport failure| F
    C -->|Truncated success body| F
    F --> G{Delivery budget remains?}
    G -->|Yes| B
    G -->|No| H[Return timeout error]

Loading

_{Reviews (2) · Last reviewed commit: "fix: retry incomplete OTLP responses wit..."}

[alongubkin]

@greptileai please re-review the latest commit. The findings from the first review have been addressed and verified with targeted tests.