Skip to content

[Due for payment 2026-03-19] [Due for payment 2026-02-27] Sentry spans taking hours/days before they are closed or cancelled #79653

Closed

Description

@rlinoz

We can see it here https://expensify.sentry.io/explore/traces/?environment=staging&environment=production&project=4510228107427840&query=span.duration%3A%3E10h&statsPeriod=24h that we have a ton of spans that takes more than 10 hours to get resolved.

This points to a problem in our implementation, but more than that it makes our metrics innacurate. Let's find out what is happening and get it fixed.

Issue OwnerCurrent Issue Owner: @mallenexpensify

Activity

  1. melvin-bot commented on Jan 21, 2026

    @melvin-bot

    @sosek108 Eep! 4 days overdue now. Issues have feelings too...

  2. rlinoz commented on Jan 21, 2026

    @rlinoz
    ContributorAuthor

    This is still true, we are trying to add logs to check what is happening, rigth @sosek108 ?

  3. melvin-bot commented on Feb 3, 2026

    @melvin-bot

    This issue has not been updated in over 14 days. @sosek108 eroding to Weekly issue.

  4. added and removed on Feb 4, 2026
  5. rlinoz commented on Feb 4, 2026

    @rlinoz
    ContributorAuthor

    This is still daily, we just added a way to cancel all spans when moving to backgroundc though, so this might be solved.

  6. self-assigned this
    on Feb 10, 2026
  7. sosek108 commented on Feb 10, 2026

    @sosek108
    Contributor

    Yes, I have this on my backlog

  8. 62 remaining items

  9. melvin-bot commented on Mar 13, 2026

    @melvin-bot

    Triggered auto assignment to @mallenexpensify (Awaiting Payment)

  10. MelvinBot commented on Mar 13, 2026

    @MelvinBot
    Contributor

    🔍 Issue Analysis

    Summary: Sentry performance spans (primarily ManualOpenReport, ActivityIndicator, and SearchRowSkeleton) were running for hours/days without being closed or cancelled, making performance metrics inaccurate. The root causes were: spans not cancelled on background/inactive transitions, spans created for sub-page navigations that had no matching endSpan, and spans created while the app was already in background state.

    Root Causes (all addressed by merged PRs):

    1. Background cancellation gap — cancelAllSpans() only fired on active→background, missing cases where appState was undefined (startup) or background (push notification launch). → Fixed by PR #81557
    2. Sub-route regex match — Navigation.ts regex had no $ end anchor, so sub-page routes like r/123/edit/policyField created ManualOpenReport spans with no corresponding endSpan(). → Fixed by PR #84595
    3. Background span creation — Spans could still be created while the app was in background/inactive state. Also lacked foreground-return cleanup. → Fixed by PR #84736

    Current Defense Layers (code as of latest merged PRs):

    • startSpan() guards against AppState !== 'active' (activeSpans.ts:22-24)
    • cancelAllSpans() fires on both background entry AND foreground return (App.ts:254-275)
    • ReportScreen unmount effect cancels pending spans (ReportScreen.tsx:641-650)
    • Navigation.navigate() regex only matches exact report-open routes (Navigation.ts:307-329)
    • startSpan() auto-cancels any existing span with the same ID before creating a new one

    Remaining Gaps / Open Questions

    1. No span TTL/timeout — There is no explicit maximum duration enforced on spans. If a span leaks through all cancellation paths (e.g., ReportScreen stays mounted but data never loads), the span persists indefinitely in the activeSpans Map until the Sentry SDK's own idle transaction timeout fires. A max-duration safety net could be valuable.

    2. endSpan before startSpan race condition — sosek108 observed cases where endSpan() is called on a span that doesn't yet exist, followed by the span being created 2 seconds later. The new startSpan() auto-cancel for duplicate IDs helps, but if the endSpan fires before the corresponding startSpan, the span will never be ended. This appears to be a narrow edge case.

    3. ActivityIndicator / image loading — Long-running ActivityIndicator spans caused by images that never finish loading remain a separate issue. PR #82927 added diagnostic reasonAttributes, but the root cause of slow image loading hasn't been fixed. This was proposed to be split into a separate issue.

    4. Web visibility edge cases — On web, AppState maps to visibilitychange. Minimizing the browser window (without switching tabs) may not trigger a visibility change on all browsers, meaning spans could continue running in a minimized window.

    Detailed Investigation

    Span Lifecycle

    The app tracks all Sentry spans in a centralized activeSpans Map (src/libs/telemetry/activeSpans.ts). Each entry contains the Sentry span object and a performance.now() start time.

    ManualOpenReport flow:

    1. Start: User taps a report in LHN (OptionRowLHN.tsx:228), Search results (Search/index.tsx:1024), or MoneyRequestReportPreview (index.tsx:109). Navigation.navigate() also starts spans for exact report routes matching /^(search\/(?:r|view)|r|e)\/(\w+)$/
    2. End (warm): If data is cached, ReportActionsView.onLayout fires → markOpenReportEnd(report, {warm: true}) → endSpan(spanId) (ReportActionsView.tsx:348)
    3. End (cold): When skeleton resolves → markOpenReportEnd(report, {warm: false}) → endSpan(spanId) (ReportActionsView.tsx:399)
    4. Cancel paths: Screen unmount, background transition, foreground return, same-report tap, empty state render

    Evidence Chain

    Root Cause Code Evidence Symptom
    No $ in regex Old: /^(?:search\/)?(?:r|e)\/(\w+)/ matched r/123/edit/policyField ManualOpenReport spans for sub-pages never ended → hours-long spans
    cancelAllSpans only on active→background Old guard: appState === 'active' was false when appState was undefined Spans survived background transitions on cold start / push notification launch
    Spans created in background No AppState check in startSpan() Spans created while app inactive were never cancelled (cancel already fired)
    Foreground return stale spans No cancelAllSpans() on →active Spans that survived backgrounding persisted until next navigation

    Relevant Code

    • App/src/libs/telemetry/activeSpans.ts:22-42 — startSpan with AppState guard and auto-cancel
    • App/src/libs/telemetry/activeSpans.ts:44-59 — endSpan with safe no-op for missing spans
    • App/src/libs/telemetry/activeSpans.ts:74-86 — cancelAllSpans and cancelSpansByPrefix
    • App/src/libs/actions/App.ts:254-275 — AppState listener with bidirectional cancel
    • App/src/libs/Navigation/Navigation.ts:307-329 — Route regex with $ anchor
    • App/src/pages/inbox/ReportScreen.tsx:641-650 — Unmount cleanup
    • App/src/libs/telemetry/markOpenReportEnd.ts:13-32 — Span end with report metadata

    Status of Deployed Fixes

    PR Version Status
    #81557 v9.3.16+ ✅ Deployed
    #82913 v9.3.23-3 ✅ Deployed, payment issued
    #84595 v9.3.36+ ✅ Merged
    #84736 v9.3.36-10 ✅ Deployed, in regression period until 2026-03-19

    Next Steps: The core span lifecycle issues have been addressed by the deployed PRs. After the regression period (2026-03-19), verify Sentry traces show the expected reduction in >10-minute spans. The ActivityIndicator/image loading issue and the endSpan-before-startSpan race condition remain as potential follow-up items for separate issues.

  11. melvin-bot commented on Mar 19, 2026

    @melvin-bot

    Payment Summary

    Upwork Job

    BugZero Checklist (@mallenexpensify)

    • (if NewFeature) I have created a PR for any necessary HelpDot updates, or I confirmed no updates are necessary.
    • I have verified the correct assignees and roles are listed above and updated the necessary manual offers
    • I have verified that there are no duplicate or incorrect contracts on Upwork for this job (https://www.upwork.com/ab/applicants//hired)
    • I have verified the PR was not reverted
    • I have applied any discounts due to bugs/regressions introduced by this PR
    • I have paid out the Upwork contracts or cancelled the ones that are incorrect
    • I have verified the payment summary above is correct
  12. mallenexpensify commented on Mar 20, 2026

    @mallenexpensify
    Contributor

    @ShridharGoel is this $125 cuz tests weren't done? I don't see em here

  13. ShridharGoel commented on Mar 20, 2026

    @ShridharGoel
    Contributor

    Yes, that sounds good

  14. mallenexpensify commented on Mar 23, 2026

    @mallenexpensify
    Contributor

    Payment Summary

    Contributor+: @ShridharGoel due $125 via NewDot

    I don't think we need a test case here, if so, comment/reopen

  15. JmillsExpensify commented on Apr 6, 2026

    @JmillsExpensify
    Contributor

    $125 approved for @ShridharGoel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Awaiting PaymentAuto-added when associated PR is deployed to productionBugSomething is broken. Auto assigns a BugZero manager.DailyKSv2

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions