You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Due for payment 2026-03-19] [Due for payment 2026-02-27] Sentry spans taking hours/days before they are closed or cancelled #79653
Summary: Sentry performance spans (primarily ManualOpenReport, ActivityIndicator, and SearchRowSkeleton) were running for hours/days without being closed or cancelled, making performance metrics inaccurate. The root causes were: spans not cancelled on background/inactive transitions, spans created for sub-page navigations that had no matching endSpan, and spans created while the app was already in background state.
Root Causes (all addressed by merged PRs):
Background cancellation gap — cancelAllSpans() only fired on active→background, missing cases where appState was undefined (startup) or background (push notification launch). → Fixed by PR #81557
Sub-route regex match — Navigation.ts regex had no $ end anchor, so sub-page routes like r/123/edit/policyField created ManualOpenReport spans with no corresponding endSpan(). → Fixed by PR #84595
Background span creation — Spans could still be created while the app was in background/inactive state. Also lacked foreground-return cleanup. → Fixed by PR #84736
Current Defense Layers (code as of latest merged PRs):
startSpan() guards against AppState !== 'active' (activeSpans.ts:22-24)
cancelAllSpans() fires on both background entry AND foreground return (App.ts:254-275)
Navigation.navigate() regex only matches exact report-open routes (Navigation.ts:307-329)
startSpan() auto-cancels any existing span with the same ID before creating a new one
Remaining Gaps / Open Questions
No span TTL/timeout — There is no explicit maximum duration enforced on spans. If a span leaks through all cancellation paths (e.g., ReportScreen stays mounted but data never loads), the span persists indefinitely in the activeSpans Map until the Sentry SDK's own idle transaction timeout fires. A max-duration safety net could be valuable.
endSpan before startSpan race condition — sosek108 observed cases where endSpan() is called on a span that doesn't yet exist, followed by the span being created 2 seconds later. The new startSpan() auto-cancel for duplicate IDs helps, but if the endSpan fires before the corresponding startSpan, the span will never be ended. This appears to be a narrow edge case.
ActivityIndicator / image loading — Long-running ActivityIndicator spans caused by images that never finish loading remain a separate issue. PR #82927 added diagnostic reasonAttributes, but the root cause of slow image loading hasn't been fixed. This was proposed to be split into a separate issue.
Web visibility edge cases — On web, AppState maps to visibilitychange. Minimizing the browser window (without switching tabs) may not trigger a visibility change on all browsers, meaning spans could continue running in a minimized window.
Detailed Investigation
Span Lifecycle
The app tracks all Sentry spans in a centralized activeSpans Map (src/libs/telemetry/activeSpans.ts). Each entry contains the Sentry span object and a performance.now() start time.
ManualOpenReport flow:
Start: User taps a report in LHN (OptionRowLHN.tsx:228), Search results (Search/index.tsx:1024), or MoneyRequestReportPreview (index.tsx:109). Navigation.navigate() also starts spans for exact report routes matching /^(search\/(?:r|view)|r|e)\/(\w+)$/
End (warm): If data is cached, ReportActionsView.onLayout fires → markOpenReportEnd(report, {warm: true}) → endSpan(spanId) (ReportActionsView.tsx:348)
End (cold): When skeleton resolves → markOpenReportEnd(report, {warm: false}) → endSpan(spanId) (ReportActionsView.tsx:399)
Next Steps: The core span lifecycle issues have been addressed by the deployed PRs. After the regression period (2026-03-19), verify Sentry traces show the expected reduction in >10-minute spans. The ActivityIndicator/image loading issue and the endSpan-before-startSpan race condition remain as potential follow-up items for separate issues.
Activity
melvin-bot commentedon Jan 21, 2026
@sosek108 Eep! 4 days overdue now. Issues have feelings too...
rlinoz commentedon Jan 21, 2026
This is still true, we are trying to add logs to check what is happening, rigth @sosek108 ?
melvin-bot commentedon Feb 3, 2026
This issue has not been updated in over 14 days. @sosek108 eroding to Weekly issue.
rlinoz commentedon Feb 4, 2026
This is still daily, we just added a way to cancel all spans when moving to backgroundc though, so this might be solved.
rlinoz commentedon Feb 10, 2026
Hey @sosek108 as we discussed things are much better after the last PR https://expensify.sentry.io/explore/traces/?cursor=0%3A0%3A1&environment=staging&field=id&field=span.name&field=span.description&field=span.duration&field=transaction&field=timestamp&field=tags%5Bcanceled%2Cnumber%5D&project=4510228107427840&query=release%3Anew.expensify%409.3.16-0%20span.duration%3A%3E10min%20%21has%3Atags%5Bcanceled%2Cnumber%5D%20span.status%3Aok&statsPeriod=24h
But apparently we don't cancel a lot of the loading spans, are you able to look into it?
sosek108 commentedon Feb 10, 2026
Yes, I have this on my backlog
62 remaining items
melvin-bot commentedon Mar 13, 2026
Triggered auto assignment to @mallenexpensify (
Awaiting Payment)MelvinBot commentedon Mar 13, 2026
🔍 Issue Analysis
Summary: Sentry performance spans (primarily
ManualOpenReport,ActivityIndicator, andSearchRowSkeleton) were running for hours/days without being closed or cancelled, making performance metrics inaccurate. The root causes were: spans not cancelled on background/inactive transitions, spans created for sub-page navigations that had no matchingendSpan, and spans created while the app was already in background state.Root Causes (all addressed by merged PRs):
cancelAllSpans()only fired onactive→background, missing cases whereappStatewasundefined(startup) orbackground(push notification launch). → Fixed by PR #81557Navigation.tsregex had no$end anchor, so sub-page routes liker/123/edit/policyFieldcreatedManualOpenReportspans with no correspondingendSpan(). → Fixed by PR #84595Current Defense Layers (code as of latest merged PRs):
startSpan()guards againstAppState !== 'active'(activeSpans.ts:22-24)cancelAllSpans()fires on both background entry AND foreground return (App.ts:254-275)ReportScreenunmount effect cancels pending spans (ReportScreen.tsx:641-650)Navigation.navigate()regex only matches exact report-open routes (Navigation.ts:307-329)startSpan()auto-cancels any existing span with the same ID before creating a new oneRemaining Gaps / Open Questions
No span TTL/timeout — There is no explicit maximum duration enforced on spans. If a span leaks through all cancellation paths (e.g.,
ReportScreenstays mounted but data never loads), the span persists indefinitely in theactiveSpansMap until the Sentry SDK's own idle transaction timeout fires. A max-duration safety net could be valuable.endSpanbeforestartSpanrace condition —sosek108observed cases whereendSpan()is called on a span that doesn't yet exist, followed by the span being created 2 seconds later. The newstartSpan()auto-cancel for duplicate IDs helps, but if theendSpanfires before the correspondingstartSpan, the span will never be ended. This appears to be a narrow edge case.ActivityIndicator / image loading — Long-running
ActivityIndicatorspans caused by images that never finish loading remain a separate issue. PR #82927 added diagnosticreasonAttributes, but the root cause of slow image loading hasn't been fixed. This was proposed to be split into a separate issue.Web visibility edge cases — On web,
AppStatemaps tovisibilitychange. Minimizing the browser window (without switching tabs) may not trigger a visibility change on all browsers, meaning spans could continue running in a minimized window.Detailed Investigation
Span Lifecycle
The app tracks all Sentry spans in a centralized
activeSpansMap (src/libs/telemetry/activeSpans.ts). Each entry contains the Sentry span object and aperformance.now()start time.ManualOpenReport flow:
OptionRowLHN.tsx:228), Search results (Search/index.tsx:1024), orMoneyRequestReportPreview(index.tsx:109).Navigation.navigate()also starts spans for exact report routes matching/^(search\/(?:r|view)|r|e)\/(\w+)$/ReportActionsView.onLayoutfires →markOpenReportEnd(report, {warm: true})→endSpan(spanId)(ReportActionsView.tsx:348)markOpenReportEnd(report, {warm: false})→endSpan(spanId)(ReportActionsView.tsx:399)Evidence Chain
$in regex/^(?:search\/)?(?:r|e)\/(\w+)/matchedr/123/edit/policyFieldManualOpenReportspans for sub-pages never ended → hours-long spanscancelAllSpansonly onactive→backgroundappState === 'active'was false whenappStatewasundefinedAppStatecheck instartSpan()cancelAllSpans()on→activeRelevant Code
App/src/libs/telemetry/activeSpans.ts:22-42—startSpanwith AppState guard and auto-cancelApp/src/libs/telemetry/activeSpans.ts:44-59—endSpanwith safe no-op for missing spansApp/src/libs/telemetry/activeSpans.ts:74-86—cancelAllSpansandcancelSpansByPrefixApp/src/libs/actions/App.ts:254-275— AppState listener with bidirectional cancelApp/src/libs/Navigation/Navigation.ts:307-329— Route regex with$anchorApp/src/pages/inbox/ReportScreen.tsx:641-650— Unmount cleanupApp/src/libs/telemetry/markOpenReportEnd.ts:13-32— Span end with report metadataStatus of Deployed Fixes
Next Steps: The core span lifecycle issues have been addressed by the deployed PRs. After the regression period (2026-03-19), verify Sentry traces show the expected reduction in >10-minute spans. The ActivityIndicator/image loading issue and the
endSpan-before-startSpanrace condition remain as potential follow-up items for separate issues.melvin-bot commentedon Mar 19, 2026
Payment Summary
Upwork Job
BugZero Checklist (@mallenexpensify)
mallenexpensify commentedon Mar 20, 2026
@ShridharGoel is this $125 cuz tests weren't done? I don't see em here
ShridharGoel commentedon Mar 20, 2026
Yes, that sounds good
mallenexpensify commentedon Mar 23, 2026
Payment Summary
Contributor+: @ShridharGoel due $125 via NewDot
I don't think we need a test case here, if so, comment/reopen
JmillsExpensify commentedon Apr 6, 2026
$125 approved for @ShridharGoel