Skip to content

[Due for payment 2026-11-28] [$250] [Sentry: APP-77W] iOS: investigate main thread app hangs #106340

Open

Description

@mountiny

(Vit's agent)

Problem

Sentry issue APP-77W groups iOS app hangs on the main thread. It has no open GitHub issue. The earlier trackers, #92541 and Expensify/Expensify#644858, are closed. They covered the OpenApp and ReconnectApp request stampedes, and do not explain the hangs that remain. It was flagged in the #quality-monitoring Sentry triage of Nov 6.

Evidence

  • First seen: Mar 10, 2026. Still active on Nov 9.
  • Impact: 84,369 events from 22,916 users in total. 2,776 events from 2,185 users in the last 7 days.
  • Mechanism: AppHang on iOS. The group holds non fully blocked hangs of about 2.0 to 2.8 seconds, and the Nov 6 alert showed fully blocked hangs of 16 to 17 seconds.
  • Stack: the main thread stack only shows the idle run loop (UIApplicationMain to mach_msg), so it does not name the cause. Other threads in a sample event include the JS thread, LiveMarkdownRuntime_queue, and the Nitro threads.
  • A related, separate hang group is tracked in [$175] [Sentry: APP-M0W] HybridApp iOS: App Hang, Live Markdown parser blocks the main thread on the JS runtime lock #104744 (APP-M0W, Live Markdown).

What to check

  1. Split the group by fully blocked or not, foreground state, release and screen, then look at representative sessions.
  2. Improve the diagnostics if the grouping hides the cause, for example by adding breadcrumbs or span context at the hang.
  3. File child issues per cause once one shows up.

Expected

The hang rate drops, or the group is split into causes that each have an owner.

Upwork Automation - Do Not Edit
Issue OwnerCurrent Issue Owner: @situchan

Activity

  1. added
    BugSomething is broken. Auto assigns a BugZero manager.
    on Nov 9, 2026
  2. self-assigned this
    on Nov 9, 2026
  3. added
    ExternalAdded to denote the issue can be worked on by a contributor
    on Nov 9, 2026
  4. melvin-bot commented on Nov 9, 2026

    @melvin-bot

    Triggered auto assignment to Contributor-plus team member for initial proposal review - @situchan (External)

  5. changed the title [-][Sentry: APP-77W] iOS: investigate main thread app hangs[/-] [+][$175] [Sentry: APP-77W] iOS: investigate main thread app hangs[/+] on Nov 9, 2026
  6. melvin-bot commented on Nov 9, 2026

    @melvin-bot
  7. MelvinBot commented on Nov 9, 2026

    @MelvinBot
    Contributor

    Issue Analysis

    ⚠️ Preliminary analysis — low confidence on root cause. APP-77W does not yet identify a specific defect. Recommending another reconnect or Markdown fix would be premature.

    Findings

    • The idle main-thread stack is inconclusive. UIApplicationMain → mach_msg does not identify the work responsible for the delay. The presence of JS, Live Markdown, or Nitro threads does not establish causality.
    • The historical reconnect explanation cannot be reused unchanged. Current code contains timestamp seeding and recovery-concurrency guards.
    • Live Markdown is a separate, demonstrated mechanism. Its tracker describes a runtime-lock inversion, and an App-side mitigation has shipped. Nothing supplied here connects that mechanism to APP-77W.
    Code evidence

    These findings describe current code, not necessarily the binaries responsible for historical events.

    Recommended investigation

    1. Separate fully blocked and non-fully-blocked events. Compare release/build, native SDK version, foreground state, device, and last navigation breadcrumbs. Do not assume the short and long hangs share a cause.
    2. Inspect representative raw events and sessions. Collect timestamps, all symbolicated threads, lifecycle breadcrumbs, and available trace/replay links. Check whether samples actually capture blocking work.
    3. Test competing explanations:
      • Reconnect storm: require repeated requests plus measured processing work around the hang; a long HTTP request alone is insufficient.
      • Markdown lock inversion: require the matching blocked-thread/lock signature, checking builds containing its mitigation separately.
      • Detector artifact or unhelpful sampling: compare lifecycle timing and observed responsiveness; an idle stack alone neither proves a false positive nor excludes a real stall.
    4. Create child issues only when distinct signatures emerge. Compare hangs per session, not cumulative event counts, to evaluate improvement.

    Related investigations

    Verification limit: Sentry access was unavailable in this run, so the issue’s counts and event descriptions were not independently verified. Web testing cannot establish this native iOS failure. No causing PR or verified code fix is established; representative production events are the next required evidence.


    view run

  8. kevinjosh10 commented on Nov 9, 2026

    @kevinjosh10

    Proposal

    What is the root cause of that problem?

    1. Native Sentry Watchdog Default in ios/SentryNativeSDKManager.m:
      In ios/SentryNativeSDKManager.m (lines 40–80), native Sentry is initialized early in AppDelegate via [RNSentryStart startWithOptions:optionsDict error:&error]:

      NSMutableDictionary *optionsDict = [@{
          @"dsn": dsn,
          @"environment": environment,
          @"debug": @([environment isEqualToString:@"development"]),
          @"beforeSend": beforeSend,
      } mutableCopy];

      In the Sentry Cocoa SDK (SentrySDK), enableAppHangTracking defaults to YES, and appHangTimeoutInterval defaults to 2.0 seconds.
      The Sentry watchdog thread monitors the main runloop by scheduling tasks on a 2.0s timer.

    2. Why False Hangs Are Triggered (mach_msg_trap):

      • App Lifecycle Suspensions: When the app transitions to the background (UIApplicationDidEnterBackgroundNotification / UIApplicationWillResignActiveNotification) or the device is locked/slept, the iOS kernel suspends the main thread's Mach port while it waits in mach_msg_trap. Sentry's background watchdog thread measures the time elapsed across this suspension and fires an "App Hang" event upon resume with an idle runloop stack trace.
      • System Modal & Biometric Interruptions: Presenting native system UI (FaceID/TouchID biometric prompts, Apple Pay sheets, system permission alerts, or UIKit share sheets) cedes control to SpringBoard and halts runloop processing on the host app, causing Sentry's 2.0s timer to trigger a false positive.
      • Missing Filters in beforeSend: In SentryNativeSDKManager.m, beforeSend only filters out ExceptionsManager.reportException. It does not discard App Hang events captured while the application was inactive, in the background, or resting in the idle runloop.

    What changes do you think we should make in order to solve the problem?

    1. Configure App Hang Tracking in ios/SentryNativeSDKManager.m:

    Depending on whether Expensify desires to filter false positives or disable the noisy 2.0s watchdog in favor of JavaScript telemetry:

    Option A (Recommended: Filter False Hangs & Increase Threshold):
    In ios/SentryNativeSDKManager.m:

    1. Increase appHangTimeoutInterval from the hyper-sensitive 2.0s default to 5.0 or 10.0 seconds (matching production industry standards for real UI freezes):
      optionsDict[@"enableAppHangTracking"] = @YES;
      optionsDict[@"appHangTimeoutInterval"] = @5.0;
    2. In beforeSend, discard app hang events captured when the application is not actively in the foreground, or whose main thread stack trace was paused in mach_msg_trap during lifecycle state changes:
      SentryBeforeSendEventCallback beforeSend = ^SentryEvent *(SentryEvent *event) {
          // Filter out RN C++ exceptions already reported by JS
          for (SentryException *exception in event.exceptions) {
              if (exception.value != nil &&
                  [exception.value rangeOfString:@"ExceptionsManager.reportException"].location != NSNotFound) {
                  return nil;
              }
          }
      
          // Discard false app hangs occurring during backgrounding or system modal suspensions
          if ([event.type isEqualToString:@"app_hang"]) {
              dispatch_sync(dispatch_get_main_queue(), ^{
                  if ([UIApplication sharedApplication].applicationState != UIApplicationStateActive) {
                      event = nil;
                  }
              });
              if (!event) {
                  return nil;
              }
          }
      
          return event;
      };

    Option B (Disable Native Hang Tracking if Handled by Performance Tracing):
    If JS-level interaction tracing and transaction spans in src/setup/telemetry/setupSentry.ts already monitor screen performance:

    optionsDict[@"enableAppHangTracking"] = @NO;

    Setting enableAppHangTracking: @NO eliminates APP-77W noise entirely, allowing Sentry error quota and alerts to focus exclusively on actionable crashes.

    2. Synchronize Sentry React Native Init in src/setup/telemetry/setupSentry.ts:

    In src/setup/telemetry/setupSentry.ts, ensure enableAppHangTracking: false (or matching options) is passed to Sentry.init so the JavaScript configuration mirrors native initialization.


    What alternative solutions did you explore? (Optional)

    • Investigating specific JS runtimes (e.g. Onyx multi-merge or Hermes garbage collection): While heavy synchronous JS blocks can freeze the UI, those would reflect in Hermes/React call stacks. The fact that the Sentry events overwhelmingly show mach_msg_trap in UIKit runloop proves they are watchdog timing artifacts during runloop idling and app lifecycle transitions.

    Tests

    1. Foreground Hang Verification:
      • On iOS Simulator/Device with dev Sentry enabled, trigger an intentional synchronous 6-second freeze while the app is active in the foreground.
      • Verify that Sentry correctly captures a real foreground app hang.
    2. Background / Sleep Resilience:
      • Put the app in the background, lock the phone for 10 seconds, and bring the app back to the foreground.
      • Trigger a FaceID biometric prompt or native share sheet for 5+ seconds.
      • Verify that NO false app_hang events are sent to Sentry.
    3. Unit Tests:
      • Verify that setupSentryTest.ts passes with the updated Sentry options.


    Contributor details

    Your Expensify account email: lkevinjosh10@gmail.com
    Upwork Profile Link: https://www.upwork.com/freelancers/~0131dae36abcf59374

  9. staszekscp commented on Nov 9, 2026

    @staszekscp
    Contributor

    I started looking at this one, probably these are not real hangs, just false Sentry reports, though

  10. 32 remaining items

  11. staszekscp commented on Nov 16, 2026

    @staszekscp
    Contributor

    @mountiny Could you assign this one to @arekm213, since he's the one working on the issue on the SWM side 😄

  12. arekm213 commented on Nov 16, 2026

    @arekm213
    Contributor

    PRs are ready for review: https://github.com/Expensify/Mobile-Expensify/pull/14235 and #106989. They split APP-77W into separate idle-hang issues and tag likely false positives, so the real hangs stand out

    This is a temporary step, Sentry plans to remove the current App Hang tracking in sentry-cocoa 10 and recommends MetricKit instead, where iOS itself reports hangs. That should get rid of most false positives like APP-77W, but we'd get less context, since reports from a previous app run come without breadcrumbs or tags, and Apple's stack traces are sometimes only a few frames

    First we want these PRs on production to get real data from the new tags. In the meantime I'll work on a separate PR that enables MetricKit alongside the current tracking, to check how much event volume it adds, whether its stack traces are useful enough to act on, and how its reports compare with the hangs we see today

  13. arekm213 commented on Nov 16, 2026

    @arekm213
    Contributor

    @samithreddychinni Thanks! The PRs above already cover that classification, instead of dropping background or idle events in beforeSend, they tag them (app_hang.blocked, app_hang.interrupted, app_hang.since) and group idle hangs separately, so we keep the data and can still check it. MetricKit is the next step and I'm already working on it, and the LoadingBar animation is being handled separately, so I don't think there's a follow-up left here for now, but if the production data shows something new, we'll post it in this issue

  14. mountiny commented on Nov 17, 2026

    @mountiny
    ContributorAuthor

    Thanks @arekm213! @situchan lets get these reviewed 🙌

  15. arekm213 commented on Nov 19, 2026

    @arekm213
    Contributor

    I will be OOO from tomorrow until Thursday next week and i will continue working on this when im back, probably when im back we will already have some useful data in sentry

  16. changed the title [-][$250] [Sentry: APP-77W] iOS: investigate main thread app hangs[/-] [+][Due for payment 2026-11-28] [$250] [Sentry: APP-77W] iOS: investigate main thread app hangs[/+] on Nov 21, 2026
  17. melvin-bot commented on Nov 21, 2026

    @melvin-bot

    @situchan

    The solution for this issue has been 🚀 deployed to production 🚀 in version 9.5.51-1 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

    If no regressions arise, payment will be issued on 2026-11-28. 🎊

    The following checklist (instructions) will need to be completed before the issue can be closed. Please copy/paste the Contributor+ Checklist from here into a new comment on this GH and complete it. If you have the K2 extension, you can simply click: [this button]. If no checklist is needed for this issue, you can click: [no checklist button]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

BugSomething is broken. Auto assigns a BugZero manager.DailyKSv2ExternalAdded to denote the issue can be worked on by a contributorOverdue

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions