Skip to content

Audit Sentry error volume and filter noise to stop paying overages #99784

Closed

Description

@melvin-bot

We're paying Sentry overage charges. We need to go through our error volume and cut what we don't act on.

Problem

Sentry event volume has grown past our plan's quota and we're being billed for overages. A large share of that volume is likely noise — errors nobody triages, network failures outside our control, third-party/extension noise, and a small number of high-frequency issues that dominate the event count.

Scope

  1. Pull the current usage breakdown from Sentry: events by project, platform, and issue over the last 60–90 days. Identify which issues account for the bulk of the quota.
  2. For the top issues by volume, decide one of: fix it, filter it, or sample it.
  3. Add inbound filters and client-side filtering for events we never act on. Candidates worth checking:
    • Network/offline errors (Network request failed, aborted fetches, timeouts)
    • Browser extension and third-party script errors
    • Unsupported/legacy browser noise
    • Known noisy libraries and expected user-facing errors already handled in product
  4. Tune beforeSend / ignoreErrors / denyUrls in the Sentry client config, and review sample rates (tracesSampleRate, profilesSampleRate, replaysSessionSampleRate) — performance and replay events count against billing too.
  5. Set spend caps and per-key rate limits so a single regression can't blow the quota again.
  6. Document what we filtered and why, so we don't silently drop something we later need.

Success criteria

  • Monthly Sentry event volume drops back under quota with no overage charges.
  • We can name every category of filtered event and justify it.
  • Remaining events are ones we actually triage.

Related: #91302 (Top Sentry crashes monthly check-in), #76128 (Sentry INP event analysis)

Reported by @rlinoz via DM with MelvinBot.

Activity

  1. self-assigned this
    on Aug 28, 2026
  2. rlinoz commented on Aug 28, 2026

    @rlinoz
    Contributor

    @MelvinBot inspect errors in Sentry and list the top 5 ones that should be logs instead of errors or could be dropped

  3. MelvinBot commented on Aug 28, 2026

    @MelvinBot
    Contributor

    ⚠️ The Sentry MCP server failed to connect on this run, so I could not pull live event counts. Everything below is ranked from code evidence in Expensify/App plus the noise categories your own audit in #91302 already declares non-actionable. Treat the ranking as a shortlist to verify against Discover, not a measured top 5.

    Top 5 candidates to downgrade or drop

    1. Known network/offline failures — drop at ingest.
    ignoreErrors contains exactly one entry, UPDATE_REQUIRED (setupSentry.ts:61). But CONST.ERROR already enumerates ~12 benign network strings (CONST/index.ts:2576-2595) that the network middleware deliberately files as Log.hmmm/Log.info (Middleware/Logging.ts:110-150). The app already says these aren't errors; Sentry's global handlers still bill them as error events. Adding that list to ignoreErrors is the single highest-leverage change.

    2. [BootSplash] splash screen is still visible — make it a log.
    Fires Sentry.captureMessage on a 10-second interval with no cap for as long as the splash is up (bootsplashTelemetry.ts:46-51). One stuck boot emits unbounded events. It's already level: 'warning' — nobody triages it as a crash. Use Sentry.logger.warn, or capture once and attach a duration.

    3. Expense API Error: <command> returned <jsonCode> — classify before sending.
    Every non-200 jsonCode on RequestMoney/TrackExpense/OpenReport goes out at level: 'error' with no allow/deny list on the code (trackExpenseCreationError.ts:100-137), so expected validation and permission responses land in the error stream. trackMFAFlowOutcome already does this correctly — routine and alternative outcomes go to Log.info, only anomalous ones become events (trackMFAFlowOutcome.ts:61). Copy that shape here.

    4. Expected-by-design exceptions in auth and Pusher — make them logs.
    Two captureException calls whose own comments say the condition is normal:

    • errorType: 'network_retry' in reauthentication — the comment states this is a spotty connection and the user is deliberately not logged out (Reauthentication.ts:263-274).
    • [Pusher] instance not found — the comment says this happens when disconnect() runs during Upgrade-Required teardown (Pusher/index.ts:284).

    5. Third-party and environmental noise — drop, don't tag.
    denyUrls only covers extension schemes (setupSentry.ts:26), and https-served third-party scripts are merely tagged third_party_code, not dropped (integrations/index.web.ts:42-47). Lazy chunk-load failures are still captured as warning-level exceptions even though the comment calls them "silent and largely environmental" (LazyModalSlot.tsx:17-22). #91302 already excludes FullStory fs.js, LinkedIn ads, MetaMask, gib.js, cross-origin Script error., and transient IDB/CacheStorage errors from triage — if they're excluded from triage, they should be excluded from billing.

    Two structural findings worth acting on alongside these

    There is no beforeSend for error events. Sentry.init wires up beforeSendTransaction and beforeSendLog but nothing for errors (setupSentry.ts:63-65). Items 1 and 5 need that hook (or ignoreErrors/denyUrls additions) to land anywhere. The existing classCallCheckNoiseFilterIntegration shows the working pattern — a processEvent returning null (classCallCheckNoiseFilter.ts:66-69).

    tracesSampleRate: 1.0 (setupSentry.ts:54) sends 100% of transactions. Profiling is already sampled at 0.1. On most Sentry bills transactions outweigh errors, so if the goal is stopping overages this is likely a bigger lever than any single error signature — worth checking before spending effort on the five above.

    Whatever you drop, drop it loudly. Every item above except #2 and #3 removes events with no trace. Per the success criteria in the issue, pair each filter with a Discover query that proves it's still matching, the way #93837 tracked third_party_code:True volume per release.


    Next Steps: Reply with @MelvinBot <your feedback> to refine this, or re-run once the Sentry MCP connection is fixed and I'll rank these against real event counts.


    view run

  4. rlinoz commented on Aug 28, 2026

    @rlinoz
    Contributor

    @MelvinBot implement 2 and 4

  5. MelvinBot commented on Aug 28, 2026

    @MelvinBot
    Contributor

    🤖 Draft PR created: #99789
    Please mark it as "Ready for review" when it is ready for review.

    Items 2 and 4 are implemented. All three call sites now send their signal to Sentry logs instead of the error stream — nothing is dropped, it just stops billing against the error quota (enableLogs: true is already set in setupSentry).

    # Change Before After
    2 bootsplashTelemetry.ts captureMessage every 10s, uncapped Sentry.logger.warn carrying stuckForMs, capped at 6 reports
    4a trackAuthenticationError.ts every error type → captureException network_retry → log with its tags preserved; all other types unchanged
    4b Pusher/index.ts + index.native.ts captureException on teardown race Sentry.logger.warn; __DEV__ still rejects loudly

    The 6-report cap on the bootsplash monitor is load-bearing, not decoration: the interval only clears when the splash state moves off VISIBLE, so a boot that never completes would otherwise report for the entire session. Six covers the first minute.

    Design notes and test results

    Why network_retry and not the whole tracker. missing_params, auth_failure, and unexpected_error still capture exceptions. Only network_retry moved, because that is the UNABLE_TO_RETRY path whose comment states the user is deliberately not logged out and the retry mechanism handles it. Its tags survive as log attributes, so it stays searchable by authentication_function and authentication_json_code. This is the same shape trackMFAFlowOutcome already uses.

    Tests run

    Check Result
    npm run typecheck passed
    npm run lint-changed passed
    npx eslint <changed files> passed — only pre-existing seatbelt warnings on untouched Pusher lines
    npm run spell-changed passed, 0 issues
    New unit tests 8/8 passed
    Existing suites over touched code (3 Pusher, ReauthenticationMiddlewareTest, forwardLogsToSentryTest, setupSentryIntegrationOrderTest) 39/39 passed

    Two new test files: tests/unit/bootsplashTelemetryTest.ts and tests/unit/trackAuthenticationErrorTest.ts.

    Not run, and why: the full npm test suite (only the suites covering touched code were run); npm run prettier (App has no prettier script or config — ESLint is the formatting gate here); storybook smoke test and React Compiler check (no .tsx or component changes).

    Not covered by a test: the Pusher change. PusherSubscribeWebTest's harness never reaches the production !socket branch, because __DEV__ is true under Jest so that path rejects instead. It is a one-line swap of the telemetry call with no surrounding logic change, and the existing Pusher suites pass.

    Items 1, 3 and 5 from the shortlist are untouched. Item 1 (adding the known network strings to ignoreErrors) is still the biggest single lever, and it needs a beforeSend hook that does not exist yet — say the word and I will take that one next.


    view run

  6. mountiny commented on Aug 30, 2026

    @mountiny
    Contributor

    Concrete filter candidates from the 14 day production crash scan on 2026-08-30. These are all error.unhandled:true and unresolved, so they sit at the top of the crash list and crowd out real crashes, but none of them are App bugs.

    Sentry ID Users (14d) Events (14d) What it actually is
    APP-AV2 642 1433 gib.js (Group-IB) cross origin frame access. Whole stack is gib.js, tagged third_party_code: True
    APP-2A9 69 843 Failed to load the gib.js script — Group-IB script blocked or offline
    APP-AWM 38 86 Same gib.js cross origin throw, / route
    APP-FF 19 46 Same gib.js cross origin throw, search route
    APP-DKT 32 665 Failed to fetch (www.google.com)
    APP-JTF 25 899 Failed to fetch (pagead2.googlesyndication.com) — ad blockers
    APP-BMR 26 77 Failed to connect to MetaMask — browser extension

    APP-AV2 alone is the single highest user count in the whole 14 day window, ahead of every real crash. The four gib.js rows together are roughly 768 users and 2400 events.

    Worth deciding whether Group-IB throws should be dropped at the SDK level via beforeSend on the third_party_code tag, or filtered as inbound rules. Either way they should not appear in unhandled crash triage.

    Separately, APP-J1W and APP-GXP (the minified reading 'se' pair) contributed roughly 2 million events in 14 days between them from about 383 users. That event to user ratio is the largest single volume contributor found in this scan and is likely worth a dedicated look for the overage question.

  7. melvin-bot commented on Sep 2, 2026

    @melvin-bot
    Author

    @rlinoz

    The solution for this issue has been 🚀 deployed to production 🚀 in version 9.4.68-1 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

  8. rlinoz commented on Sep 4, 2026

    @rlinoz
    Contributor

    missing mobile-expensify PR

  9. 5 remaining items

  10. melvin-bot commented on Sep 10, 2026

    @melvin-bot
    Author

    @rlinoz Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

  11. rlinoz commented on Sep 11, 2026

    @rlinoz
    Contributor

    Next up are certificate pinning and heartbeat, will get a PR for it next week.

  12. rlinoz commented on Sep 14, 2026

    @rlinoz
    Contributor

    PR up for certificate pinning

  13. rlinoz commented on Sep 16, 2026

    @rlinoz
    Contributor

    Not overdue, ongoing PRs and monitoring.

  14. melvin-bot commented on Sep 24, 2026

    @melvin-bot
    Author

    @rlinoz Whoops! This issue is 2 days overdue. Let's get this updated quick!

  15. rlinoz commented on Sep 28, 2026

    @rlinoz
    Contributor

    We are on track to not hit overages.

  16. melvin-bot commented on Oct 5, 2026

    @melvin-bot
    Author

    @rlinoz Whoops! This issue is 2 days overdue. Let's get this updated quick!

  17. rlinoz commented on Oct 6, 2026

    @rlinoz
    Contributor

    We are back within our error limits, so we can close this one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions