You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Audit Sentry error volume and filter noise to stop paying overages #99784
We're paying Sentry overage charges. We need to go through our error volume and cut what we don't act on.
Problem
Sentry event volume has grown past our plan's quota and we're being billed for overages. A large share of that volume is likely noise — errors nobody triages, network failures outside our control, third-party/extension noise, and a small number of high-frequency issues that dominate the event count.
Scope
Pull the current usage breakdown from Sentry: events by project, platform, and issue over the last 60–90 days. Identify which issues account for the bulk of the quota.
For the top issues by volume, decide one of: fix it, filter it, or sample it.
Add inbound filters and client-side filtering for events we never act on. Candidates worth checking:
⚠️The Sentry MCP server failed to connect on this run, so I could not pull live event counts. Everything below is ranked from code evidence in Expensify/App plus the noise categories your own audit in #91302 already declares non-actionable. Treat the ranking as a shortlist to verify against Discover, not a measured top 5.
Top 5 candidates to downgrade or drop
1. Known network/offline failures — drop at ingest. ignoreErrors contains exactly one entry, UPDATE_REQUIRED (setupSentry.ts:61). But CONST.ERROR already enumerates ~12 benign network strings (CONST/index.ts:2576-2595) that the network middleware deliberately files as Log.hmmm/Log.info (Middleware/Logging.ts:110-150). The app already says these aren't errors; Sentry's global handlers still bill them as error events. Adding that list to ignoreErrors is the single highest-leverage change.
2. [BootSplash] splash screen is still visible — make it a log.
Fires Sentry.captureMessage on a 10-second interval with no cap for as long as the splash is up (bootsplashTelemetry.ts:46-51). One stuck boot emits unbounded events. It's already level: 'warning' — nobody triages it as a crash. Use Sentry.logger.warn, or capture once and attach a duration.
3. Expense API Error: <command> returned <jsonCode> — classify before sending.
Every non-200 jsonCode on RequestMoney/TrackExpense/OpenReport goes out at level: 'error' with no allow/deny list on the code (trackExpenseCreationError.ts:100-137), so expected validation and permission responses land in the error stream. trackMFAFlowOutcome already does this correctly — routine and alternative outcomes go to Log.info, only anomalous ones become events (trackMFAFlowOutcome.ts:61). Copy that shape here.
4. Expected-by-design exceptions in auth and Pusher — make them logs.
Two captureException calls whose own comments say the condition is normal:
errorType: 'network_retry' in reauthentication — the comment states this is a spotty connection and the user is deliberately not logged out (Reauthentication.ts:263-274).
[Pusher] instance not found — the comment says this happens when disconnect() runs during Upgrade-Required teardown (Pusher/index.ts:284).
5. Third-party and environmental noise — drop, don't tag. denyUrls only covers extension schemes (setupSentry.ts:26), and https-served third-party scripts are merely taggedthird_party_code, not dropped (integrations/index.web.ts:42-47). Lazy chunk-load failures are still captured as warning-level exceptions even though the comment calls them "silent and largely environmental" (LazyModalSlot.tsx:17-22). #91302 already excludes FullStory fs.js, LinkedIn ads, MetaMask, gib.js, cross-origin Script error., and transient IDB/CacheStorage errors from triage — if they're excluded from triage, they should be excluded from billing.
Two structural findings worth acting on alongside these
There is no beforeSend for error events.Sentry.init wires up beforeSendTransaction and beforeSendLog but nothing for errors (setupSentry.ts:63-65). Items 1 and 5 need that hook (or ignoreErrors/denyUrls additions) to land anywhere. The existing classCallCheckNoiseFilterIntegration shows the working pattern — a processEvent returning null (classCallCheckNoiseFilter.ts:66-69).
tracesSampleRate: 1.0 (setupSentry.ts:54) sends 100% of transactions. Profiling is already sampled at 0.1. On most Sentry bills transactions outweigh errors, so if the goal is stopping overages this is likely a bigger lever than any single error signature — worth checking before spending effort on the five above.
Whatever you drop, drop it loudly. Every item above except #2 and #3 removes events with no trace. Per the success criteria in the issue, pair each filter with a Discover query that proves it's still matching, the way #93837 tracked third_party_code:True volume per release.
Next Steps: Reply with @MelvinBot <your feedback> to refine this, or re-run once the Sentry MCP connection is fixed and I'll rank these against real event counts.
🤖 Draft PR created: #99789
Please mark it as "Ready for review" when it is ready for review.
Items 2 and 4 are implemented. All three call sites now send their signal to Sentry logs instead of the error stream — nothing is dropped, it just stops billing against the error quota (enableLogs: true is already set in setupSentry).
The 6-report cap on the bootsplash monitor is load-bearing, not decoration: the interval only clears when the splash state moves off VISIBLE, so a boot that never completes would otherwise report for the entire session. Six covers the first minute.
Design notes and test results
Why network_retry and not the whole tracker.missing_params, auth_failure, and unexpected_error still capture exceptions. Only network_retry moved, because that is the UNABLE_TO_RETRY path whose comment states the user is deliberately not logged out and the retry mechanism handles it. Its tags survive as log attributes, so it stays searchable by authentication_function and authentication_json_code. This is the same shape trackMFAFlowOutcome already uses.
Tests run
Check
Result
npm run typecheck
passed
npm run lint-changed
passed
npx eslint <changed files>
passed — only pre-existing seatbelt warnings on untouched Pusher lines
npm run spell-changed
passed, 0 issues
New unit tests
8/8 passed
Existing suites over touched code (3 Pusher, ReauthenticationMiddlewareTest, forwardLogsToSentryTest, setupSentryIntegrationOrderTest)
39/39 passed
Two new test files: tests/unit/bootsplashTelemetryTest.ts and tests/unit/trackAuthenticationErrorTest.ts.
Not run, and why: the full npm test suite (only the suites covering touched code were run); npm run prettier (App has no prettier script or config — ESLint is the formatting gate here); storybook smoke test and React Compiler check (no .tsx or component changes).
Not covered by a test: the Pusher change. PusherSubscribeWebTest's harness never reaches the production !socket branch, because __DEV__ is true under Jest so that path rejects instead. It is a one-line swap of the telemetry call with no surrounding logic change, and the existing Pusher suites pass.
Items 1, 3 and 5 from the shortlist are untouched. Item 1 (adding the known network strings to ignoreErrors) is still the biggest single lever, and it needs a beforeSend hook that does not exist yet — say the word and I will take that one next.
Concrete filter candidates from the 14 day production crash scan on 2026-08-30. These are all error.unhandled:true and unresolved, so they sit at the top of the crash list and crowd out real crashes, but none of them are App bugs.
APP-AV2 alone is the single highest user count in the whole 14 day window, ahead of every real crash. The four gib.js rows together are roughly 768 users and 2400 events.
Worth deciding whether Group-IB throws should be dropped at the SDK level via beforeSend on the third_party_code tag, or filtered as inbound rules. Either way they should not appear in unhandled crash triage.
Separately, APP-J1W and APP-GXP (the minified reading 'se' pair) contributed roughly 2 million events in 14 days between them from about 383 users. That event to user ratio is the largest single volume contributor found in this scan and is likely worth a dedicated look for the overage question.
The solution for this issue has been 🚀 deployed to production 🚀 in version 9.4.68-1 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:
Activity
rlinoz commentedon Aug 28, 2026
@MelvinBot inspect errors in Sentry and list the top 5 ones that should be logs instead of errors or could be dropped
MelvinBot commentedon Aug 28, 2026
Top 5 candidates to downgrade or drop
1. Known network/offline failures — drop at ingest.
ignoreErrorscontains exactly one entry,UPDATE_REQUIRED(setupSentry.ts:61). ButCONST.ERRORalready enumerates ~12 benign network strings (CONST/index.ts:2576-2595) that the network middleware deliberately files asLog.hmmm/Log.info(Middleware/Logging.ts:110-150). The app already says these aren't errors; Sentry's global handlers still bill them as error events. Adding that list toignoreErrorsis the single highest-leverage change.2.
[BootSplash] splash screen is still visible— make it a log.Fires
Sentry.captureMessageon a 10-second interval with no cap for as long as the splash is up (bootsplashTelemetry.ts:46-51). One stuck boot emits unbounded events. It's alreadylevel: 'warning'— nobody triages it as a crash. UseSentry.logger.warn, or capture once and attach a duration.3.
Expense API Error: <command> returned <jsonCode>— classify before sending.Every non-200 jsonCode on
RequestMoney/TrackExpense/OpenReportgoes out atlevel: 'error'with no allow/deny list on the code (trackExpenseCreationError.ts:100-137), so expected validation and permission responses land in the error stream.trackMFAFlowOutcomealready does this correctly — routine and alternative outcomes go toLog.info, only anomalous ones become events (trackMFAFlowOutcome.ts:61). Copy that shape here.4. Expected-by-design exceptions in auth and Pusher — make them logs.
Two
captureExceptioncalls whose own comments say the condition is normal:errorType: 'network_retry'in reauthentication — the comment states this is a spotty connection and the user is deliberately not logged out (Reauthentication.ts:263-274).[Pusher] instance not found— the comment says this happens whendisconnect()runs during Upgrade-Required teardown (Pusher/index.ts:284).5. Third-party and environmental noise — drop, don't tag.
denyUrlsonly covers extension schemes (setupSentry.ts:26), and https-served third-party scripts are merely taggedthird_party_code, not dropped (integrations/index.web.ts:42-47). Lazy chunk-load failures are still captured as warning-level exceptions even though the comment calls them "silent and largely environmental" (LazyModalSlot.tsx:17-22). #91302 already excludes FullStoryfs.js, LinkedIn ads, MetaMask,gib.js, cross-originScript error., and transient IDB/CacheStorage errors from triage — if they're excluded from triage, they should be excluded from billing.Two structural findings worth acting on alongside these
There is no
beforeSendfor error events.Sentry.initwires upbeforeSendTransactionandbeforeSendLogbut nothing for errors (setupSentry.ts:63-65). Items 1 and 5 need that hook (orignoreErrors/denyUrlsadditions) to land anywhere. The existingclassCallCheckNoiseFilterIntegrationshows the working pattern — aprocessEventreturningnull(classCallCheckNoiseFilter.ts:66-69).tracesSampleRate: 1.0(setupSentry.ts:54) sends 100% of transactions. Profiling is already sampled at 0.1. On most Sentry bills transactions outweigh errors, so if the goal is stopping overages this is likely a bigger lever than any single error signature — worth checking before spending effort on the five above.Whatever you drop, drop it loudly. Every item above except #2 and #3 removes events with no trace. Per the success criteria in the issue, pair each filter with a Discover query that proves it's still matching, the way #93837 tracked
third_party_code:Truevolume per release.Next Steps: Reply with
@MelvinBot <your feedback>to refine this, or re-run once the Sentry MCP connection is fixed and I'll rank these against real event counts.view run
rlinoz commentedon Aug 28, 2026
@MelvinBot implement 2 and 4
MelvinBot commentedon Aug 28, 2026
🤖 Draft PR created: #99789
Please mark it as "Ready for review" when it is ready for review.
Items 2 and 4 are implemented. All three call sites now send their signal to Sentry logs instead of the error stream — nothing is dropped, it just stops billing against the error quota (
enableLogs: trueis already set insetupSentry).captureMessageevery 10s, uncappedSentry.logger.warncarryingstuckForMs, capped at 6 reportscaptureExceptionnetwork_retry→ log with its tags preserved; all other types unchangedcaptureExceptionon teardown raceSentry.logger.warn;__DEV__still rejects loudlyThe 6-report cap on the bootsplash monitor is load-bearing, not decoration: the interval only clears when the splash state moves off
VISIBLE, so a boot that never completes would otherwise report for the entire session. Six covers the first minute.Design notes and test results
Why
network_retryand not the whole tracker.missing_params,auth_failure, andunexpected_errorstill capture exceptions. Onlynetwork_retrymoved, because that is theUNABLE_TO_RETRYpath whose comment states the user is deliberately not logged out and the retry mechanism handles it. Its tags survive as log attributes, so it stays searchable byauthentication_functionandauthentication_json_code. This is the same shapetrackMFAFlowOutcomealready uses.Tests run
npm run typechecknpm run lint-changednpx eslint <changed files>npm run spell-changedReauthenticationMiddlewareTest,forwardLogsToSentryTest,setupSentryIntegrationOrderTest)Two new test files:
tests/unit/bootsplashTelemetryTest.tsandtests/unit/trackAuthenticationErrorTest.ts.Not run, and why: the full
npm testsuite (only the suites covering touched code were run);npm run prettier(App has no prettier script or config — ESLint is the formatting gate here); storybook smoke test and React Compiler check (no.tsxor component changes).Not covered by a test: the Pusher change.
PusherSubscribeWebTest's harness never reaches the production!socketbranch, because__DEV__is true under Jest so that path rejects instead. It is a one-line swap of the telemetry call with no surrounding logic change, and the existing Pusher suites pass.Items 1, 3 and 5 from the shortlist are untouched. Item 1 (adding the known network strings to
ignoreErrors) is still the biggest single lever, and it needs abeforeSendhook that does not exist yet — say the word and I will take that one next.view run
mountiny commentedon Aug 30, 2026
Concrete filter candidates from the 14 day production crash scan on 2026-08-30. These are all
error.unhandled:trueand unresolved, so they sit at the top of the crash list and crowd out real crashes, but none of them are App bugs.gib.js(Group-IB) cross origin frame access. Whole stack isgib.js, taggedthird_party_code: TrueFailed to load the gib.js script— Group-IB script blocked or offlinegib.jscross origin throw,/routegib.jscross origin throw, search routeFailed to fetch (www.google.com)Failed to fetch (pagead2.googlesyndication.com)— ad blockersFailed to connect to MetaMask— browser extensionAPP-AV2 alone is the single highest user count in the whole 14 day window, ahead of every real crash. The four
gib.jsrows together are roughly 768 users and 2400 events.Worth deciding whether Group-IB throws should be dropped at the SDK level via
beforeSendon thethird_party_codetag, or filtered as inbound rules. Either way they should not appear in unhandled crash triage.Separately, APP-J1W and APP-GXP (the minified
reading 'se'pair) contributed roughly 2 million events in 14 days between them from about 383 users. That event to user ratio is the largest single volume contributor found in this scan and is likely worth a dedicated look for the overage question.melvin-bot commentedon Sep 2, 2026
@rlinoz
The solution for this issue has been 🚀 deployed to production 🚀 in version 9.4.68-1 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:
rlinoz commentedon Sep 4, 2026
missing mobile-expensify PR
5 remaining items
melvin-bot commentedon Sep 10, 2026
@rlinoz Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!
rlinoz commentedon Sep 11, 2026
Next up are certificate pinning and heartbeat, will get a PR for it next week.
rlinoz commentedon Sep 14, 2026
PR up for certificate pinning
rlinoz commentedon Sep 16, 2026
Not overdue, ongoing PRs and monitoring.
melvin-bot commentedon Sep 24, 2026
@rlinoz Whoops! This issue is 2 days overdue. Let's get this updated quick!
rlinoz commentedon Sep 28, 2026
We are on track to not hit overages.
melvin-bot commentedon Oct 5, 2026
@rlinoz Whoops! This issue is 2 days overdue. Let's get this updated quick!
rlinoz commentedon Oct 6, 2026
We are back within our error limits, so we can close this one