[Performance] High monitor lock contention in DefaultHeartBeatController and blockingRegister during cold startups on mid-range devices #8500

[READ] Step 1: Are you in the right place?

Yes, this issue describes performance overhead, massive thread lock contention, and CPU core starvation originating directly from the firebase-common heartbeats storage and initialization path in this repository.

[REQUIRED] Step 2: Describe your environment

• Android Studio version: Android Studio Ladybug (2024.2.1) (Profiling conducted on physical Moto G53 5G, Snapdragon 695, Android 14, API 34).
• Firebase Component: firebase-common (DefaultHeartBeatController & JavaDataStorage interactions affecting firebase-installations, firebase-messaging, firebase-perf, and firebase-sessions).
• Component version: com.google.firebase:firebase-common:22.1.0 (as resolved under Firebase BoM 34.15.0).

[REQUIRED] Step 3: Describe the problem

While the deterministic deadlock reported in previous issues (such as nested runBlocking calls within DataStore edits) was successfully resolved in PR #8182 (firebase-common:22.1.0), severe monitor lock contention still plagues the heartbeat storage path under high-concurrency cold startups (First Install scenario).

During early application startup, multiple modular Firebase components (FCM, Remote Config, Performance, Sessions) attempt to initialize and write/read metadata in parallel. On mid-range devices like the Moto G53 5G, this concurrent access to DefaultHeartBeatController triggers a massive thread synchronization storm.

Background thread lock queues of up to 938 ms are created. This heavy context-switching overhead saturates the CPU, forcing the OS scheduler to evict high-priority UI threads (RenderThread) to slower efficiency cores (Little Cores). This results in severe UI rendering delays, inflating EGLContext creation from a typical <5 ms to 220 ms, and triggering severe frame jank (588 ms Choreographer#doFrame).

Steps to reproduce:

1. Integrate Firebase BoM 34.15.0 (with firebase-common:22.1.0) into a medium-to-large multi-module application using multiple Firebase SDKs (FCM, Remote Config, Performance, Sessions).
2. Perform a First Install run (cold start with no cached preferences or metadata) on a mid-range octa-core device (e.g., Moto G53 5G, Snapdragon 695 with 2x Gold cores + 6x Silver cores on Android 14).
3. Capture a system trace using Perfetto spanning the application initialization and first frame rendering.
4. Analyze the trace in the Perfetto UI or via trace_processor_shell. Observe massive concurrent Thread::Init slices, thread lock contentions, and DefaultHeartBeatController monitor bottlenecks on background threads.

Perfetto Trace Evidence (Extracted via trace_processor):

Across a batch of 16 Baseline Profile test runs, 100% of the traces exhibited severe firebase-common monitor and thread lock contention, creating severe bottleneck queues:

-- Querying monitor contentions and thread lock bottlenecks on background threads:
SELECT name, ts, dur/1000000.0 as dur_ms 
FROM slice 
WHERE name LIKE '%monitor contention%' AND (name LIKE '%firebase%' OR name LIKE '%Firebase%')

-- Output Examples (Worst-Case Peak timings from RC29.3 benchmark batch):
1. Thread: "Firebase Background Thread #3" (TID 9955)
   Slice: "monitor contention with owner Firebase Background Thread #3 at void com.google.firebase.heartbeatinfo.DefaultHeartBeatController..."
   Duration: 938.67 ms (CRITICAL)

2. Thread: "Firebase-Messag" (TID 9933)
   Slice: "Lock contention on a monitor lock (blocking from com.google.firebase.messaging.FirebaseMessaging.blockingRegister)"
   Duration: 569.04 ms

3. Thread: "Firebase Background Thread #1" (TID 9915)
   Slice: "monitor contention on DefaultHeartBeatController"
   Duration: 754.75 ms

4. Thread: "Firebase Background Thread #0" (TID 9911)
   Slice: "monitor contention on DefaultHeartBeatController"
   Duration: 325.50 ms

Secondary Impact: CPU Core Starvation and Render Thread Degradation

Because of this parallel thread storm (including Thread::Init running 130+ times for newly spawned SharedPreferences and Firebase worker threads), the CPU Gold cores are fully choked.

CPU scheduling analysis (sched table) shows:

• The RenderThread is starved of big cores and is scheduled 68.9% of its active CPU time on Little Cores (CPUs 0-5) (487.32 ms out of 706.79 ms).
• This starvation degrades native GPU driver initialization, causing Creating EGLContext on the RenderThread to take an astronomical 220.65 ms (blocking the Main Thread's first layout draw for 259.99 ms).

Relevant Code:

Even though PR #8182 eliminated the cyclic deadlock, the underlying design of HeartBeatInfoStorage and DefaultHeartBeatController relies heavily on global Java monitor locks (synchronized methods) wrapping asynchronous/blocking data access under high concurrency.

When 5-8 sub-modules of Firebase hit these methods in parallel on different background workers at the exact same millisecond on a cold start, they queue up linearly on DefaultHeartBeatController:

// Sychronized methods in HeartBeatInfoStorage.java still trigger massive queues under high parallel load
synchronized void storeHeartBeat(long millis, String userAgentString) {
    // Highly concurrent threads queue up here while background DataStore transactions are in-flight,
    // causing up to ~938ms of monitor contention even if they eventually unlock without deadlocking.
    ...
}