Fix runtime data races, memory leak, and shutdown safety - #1429
Fix runtime data races, memory leak, and shutdown safety#1429bmehta001 wants to merge 43 commits into
Conversation
- HttpClient_Apple: scope Cancel() to m_dataTask only instead of blanket-cancelling every task on the shared session. Fix torn read on m_requests.empty() in CancelAllRequests spin loop. - HttpClientManager: fix torn read on m_httpCallbacks.empty() in cancelAllRequests spin loop — read under lock. - HttpResponseDecoder: add missing delete ctx->httpResponse before nullptr in Abort and RetryNetwork paths (memory leak). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Only delete queued tasks after successful join (not after detach, where the thread may still access them — undefined behavior) - Replace catch(...) with std::system_error and std::exception handlers that log error code and message - Log pending queue sizes in both join and detach paths Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both variables are read and written from different threads during normal upload scheduling. Declare as std::atomic to eliminate data races per the C++ memory model. Add .load() for variadic LOG_TRACE calls. Add comment explaining why unlocked stores in uploadAsync are safe. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove LOG_TRACE from Logger destructor — it triggers a crash on iOS simulator when the recursive_mutex used by logging has already been destroyed during static destruction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3f0289c to
de46cb2
Compare
Reject new worker-thread tasks once shutdown starts so queue cleanup cannot race with late producers, and move the TPM scheduled-upload state back under a single mutex so latency/next-upload decisions stay consistent without mixed atomic and mutex access. Files changed: - lib/pal/WorkerThread.cpp - lib/tpm/TransmissionPolicyManager.cpp - lib/tpm/TransmissionPolicyManager.hpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR targets runtime correctness in the SDK by addressing thread-safety issues, shutdown safety, and a per-request memory leak in HTTP response handling.
Changes:
- Tighten HTTP request cancellation scoping and fix torn reads in request/callback tracking loops.
- Fix a
SimpleHttpResponseleak on aborted/network-failure decode paths. - Rework worker-thread shutdown behavior to avoid unsafe queue cleanup after
detach()and improve error logging. - Refactor
TransmissionPolicyManagerscheduling state synchronization (mutex + newcancelUploadTaskLocked()helper).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| lib/tpm/TransmissionPolicyManager.hpp | Changes upload-scheduling state fields and adds a locked cancellation helper declaration. |
| lib/tpm/TransmissionPolicyManager.cpp | Moves upload-scheduling state access under a mutex and adjusts cancellation/scheduling flow. |
| lib/pal/WorkerThread.cpp | Makes shutdown/join behavior safer and improves exception handling/logging during join/detach. |
| lib/http/HttpResponseDecoder.cpp | Deletes ctx->httpResponse on Abort/RetryNetwork paths to prevent leaks. |
| lib/http/HttpClient_Apple.mm | Limits cancellation to the instance’s task and fixes a torn read in the shutdown wait loop. |
| lib/http/HttpClientManager.cpp | Fixes a torn read in the shutdown wait loop by locking around empty-check. |
| lib/api/Logger.cpp | Removes destructor logging to avoid iOS static-destruction-order crash. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Keep the scheduled-upload state mutex-based, but stop holding m_scheduledUploadMutex across DeferredCallbackHandle::Cancel so shutdown and pause paths do not block uploadAsync behind the same lock. While touching the path, use std::chrono::milliseconds for the bandwidth-controller reschedule call so ENABLE_BW_CONTROLLER builds cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Keep forced upload scheduling atomic around no-wait cancellation and preserve HTTP responses until downstream abort/network-failure handlers finish. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When scheduleUpload is called with force=true (or zero delay) and the previously scheduled upload task is currently executing on the worker, the no-wait cancel returns false and m_isUploadScheduled stays set. The existing m_isUploadScheduled check then skipped scheduling a new task, silently dropping the requested latency for force-scheduled profile changes. Propagate the requested latency to m_runningLatency under the same mutex when this race occurs. uploadAsync re-reads m_runningLatency inside its own LOCKGUARD, so a task that hasn't yet entered that critical section will pick up the new latency. If uploadAsync has already cleared m_isUploadScheduled (past its LOCKGUARD), the existing fallthrough at line 184 schedules a fresh task with the new latency. Add a regression test using a fake dispatcher whose Cancel always returns false, asserting that a force-scheduled call updates m_runningLatency without enqueueing a duplicate task. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the existing LOCKGUARD helper because scheduled upload cancellation does not need movable lock ownership. Consolidate the duplicated Issue 388 cancellation note so the PR keeps the remaining limitation documented without repeating the same TODO. Files changed: - lib/tpm/TransmissionPolicyManager.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace tight current-time assertions with a direct comparison against the original delayed schedule time. This keeps coverage for the forced immediate upload race while reducing timing sensitivity in CI. Files changed: - tests/unittests/TransmissionPolicyManagerTests.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restore the existing Issue 388 wording in the remaining cancellation comment while keeping the duplicated helper comment removed. Files changed: - lib/tpm/TransmissionPolicyManager.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix printf-style logging arguments for scheduled upload delays and queued worker task pointers. Ensure the blocking cancel test releases the dispatcher before failing so async futures cannot hang the test runner. Files changed: - lib/pal/WorkerThread.cpp - lib/tpm/TransmissionPolicyManager.cpp - tests/unittests/TransmissionPolicyManagerTests.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
4178021 to
d357260
Compare
Reword the comment to describe the test without citing a tracking number; no code change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reword the three `enabled` comments to describe the behavior without citing a tracking number; no code change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # lib/pal/WorkerThread.cpp # tests/unittests/TaskDispatcherCAPITests.cpp
…own thread The process-wide PAL WorkerThread is shared by reference count. A task running on the worker thread can drop the last reference (e.g. by tearing down its LogManager/PAL), which ran ~WorkerThread -> Join() synchronously inside the task: Join() detached the thread and returned, freeing the object while threadFunc was still on the stack below the task. threadFunc then kept touching freed members (m_itemInProgress, the locks, and the queues it drains at shutdown) -- a use-after-free / heap corruption confirmed by AddressSanitizer. Give the worker a custom shared_ptr deleter: when the last reference is released on the worker thread itself, detach and defer destruction to the thread, which deletes itself only after its loop has broken and all member access is done. On any other thread the object is deleted immediately as before (~WorkerThread joins the worker first). Add a PalTests regression test that drops the last reference from within a task running on the worker thread; it is clean under AddressSanitizer with the fix and reports heap-use-after-free without it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
onLastReferenceReleased() decided whether it was running on its own worker thread via m_hThread.get_id(), which returns the default not-a-thread id after detach(). If Join() had already run on the worker thread (its self-path detaches m_hThread), a later last-reference drop on that same thread would miss the self-check and delete the object while threadFunc was still executing below it -- the same UAF this change set fixes. Capture the worker's id in an atomic at threadFunc start and compare against that instead, so detection is correct regardless of detach ordering. Not reachable through current SDK code (the default WorkerThread is never explicitly Join()-ed), so this is defense-in-depth. Validated: ASAN dispatcher tests (PalTests incl WorkerThreadSelfDisposeOnOwnThreadIsSafe, TaskDispatcherCAPITests) all pass clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two issues raised on the previous commit: - std::atomic<std::thread::id> is not portable (std::thread::id is not guaranteed trivially copyable). Store m_workerId as a plain std::thread::id guarded by the existing recursive m_lock instead; the self-dispose check reads it under the lock. - Passing std::thread::id to LOG_INFO's printf-style '%u' is undefined behavior (varargs). Format the id with std::hash<std::thread::id> and '%zu' at both log sites (the constructor's 'Started new thread' and threadFunc's 'Running thread'). This was pre-existing; the surrounding change touches these lines. Validated: ASAN dispatcher tests (PalTests incl WorkerThreadSelfDisposeOnOwnThreadIsSafe, TaskDispatcherCAPITests) 15/15 pass clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep scheduled-task rejection detection internal by tracking the task lifetime across Queue(), so scheduleTask() returns a no-op handle if the dispatcher deletes the task during shutdown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TaskDispatcher.hpp now keeps DeferredCallbackHandle tied to TaskLifetimeState instead of a raw Task*. That lets scheduleTask() handles observe when the task is dropped or finishes normally, so a later Cancel() becomes a safe no-op instead of reusing a stale pointer. WorkerThread.cpp now centralizes shutdown sentinel enqueueing and pending-task drain/delete logic in shared helpers so Join() and the self-detach shutdown path cannot drift. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b12c5862-01e3-45e4-bf91-6389c20cae41
…xception Flush() paired ILogManager::StartActivity()/EndActivity() manually -- StartActivity() at the top, EndActivity() on the last line -- with no RAII guard and no try/catch in between. StoreRecords(), the optional checkpoint Flush(), and IOfflineStorageObserver::OnStorageRecordsSaved() are all real throw surfaces (disk I/O, a full/locked DB, or an observer implementation). If any of them threw, EndActivity() was skipped and m_pause_active_count was permanently leaked, so every later FlushAndTeardown()'s PauseActivity()+WaitPause() would deadlock waiting for a count that could never reach zero. This reproduced as a live macOS deadlock on main. Add ActivityGuard, an RAII wrapper matching the existing safe pattern already used by PauseGuard (TransmissionPolicyManager.cpp) and ActiveLoggerCall (Logger.cpp): its destructor calls EndActivity() on every exit path, including exception unwinding. Flush() now constructs the guard instead of calling StartActivity() directly, checks IsActive() instead of the raw bool, and no longer calls EndActivity() explicitly -- the guard's destructor handles it uniformly for both the normal-completion and the StartActivity()-returned-false early-return paths. Validated: full WSL Release build + complete UnitTests suite, 536/536 passed. No exception-injection regression test was added (Flush() has no existing throwing-observer test harness to extend); the fix's correctness rests on C++'s standard guaranteed-destructor-during-unwinding semantics, the same guarantee the two existing PauseGuard/ActiveLoggerCall call sites already rely on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b12c5862-01e3-45e4-bf91-6389c20cae41
…-order hazard LogManagerFactory::instance() and PAL::GetPAL() used ordinary function-local statics. Their destruction order relative to LogManagerProvider::Release() and PAL::shutdown() (both called during process teardown) is unspecified, since PAL in particular is constructed lazily on first use rather than at a fixed point relative to these teardown calls. A downstream consumer (onnxruntime-genai, see microsoft/onnxruntime-genai#2363) hit this in production as intermittent EXC_BAD_ACCESS crashes on macOS-arm64 at process exit: LogManagerFactory's registries and PAL's ISystemInformation shared_ptr member were sometimes already destroyed by the time teardown code tried to use them, and worked around it in their vendored copy of this SDK by leaking both singletons. Apply the same fix upstream: static T& x = *new T(); deliberately never destroys the object, so its members stay valid for the rest of the process regardless of teardown timing. Both objects are small and fixed-size (one per process), and PAL::shutdown() / Release() already perform the real resource teardown explicitly, so this only avoids the destructor-ordering hazard, not a resource leak in the ordinary sense. Validated: WSL build, 536/536 UnitTests pass.
Fixes runtime thread-safety, shutdown-safety, response-lifetime, and resource-leak issues that affect normal SDK operation. Split out from #1415 per reviewer request so runtime behavior changes stay separate from CI/build/test fixes.
Fix HTTP handle cleanup
HttpClient_WinInet.cpp
m_hWinInetSessionwas closed only inside theif (m_hWinInetRequest != nullptr)block, so ifHttpOpenRequestAfailed afterInternetConnectAsucceeded (session set, request null) the session handle leaked.Fix HTTP response lifetime on abort/network-failure paths
HttpResponseDecoder.cpp
ctx->httpResponsethroughrequestAborted(ctx)andtemporaryNetworkFailure(ctx)so downstream storage/statistics handlers can still read status and headers.EventsUploadContext, whoseclear()path deletes the response.HttpResponseDecoderTests.cpp
Fix WorkerThread shutdown safety, dropped-task handles, and task leak
TaskDispatcher.hpp
scheduleTask(...)return a no-opDeferredCallbackHandlewhen the dispatcher synchronously drops/deletes the task duringQueue()(for example on a shutdown-drop path), instead of returning a handle that dangles.Queue().WorkerThread.cpp
m_shuttingDown): lateQueue()calls are rejected (and the task deleted) once teardown starts.Join()the owning thread deletes them; on the self-dispose path the worker drains and deletes its own remaining tasks before exiting, so neither path leaks.TaskDispatcherCAPITests.cpp
scheduleTask(...)returns a no-op handle when a dispatcher drops the task synchronously duringQueue().PalTests.cpp
Join()returns a no-op handle, and that releasing the last worker-thread reference from a task on that same worker thread does not use-after-free.Fix Flush teardown deadlock
OfflineStorageHandler.cpp
Flush()could early-return (whenStartActivity()fails during teardown) without posting flush completion, soWaitForFlush()blocked forever. Signal completion on the early-return path so teardown cannot deadlock.Flush()also pairedStartActivity()/EndActivity()manually (StartActivity()at the top,EndActivity()on the last line) with no exception safety in between.StoreRecords(), the optional checkpointFlush(), andIOfflineStorageObserver::OnStorageRecordsSaved()are all real throw surfaces (disk I/O, a full/locked DB, or an observer implementation); if any of them threw,EndActivity()was skipped and the pause-activity count was permanently leaked, deadlocking every laterFlushAndTeardown()'sPauseActivity()+WaitPause(). This reproduced as a live macOS deadlock onmain. AddedActivityGuard, an RAII wrapper matching the existing safe pattern already used byPauseGuard(TransmissionPolicyManager.cpp) andActiveLoggerCall(Logger.cpp): its destructor callsEndActivity()on every exit path, including exception unwinding.BasicFuncTests.cpp
CFG_INT_MAX_TEARDOWN_TIME = 0, large payloads against the slow endpoint) and asserts shutdown completes cleanly; it also asserts the/slow/endpoint rewrite actually happened so the coverage can’t silently lapse.Fix static-destruction-order crashes in the two process-wide singletons
LogManagerFactory.hpp / PAL.cpp
LogManagerFactory::instance()andPAL::GetPAL()were ordinary function-local statics. Their destruction order relative toLogManagerProvider::Release()andPAL::shutdown()(both invoked during process teardown) is unspecified —PALin particular is constructed lazily on first use rather than at a fixed point, so whether it outlives the teardown call that needs it depends on runtime timing, not source order.EXC_BAD_ACCESScrashes on macOS-arm64 at process exit —LogManagerFactory's registries andPAL'sISystemInformationmember were sometimes already destroyed by the time teardown code tried to use them — and worked around it in their vendored copy of this SDK by leaking both singletons.static T& x = *new T();deliberately never destroys the object, so it stays valid for the rest of the process regardless of teardown timing. Both objects are small and process-lifetime singletons (one instance ever), andPAL::shutdown()/Release()already perform the real resource teardown explicitly, so this only removes the destructor-ordering hazard, not a resource leak in the ordinary sense.Make TransmissionPolicyManager scheduling consistently mutex-guarded
TransmissionPolicyManager.cpp / .hpp
m_isUploadScheduled,m_runningLatency, andm_scheduledUploadTimeconsistently withm_scheduledUploadMutex.m_scheduledUploadMutexacross potentially blocking cancellation during stop/shutdown.std::chrono::millisecondsvalue in the bandwidth-controller reschedule path, and caststd::chronocounts tolong longin the%lldLOG_TRACEcalls (the rep islongon LP64, a-Wformatmismatch in logging-enabled builds).TransmissionPolicyManagerTests.cpp
Fix Logger static-destruction-order crash
Logger.cpp
Logger::~Logger()because it can run after logging infrastructure has already been destroyed, causing crashes during static teardown.Known parity gap (separate repo, follow-up):
AIHttpResponseDecoder::handleDecodein thelib/modulessubmodule (lib/modules/azmon/AIHttpResponseDecoder.cpp:105,118) still setsctx->httpResponse = nullptr;on the aborted/failure paths — the same response-lifetime leak fixed here inlib/http/HttpResponseDecoder.cpp. It lives in a different repository (thelib/modulessubmodule), so it must be fixed there and pulled in via a submodule bump; it is out of scope for this PR.