fix: prevent EDEADLK self-join in ~CurlHttpOperation on async-thread destruction - #1481
fix: prevent EDEADLK self-join in ~CurlHttpOperation on async-thread destruction#1481bmehta001 wants to merge 37 commits into
Conversation
Under -Werror on Linux/macOS, the modules-repo CI (build-posix-latest-exp)
has been failing for ~2 weeks with:
config-default.h:36: error: 'HAVE_MAT_LIVEEVENTINSPECTOR' macro redefined
[-Werror,-Wmacro-redefined]
config-default.h:37: error: 'HAVE_MAT_PRIVACYGUARD' macro redefined
tests/functests/CMakeLists.txt and tests/unittests/CMakeLists.txt add
-DHAVE_MAT_LIVEEVENTINSPECTOR / -DHAVE_MAT_PRIVACYGUARD on the command
line when BUILD_LIVEEVENTINSPECTOR / BUILD_PRIVACYGUARD (default YES) and
the respective module dir exists. The three config-default headers then
redefined them unconditionally, which is fatal under -Werror (added by
microsoft#1415).
Wrapping the two defines in #ifndef in all three config-default*.h
headers preserves all existing behavior:
- Without command-line -D: macros get defined here as before.
- With command-line -D: header skips the redefinition, no warning.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…destruction
The modules-repo CI test ECSClientFuncTests.GetConfigs (and every
test in the ECSClientFuncTests suite) crashed on Linux/macOS with:
terminate called after throwing an instance of 'std::system_error'
what(): Resource deadlock avoided
Aborted (core dumped)
Root cause
==========
SendAsync() runs Send() + the user callback on a std::async worker
thread. The callback owns a strong ref to CurlHttpOperation, so when
it releases the last ref the ~CurlHttpOperation destructor runs on
the async thread itself.
libstdc++'s std::future<>::~future implicitly calls
_Async_state_impl::~_Async_state_impl, which calls _M_complete_async
-> _M_join via std::call_once. On the async thread that's a self-join;
call_once throws std::system_error(EDEADLK). Because the throw escapes
a noexcept destructor, terminate() aborts the process. A try/catch
around the future cannot rescue this — destructors of std::future are
noexcept.
Fix
===
Move the future onto a detached helper thread before its destructor
runs. The helper is by definition NOT the async thread (we'd only be
on the async thread if its work already finished), so the implicit
join completes immediately. On the common path (destruction from the
caller thread) it costs one short-lived thread spawn that exits in
microseconds.
Verified locally with sister + modules linked: all 113 FuncTests pass,
including all 25 ECSClientFuncTests (which include the formerly-fatal
GetConfigs).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ession)
Code review found that the previous fix detached the async future's join for
EVERY destruction. That removed the cross-thread lifetime guarantee the old
result.wait() provided: when the operation is destroyed from another thread
while the async Send() is still running, the destructor would proceed to
curl_easy_cleanup()/ReleaseResponse() and destroy the by-reference request body
while the worker thread is still using them -> use-after-free.
Restore the guarantee while keeping the EDEADLK self-join fix:
- Record the async task's thread id (atomic) when SendAsync's task starts.
- In the destructor, compare std::this_thread::get_id():
* self-join (destroyed from within our own async callback, e.g. EraseRequest
drops the last reference): the work is necessarily complete, so defer the
future's join to a detached helper thread instead of joining on this (the
async) thread, avoiding EDEADLK.
* cross-thread: result.wait() to keep the curl handle, response buffer and
by-reference request body alive until the async Send() finishes.
- Heap-allocate the deferred future first so a rare std::thread spawn failure
leaks the already-finished future rather than self-joining (EDEADLK) or
letting std::system_error escape this noexcept destructor (std::terminate).
- Refresh the stale HttpClient_Curl.cpp lifetime comment.
Logic validated with a standalone C++11 repro under AddressSanitizer: the
cross-thread path waits (no UAF) and the self-join path does not deadlock.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR addresses a Linux/macOS crash caused by std::future’s implicit join throwing on self-join when ~CurlHttpOperation runs on the same async thread created by std::async.
Changes:
- Adds tracking of the async task’s thread id to detect destructor execution on the async thread.
- Updates
~CurlHttpOperationto avoid self-join by deferringstd::futuredestruction to a detached helper thread. - Updates comments in the curl HTTP client to document the lifetime/join behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| lib/http/HttpClient_Curl.hpp | Adds async thread id tracking and modifies destructor logic to avoid std::future self-join aborts. |
| lib/http/HttpClient_Curl.cpp | Updates documentation comment describing the lifetime guarantees of CurlHttpOperation across async send. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…row-new failure - Add #include <new> so std::nothrow is not relied on transitively (review). - If new (std::nothrow) returns nullptr (OOM), result stays valid and would self-join (EDEADLK) at end of the noexcept dtor; abort() as a last resort instead of falling through to that, per review. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…fix lifetime comments - Replace std::atomic<std::thread::id> (not guaranteed supported across standard libraries) with a plain std::thread::id published via an std::atomic<bool> flag using release/acquire ordering. - Correct the lifetime comments: the operation's last shared_ptr is held by the owning CurlHttpRequest (via SetOperation), not by EraseRequest (which only removes the raw id from m_requests). The self-join occurs when the async callback leads to that request being destroyed on the async thread (OnHttpResponse -> EventsUploadContext::clear()). Re-validated the wait-vs-detach logic with a standalone C++11 repro under AddressSanitizer + UBSan. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… reset flag on reuse - Reword the self-join comment: in that case Send() has returned (we are in its callback) but the async task itself has not yet returned (the destructor runs inside it), so the deferred helper's ~future join completes only after this destructor unwinds. Avoids implying the async task is already finished. - Reset m_asyncThreadIdSet to false at the start of SendAsync so self-join detection stays correct if the operation were ever reused (it is single-use today: one SendAsync per request). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…microsoft#1481) The EDEADLK self-join was a symptom of using std::async(std::launch::async) for the HTTP send: the returned std::future joins its worker thread on destruction, so when the async callback caused the operation to be destroyed on that same worker thread (OnHttpResponse -> EventsUploadContext::clear()), ~future self-joined and aborted the process out of the noexcept destructor. Rather than detect-and-defer that self-join (the previous approach: published thread id + atomic flag + heap-move the future to a detached helper, with OOM/ thread-exhaustion fallbacks), remove the joining future entirely: - CurlHttpOperation now derives from enable_shared_from_this. SendAsync runs Send() on a detached std::thread that holds a shared_ptr keepalive to the operation, so the operation (and its curl handle, response buffer, and by-reference request body) stays alive until the worker finishes -- the same lifetime guarantee the destructor's result.wait() used to provide. - There is no future, so ~CurlHttpOperation never joins anything and is safe on any thread, including the worker thread itself. The destructor drops to plain curl cleanup. - Removes the future member, the m_asyncThreadId/m_asyncThreadIdSet machinery, and the <future>/<new> includes. Net -54 lines in the client. Adds HttpClientCurlTests.SendAsync_DestroyOnWorkerThread_NoSelfJoin, which drops the last external reference from inside the callback (on the worker thread) -- the exact microsoft#1481 trigger. It aborts the process on the old std::async code and passes on this fix. Verified on Linux GCC 13: all HttpClientCurlTests (12) pass including the new regression; the full FuncTests suite (39) passes with the curl client. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… into bhamehta/fix-curl-async-self-join
…xceptions, tidy test - requestBody use-after-free (comments 1 & 3): the old blocking destructor kept the by-reference body alive because destroying the request waited for Send(). With the self-keepalive worker the operation can outlive the request, so a reference into CurlHttpRequest::m_body could dangle mid-send. CurlHttpOperation now takes the body by value and owns it, so it is valid for the operation's whole lifetime regardless of when the request is released. Costs one body copy per request (the prior zero-copy relied on the blocking wait that caused microsoft#1481). - Detached-worker exceptions (comment 2): an exception escaping Send()/callback would call std::terminate, whereas the old std::async captured (and effectively swallowed) it. Wrap the worker body in try/catch to preserve the non-terminating behavior. - Test (comment 4): replace the raw new/delete shared_ptr box with a shared_ptr<shared_ptr<CurlHttpOperation>> whose contained pointer is reset in the callback, so it cannot leak if SendAsync throws. Verified on Linux GCC 13: all HttpClientCurlTests (12) pass including the self-join regression; full FuncTests (39) pass with the by-value body. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RunSendAndCallback sets m_completed regardless of the send result (including an immediate curl_easy_init failure), so OnDestroy is dispatched only when the operation is destroyed without SendAsync ever having run -- not on construction failure. Reword the comment to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t path Mirror the stronger teardown from SendAsync_NoOnDestroyDispatchAfterCompletion: on the (unexpected) timeout path, wait for weakOp to expire after Abort so the detached worker cannot outlive fixture teardown (m_client/curl_global_cleanup, m_headers, m_body) and cause secondary crashes that obscure the real failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…s path The callback sets the promise, but the detached worker still holds its self- reference until RunSendAndCallback returns. Wait (bounded) for weakOp to expire before the test returns so the operation's curl_easy_cleanup cannot race with fixture teardown, matching the other async regression test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ests Extract a shared DrainOperation helper that waits for the operation to be destroyed and aborts a stuck worker as a fallback, then hard-asserts it is gone. Both async regression tests now ensure the detached worker (and its curl_easy_cleanup) cannot outlive fixture teardown (m_client -> curl_global_cleanup) on either the success or timeout path, rather than returning while the worker might still run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These directly-constructed operations are not tracked by HttpClient_Curl::m_activeOps, so nothing else bounds the race between a lingering worker's curl_easy_cleanup and the fixture's curl_global_cleanup. DrainOperationOrDie now aborts a stuck worker and, if the operation is still alive afterward (a genuine keepalive/abort regression), records a failure and std::abort()s rather than returning into fixture teardown with an in-flight curl worker. In practice the .invalid host fails DNS in milliseconds so the operation is always gone immediately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move curl request tracking into shared state captured by detached workers so late callbacks do not dereference HttpClient_Curl after the shutdown drain times out. Preserve the bounded drain before curl_global_cleanup and abandon late callbacks/logging when shutdown cannot safely wait. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the detached self-keepalive and shutdown-tracker design with an owned worker thread. Normal destruction joins the worker; callback-thread destruction detaches it to avoid EDEADLK, while completion is published before callbacks can release the operation. Files: - lib/http/HttpClient_Curl.hpp: own, publish, join, and self-detach the worker safely - lib/http/HttpClient_Curl.cpp: restore the direct client lifetime model - tests/unittests/HttpClientCurlTests.cpp: cover self-destruction and late OnDestroy suppression Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b12c5862-01e3-45e4-bf91-6389c20cae41
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
lib/http/HttpClient_Curl.hpp:352
SendAsync()assignsm_worker = std::thread(...)without guarding thread construction.std::threadconstruction can throw (e.g.,std::system_erroron resource exhaustion, orstd::bad_allocwhile allocating/copying the callable), and this will currently escapeSendAsync();HttpClient_Curl::SendRequestAsync()does not catch, so this can terminate the process and/or violate the expectation that async sends always eventually invoke the completion path. Consider wrapping thread creation in a try/catch and falling back to runningSend()+ invoking the completion callback synchronously (or reporting a failure result and invoking the callback), so nothing escapes fromSendAsync().
m_worker = std::thread([this, callback]() {
{
std::lock_guard<std::mutex> startGuard(m_workerStartMtx);
}
lib/http/HttpClient_Curl.hpp:200
- The destructor now suppresses
DispatchEvent(OnDestroy)oncem_completedis true. This means theOnDestroystate event is no longer guaranteed to be delivered for successfully completed operations, which contradicts the HTTP state machine documented inlib/include/public/IHttpClient.hpp("destroy -> [OnDestroy]"). IfOnDestroymust be suppressed to avoid calling into a callback that may have been deleted by the completion path, consider either (1) updating the documented contract, or (2) redesigning state-event delivery soOnDestroycan still be reported without dereferencing a potentially-freedIHttpResponseCallback(e.g., route state events to a stable sink that outlives the request/callback).
// The completion callback may destroy m_callback. SendAsync marks completion
// before invoking it, so do not dispatch through that pointer afterward.
if (!m_completed.load(std::memory_order_acquire))
{
DispatchEvent(OnDestroy);
Keep OnDestroy delivery exactly once while the response callback is still valid, and complete requests synchronously when callable copying or thread creation fails. Track send attempts independently of thread joinability so failed construction cannot make an operation reusable. Files: - lib/http/HttpClient_Curl.hpp: centralize terminal event/callback delivery and harden thread startup - tests/unittests/HttpClientCurlTests.cpp: verify self-destruction, terminal event delivery, construction failure, and single-use behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b12c5862-01e3-45e4-bf91-6389c20cae41
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
tests/unittests/HttpClientCurlTests.cpp:22
- This test file uses
std::function(later inSendAsync_CallbackCopyFailureStillCompletes) but does not include<functional>, relying on transitive includes. Adding the direct include makes the test more robust to future header/include changes.
#include <future>
#include <chrono>
#include <memory>
#include <atomic>
#include <cstdlib>
#include <stdexcept>
#include <utility>
Retain both microsoft#1481's worker-lifetime regression coverage and main's response-size-cap coverage in HttpClientCurlTests.cpp. Update the response-cap fixture comment so it no longer describes the fixed std::async self-join behavior. Files: - tests/unittests/HttpClientCurlTests.cpp: resolve additive curl-test conflict Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b12c5862-01e3-45e4-bf91-6389c20cae41
Problem
Linux and macOS module tests abort with:
Root cause
CurlHttpOperation::SendAsyncusedstd::asyncand stored itsstd::futurein the operation. The completion callback can release the request's last reference to the operation on the async worker itself. The operation destructor then destroys the future on that same worker; libstdc++ tries to join the async thread from itself, throwsEDEADLK, and terminates through the destructor'snoexceptboundary.Fix
Replace the joining future with one worker thread owned by
CurlHttpOperation:std::threadhas been published to the member;OnDestroyis delivered exactly once before the completion callback, while theIHttpResponseCallbackis still valid;std::threadwould terminate the process;CURLE_FAILED_INITthrough the normal completion callback instead of escaping;The request body remains referenced rather than copied:
CurlHttpRequestdestroys its operation member before inherited request-body storage, and cross-thread operation destruction joins the worker before that storage is released.This removes the detached self-keepalive, shared client state, in-flight tracker, shutdown timeout/abandonment path, body copy, and synchronous fallback introduced by earlier revisions. There is no public SDK API change.
Verification
mainfull unit suite: 536/536 passed.maincurl and response-cap groups: 15/15 passed.mainself-join regression stress: 200 repeated runs passed.maincallable-copy/thread-start failure stress: 200 repeated runs passed.mainmerge: 527/527 passed; subsequent targeted AddressSanitizer curl and stress runs also passed.The focused regression drops the operation's last owner from inside its worker callback, reproducing the original destruction path directly.