Skip to content

fix: prevent EDEADLK self-join in ~CurlHttpOperation on async-thread destruction - #1481

Open
bmehta001 wants to merge 37 commits into
microsoft:mainfrom
bmehta001:bhamehta/fix-curl-async-self-join
Open

fix: prevent EDEADLK self-join in ~CurlHttpOperation on async-thread destruction#1481
bmehta001 wants to merge 37 commits into
microsoft:mainfrom
bmehta001:bhamehta/fix-curl-async-self-join

Conversation

@bmehta001

@bmehta001 bmehta001 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Problem

Linux and macOS module tests abort with:

terminate called after throwing an instance of 'std::system_error'
  what(): Resource deadlock avoided

Root cause

CurlHttpOperation::SendAsync used std::async and stored its std::future in the operation. The completion callback can release the request's last reference to the operation on the async worker itself. The operation destructor then destroys the future on that same worker; libstdc++ tries to join the async thread from itself, throws EDEADLK, and terminates through the destructor's noexcept boundary.

Fix

Replace the joining future with one worker thread owned by CurlHttpOperation:

  • normal destruction joins the worker, preserving the original lifetime barrier;
  • destruction from the worker callback detaches instead of self-joining;
  • a small start gate ensures a fast worker cannot destroy the operation before its std::thread has been published to the member;
  • OnDestroy is delivered exactly once before the completion callback, while the IHttpResponseCallback is still valid;
  • worker exceptions are contained because an exception escaping a raw std::thread would terminate the process;
  • callable-copy or thread-creation failures report CURLE_FAILED_INIT through the normal completion callback instead of escaping;
  • a permanent send-attempt flag keeps operations single-use even when worker construction fails.

The request body remains referenced rather than copied: CurlHttpRequest destroys its operation member before inherited request-body storage, and cross-thread operation destruction joins the worker before that storage is released.

This removes the detached self-keepalive, shared client state, in-flight tracker, shutdown timeout/abandonment path, body copy, and synchronous fallback introduced by earlier revisions. There is no public SDK API change.

Verification

  • Current-main full unit suite: 536/536 passed.
  • Current-main curl and response-cap groups: 15/15 passed.
  • Current-main self-join regression stress: 200 repeated runs passed.
  • Current-main callable-copy/thread-start failure stress: 200 repeated runs passed.
  • AddressSanitizer full suite before the conflict-free main merge: 527/527 passed; subsequent targeted AddressSanitizer curl and stress runs also passed.
  • No AddressSanitizer findings.
  • Final material review: no significant findings.

The focused regression drops the operation's last owner from inside its worker callback, reproducing the original destruction path directly.

bmehta001 and others added 2 commits June 10, 2026 13:48
Under -Werror on Linux/macOS, the modules-repo CI (build-posix-latest-exp)
has been failing for ~2 weeks with:

  config-default.h:36: error: 'HAVE_MAT_LIVEEVENTINSPECTOR' macro redefined
                              [-Werror,-Wmacro-redefined]
  config-default.h:37: error: 'HAVE_MAT_PRIVACYGUARD' macro redefined

tests/functests/CMakeLists.txt and tests/unittests/CMakeLists.txt add
-DHAVE_MAT_LIVEEVENTINSPECTOR / -DHAVE_MAT_PRIVACYGUARD on the command
line when BUILD_LIVEEVENTINSPECTOR / BUILD_PRIVACYGUARD (default YES) and
the respective module dir exists. The three config-default headers then
redefined them unconditionally, which is fatal under -Werror (added by
microsoft#1415).

Wrapping the two defines in #ifndef in all three config-default*.h
headers preserves all existing behavior:
- Without command-line -D: macros get defined here as before.
- With command-line -D: header skips the redefinition, no warning.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…destruction

The modules-repo CI test ECSClientFuncTests.GetConfigs (and every
test in the ECSClientFuncTests suite) crashed on Linux/macOS with:

  terminate called after throwing an instance of 'std::system_error'
    what():  Resource deadlock avoided
  Aborted (core dumped)

Root cause
==========
SendAsync() runs Send() + the user callback on a std::async worker
thread. The callback owns a strong ref to CurlHttpOperation, so when
it releases the last ref the ~CurlHttpOperation destructor runs on
the async thread itself.

libstdc++'s std::future<>::~future implicitly calls
_Async_state_impl::~_Async_state_impl, which calls _M_complete_async
-> _M_join via std::call_once. On the async thread that's a self-join;
call_once throws std::system_error(EDEADLK). Because the throw escapes
a noexcept destructor, terminate() aborts the process. A try/catch
around the future cannot rescue this — destructors of std::future are
noexcept.

Fix
===
Move the future onto a detached helper thread before its destructor
runs. The helper is by definition NOT the async thread (we'd only be
on the async thread if its work already finished), so the implicit
join completes immediately. On the common path (destruction from the
caller thread) it costs one short-lived thread spawn that exits in
microseconds.

Verified locally with sister + modules linked: all 113 FuncTests pass,
including all 25 ECSClientFuncTests (which include the formerly-fatal
GetConfigs).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bmehta001
bmehta001 requested a review from a team as a code owner June 11, 2026 08:17
bmehta001 and others added 2 commits June 11, 2026 11:13
…ession)

Code review found that the previous fix detached the async future's join for
EVERY destruction. That removed the cross-thread lifetime guarantee the old
result.wait() provided: when the operation is destroyed from another thread
while the async Send() is still running, the destructor would proceed to
curl_easy_cleanup()/ReleaseResponse() and destroy the by-reference request body
while the worker thread is still using them -> use-after-free.

Restore the guarantee while keeping the EDEADLK self-join fix:
- Record the async task's thread id (atomic) when SendAsync's task starts.
- In the destructor, compare std::this_thread::get_id():
  * self-join (destroyed from within our own async callback, e.g. EraseRequest
    drops the last reference): the work is necessarily complete, so defer the
    future's join to a detached helper thread instead of joining on this (the
    async) thread, avoiding EDEADLK.
  * cross-thread: result.wait() to keep the curl handle, response buffer and
    by-reference request body alive until the async Send() finishes.
- Heap-allocate the deferred future first so a rare std::thread spawn failure
  leaks the already-finished future rather than self-joining (EDEADLK) or
  letting std::system_error escape this noexcept destructor (std::terminate).
- Refresh the stale HttpClient_Curl.cpp lifetime comment.

Logic validated with a standalone C++11 repro under AddressSanitizer: the
cross-thread path waits (no UAF) and the self-join path does not deadlock.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a Linux/macOS crash caused by std::future’s implicit join throwing on self-join when ~CurlHttpOperation runs on the same async thread created by std::async.

Changes:

  • Adds tracking of the async task’s thread id to detect destructor execution on the async thread.
  • Updates ~CurlHttpOperation to avoid self-join by deferring std::future destruction to a detached helper thread.
  • Updates comments in the curl HTTP client to document the lifetime/join behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
lib/http/HttpClient_Curl.hpp Adds async thread id tracking and modifies destructor logic to avoid std::future self-join aborts.
lib/http/HttpClient_Curl.cpp Updates documentation comment describing the lifetime guarantees of CurlHttpOperation across async send.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread lib/http/HttpClient_Curl.hpp
Comment thread lib/http/HttpClient_Curl.hpp Outdated
…row-new failure

- Add #include <new> so std::nothrow is not relied on transitively (review).
- If new (std::nothrow) returns nullptr (OOM), result stays valid and would
  self-join (EDEADLK) at end of the noexcept dtor; abort() as a last resort
  instead of falling through to that, per review.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

Comment thread lib/http/HttpClient_Curl.hpp Outdated
Comment thread lib/http/HttpClient_Curl.hpp Outdated
Comment thread lib/http/HttpClient_Curl.hpp Outdated
Comment thread lib/http/HttpClient_Curl.cpp Outdated
…fix lifetime comments

- Replace std::atomic<std::thread::id> (not guaranteed supported across standard
  libraries) with a plain std::thread::id published via an std::atomic<bool>
  flag using release/acquire ordering.
- Correct the lifetime comments: the operation's last shared_ptr is held by the
  owning CurlHttpRequest (via SetOperation), not by EraseRequest (which only
  removes the raw id from m_requests). The self-join occurs when the async
  callback leads to that request being destroyed on the async thread
  (OnHttpResponse -> EventsUploadContext::clear()).

Re-validated the wait-vs-detach logic with a standalone C++11 repro under
AddressSanitizer + UBSan.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread lib/http/HttpClient_Curl.hpp Outdated
Comment thread lib/http/HttpClient_Curl.hpp Outdated
… reset flag on reuse

- Reword the self-join comment: in that case Send() has returned (we are in its
  callback) but the async task itself has not yet returned (the destructor runs
  inside it), so the deferred helper's ~future join completes only after this
  destructor unwinds. Avoids implying the async task is already finished.
- Reset m_asyncThreadIdSet to false at the start of SendAsync so self-join
  detection stays correct if the operation were ever reused (it is single-use
  today: one SendAsync per request).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@bmehta001 bmehta001 self-assigned this Jul 8, 2026
bmehta001 and others added 3 commits July 8, 2026 15:58
…microsoft#1481)

The EDEADLK self-join was a symptom of using std::async(std::launch::async) for
the HTTP send: the returned std::future joins its worker thread on destruction, so
when the async callback caused the operation to be destroyed on that same worker
thread (OnHttpResponse -> EventsUploadContext::clear()), ~future self-joined and
aborted the process out of the noexcept destructor.

Rather than detect-and-defer that self-join (the previous approach: published
thread id + atomic flag + heap-move the future to a detached helper, with OOM/
thread-exhaustion fallbacks), remove the joining future entirely:

- CurlHttpOperation now derives from enable_shared_from_this. SendAsync runs Send()
  on a detached std::thread that holds a shared_ptr keepalive to the operation, so
  the operation (and its curl handle, response buffer, and by-reference request
  body) stays alive until the worker finishes -- the same lifetime guarantee the
  destructor's result.wait() used to provide.
- There is no future, so ~CurlHttpOperation never joins anything and is safe on any
  thread, including the worker thread itself. The destructor drops to plain curl
  cleanup.
- Removes the future member, the m_asyncThreadId/m_asyncThreadIdSet machinery, and
  the <future>/<new> includes. Net -54 lines in the client.

Adds HttpClientCurlTests.SendAsync_DestroyOnWorkerThread_NoSelfJoin, which drops the
last external reference from inside the callback (on the worker thread) -- the exact
microsoft#1481 trigger. It aborts the process on the old std::async code and passes on this
fix.

Verified on Linux GCC 13: all HttpClientCurlTests (12) pass including the new
regression; the full FuncTests suite (39) passes with the curl client.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

Comment thread lib/http/HttpClient_Curl.hpp Outdated
Comment thread lib/http/HttpClient_Curl.hpp Outdated
Comment thread lib/http/HttpClient_Curl.cpp Outdated
Comment thread tests/unittests/HttpClientCurlTests.cpp Outdated
…xceptions, tidy test

- requestBody use-after-free (comments 1 & 3): the old blocking destructor kept the
  by-reference body alive because destroying the request waited for Send(). With the
  self-keepalive worker the operation can outlive the request, so a reference into
  CurlHttpRequest::m_body could dangle mid-send. CurlHttpOperation now takes the body
  by value and owns it, so it is valid for the operation's whole lifetime regardless
  of when the request is released. Costs one body copy per request (the prior
  zero-copy relied on the blocking wait that caused microsoft#1481).
- Detached-worker exceptions (comment 2): an exception escaping Send()/callback would
  call std::terminate, whereas the old std::async captured (and effectively swallowed)
  it. Wrap the worker body in try/catch to preserve the non-terminating behavior.
- Test (comment 4): replace the raw new/delete shared_ptr box with a
  shared_ptr<shared_ptr<CurlHttpOperation>> whose contained pointer is reset in the
  callback, so it cannot leak if SendAsync throws.

Verified on Linux GCC 13: all HttpClientCurlTests (12) pass including the self-join
regression; full FuncTests (39) pass with the by-value body.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread lib/http/HttpClient_Curl.hpp Outdated
RunSendAndCallback sets m_completed regardless of the send result (including an
immediate curl_easy_init failure), so OnDestroy is dispatched only when the
operation is destroyed without SendAsync ever having run -- not on construction
failure. Reword the comment to match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread tests/unittests/HttpClientCurlTests.cpp Outdated
…t path

Mirror the stronger teardown from SendAsync_NoOnDestroyDispatchAfterCompletion: on
the (unexpected) timeout path, wait for weakOp to expire after Abort so the detached
worker cannot outlive fixture teardown (m_client/curl_global_cleanup, m_headers,
m_body) and cause secondary crashes that obscure the real failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread tests/unittests/HttpClientCurlTests.cpp Outdated
…s path

The callback sets the promise, but the detached worker still holds its self-
reference until RunSendAndCallback returns. Wait (bounded) for weakOp to expire
before the test returns so the operation's curl_easy_cleanup cannot race with
fixture teardown, matching the other async regression test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comment thread tests/unittests/HttpClientCurlTests.cpp Outdated
Comment thread tests/unittests/HttpClientCurlTests.cpp Outdated
…ests

Extract a shared DrainOperation helper that waits for the operation to be destroyed
and aborts a stuck worker as a fallback, then hard-asserts it is gone. Both async
regression tests now ensure the detached worker (and its curl_easy_cleanup) cannot
outlive fixture teardown (m_client -> curl_global_cleanup) on either the success or
timeout path, rather than returning while the worker might still run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread tests/unittests/HttpClientCurlTests.cpp Outdated
These directly-constructed operations are not tracked by HttpClient_Curl::m_activeOps,
so nothing else bounds the race between a lingering worker's curl_easy_cleanup and the
fixture's curl_global_cleanup. DrainOperationOrDie now aborts a stuck worker and, if the
operation is still alive afterward (a genuine keepalive/abort regression), records a
failure and std::abort()s rather than returning into fixture teardown with an in-flight
curl worker. In practice the .invalid host fails DNS in milliseconds so the operation is
always gone immediately.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

bmehta001 and others added 2 commits July 12, 2026 19:48
Move curl request tracking into shared state captured by detached workers so late callbacks do not dereference HttpClient_Curl after the shutdown drain times out. Preserve the bounded drain before curl_global_cleanup and abandon late callbacks/logging when shutdown cannot safely wait.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the detached self-keepalive and shutdown-tracker design with an owned worker thread. Normal destruction joins the worker; callback-thread destruction detaches it to avoid EDEADLK, while completion is published before callbacks can release the operation.

Files:
- lib/http/HttpClient_Curl.hpp: own, publish, join, and self-detach the worker safely
- lib/http/HttpClient_Curl.cpp: restore the direct client lifetime model
- tests/unittests/HttpClientCurlTests.cpp: cover self-destruction and late OnDestroy suppression

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b12c5862-01e3-45e4-bf91-6389c20cae41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

lib/http/HttpClient_Curl.hpp:352

  • SendAsync() assigns m_worker = std::thread(...) without guarding thread construction. std::thread construction can throw (e.g., std::system_error on resource exhaustion, or std::bad_alloc while allocating/copying the callable), and this will currently escape SendAsync(); HttpClient_Curl::SendRequestAsync() does not catch, so this can terminate the process and/or violate the expectation that async sends always eventually invoke the completion path. Consider wrapping thread creation in a try/catch and falling back to running Send() + invoking the completion callback synchronously (or reporting a failure result and invoking the callback), so nothing escapes from SendAsync().
        m_worker = std::thread([this, callback]() {
            {
                std::lock_guard<std::mutex> startGuard(m_workerStartMtx);
            }

lib/http/HttpClient_Curl.hpp:200

  • The destructor now suppresses DispatchEvent(OnDestroy) once m_completed is true. This means the OnDestroy state event is no longer guaranteed to be delivered for successfully completed operations, which contradicts the HTTP state machine documented in lib/include/public/IHttpClient.hpp ("destroy -> [OnDestroy]"). If OnDestroy must be suppressed to avoid calling into a callback that may have been deleted by the completion path, consider either (1) updating the documented contract, or (2) redesigning state-event delivery so OnDestroy can still be reported without dereferencing a potentially-freed IHttpResponseCallback (e.g., route state events to a stable sink that outlives the request/callback).
        // The completion callback may destroy m_callback. SendAsync marks completion
        // before invoking it, so do not dispatch through that pointer afterward.
        if (!m_completed.load(std::memory_order_acquire))
        {
            DispatchEvent(OnDestroy);

Keep OnDestroy delivery exactly once while the response callback is still valid, and complete requests synchronously when callable copying or thread creation fails. Track send attempts independently of thread joinability so failed construction cannot make an operation reusable.

Files:
- lib/http/HttpClient_Curl.hpp: centralize terminal event/callback delivery and harden thread startup
- tests/unittests/HttpClientCurlTests.cpp: verify self-destruction, terminal event delivery, construction failure, and single-use behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b12c5862-01e3-45e4-bf91-6389c20cae41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tests/unittests/HttpClientCurlTests.cpp:22

  • This test file uses std::function (later in SendAsync_CallbackCopyFailureStillCompletes) but does not include <functional>, relying on transitive includes. Adding the direct include makes the test more robust to future header/include changes.
#include <future>
#include <chrono>
#include <memory>
#include <atomic>
#include <cstdlib>
#include <stdexcept>
#include <utility>

Retain both microsoft#1481's worker-lifetime regression coverage and main's response-size-cap coverage in HttpClientCurlTests.cpp. Update the response-cap fixture comment so it no longer describes the fixed std::async self-join behavior.

Files:
- tests/unittests/HttpClientCurlTests.cpp: resolve additive curl-test conflict

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b12c5862-01e3-45e4-bf91-6389c20cae41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants