Skip to content

Fix JSONRPC server shutdown race - #13461

Merged
bneradt merged 1 commit into
apache:masterfrom
bneradt:fix-jsonrpc-server-shutdown-race
Jul 30, 2026
Merged

Fix JSONRPC server shutdown race#13461
bneradt merged 1 commit into
apache:masterfrom
bneradt:fix-jsonrpc-server-shutdown-race

Conversation

@bneradt

@bneradt bneradt commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

JSONRPC server shutdown can race with worker thread startup. When the
worker starts after stop_thread(), it restores the running flag and
polls a closed socket indefinitely. This causes test_jsonrpcserver and
process shutdown to hang.

This marks the socket server as running before the worker is created,
so a concurrent stop cannot be overwritten. It also passes the owning
server to the worker instead of relying on the mutable global server
pointer.

JSONRPC server shutdown can race with worker thread startup. When the
worker starts after stop_thread(), it restores the running flag and
polls a closed socket indefinitely. This causes test_jsonrpcserver and
process shutdown to hang.

This marks the socket server as running before the worker is created,
so a concurrent stop cannot be overwritten. It also passes the owning
server to the worker instead of relying on the mutable global server
pointer.
@bneradt bneradt added this to the 11.0.0 milestone Jul 30, 2026
Copilot AI review requested due to automatic review settings July 30, 2026 19:32
@bneradt bneradt self-assigned this Jul 30, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@bneradt bneradt added JSONRPC JSONRPC 2.0 related work. Bug labels Jul 30, 2026
@brbzull0
brbzull0 requested a review from Copilot July 30, 2026 19:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/mgmt/rpc/server/RPCServer.cc:67

  • The worker thread runs the optional cb_init (e.g. TSThreadInit) and stores the returned handle in _rpcThread, but the matching cb_destroy is never invoked. For TSThreadDestroy specifically, the API requires the same thread that created the TSThread to destroy it (see src/api/InkIOCoreAPI.cc:202-205), so destroying it from stop_thread (caller thread) would be incorrect. This should be cleaned up in the worker thread after _socketImpl->run() returns to avoid leaking per-thread state and to respect the required destroy-thread affinity.
  if (server->_init) {
    server->_rpcThread = server->_init();
  }
  server->_socketImpl->run();
  Dbg(dbg_ctl, "Socket stopped");

@bneradt
bneradt merged commit 8b9a007 into apache:master Jul 30, 2026
15 checks passed
@bneradt
bneradt deleted the fix-jsonrpc-server-shutdown-race branch July 30, 2026 21:09
@github-project-automation github-project-automation Bot moved this to For v10.2.0 in ATS v10.2.x Jul 30, 2026
cmcfarlen pushed a commit to cmcfarlen/trafficserver that referenced this pull request Jul 30, 2026
JSONRPC server shutdown can race with worker thread startup. When the
worker starts after stop_thread(), it restores the running flag and
polls a closed socket indefinitely. This causes test_jsonrpcserver and
process shutdown to hang.

This marks the socket server as running before the worker is created,
so a concurrent stop cannot be overwritten. It also passes the owning
server to the worker instead of relying on the mutable global server
pointer.

(cherry picked from commit 8b9a007)
@cmcfarlen cmcfarlen modified the milestones: 11.0.0, 10.2.0 Jul 31, 2026
@cmcfarlen cmcfarlen moved this from For v10.2.0 to Picked v10.2.0 in ATS v10.2.x Jul 31, 2026
@cmcfarlen

Copy link
Copy Markdown
Contributor

Cherry-picked to the 10.2.x branch as 1427ebf for the 10.2.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug JSONRPC JSONRPC 2.0 related work.

Projects

Status: Picked v10.2.0

Development

Successfully merging this pull request may close these issues.

3 participants