[copybara-service]

tcp: Use SYN cookies after shrinking a listener's backlog.

Calling Listen again on a listening endpoint can reduce the accept queue capacity while preserving existing pending handshakes. The resize check only considers completed connections in the accept queue, so the pending count can exceed the new SYN backlog of capacity - 1.

The SYN cookie guard currently checks only for equality. After a reduction below the pending count, new SYNs bypass this guard and allocate additional pending handshake state instead of using SYN cookies.

Change the guard to >=. The capacity - 1 relationship accounts for the extra accept queue slot added by the syscall layer; it does not imply that existing pending handshakes must always fit within a reduced backlog. The limit controls admission of new pending handshakes. When the completed accept queue has space, use SYN cookies whenever the pending count is at or above that limit.

Keep the existing Listen resize policy and completed-queue checks unchanged. Enforcing pending <= capacity - 1 in Listen would instead require rejecting otherwise permitted reductions, retaining a larger capacity than requested, or aborting existing handshakes. None of those behavior changes is needed for this fix. Do not move or duplicate the syscall layer's +1 adjustment.

This matches the queue-full comparison in the Linux revision referenced by the existing comment: inet_csk_reqsk_queue_is_full() uses >=, while inet_listen() updates the backlog of an already-listening socket without rejecting the update based on the pending request count. This is a comparison of these specific code paths, not a claim of complete Listen compatibility.

Update the guard's comment to explain both capacity - 1 and why the pending count may already exceed the limit. Add
TestListenerShrinkingBacklogUsesCookies: create two pending handshakes with Endpoint.Listen(4), reduce the capacity with Endpoint.Listen(2), and verify that the next SYN uses a cookie without adding a pending endpoint. Verify that the cookie ACK still produces an accept-ready connection with the expected peer. Use a manual stack clock, port and ACK matching, and listener-mutex synchronization to keep state and statistics checks deterministic with respect to handshake timers and segment processing.

Linux references: backlog update and pending queue-full comparison.

Assisted-by: ChatGPT
FUTURE_COPYBARA_INTEGRATE_REVIEW=#15038 from 134130:134130/fix-listener-backlog 2269c07