Skip to content

Stabilize post and slow_post autests under parallel runs - #12886

Merged
bryancall merged 5 commits into
apache:masterfrom
bryancall:fix-thread-config-port-readiness
Feb 19, 2026
Merged

Stabilize post and slow_post autests under parallel runs#12886
bryancall merged 5 commits into
apache:masterfrom
bryancall:fix-thread-config-port-readiness

Conversation

@bryancall

@bryancall bryancall commented Feb 13, 2026

Copy link
Copy Markdown
Contributor

Problem

The post-early-return and slow_post/server_abort autests fail intermittently under parallel execution on Fedora 43. The nc -l based mock server only accepts a single TCP connection, which gets consumed by the readiness probe, leaving nothing for ATS to connect to.

Changes

  • Stabilize thread_config startup ordering -- gate each check_threads invocation on ATS port readiness using When.PortOpen(ts.Variables.port) to avoid races where validation runs before traffic_server is ready.
  • Stabilize post and slow_post autests -- add explicit origin readiness checks in post-early-return and accept the current microserver exception variant in server_abort for Fedora 43 compatibility.
  • Revert thread_config PortOpen change per review -- keep ATS startup gating on existing fully-initialized readiness behavior. Move post-early-return upstream readiness to server process Ready fields instead of StartBefore calls.
  • Replace nc-based server with mock_origin.py -- server1.sh only accepts a single TCP connection, consumed by the When.PortOpen() readiness probe. mock_origin.py absorbs probes, sends the configured HTTP response, and drains request data before closing to prevent TCP RST that causes 502 on H2 streams.
  • Remove unused server1.sh -- replaced by mock_origin.py.

Testing

  • 5/5 parallel ASAN runs on hera2
  • All 15 CI platforms green

Gate each check_threads invocation on ATS port readiness using
When.PortOpen(ts.Variables.port) to avoid races where validation runs before
traffic_server is ready.
@bryancall
bryancall requested a review from Copilot February 13, 2026 18:26
@bryancall bryancall self-assigned this Feb 13, 2026
@bryancall bryancall added this to the 11.0.0 milestone Feb 13, 2026
Add explicit origin readiness checks in post-early-return and accept the current microserver exception variant in server_abort so these tests pass consistently on Fedora 43.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves determinism in the thread_config gold test by ensuring the thread-validation helper (check_threads.py) does not run until the corresponding ATS instance is actually accepting connections on its configured port, eliminating a startup-order race in parallel autest runs.

Changes:

  • Add per-run readiness gating so each check_threads.py invocation waits on When.PortOpen(ts.Variables.port) before proceeding.
  • Apply the same readiness condition consistently across all thread-configuration variants in the test.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/gold_tests/post/post-early-return.test.py Outdated
Comment thread tests/gold_tests/thread_config/thread_config.test.py Outdated
Revert the thread_config PortOpen readiness change and keep ATS process
startup gating on its existing fully-initialized readiness behavior. For
post-early-return, move upstream readiness to server process Ready fields
instead of passing ready via StartBefore calls.
@bryancall bryancall changed the title Stabilize thread_config autest startup ordering Stabilize post and slow_post autests under parallel runs Feb 13, 2026
bneradt
bneradt previously approved these changes Feb 13, 2026
@apache apache deleted a comment from bryancall Feb 13, 2026
@maskit

maskit commented Feb 16, 2026

Copy link
Copy Markdown
Member

[approve ci autest 1]

The nc-l based server1.sh only accepts a single TCP connection, which
is consumed by the When.PortOpen() readiness probe, leaving nothing for
ATS to connect to.  Replace with a new shared mock_origin.py tool that:

  - Absorbs PortOpen readiness probes (connect/disconnect with no data)
  - Sends the configured HTTP response after reading request headers
  - Drains remaining request data before closing, preventing TCP RST
    that causes ATS to return 502 on HTTP/2 streams

Tested 5/5 parallel ASAN runs passing on hera2.
@bryancall

Copy link
Copy Markdown
Contributor Author

The consistent AuTest 1of4 failure was caused by a fundamental incompatibility between nc -l and When.PortOpen():

nc -l only accepts a single TCP connection. The PortOpen readiness probe connects to check the port is open, consuming that one connection. When ATS then tries to proxy the request to the origin, the port is dead → 502 every time. This is why it failed deterministically on all 4 CI retries.

The fix (pushed in 3cf53d3) replaces server1.sh with a new shared tests/tools/mock_origin.py that:

  1. Uses Python socket.listen() which handles multiple accept() calls, so readiness probes are absorbed silently
  2. Sends the configured HTTP response after reading request headers
  3. Drains remaining request data before closing — without this, closing the connection mid-POST causes a TCP RST that makes ATS return 502 on HTTP/2 streams (this was a second issue discovered during testing)

mock_origin.py is a reusable tool that can replace all the ad-hoc nc -l server scripts in the test suite (post/server1.sh, chunked_encoding/server2-4.sh, post_slow_server/server.sh). It supports configurable status codes, response bodies (content-length or chunked), and response delays via CLI args.

Tested 5/5 parallel ASAN runs passing on a local dev machine. The previous mock_origin.py without the drain fix was failing ~2/3 parallel runs on the HTTP/2 test case.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@maskit

maskit commented Feb 17, 2026

Copy link
Copy Markdown
Member

Fatal: ../src/api/InkAPITest.cc:916: failed assertion (event == TS_EVENT_NET_ACCEPT) || (event == TS_EVENT_NET_ACCEPT_FAILED)

I see this error on my PR as well.

@maskit

maskit commented Feb 17, 2026

Copy link
Copy Markdown
Member

Can we remove server1.sh ?

@bryancall bryancall closed this Feb 18, 2026
@bryancall bryancall reopened this Feb 18, 2026
@bryancall bryancall closed this Feb 18, 2026
@bryancall bryancall reopened this Feb 18, 2026
@bryancall

Copy link
Copy Markdown
Contributor Author

Yes, it can be removed. I will push a new commit to delete it.

@bryancall
bryancall merged commit a1ac1db into apache:master Feb 19, 2026
15 checks passed
cmcfarlen pushed a commit that referenced this pull request Jun 9, 2026
* Stabilize thread_config autest startup ordering -- gate check_threads
  on ATS port readiness to avoid validation races.
* Stabilize post/slow_post autests -- add explicit origin readiness checks
  in post-early-return; accept microserver exception variant for Fedora 43.
* Revert thread_config PortOpen change per review -- keep ATS startup gating
  on existing readiness behavior, move upstream readiness to server Ready fields.
* Replace nc-based server with mock_origin.py -- server1.sh only accepts one
  TCP connection (consumed by readiness probe). mock_origin.py absorbs probes,
  sends configured response, and drains request data to prevent TCP RST / 502.
* Remove unused server1.sh replaced by mock_origin.py

(cherry picked from commit a1ac1db)
@cmcfarlen cmcfarlen modified the milestones: 11.0.0, 10.2.0 Jun 16, 2026
cmcfarlen pushed a commit that referenced this pull request Aug 10, 2026
…3525)

* Stabilize thread_config autest startup ordering -- gate check_threads
  on ATS port readiness to avoid validation races.
* Stabilize post/slow_post autests -- add explicit origin readiness checks
  in post-early-return; accept microserver exception variant for Fedora 43.
* Revert thread_config PortOpen change per review -- keep ATS startup gating
  on existing readiness behavior, move upstream readiness to server Ready fields.
* Replace nc-based server with mock_origin.py -- server1.sh only accepts one
  TCP connection (consumed by readiness probe). mock_origin.py absorbs probes,
  sends configured response, and drains request data to prevent TCP RST / 502.
* Remove unused server1.sh replaced by mock_origin.py

(cherry picked from commit a1ac1db)

Co-authored-by: Bryan Call <bcall@apache.org>
@cmcfarlen cmcfarlen modified the milestones: 10.2.0, Backported Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants