[autest] thread_config: add startup polling and skip test on non-Linux by moonchen · Pull Request #12940 · apache/trafficserver

moonchen · 2026-03-04T20:19:14Z

check_threads.py now uses a short bounded poll/retry window so thread-count validation doesn’t fail on startup races.

The test is also skipped on non-Linux because per-thread introspection used by this check is not available there.

check_threads.py now uses a short bounded poll/retry window so thread-count validation doesn’t fail on startup races; the test is also skipped on macOS because per-thread introspection used by this check is not reliably available there.

Copilot

Pull request overview

This PR updates the thread_config gold test to be more reliable during ATS startup by adding a bounded poll/retry window for thread-count validation, and it restricts the test to platforms where per-thread introspection is supported.

Changes:

Add a bounded poll/retry loop in check_threads.py to avoid thread-count validation failing due to startup races.
Skip the thread_config gold test on non-Linux platforms via SkipUnless(IsPlatform("linux")).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
tests/gold_tests/thread_config/thread_config.test.py	Skips the thread configuration test unless running on Linux.
tests/gold_tests/thread_config/check_threads.py	Adds retry-based thread counting and expands ATS process matching logic for more robust detection.

brbzull0

It seems we are addressing the same issue #12916
I'm kinda inclined to yours as it catches a few things mine doesn't.

bryancall

Co-pilot made some recommendations. I think the main one is the exit code. Besides that, looks good.

bryancall

The retry logic looks good overall, but there's an exit code ambiguity that should be fixed:

_count_threads_once() returns code 1 when p.threads() fails ("Could not inspect ATS process threads"), but count_threads() treats code 1 as a retryable "process not found yet" condition in retry_codes = {1, 4, 6, 9, 11}. This means a real permission/psutil error gets silently retried for the full 10s window instead of failing immediately.

Please use a distinct exit code (e.g. 12) for the p.threads() failure case so it's not masked by the retry loop.

moonchen · 2026-03-16T21:51:14Z

The retry logic looks good overall, but there's an exit code ambiguity that should be fixed:

_count_threads_once() returns code 1 when p.threads() fails ("Could not inspect ATS process threads"), but count_threads() treats code 1 as a retryable "process not found yet" condition in retry_codes = {1, 4, 6, 9, 11}. This means a real permission/psutil error gets silently retried for the full 10s window instead of failing immediately.

Please use a distinct exit code (e.g. 12) for the p.threads() failure case so it's not masked by the retry loop.

I've added a distinct exit code 12 for the condition when psutil gets an access denied, so the test fails immediately.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

You can also share your feedback on Copilot code review. Take the survey.

cmcfarlen · 2026-03-18T12:41:40Z

Cherry-picked to 10.2.x

#12940) * thread_config: add startup polling for thread checks and skip on non-Linux check_threads.py now uses a short bounded poll/retry window so thread-count validation does not fail on startup races; the test is also skipped on non-Linux platforms because per-thread introspection used by this check is not reliably available there. * thread_config: stop retrying when Process.threads() access is denied (cherry picked from commit 6dfaadd)

apache#12940) * thread_config: add startup polling for thread checks and skip on non-Linux check_threads.py now uses a short bounded poll/retry window so thread-count validation does not fail on startup races; the test is also skipped on non-Linux platforms because per-thread introspection used by this check is not reliably available there. * thread_config: stop retrying when Process.threads() access is denied (cherry picked from commit 6dfaadd)

* curl 8.20 test update: curl PROXY destination changes (#13239) curl 8.20 intentionally mirrors --haproxy-clientip into both PROXY addresses to keep the header address family consistent. The TSVConnPPInfo AuTest still expected the older destination address, so jobs with newer curl failed even though ATS preserved the PROXY metadata it received. This relaxes the destination-address expectation to accept either curl behavior while continuing to verify the source address and PROXY metadata. This also wraps the long curl command strings while leaving the test's request flow unchanged. (cherry picked from commit ad0ce02) * fedora:44: Trim remap ACL reload waits (#13237) The remap ACL AuTests run hundreds of reload scenarios in a single case, and the Fedora 44 shard is sensitive to extra reload-wait overhead, causing the tests to hang. Their reload sentinel also counted only explicit reloads, even though the log contains the startup load marker too. This replaces the long-lived sleep Ready helper with a short command that exits once the expected reload marker count is present. This also waits for the startup marker plus the explicit reload count, so each scenario observes the reload it just requested. (cherry picked from commit c52eeda) * Align AuTests with latest proxy-verifier checks (#12986) The latest proxy-verifier now fails a run when a verifier server or client is given proxy-side checks for traffic that ATS never produces. Most tests only needed stale proxy-request or proxy-response nodes removed, but the shared replay cases below need server-specific files so each verifier only owns traffic it can actually observe. - disable_pristine_host_hdr_server_canary_false.replay.yaml keeps the canary server on uuid 1 when pristine_host_hdr stays enabled. - disable_pristine_host_hdr_server_canary_true.replay.yaml keeps the canary server on uuid 1 when the Host header is rewritten. - disable_pristine_host_hdr_server_stable.replay.yaml keeps the stable server on uuid 2, which the canary server never receives. - escalate_original_server_default.replay.yaml keeps only the requests that the default-mode origin server really handles. - escalate_failover_server_default.replay.yaml keeps only the GET requests that default-mode escalation sends to failover. - escalate_original_server_non_get.replay.yaml keeps the origin-side subset when --escalate-non-get-methods is enabled. - escalate_failover_server_non_get.replay.yaml keeps the failover subset, including the escalated HEAD request in that mode. - ja4_fingerprint_basic_server.replay.yaml limits the non-preserve test to its one real request instead of preserve-only checks. - traffic_dump_server.yaml keeps origin verification only for the sessions that really reach origin in the main traffic_dump test. - traffic_dump_ip_filter_server.yaml keeps only the /one request used by the traffic_dump IP filter test. These per-server replays preserve fallback server-response directives and client-side coverage while dropping only the proxy-side checks that latest proxy-verifier now correctly reports as unprocessed. (cherry picked from commit ecf505c) * Fix 10.1.x PV replay expectations Adjust two replay expectations after backporting the proxy-verifier 3.1 cleanup. The 10.1.x branch has fewer escalate transactions, and duplicate header checks now observe the combined header value. * [autest] thread_config: add startup polling and skip test on non-Linux (#12940) * thread_config: add startup polling for thread checks and skip on non-Linux check_threads.py now uses a short bounded poll/retry window so thread-count validation does not fail on startup races; the test is also skipped on non-Linux platforms because per-thread introspection used by this check is not reliably available there. * thread_config: stop retrying when Process.threads() access is denied (cherry picked from commit 6dfaadd) * Fix flaky autests for timeout, sigusr2, and thread_config (#13012) * Handle SIGPIPE in ssl-delay-server to prevent helper death when a client disconnects during TLS handshake delay. * Retry accept() on EINTR under heavy parallel load instead of treating the interruption as a fatal error. * Fix accept() error check to use < 0 instead of <= 0 since fd 0 is a valid descriptor when stdin is closed. * Add cmdline matching fallback in check_threads.py for ASAN where the process CWD differs from expected ts_path. (cherry picked from commit ff31470) * Fix 10.1.x JA4 AuTest setup Initialize the branch-local JA4 test helper's preserve flag after backporting replay changes from master. Without it the test fails during Python load before the actual replay runs. * Proxy Verifier: use concise stack protocol specification (#13003) Proxy Verifier v3.0.0 has a more concise `stack` configurable for `protocol` specification. This makes use of that over the more verbose full `protocol` sequence. * tests/gold_tests/headers tests: use ATSReplayTest (#13033) Move the replay-friendly headers gold tests to ATSReplayTest wrappers and describe their ATS, origin, and client setup in replay YAML. This keeps the cache, range, redirect, HSTS, and alternate handling coverage while making the tests easier to read and maintain, and removes the old gold files left orphaned by the conversion. (cherry picked from commit 49cb7c8) * Adapt headers replay tests for 10.1.x --------- Co-authored-by: bneradt <bneradt@yahooinc.com> Co-authored-by: Mo Chen <mochen@apache.org> Co-authored-by: Bryan Call <bcall@apache.org>

moonchen requested review from bneradt and Copilot March 4, 2026 20:20

moonchen self-assigned this Mar 4, 2026

moonchen added the AuTest label Mar 4, 2026

moonchen added this to 11-Dev Mar 4, 2026

moonchen added this to the 11.0.0 milestone Mar 4, 2026

Copilot started reviewing on behalf of moonchen March 4, 2026 20:21 View session

Copilot AI reviewed Mar 4, 2026

View reviewed changes

Comment thread tests/gold_tests/thread_config/check_threads.py Outdated

Comment thread tests/gold_tests/thread_config/check_threads.py

Comment thread tests/gold_tests/thread_config/thread_config.test.py

brbzull0 reviewed Mar 5, 2026

View reviewed changes

bryancall self-requested a review March 9, 2026 22:22

bryancall removed this from 11-Dev Mar 9, 2026

bryancall added this to ATS v10.2.x Mar 9, 2026

bryancall requested changes Mar 12, 2026

View reviewed changes

Stop retrying if Process.threads() gets denied

488e964

moonchen requested a review from bryancall March 16, 2026 21:51

bryancall requested a review from Copilot March 16, 2026 22:38

Copilot started reviewing on behalf of bryancall March 16, 2026 22:38 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

bryancall changed the title ~~[autest] thread_config: fix race and skip on macOS~~ [autest] thread_config: add startup polling and skip test on non-Linux Mar 17, 2026

bryancall approved these changes Mar 17, 2026

View reviewed changes

bryancall merged commit 6dfaadd into apache:master Mar 17, 2026
19 checks passed

github-project-automation Bot moved this to For v10.2.0 in ATS v10.2.x Mar 17, 2026

cmcfarlen moved this from For v10.2.0 to Picked v10.2.0 in ATS v10.2.x Mar 18, 2026

cmcfarlen modified the milestones: 11.0.0, 10.2.0 Mar 18, 2026

brbzull0 mentioned this pull request Apr 10, 2026

thread_config test - add visibility and some defensive improvements. #12916

Closed

Navigation Menu

Uh oh!

[autest] thread_config: add startup polling and skip test on non-Linux - #12940

[autest] thread_config: add startup polling and skip test on non-Linux#12940
bryancall merged 2 commits into
apache:masterfrom
moonchen:fix-check-threads-race

moonchen commented Mar 4, 2026 •

edited by bryancall

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brbzull0 left a comment

Uh oh!

bryancall left a comment

Uh oh!

bryancall left a comment

Uh oh!

moonchen commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

cmcfarlen commented Mar 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

moonchen commented Mar 4, 2026 • edited by bryancall Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brbzull0 left a comment

Choose a reason for hiding this comment

Uh oh!

bryancall left a comment

Choose a reason for hiding this comment

Uh oh!

bryancall left a comment

Choose a reason for hiding this comment

Uh oh!

moonchen commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

cmcfarlen commented Mar 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

moonchen commented Mar 4, 2026 •

edited by bryancall

Loading