strange "Error. No more retries." problem

[esmq2092]

the origin server has round-robin DNS entries which resolve to 4 ip addresses,

i have the following settings
connect_attempts_rr_retries set to 1
connect_attempts_max_retries set to 4

when 3 ips goes down，there is a small chance that ats will return 502，

turn on debug log, it says

[handle_response_from_server] Error. No more retries.

but actually ats only try 2 or 3 of ips, no all of the origins.

the following debug shows that ats only try 3 ips, why not try the fourth？

---

[Sep 6 16:20:15.803] Server {0x2afd53046700} DEBUG: (http) [196] open connection to test.xx.example.com.cats.foobar.com: 22.33.4.147:80
[Sep 6 16:20:15.803] Server {0x2afd53046700} DEBUG: (http) calling netProcessor.connect_re
[Sep 6 16:20:15.803] Server {0x2afd53046700} DEBUG: (http_ss) [196] max number of connections: 6000
[Sep 6 16:20:15.803] Server {0x2afd53046700} DEBUG: (http_ss) [167] new connection, ip: 22.33.4.147:80, count: 1
[Sep 6 16:20:26.216] Server {0x2afd53046700} DEBUG: (http_ss) [167] connection closed, ip: 22.33.4.147:80, count: 0
[Sep 6 16:20:26.216] Server {0x2afd53046700} DEBUG: (http_trans) [196] [0] failed to connect to 22.33.4.147
[Sep 6 16:20:26.216] Server {0x2afd53046700} DEBUG: (http) [196] open connection to test.xx.example.com.cats.foobar.com: 55.66.158.21:80
[Sep 6 16:20:26.216] Server {0x2afd53046700} DEBUG: (http) calling netProcessor.connect_re
[Sep 6 16:20:26.217] Server {0x2afd53046700} DEBUG: (http_ss) [196] max number of connections: 6000
[Sep 6 16:20:26.217] Server {0x2afd53046700} DEBUG: (http_ss) [169] new connection, ip: 55.66.158.21:80, count: 1
[Sep 6 16:20:37.213] Server {0x2afd53046700} DEBUG: (http_ss) [169] connection closed, ip: 55.66.158.21:80, count: 0
[Sep 6 16:20:37.213] Server {0x2afd53046700} DEBUG: (http_trans) [196] [1] failed to connect to 55.66.158.21
[Sep 6 16:20:37.213] Server {0x2afd53046700} DEBUG: (http) [196] open connection to test.xx.example.com.cats.foobar.com: 55.66.158.20:80
[Sep 6 16:20:37.213] Server {0x2afd53046700} DEBUG: (http) calling netProcessor.connect_re
[Sep 6 16:20:37.214] Server {0x2afd53046700} DEBUG: (http_ss) [196] max number of connections: 6000
[Sep 6 16:20:37.214] Server {0x2afd53046700} DEBUG: (http_ss) [171] new connection, ip: 55.66.158.20:80, count: 1
[Sep 6 16:20:48.202] Server {0x2afd53046700} DEBUG: (http_ss) [171] connection closed, ip: 55.66.158.20:80, count: 0
[Sep 6 16:20:48.202] Server {0x2afd53046700} DEBUG: (http_trans) [196] [handle_response_from_server] Error. No more retries.
[Sep 6 16:20:48.202] Server {0x2afd53046700} DEBUG: (http_trans) [196] [handle_server_connection_not_open] (hscno)
[Sep 6 16:20:48.202] Server {0x2afd53046700} DEBUG: (http_seq) [196] [HttpTransact::handle_server_connection_not_open]

[bryancall]

The connect-attempt retry path, including round-robin handling, has been comprehensively reworked since this was filed. PR #13102 ("Fix connect attempt retries") notes that the three retry settings (connect_attempts_max_retries, connect_attempts_max_retries_down_server, and connect_attempts_rr_retries) were not implemented correctly before that change, and it fixes both the round-robin member selection and the per-host versus global retry counting that caused Traffic Server to give up before exhausting all round-robin addresses. That fix shipped first in release 10.2.0, and a follow-up added an automated test covering round-robin retries and max retries. I am closing this as fixed. If you can still reproduce the early give-up on 10.2.0 or later, please reopen with a fresh reproduction and the retry settings you are using.