fix(admin): keep cluster election scheduled after failures (#6495) - #7406
sunnysabor wants to merge 2 commits into
Conversation
|
CI attribution: |
|
CI follow-up: the remaining checks have now completed. All triggered Kubernetes/integration jobs, database storage E2E jobs, protocol E2E cases, Docker image build, license/header checks, and Java analysis passed. The only failing check is still |
Aias00
left a comment
There was a problem hiding this comment.
Same class of bug as #7414, correctly diagnosed: ScheduledThreadPoolExecutor.scheduleAtFixedRate permanently cancels the period the moment the task throws, so a single transient lock failure used to kill master election for the remaining life of the process. Wrapping the invocation and preserving the cause in ShenyuException(message, e) instead of discarding the stack trace are both right.
One problem with the new test — ShenyuClusterServiceTest#testMasterSelectionContinuesAfterFailure can fail on a loaded machine:
assertTrue(attemptsFinished.await(5, TimeUnit.SECONDS));
verify(selectMasterService, atLeast(2)).selectMaster(...);
verify(selectMasterService, times(2)).releaseMaster(); // <-- racyselectPeriod is 1 second and nothing stops the scheduler between the latch opening and the verify calls. A third tick can land in that window, and releaseMaster() runs for every attempt (it is in the finally of doSelectMaster), so times(2) turns into a failure purely on timing. You already used atLeast(2) for selectMaster, which is the correct tolerance; releaseMaster needs the same, or better, stop the clock before asserting:
assertTrue(attemptsFinished.await(5, TimeUnit.SECONDS));
executorService.shutdownNow();
assertTrue(executorService.awaitTermination(5, TimeUnit.SECONDS));
verify(selectMasterService, atLeast(2)).selectMaster("127.0.0.1", "9195", "/shenyu");
verify(selectMasterService, times(2)).releaseMaster();shutdownNow() is idempotent, so leaving the one in the finally as a safety net is fine.
Minor: the wrapper catches RuntimeException while the equivalent guard added in InstanceCheckService (#7414) catches Exception. Nothing here throws a checked exception today so both work, but settling on one convention across the codebase would be easier to maintain.
Signed-off-by: jerry <1394367234@qq.com>
2314a6f to
fe42651
Compare
|
Addressed in signed-off commit Validation passed on the rebased current
The target test passed (1 test), the 38-module dependency reactor completed, Checkstyle reported 0 violations, and |
|
CI update for head |
Fixes #6495.
Summary
A transient exception from
doSelectMasterescaped thescheduleAtFixedRatecallback and cancelled every later master-election attempt. Catch runtime failures at the scheduled-task boundary so a failed election is logged and the next configured period can retry. The existing service cleanup and lock release behavior remains in place; fatalErrors are not swallowed.Tests
ShenyuClusterServiceTest#testMasterSelectionContinuesAfterFailure: the first election attempt throws, the following scheduled attempt runs, and failure cleanup/lock release are verified../mvnw -B -ntp -pl shenyu-admin -am -Dtest=ShenyuClusterServiceTest -DfailIfNoTests=false test— passed (38-module reactor, 1 test, Checkstyle clean).