Skip to content

fix(admin): keep cluster election scheduled after failures (#6495) - #7406

Open
sunnysabor wants to merge 2 commits into
apache:masterfrom
sunnysabor:fix/6495-master-election-retry
Open

sunnysabor wants to merge 2 commits into
apache:masterfrom
sunnysabor:fix/6495-master-election-retry

Conversation

@sunnysabor

Copy link
Copy Markdown
Contributor

Fixes #6495.

Summary

A transient exception from doSelectMaster escaped the scheduleAtFixedRate callback and cancelled every later master-election attempt. Catch runtime failures at the scheduled-task boundary so a failed election is logged and the next configured period can retry. The existing service cleanup and lock release behavior remains in place; fatal Errors are not swallowed.

Tests

  • Added ShenyuClusterServiceTest#testMasterSelectionContinuesAfterFailure: the first election attempt throws, the following scheduled attempt runs, and failure cleanup/lock release are verified.
  • ./mvnw -B -ntp -pl shenyu-admin -am -Dtest=ShenyuClusterServiceTest -DfailIfNoTests=false test — passed (38-module reactor, 1 test, Checkstyle clean).

@sunnysabor

Copy link
Copy Markdown
Contributor Author

CI attribution: pr_build fails before reaching the changed shenyu-admin module in shenyu-client-core at HeartbeatFailureIsolationTest.failureDoesNotSuppressOtherUrisOrSubsequentTicks (NoSuchElementException, line 57). The PR diff contains only the admin service and its new test. I reproduced the same failure on a clean upstream/master worktree (09c6a528) with ./mvnw -B -ntp -pl shenyu-client/shenyu-client-core -am -Dtest=HeartbeatFailureIsolationTest -DfailIfNoTests=false test. The new ShenyuClusterServiceTest passed three times with the 38-module admin reactor and Checkstyle clean. The other triggered CI jobs are still running; I will update if any report a failure relevant to this change.

@sunnysabor

Copy link
Copy Markdown
Contributor Author

CI follow-up: the remaining checks have now completed. All triggered Kubernetes/integration jobs, database storage E2E jobs, protocol E2E cases, Docker image build, license/header checks, and Java analysis passed. The only failing check is still pr_build, at HeartbeatFailureIsolationTest.failureDoesNotSuppressOtherUrisOrSubsequentTicks in shenyu-client-core (NoSuchElementException, line 57); I reproduced this on clean upstream/master (09c6a528) with the focused module-reactor command recorded above. The aggregate build gate fails only because pr_build failed. The changed admin test passed three times locally through the 38-module reactor with Checkstyle clean.

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same class of bug as #7414, correctly diagnosed: ScheduledThreadPoolExecutor.scheduleAtFixedRate permanently cancels the period the moment the task throws, so a single transient lock failure used to kill master election for the remaining life of the process. Wrapping the invocation and preserving the cause in ShenyuException(message, e) instead of discarding the stack trace are both right.

One problem with the new test — ShenyuClusterServiceTest#testMasterSelectionContinuesAfterFailure can fail on a loaded machine:

assertTrue(attemptsFinished.await(5, TimeUnit.SECONDS));
verify(selectMasterService, atLeast(2)).selectMaster(...);
verify(selectMasterService, times(2)).releaseMaster();   // <-- racy

selectPeriod is 1 second and nothing stops the scheduler between the latch opening and the verify calls. A third tick can land in that window, and releaseMaster() runs for every attempt (it is in the finally of doSelectMaster), so times(2) turns into a failure purely on timing. You already used atLeast(2) for selectMaster, which is the correct tolerance; releaseMaster needs the same, or better, stop the clock before asserting:

assertTrue(attemptsFinished.await(5, TimeUnit.SECONDS));
executorService.shutdownNow();
assertTrue(executorService.awaitTermination(5, TimeUnit.SECONDS));
verify(selectMasterService, atLeast(2)).selectMaster("127.0.0.1", "9195", "/shenyu");
verify(selectMasterService, times(2)).releaseMaster();

shutdownNow() is idempotent, so leaving the one in the finally as a safety net is fine.

Minor: the wrapper catches RuntimeException while the equivalent guard added in InstanceCheckService (#7414) catches Exception. Nothing here throws a checked exception today so both work, but settling on one convention across the codebase would be easier to maintain.

@sunnysabor
sunnysabor force-pushed the fix/6495-master-election-retry branch from 2314a6f to fe42651 Compare October 2, 2026 04:07
@sunnysabor

Copy link
Copy Markdown
Contributor Author

Addressed in signed-off commit fe42651e5. The periodic executor remains active after the latch opens, so releaseMaster() is now verified with atLeast(2) just like selectMaster(); later scheduled ticks can no longer make the test fail on an exact call count. I also aligned the wrapper with the Exception handling convention used in the related instance-sync fix.

Validation passed on the rebased current upstream/master base:

./mvnw -B -ntp -pl shenyu-admin -am -Dtest=ShenyuClusterServiceTest -Dsurefire.failIfNoSpecifiedTests=false -Djacoco.skip=true -Dmaven.javadoc.skip=true -Drat.skip=true test

The target test passed (1 test), the 38-module dependency reactor completed, Checkstyle reported 0 violations, and git diff --check passed.

@sunnysabor

Copy link
Copy Markdown
Contributor Author

CI update for head fe42651e56dbe4838a55225d7ad21b3563acd5cc: pr_build and its dependent aggregate build failed before reaching shenyu-admin on the known shenyu-client-core HeartbeatFailureIsolationTest.failureDoesNotSuppressOtherUrisOrSubsequentTicks error (NoSuchElementException, line 57; #7401). This module is unchanged in this PR, and the failure has already been reproduced on clean upstream/master and is covered by the separate #7401 fix PRs #7402/#7403. License/header, changes, and migration checks passed. The remaining hosted integration, K8s, Docker, and Java analysis jobs are still pending; I will report them once terminal.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Cluster master election scheduler can stop after one renewal exception

2 participants