Repository navigation
Flaky test: TestUpdaterRollbackAndPauseNoMonitor #2137
Description
--- FAIL: TestUpdaterRollbackAndPauseNoMonitor (1.05s)
Error Trace: testutils.go:26
update_test.go:210
update_test.go:23
Error: no task creation
Seen here: https://circleci.com/gh/docker/swarmkit/7016
Activity
[-]Flaky test: https://circleci.com/gh/docker/swarmkit/7016[/-][+]Flaky test: TestUpdaterRollbackAndPauseNoMonitor[/+]ijc commentedon Jun 8, 2017
I saw this a bunch of times when testing #2221/#2224. Twice on master (out of 20 iterations, labelled
beforein the logs below), once with the fix for #2224 and an updated grpc (out of 20 iterations, labelledfixin the logs below) and three times with #2222+#2223+#2224 and an updated grpc (out of 20 iterations, labelledfix+in the logs below).In every case the failure was preceded by at least one (sometimes two) messages of the general form:
aaronlehmann commentedon Jun 13, 2017
The test is occasionally exposing a race condition. A task failure separately triggers the update rollback and makes the orchestrator change the desired state of the task to "shutdown". Generally the desired state change happens first, but if the failure happens very soon after the task was started, it's possible for the rollback to begin first. In this case, the rollback process will try to shut down a task which has already been shut down, and to avoid the possibility of creating an extra replica, it won't start a replacement task for something it hasn't shut down itself.
#2242 has more information on the problem and possible solutions (this is case (2) in that issue).
aaronlehmann commentedon Jun 13, 2017
(Note that this test sets the restart condition to "none", otherwise the orchestrator would restart the failed task, and we'd still have the expected number of replicas.)