You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Restart worker via Nanny on connection failure #6387
With #6361, any temporary network disconnect will shut down the worker.
If we have a Nanny around, we could at least let the Nanny restart the worker, so it can try to reconnect from a fresh state. Currently, we tell the Nanny to close gracefully.
There are a couple of things to think about:
Make sure that, if the scheduler has asked us to close and then closes the connection, the Nanny doesn't try to restart
The Nanny also reports worker loss to the scheduler. This shouldn't happen until the worker process is already closed, which means the network connection should already be closed, so the scheduler should have already removed the worker itself, but think about race conditions here.
Activity
Remove `Worker.terminate`
Restart worker if it's unrecognized by scheduler
fjetter commentedon Nov 25, 2022
I think this is blocked by #7323