Forward multiprocessing.current_process().authkey to workers #9122

Activity

1. moi90 commented on Oct 8, 2025

   

   moi90

   on Oct 8, 2025

   ContributorAuthor

   More actions

   For now, I solved this with a WorkerPlugin:

   class AuthkeyFromEnvPlugin(distributed.diagnostics.plugin.WorkerPlugin):
       """
       A Dask WorkerPlugin to set the authkey for multiprocessing on the worker process.
   
       By default, `sbatch` forwards all of the user's environment to the job.
       """
   
       def setup(self, worker):
           import multiprocessing.process
   
           print("Setting authkey on worker process")
   
           multiprocessing.process.current_process().authkey = os.environ[
               "MULTIPROCESSING_AUTHKEY"
           ].encode("ascii")
   
   # Generate a random authkey for the main process and all workers
   authkey = secrets.token_hex(32)
   os.environ["MULTIPROCESSING_AUTHKEY"] = authkey
   multiprocessing.process.current_process().authkey = authkey.encode("ascii")
   
   with (
       SLURMCluster(
           memory="1g",
           processes=1,
           cores=1,
           queue=queue,
           account=account,
           interface=interface,
           name="richer-progress-test",
       ) as cluster,
       Client(cluster) as client,
   ):
       client.register_plugin(AuthkeyFromEnvPlugin())
   
       ...

   Maybe this can be integrated into dask-jobqueue (or maybe distributed)?
2. jacobtomlinson

   transferred this issue fromdask/dask-jobqueueon Oct 9, 2025
3. github-actions

   added

   needs triage

   on Oct 9, 2025
4. jacobtomlinson commented on Oct 9, 2025

   

   jacobtomlinson

   on Oct 9, 2025

   Member

   More actions

   Thanks for writing this up and also documenting your workaround. This problem feels broad enough that we should put this in distributed. Do you have any interest in opening a PR that handles this as part of the core distributed functionality?

   👍React with 👍1Reacted by Simon-Martin Schröder
5. jacobtomlinson

   added

   bugSomething is brokenSomething is broken

   and removed

   needs triage

   on Oct 9, 2025
6. moi90 commented on Oct 10, 2025

   

   moi90

   on Oct 10, 2025

   ContributorAuthor

   More actions

   Sure!