test_work_stealing_on_scaling_up failing #336

Activity

1. fjetter commented on Sep 16, 2022

   

   fjetter

   on Sep 16, 2022

   Last edited by fjetter

   Contributor

   More actions

   I'd like to point out that this is not a flakiness problem since it now fails very reliably. There appears to be a problem with the software environments

     +-------------+-----------------------+-----------+----------+
     | Package     | client                | scheduler | workers  |
     +-------------+-----------------------+-----------+----------+
     | dask        | 2022.9.0+22.g982376ef | 2022.6.0  | 2022.6.0 |
     | distributed | 2022.9.0+13.g1fd07f03 | 2022.6.0  | 2022.6.0 |
     +-------------+-----------------------+-----------+----------+
   

   Did this fail before package_sync as well? The CI jobs you are pointing to are using package_sync.
2. fjetter commented on Sep 16, 2022

   

   fjetter

   on Sep 16, 2022

   Contributor

   More actions

   This test is using cluster.scale to upscale an existing cluster. @shughes-uk is this supported with package sync?
3. fjetter

   changed the title [-]`test_work_stealing_on_scaling_up` flaky[/-] [+]`test_work_stealing_on_scaling_up` failing[/+] on Sep 16, 2022
4. hendrikmakait commented on Sep 16, 2022

   

   hendrikmakait

   on Sep 16, 2022

   Last edited by hendrikmakait

   Contributor

   More actions

   Note the difference between the mismatch on CI

     +-------------+-----------------------+-----------+----------+
     | Package     | client                | scheduler | workers  |
     +-------------+-----------------------+-----------+----------+
     | dask        | 2022.9.0+25.g1a8533fd | 2022.6.0  | 2022.6.0 |
     | distributed | 2022.9.0+13.g1fd07f03 | 2022.6.0  | 2022.6.0 |
     +-------------+-----------------------+-----------+----------+
   

   and the mismatch for worker i-04790bd562d506836 on the corresponding cluster

   Sep 16 00:34:42 ip-10-0-11-96 cloud-init[1125]: +-------------+-------------+-----------+---------------------------------------+
   Sep 16 00:34:42 ip-10-0-11-96 cloud-init[1125]: | Package     | This Worker | scheduler | workers                               |
   Sep 16 00:34:42 ip-10-0-11-96 cloud-init[1125]: +-------------+-------------+-----------+---------------------------------------+
   Sep 16 00:34:42 ip-10-0-11-96 cloud-init[1125]: | dask        | 2022.6.0    | 2022.6.0  | {'2022.6.0', '2022.9.0+25.g1a8533fd'} |
   Sep 16 00:34:42 ip-10-0-11-96 cloud-init[1125]: | distributed | 2022.6.0    | 2022.6.0  | {'2022.6.0', '2022.9.0+13.g1fd07f03'} |
   Sep 16 00:34:42 ip-10-0-11-96 cloud-init[1125]: +-------------+-------------+-----------+---------------------------------------+
   
5. ntabris commented on Sep 16, 2022

   

   ntabris

   on Sep 16, 2022

   Member

   More actions

   possibly same problem as https://github.com/coiled/platform/issues/84
6. ian-r-rose commented on Sep 16, 2022

   

   ian-r-rose

   on Sep 16, 2022

   ContributorAuthor

   More actions

   Thanks @ntabris, is this deployed yet? If not, I'd propose skipping this test until then.
7. ntabris commented on Sep 16, 2022

   

   ntabris

   on Sep 16, 2022

   Member

   More actions

   is this deployed yet?

   No, we were waiting on pyzmq wheels for linux x86 which are now out. I'd expect Coiled release today or Monday.

   👍React with 👍1Reacted by Florian Jetter
8. shughes-uk commented on Sep 17, 2022

   

   shughes-uk

   on Sep 17, 2022

   Contributor

   More actions

   Should be deployed now
9. ian-r-rose

   closed this as completedin #351on Sep 20, 2022