Repository navigation

Constrained spill - #5543

Merged

crusaderky merged 127 commits into

dask:mainfrom

ncclementi:constrained_spill

Feb 16, 2022

Merged

Constrained spill #5543
crusaderky merged 127 commits into
dask:mainfrom
ncclementi:constrained_spill

Commits on Nov 19, 2021
- Constrain spill to disk to predefined capacity
  crusaderky
  committed
  cd07540
Commits on Nov 23, 2021
- this works except when key > target and max spill
  ncclementi
  committed
  67d067e
Commits on Nov 24, 2021
- Update distributed/spill.py
  
  ncclementi
  and
  crusaderky
  authored
  d12e3d4
- use len for sieze of bytes
  ncclementi
  committed
  25f3e63
- Merge branch 'constrained_spill' of github.com:ncclementi/distributed into constrained_spill
  ncclementi
  committed
  6e545d0
Commits on Nov 29, 2021
- modify spill buffer to use new lru
  ncclementi
  committed
  4e95614
- remove comment use for notes
  ncclementi
  committed
  0fd78a8
- add max-spill to config
  ncclementi
  committed
  87f1675
- add max-spill to worker
  ncclementi
  committed
  42a34c4
- use False as default for max_spill on spillbuffer
  ncclementi
  committed
  9bc1659
- adapt test_spill to use new SpillBuffer
  ncclementi
  committed
  d27070a
- add tests for spill buffer with max_spill limit
  ncclementi
  committed
  93b3f7b
- add not to trigger warning properly on zict version
  ncclementi
  committed
  0f1c728
- extend SpillBuffer docstring to mention max_spill
  ncclementi
  committed
  344b974
Commits on Dec 1, 2021
- use consitently slow/fast
  ncclementi
  committed
  4980086
- Update max_spill annotation on worker
  
  ncclementi
  and
  crusaderky
  authored
  5f9226a
- update max_spill reading
  
  ncclementi
  and
  crusaderky
  authored
  410e7d4
- Update max-spill in distributed-schema.yaml
  
  ncclementi
  and
  crusaderky
  authored
  0a82455
- Update max spill definition/comment
  
  ncclementi
  and
  crusaderky
  authored
  c723d00
- fix import for type checking
  
  ncclementi
  and
  crusaderky
  authored
  291adfc
- Merge branch 'constrained_spill' of github.com:ncclementi/distributed into constrained_spill
  ncclementi
  committed
  ab04c6d
- add comment to fix zict version warning later
  ncclementi
  committed
  bcb4d3a
- add comment to fix zict version later
  ncclementi
  committed
  225a153
- remove unnecessary comments
  ncclementi
  committed
  ed7edf4
- add test cases for overwrite key in fast/slow bigger than max_spill
  ncclementi
  committed
  1390bc4
- add comment
  ncclementi
  committed
  b9bf50a
- add spill logging warning catching to tests
  ncclementi
  committed
  977fdd1
- use count str method to check for spill raises
  ncclementi
  committed
  edaaec2
Commits on Dec 2, 2021
- use async context manager in test_spill_to_disk
  ncclementi
  committed
  8c5ba04
- add spill disk constrained worker test
  ncclementi
  committed
  5870c1b
Commits on Dec 3, 2021
- include possibility of max_spill = 0
  ncclementi
  committed
  351c8be
- modify test for zict > 2.0 where we keep keys when exception happens
  ncclementi
  committed
  f5a7038
- fix extra line of code
  ncclementi
  committed
  68aa83f
- fix version check on spill.py
  ncclementi
  committed
  5c7fcea
- pin zict temporarily to zict slow_raise branch
  ncclementi
  committed
  84a43c5
Commits on Dec 15, 2021
- attempt to fix test_fail_write_to_disk (broken)
  ncclementi
  committed
  6744431
Commits on Dec 16, 2021
- attempt to test bad key
  ncclementi
  committed
  52fd06c
- wrong limit size
  ncclementi
  committed
  4f62f69
- remove breakpoint
  ncclementi
  committed
  f796811
Commits on Dec 17, 2021
- Update distributed/spill.py
  
  ncclementi
  and
  crusaderky
  authored
  009fc74
- Update distributed/spill.py
  
  ncclementi
  and
  crusaderky
  authored
  e167e5b
- Update distributed/spill.py
  
  ncclementi
  and
  crusaderky
  authored
  0fb62e9
- Update distributed/spill.py
  
  ncclementi
  and
  crusaderky
  authored
  8608d7b
- Update distributed/tests/test_spill.py
  
  ncclementi
  and
  crusaderky
  authored
  798a97a
- Update distributed/tests/test_spill.py
  
  ncclementi
  and
  crusaderky
  authored
  844d02a
- Update distributed/tests/test_spill.py
  
  ncclementi
  and
  crusaderky
  authored
  8b674fd
- Apply suggestions from code review
  
  ncclementi
  and
  crusaderky
  authored
  52b46bd
- include review comments
  ncclementi
  committed
  0fcd218
Commits on Jan 11, 2022
- fix exception raising/catching related to comms
  ncclementi
  committed
  a9bf0f9
- fix typos
  ncclementi
  committed
  9b2c217
- add tests
  ncclementi
  committed
  d9642b1
Commits on Jan 12, 2022
- add skipif windows and modify comment
  ncclementi
  committed
  f60f9e8
Commits on Jan 13, 2022
- fix OSError catching when writing to disk directly
  ncclementi
  committed
  f10798d
- remove prints
  ncclementi
  committed
  d38088f
Commits on Jan 14, 2022
- Merge branch 'main' of github.com:dask/distributed into constrained_spill
  ncclementi
  committed
  7cf9415
- fix typos in distributed-schema.yaml
  ncclementi
  committed
  77fcf1c
Commits on Jan 18, 2022
- fix try except when adding to dict
  ncclementi
  committed
  f561f31
- Apply suggestions from code review
  
  ncclementi
  and
  crusaderky
  authored
  65815c1
- go back to loosversion
  ncclementi
  committed
  fab2c34
- fix some asserts according to new logs
  ncclementi
  committed
  34851e9
- limit logging by trcking logged keys
  ncclementi
  committed
  ff7ed20
- add comment
  ncclementi
  committed
  3db086a
- better typing
  ncclementi
  committed
  44ecf6a
Commits on Jan 21, 2022
- add minimum loggin time interval and cleanup
  ncclementi
  committed
  363a646
- replace LooseVersion for parse_version
  ncclementi
  committed
  545ede8
Commits on Feb 2, 2022
- Update spill docstring
  
  ncclementi
  and
  crusaderky
  authored
  7c181c1
- fix logging tracking
  
  ncclementi
  and
  crusaderky
  authored
  81af9ca
- update logging tracking
  
  ncclementi
  and
  crusaderky
  authored
  2227414
- update logging tracking
  
  ncclementi
  and
  crusaderky
  authored
  fb0984a
- fix condition based on new variable name
  
  ncclementi
  and
  crusaderky
  authored
  1534d3a
- when pickle error log
  
  ncclementi
  and
  crusaderky
  authored
  3a15ab3
- add checks
  
  ncclementi
  and
  crusaderky
  authored
  026b19a
- clean Slow
  
  ncclementi
  and
  crusaderky
  authored
  cdc6075
- use importorskip for zict
  
  ncclementi
  and
  crusaderky
  authored
  30ababd
- simplify decorator for zict version
  
  ncclementi
  and
  crusaderky
  authored
  8bb8e5a
- update comments test_spill
  
  ncclementi
  and
  crusaderky
  authored
  7f43b31
Commits on Feb 3, 2022
- update comment
  
  ncclementi
  and
  crusaderky
  authored
  a7d1df6
- update comment
  
  ncclementi
  and
  crusaderky
  authored
  2f25e3a
- fix typo itroduced in suggestions
  ncclementi
  committed
  98eec51
- avoid skipif code duplication
  ncclementi
  committed
  f96b5d5
- fix type: keys are converted from hashable to str when in transit
  ncclementi
  committed
  9a2159f
- undo a type changed by mistake
  ncclementi
  committed
  112aab4
- use better exception in pytest raise
  ncclementi
  committed
  d7915bb
- add missing case test_spill
  ncclementi
  committed
  8a42f01
- add asserts for fast and slow in fail to serialize test
  ncclementi
  committed
  f2596bb
- use os instead of subprocess call to modify tmpdir permissions
  ncclementi
  committed
  c6d90e3
Commits on Feb 7, 2022
- use a better size to avoid confusion
  ncclementi
  committed
  8fba457
- add missing case on worker test constrained spill
  ncclementi
  committed
  8ab7ee7
- use better exception
  ncclementi
  committed
  6616991
- re-write Bad class
  ncclementi
  committed
  c875000
Commits on Feb 9, 2022
- handle key eviction when no target
  ncclementi
  committed
  b2f3bf3
- add test for spill buffer evict method
  ncclementi
  committed
  4254ebe
Commits on Feb 10, 2022
- add test_fail_to_write_disk_evict
  ncclementi
  committed
  0a5a458
- add docs to spill buffer setitem and evict
  ncclementi
  committed
  bce8f0c
- Merge branch 'main' of github.com:dask/distributed into constrained_spill
  ncclementi
  committed
  1322dd4
- Apply suggestions from code review to sppill.py
  
  ncclementi
  and
  crusaderky
  authored
  f1fd40e
- Apply suggestion in test_spill
  
  ncclementi
  and
  crusaderky
  authored
  4675f7a
- Apply more suggestion in test_spill.py
  
  ncclementi
  and
  crusaderky
  authored
  ccaffaa
- Apply more suggestion in worker and test_worker.py
  
  ncclementi
  and
  crusaderky
  authored
  f478089
- fix merging issue
  ncclementi
  committed
  c446c29
- fix test assertion
  ncclementi
  committed
  6da3fd8
- add test and problems
  ncclementi
  committed
  4c861b3
Commits on Feb 11, 2022
- Update distributed/tests/test_spill.py
  crusaderky
  authored
  407d609
- Update distributed/tests/test_spill.py
  crusaderky
  authored
  36fa72a
- Update distributed/tests/test_worker.py
  crusaderky
  authored
  838b513
- Merge branch 'main' of github.com:dask/distributed into constrained_spill
  ncclementi
  committed
  0e85804
- Apply suggestions from code review
  
  ncclementi
  and
  crusaderky
  authored
  1283044
- Merge branch 'constrained_spill' of github.com:ncclementi/distributed into constrained_spill
  ncclementi
  committed
  ee41e1c
- Code review
  crusaderky
  committed
  645782c
- lint
  crusaderky
  committed
  60dabf5
- fix flaky test
  crusaderky
  committed
  bfefc27
- Merge pull request #1 from crusaderky/constrained_spill_2
  ncclementi
  authored
  2299a74
Commits on Feb 14, 2022
- Update distributed/spill.py
  crusaderky
  authored
  25d0ff8
- Update distributed/tests/test_worker.py
  crusaderky
  authored
  a5e45cb
- Update distributed/tests/test_worker.py
  crusaderky
  authored
  19e16ad
- Further code review
  crusaderky
  committed
  55a2b49
- Merge pull request #3 from crusaderky/constrained_spill_2
  ncclementi
  authored
  9d68819
- Update continuous_integration/environment-3.7.yaml
  crusaderky
  authored
  d01d139
- Update continuous_integration/environment-3.8.yaml
  crusaderky
  authored
  4591390
- Merge branch 'constrained_spill' of github.com:ncclementi/distributed into constrained_spill
  ncclementi
  committed
  998354c
- Merge branch 'main' of github.com:dask/distributed into constrained_spill
  ncclementi
  committed
  d9014e5
- compatibility with zict 2.0
  ncclementi
  committed
  5dd0fe2
Commits on Feb 15, 2022
- Cleaner implementation of requires zict 210
  
  ncclementi
  and
  crusaderky
  authored
  0cbba2d
- Update distributed/spill.py
  crusaderky
  authored
  a7342c4
- Update distributed/spill.py
  crusaderky
  authored
  2e55210
- Update distributed/spill.py
  crusaderky
  authored
  2bca7d5
Commits on Feb 16, 2022
- Update continuous_integration/environment-3.9.yaml
  crusaderky
  authored
  e27c7e3

Commits

Commits on Nov 19, 2021

Commits on Nov 23, 2021

Commits on Nov 24, 2021

Commits on Nov 29, 2021

Commits on Dec 1, 2021

Commits on Dec 2, 2021

Commits on Dec 3, 2021

Commits on Dec 15, 2021

Commits on Dec 16, 2021

Commits on Dec 17, 2021

Commits on Jan 11, 2022

Commits on Jan 12, 2022

Commits on Jan 13, 2022

Commits on Jan 14, 2022

Commits on Jan 18, 2022

Commits on Jan 21, 2022

Commits on Feb 2, 2022

Commits on Feb 3, 2022

Commits on Feb 7, 2022

Commits on Feb 9, 2022

Commits on Feb 10, 2022

Commits on Feb 11, 2022

Commits on Feb 14, 2022