Skip to content

[WIP][Prototype] Experimental repartition_by_hash using dynamic task-generation - #5087

Closed
rjzamora wants to merge 13 commits into
NVIDIA:branch-0.14from
rjzamora:task-generator-shuffle
Closed

rjzamora wants to merge 13 commits into
NVIDIA:branch-0.14from
rjzamora:task-generator-shuffle

Conversation

@rjzamora

@rjzamora rjzamora commented May 4, 2020 •

Copy link
Copy Markdown
Contributor

Requires distributed#3765
Based on dask#6173

Warning!! This is currently very rough.

Usage:

ddf.repartition(
    columns=[...],
    ignore_index=True,
    dynamic_tasks=True,
)

cc @madsbk

@GPUtester

Copy link
Copy Markdown
Collaborator

Please update the changelog in order to start CI tests.

View the gpuCI docs here.

@rjzamora rjzamora changed the title [WIP][Prototype] Experimental reaprtition_by_hash using dynamic task-generation [WIP][Prototype] Experimental repartition_by_hash using dynamic task-generation May 4, 2020
Comment thread python/dask_cudf/dask_cudf/sorting.py Resolved
@codecov

codecov Bot commented May 4, 2020 •

Copy link
Copy Markdown

Codecov Report

Merging #5087 into branch-0.14 will decrease coverage by 0.38%.
The diff coverage is 36.70%.

Impacted file tree graph

@@               Coverage Diff               @@
##           branch-0.14    #5087      +/-   ##
===============================================
- Coverage        88.44%   88.06%   -0.39%     
===============================================
  Files               54       54              
  Lines            10267    10322      +55     
===============================================
+ Hits              9081     9090       +9     
- Misses            1186     1232      +46     
Impacted Files Coverage Δ
python/dask_cudf/dask_cudf/core.py 72.14% <ø> (ø)
python/dask_cudf/dask_cudf/sorting.py 78.83% <36.70%> (-15.28%) ⬇️
python/cudf/cudf/core/dataframe.py 89.45% <0.00%> (ø)
python/cudf/cudf/core/column/column.py 86.46% <0.00%> (+0.27%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f8a225...70035c5. Read the comment docs.

@harrism harrism added dask-cudf 2 - In Progress Currently a work in progress labels Jun 16, 2020
@harrism

harrism commented Jun 16, 2020

Copy link
Copy Markdown
Contributor

@rjzamora is this still needed?

@rjzamora

Copy link
Copy Markdown
Contributor Author

@rjzamora is this still needed?

We should be able to close this, but I may need to keep the branch alive until Mads has a long-term solution in place. @beckernick @madsbk - Do we need to keep this branch alive?

@rjzamora rjzamora closed this Jun 16, 2020
@vyasr vyasr added dask Dask issue and removed dask-cudf labels Feb 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2 - In Progress Currently a work in progress dask Dask issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants