implement !peersync case in gds_kernel_loopback_latency via CUDA stream callbacks by drossetti · Pull Request #63 · gpudirect/libgdsync

drossetti · 2018-05-16T00:11:46Z

These changes help evaluating the relative performance of the direct Async (GPU->NIC) vs offloaded (GPU->CPU->NIC) path.

e-ago · 2018-05-16T17:10:37Z

On ivys2/3, with CUDA 9.1, r396.21 gds_kernel_latency -P -U (without peersync and using descriptors) gets stuck:

INFO: switching PeerSync OFF
INFO: use_desc_apis=1
[0] pid=15998 client:drossetti-ivy3.nvidia.com
[0] picking 1st available device
USE_GPU=0(0)
[1] pid=23278 server:drossetti-ivy2.nvidia.com
[1] picking 1st available device
USE_GPU=0(0)
GPU_ENABLE_DEBUG=0
GPU_ENABLE_DEBUG=0
[1]  local address:  LID 0x0003, QPN 0x000753, PSN 0x10101b: GID
[0]  local address:  LID 0x000c, QPN 0x000759, PSN 0x734733: GID
[0] remote address: LID 0x0003, QPN 0x000753, PSN 0x10101b, GID ::
[1] remote address: LID 0x000c, QPN 0x000759, PSN 0x734733, GID ::

pre-posted 0 sequences in 2 batches
GPU kernel calc buf size: 131072
iters=1000 tx/rx_depth=1024

testing....

drossetti · 2018-05-16T22:44:48Z

@e-ago I only modified gds_kernel_loopback_latency.
So this regression should be preexisting.
Can you verify that gds_kernel_loopback_latency works in the other branches? the sequence is:
master, new_memops, nvtx (not worth the pain), streamcb

…pu. add atomic helpers

…ile --size=0 launches a void kernel. Implement !peersync case in terms of CUDA stream callbacks, relies on CPU posting descriptors via gds_post_descriptors(). use NVTX macros in stream callback.

drossetti · 2018-05-16T23:01:25Z

just verified that gds_kernel_loopback_latency works on each individual machine

drossetti@drossetti-ivy2 15:59 (117) libgdsync>./gds_kernel_loopback_latency.sh
libgdsync build version 0x00020002, major=2 minor=2
libgdsync queried version 0x00020002
INFO: use_gpumem=1
INFO: gpu id=0
INFO: kernel calc size=8388608
INFO: message size=4096
INFO: max_batch_len=20
INFO: use_desc_apis=1
[0] pid=22932 server:localhost
[0] requested IB device: <mlx5_0>
use gpumem: 1
GPU_ENABLE_DEBUG=0
INFO: [gpu_init] GPU id:0 dev:0 name:Tesla K40m pci 2:0
INFO: [gpu_init] creating CUDA Primary Ctx on device:0 id:0
allocating GPU memory buf
ctx buf=0x23045c0000
  local address:  LID 0x000c, QPN 0x00075b, PSN 0x2aed42: GID ::
000c:00075b:2aed42:
000c:00075b:2aed42
  remote address: LID 0x000c, QPN 0x00075b, PSN 0x2aed42, GID ::
[22932] GDS WARN  gds_fill_poll() RDMA consistency for pre-launched GPU work is not guaranteed at the moment
[0] batch 0: posted 20 sequences
[0] batch 1: posted 20 sequences
[0] batch 2: posted 20 sequences
pre-posting took 712.00 usec

batch info: rx+kernel+tx 20 per batch
pre-posted 60 sequences in 2 batches
GPU kernel calc buf size: 8388608
iters=5000 tx/rx_depth=1024

testing....
[0] 40960000 bytes in 2.55 seconds = 128.73 Mbit/sec
[0] 5000 iters in 2.55 seconds = 509.08 usec/iter


drossetti@drossetti-ivy3 16:00 (71) libgdsync>./gds_kernel_loopback_latency.sh
libgdsync build version 0x00020002, major=2 minor=2
libgdsync queried version 0x00020002
INFO: use_gpumem=1
INFO: gpu id=0
INFO: kernel calc size=8388608
INFO: message size=4096
INFO: max_batch_len=20
INFO: use_desc_apis=1
[0] pid=24586 server:localhost
[0] requested IB device: <mlx5_0>
use gpumem: 1
GPU_ENABLE_DEBUG=0
INFO: [gpu_init] GPU id:0 dev:0 name:Tesla K40m pci 2:0
INFO: [gpu_init] creating CUDA Primary Ctx on device:0 id:0
allocating GPU memory buf
ctx buf=0x23045c0000
  local address:  LID 0x0003, QPN 0x000755, PSN 0x6dda97: GID ::
0003:000755:6dda97:
0003:000755:6dda97
  remote address: LID 0x0003, QPN 0x000755, PSN 0x6dda97, GID ::
[24586] GDS WARN  gds_fill_poll() RDMA consistency for pre-launched GPU work is not guaranteed at the moment
[0] batch 0: posted 20 sequences
[0] batch 1: posted 20 sequences
[0] batch 2: posted 20 sequences
pre-posting took 693.00 usec

batch info: rx+kernel+tx 20 per batch
pre-posted 60 sequences in 2 batches
GPU kernel calc buf size: 8388608
iters=5000 tx/rx_depth=1024

testing....
[0] 40960000 bytes in 2.54 seconds = 129.00 Mbit/sec
[0] 5000 iters in 2.54 seconds = 508.03 usec/iter

drossetti · 2018-05-16T23:03:32Z

also in !peersync (-P) mode

drossetti@drossetti-ivy2 15:59 (118) libgdsync>./gds_kernel_loopback_latency.sh
libgdsync build version 0x00020002, major=2 minor=2
libgdsync queried version 0x00020002
INFO: use_gpumem=1
INFO: gpu id=0
INFO: kernel calc size=8388608
INFO: message size=4096
INFO: max_batch_len=20
INFO: use_desc_apis=1
INFO: switching PeerSync OFF
WARNING: PeerSync OFF is approximated using CUDA stream callbacks
[0] pid=22954 server:localhost
[0] requested IB device: <mlx5_0>
use gpumem: 1
GPU_ENABLE_DEBUG=0
INFO: [gpu_init] GPU id:0 dev:0 name:Tesla K40m pci 2:0
INFO: [gpu_init] creating CUDA Primary Ctx on device:0 id:0
allocating GPU memory buf
ctx buf=0x23045c0000
  local address:  LID 0x000c, QPN 0x00075c, PSN 0x8ad90a: GID ::
000c:00075c:8ad90a:
000c:00075c:8ad90a
  remote address: LID 0x000c, QPN 0x00075c, PSN 0x8ad90a, GID ::
[0] batch 0: posted 20 sequences
[0] batch 1: posted 20 sequences
[0] batch 2: posted 20 sequences
pre-posting took 1054.00 usec

batch info: rx+kernel+tx 20 per batch
pre-posted 60 sequences in 2 batches
GPU kernel calc buf size: 8388608
iters=5000 tx/rx_depth=1024
testing....
[0] 40960000 bytes in 5.00 seconds = 65.54 Mbit/sec
[0] 5000 iters in 5.00 seconds = 999.96 usec/iter


drossetti@drossetti-ivy3 16:00 (72) libgdsync>./gds_kernel_loopback_latency.sh
libgdsync build version 0x00020002, major=2 minor=2
libgdsync queried version 0x00020002
INFO: use_gpumem=1
INFO: gpu id=0
INFO: kernel calc size=8388608
INFO: message size=4096
INFO: max_batch_len=20
INFO: use_desc_apis=1
INFO: switching PeerSync OFF
WARNING: PeerSync OFF is approximated using CUDA stream callbacks
[0] pid=24607 server:localhost
[0] requested IB device: <mlx5_0>
use gpumem: 1
GPU_ENABLE_DEBUG=0
INFO: [gpu_init] GPU id:0 dev:0 name:Tesla K40m pci 2:0
INFO: [gpu_init] creating CUDA Primary Ctx on device:0 id:0
allocating GPU memory buf
ctx buf=0x23045c0000
  local address:  LID 0x0003, QPN 0x000756, PSN 0xf150fd: GID ::
0003:000756:f150fd:
0003:000756:f150fd
  remote address: LID 0x0003, QPN 0x000756, PSN 0xf150fd, GID ::
[0] batch 0: posted 20 sequences
[0] batch 1: posted 20 sequences
[0] batch 2: posted 20 sequences
pre-posting took 1118.00 usec
batch info: rx+kernel+tx 20 per batch
pre-posted 60 sequences in 2 batches
GPU kernel calc buf size: 8388608
iters=5000 tx/rx_depth=1024
testing....
[0] 40960000 bytes in 5.00 seconds = 65.54 Mbit/sec
[0] 5000 iters in 5.00 seconds = 999.87 usec/iter

drossetti · 2018-05-16T23:19:59Z

confirming that gds_kernel_latency works in peersync mode:

# running NP=2, ./scripts/wrapper.sh /ivylogin/home/drossetti/work/peersync/local-ofed4.2/bin/gds_kernel_latency -S 16384 -s 128 -B 20 -n 5000 -U -K 1, stdout:1
+ /opt/openmpi/v1.10.2/cuda7.5/bin/mpirun -mca pml ob1 --mca mtl '^mxm' --mca btl openib,self --mca btl_openib_want_cuda_gdr 1 --mca btl_openib_warn_default_gid_prefix 0 -x QUDA_ENABLE_P2P -x QUDA_RESOURCE_PATH -x QUDA_USE_COMM_ASYNC_STREAM -x QUDA_USE_COMM_ASYNC_PREPARED -x QUDA_ASYNC_ENABLE_DEBUG -x GDS_ENABLE_DEBUG -x GDS_ENABLE_DUMP_MEMOPS -x GDS_DISABLE_WRITE64 -x GDS_SIMULATE_WRITE64 -x GDS_DISABLE_INLINECOPY -x 
GDS_DISABLE_MEMBAR -x GDS_DISABLE_WEAK_CONSISTENCY -x MP_ENABLE_DEBUG -x MP_ENABLE_WARN -x MP_EVENT_ASYNC -x MP_DBREC_ON_GPU -x MP_TX_CQ_ON_GPU -x MP_RX_CQ_ON_GPU -x MP_ENABLE_IPC -x MP_USE_IB_HCA -x CUDA_VISIBLE_DEVICES -x LD_LIBRARY_PATH -x PATH -x SIZE -x MAX_SIZE -x KERNEL_TIME -x CALC_SIZE -x COMM_COMP_RATIO -x USE_SINGLE_STREAM -x USE_GPU_ASYNC -x OMP_NUM_THREADS -x COMM_USE_ASYNC -x COMM_USE_COMM -x COMM_USE_GPU_COMM -x COMM_USE_GDRDMA -x HPGMG_ENABLE_DEBUG -x MLX5_DEBUG_MASK -x PX -x PY -x USE_GPU_COMM_BUFFERS -x USE_MPI --map-by node -hostfile ./scripts/hostfile -np 2 ./scripts/wrapper.sh /ivylogin/home/drossetti/work/peersync/local-ofed4.2/bin/gds_kernel_latency -S 16384 -s 128 -B 20 -n 5000 -U -K 1
...
libgdsync build version 0x00020002, major=2 minor=2
libgdsync queried version 0x00020002
INFO: max_batch_len=20
INFO: use_desc_apis=1
INFO: GDS_QPT UD
libgdsync build version 0x00020002, major=2 minor=2
libgdsync queried version 0x00020002
INFO: max_batch_len=20
INFO: use_desc_apis=1
INFO: GDS_QPT UD
[0] pid=23039 client:drossetti-ivy3.nvidia.com
[0] USE_HCA: <mlx5_0>
[1] pid=24715 server:drossetti-ivy2.nvidia.com
[1] USE_HCA: <mlx5_0>
INFO: [gpu_init] GPU id:0 dev:0 name:Tesla K40m pci 2:0
INFO: [gpu_init] creating CUDA Primary Ctx on device:0 id:0
GPU_ENABLE_DEBUG=0
INFO: [gpu_init] GPU id:0 dev:0 name:Tesla K40m pci 2:0
INFO: [gpu_init] creating CUDA Primary Ctx on device:0 id:0
INFO: [pp_init_ctx] allocated ctx buffer 0x23045c0000
INFO: [pp_init_ctx] allocated ctx buffer 0x23045c0000
[0]  local address:  LID 0x000c, QPN 0x00077c, PSN 0x262eda: GID 
[0] remote address: LID 0x0003, QPN 0x000776, PSN 0x66d084, GID ::
[23039] GDS WARN  gds_fill_poll() RDMA consistency for pre-launched GPU work is not guaranteed at the moment
[1]  local address:  LID 0x0003, QPN 0x000776, PSN 0x66d084: GID 
[1] remote address: LID 0x000c, QPN 0x00077c, PSN 0x262eda, GID ::
[24715] GDS WARN  gds_fill_poll() RDMA consistency for pre-launched GPU work is not guaranteed at the moment
[0] batch 0: posted 20 sequences
[0] batch 1: posted 20 sequences
[0] batch 2: posted 20 sequences
pre-posting took 827.00 usec

batch info: rx+kernel+tx 20 per batch
pre-posted 60 sequences in 2 batches
GPU kernel calc buf size: 16384
iters=5000 tx/rx_depth=1024

testing....
[1] batch 0: posted 20 sequences
[1] batch 1: posted 20 sequences
[1] batch 2: posted 20 sequences
pre-posting took 691.00 usec
[0] 1280000 bytes in 0.13 seconds = 78.13 Mbit/sec
[0] 5000 iters in 0.13 seconds = 26.21 usec/iter
[1] 1280000 bytes in 0.13 seconds = 78.11 Mbit/sec
[1] 5000 iters in 0.13 seconds = 26.22 usec/iter

drossetti · 2018-05-16T23:28:21Z

running gds_kernel_latency works in !peersync mode over the following branches:

streamcb: FAIL
new_memops: FAIL
master: OK

FAIL=stuck after pre-submisison, top shows user-mode process taking 100% of CPU core

drossetti · 2018-05-16T23:41:56Z

bisecting:
fails at bbaf2e2 don't use extended memops by default (Davide Rossetti)
fails at 186b2b4 fix merge breakages (Davide Rossetti)
does not build at 9e1731a cleanups, passing on K40m (Davide Rossetti)
does not build at 9a36de6 translate CUDA_ERROR_NOT_SUPPORTED, propagate error ...
does not build at b95132 introduce GDSCHECK (Davide Rossetti)
does not build at 2e4426c add support for WRITE_MEMORY
works at 4a4a2be bump version to 2.2 (Davide Rossetti)
works at e8461b6 fix bug such that GDS_DISABLE_WEAK_CONSISTENCY was not effective
works at 69fe4e9 fixes for experimental CUDA MemOps (Davide Rossetti)

drossetti · 2018-05-17T00:38:06Z

rewriting new_memops branch to fix the compile breakages earlier

drossetti · 2018-05-17T01:09:31Z

bisected the !peersync problem, it is due to:
e94c87a (HEAD) use gpu debug/error/tracing functions. refactor code to support non-peersync mode (still not working)
it seems like !peersync case was never supposed to work in gds_kernel_latency, and it is implemented in a different way, i.e. not with CUDA stream callbacks

drossetti · 2018-05-17T01:12:59Z

filed issue #64 to track future work

drossetti · 2018-05-17T01:13:34Z

@e-ago you should use gds_kernel_loopback_latency to test the feature added by this PR

e-ago · 2018-05-17T09:59:33Z

Tested again on ivys, I can confirm that:

gds_kernel_loopback_latency, peersync OK, !peersync OK
gds_kernel_latency, peersync OK, !peersync FAIL ( in gds_kernel_latency !peersync case does not work #64 )
gds_sanity and gds_poll_lat, FAIL ( gds_sanity and gds_poll_lat fail #60 )

On the master branch, gds_kernel_latency passes all tests.

drossetti · 2018-05-17T21:46:18Z

On the master branch, gds_kernel_latency passes all tests.

that is because on gds_kernel_latency really !peersync was basically a no-op.

drossetti requested a review from e-ago May 16, 2018 00:12

drossetti added 9 commits May 16, 2018 15:49

introduce gds_post_descriptors

e7e4589

fix decl of gds_post_descriptors

08f0ab1

implement missing cases in gds_post_descriptors and gds_post_ops_on_c…

9b93e8c

…pu. add atomic helpers

fix typo in comment

a75b939

add gpu_warnc & gpu_warn_once

85ee32c

make gpu_launch_calc_kernel_on_stream calculate grid

0577d24

warn once if size==0

6c27f5a

route dbg macro to gpu_dbg. introduce --skip-kernel-launch option, wh…

5eb44b0

…ile --size=0 launches a void kernel. Implement !peersync case in terms of CUDA stream callbacks, relies on CPU posting descriptors via gds_post_descriptors(). use NVTX macros in stream callback.

fix help test for -P option

33bd1b9

drossetti force-pushed the streamcb branch from e65b549 to 33bd1b9 Compare May 16, 2018 22:55

drossetti changed the title ~~implement !peersync case via CUDA stream callbacks~~ implement !peersync case in gds_kernel_loopback_latency via CUDA stream callbacks May 17, 2018

e-ago merged commit 63f5955 into nvtx May 17, 2018

e-ago mentioned this pull request May 17, 2018

add support for NVTX style profiling annotations #62

Merged

drossetti mentioned this pull request May 18, 2018

fix kernel latency tests #65

Merged

Conversation

drossetti commented May 16, 2018

Uh oh!

e-ago commented May 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drossetti commented May 16, 2018

Uh oh!

drossetti commented May 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drossetti commented May 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drossetti commented May 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drossetti commented May 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drossetti commented May 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drossetti commented May 17, 2018

Uh oh!

drossetti commented May 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drossetti commented May 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drossetti commented May 17, 2018

Uh oh!

e-ago commented May 17, 2018

Uh oh!

drossetti commented May 17, 2018

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

e-ago commented May 16, 2018 •

edited

Loading

drossetti commented May 16, 2018 •

edited

Loading

drossetti commented May 16, 2018 •

edited

Loading

drossetti commented May 16, 2018 •

edited

Loading

drossetti commented May 16, 2018 •

edited

Loading

drossetti commented May 16, 2018 •

edited

Loading

drossetti commented May 17, 2018 •

edited

Loading

drossetti commented May 17, 2018 •

edited

Loading