I'm noticing that the performance is not affected at all when using multiple GPUs at a time, although the code seems to use DistributedDataParallel, I'm confused.
The number of batches per epoch is the same, and the seconds per iteration are the same ( I use tqdm to measure it)
There is one change in my code however, I set DDP(....., find_unused_parameters=True), I've tried it both, enabled and disabled, and still the speed is the same.
for quick checking, I advise just using tqdm around the epoch loop
for batch_idx, (x, x_lengths, spec, spec_lengths, y, y_lengths, speakers_or_embeds) in tqdm((train_loader),
'', len(train_loader), disable=writers is None):Speed on VCTK dataset, for multiGPU (4x RTX 8000):
- 293.8784s/epoch
- 1.63sec/batch
Find the environment information bellow
$ python -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.6.0
Is debug build: No
CUDA used to build PyTorch: 10.2
OS: Pop!_OS 20.04 LTS
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
CMake version: Could not collect
Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000
GPU 2: Quadro RTX 8000
GPU 3: Quadro RTX 8000
Nvidia driver version: 470.63.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.6.0
[pip3] torchvision==0.7.0
[conda] numpy 1.18.5 pypi_0 pypi
[conda] torch 1.6.0 pypi_0 pypi
[conda] torchvision 0.7.0 pypi_0 pypi
I'm noticing that the performance is not affected at all when using multiple GPUs at a time, although the code seems to use DistributedDataParallel, I'm confused.
The number of batches per epoch is the same, and the seconds per iteration are the same ( I use tqdm to measure it)
There is one change in my code however, I set
DDP(....., find_unused_parameters=True), I've tried it both, enabled and disabled, and still the speed is the same.for quick checking, I advise just using tqdm around the epoch loop
Speed on VCTK dataset, for multiGPU (4x RTX 8000):
Find the environment information bellow
$ python -m torch.utils.collect_env Collecting environment information... PyTorch version: 1.6.0 Is debug build: No CUDA used to build PyTorch: 10.2 OS: Pop!_OS 20.04 LTS GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 CMake version: Could not collect Python version: 3.7 Is CUDA available: Yes CUDA runtime version: Could not collect GPU models and configuration: GPU 0: Quadro RTX 8000 GPU 1: Quadro RTX 8000 GPU 2: Quadro RTX 8000 GPU 3: Quadro RTX 8000 Nvidia driver version: 470.63.01 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5 Versions of relevant libraries: [pip3] numpy==1.18.5 [pip3] torch==1.6.0 [pip3] torchvision==0.7.0 [conda] numpy 1.18.5 pypi_0 pypi [conda] torch 1.6.0 pypi_0 pypi [conda] torchvision 0.7.0 pypi_0 pypi