pytorch high memory usage but low volatile gpu-util

在使用GPU訓練神經網絡模型時，可能會出現GPU利用率較低的情況：

 

 

可以通過以下幾種方式解決：

1：

這個nvidia forum中提到，應該是GPU默認打開了ECC(error correcting code, 錯誤檢查和糾正),會占用顯存和降低顯卡性能，打開Persistence Mode Enabled（用root執行nvidia-smi -pm 1）后5、6號顯卡的顯卡使用率恢復正常水平，問題解決。

2：對於DataLoader函數而言：

torch.utils.data.DataLoader(dataset, batch_size=1, shuffle=False, sampler=None, 
　　　　　　　　　　　　　　　　　batch_sampler=None, num_workers=0, collate_fn=None, 
                            pin_memory=False, drop_last=False, timeout=0, 
                            worker_init_fn=None, multiprocessing_context=None)

 

 

增大num_workers (2, 4, 8, 16, 32)

num_workers (int, optional) – how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process. (default: 0)

設置pin_memory 為True

pin_memory (bool, optional) – If True, the data loader will copy Tensors into CUDA pinned memory before returning them. If your data elements are a custom type, or your collate_fn returns a batch that is a custom type, see the example below.

 

3:檢查cuda版本是否和pytorch對齊：

Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.version.cuda
'10.0.130'
>>> exit()
(py36) [tianqx15@localhost ~]$  vi .bashrc
(py36) [tianqx15@localhost ~]$  source .bashrc
(py36) [tianqx15@localhost ~]$  nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

 

如果不對應，更新cuda或torch:

 export CUDA_HOME=/usr/local/cuda-10.0
 export PATH=/usr/local/cuda-10.0/bin${PATH:+:${PATH}}
 export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}