打開nccl的錯誤信息：
export NCCL_DEBUG=WARN
錯誤1.docker容器內運行pytorch多gpu報錯 RuntimeError: NCCL Error 2: unhandled system error
在啟動容器的時候加上 -e NVIDIA_VISIBLE_DEVICES=0,1,2,3
docker run --runtime=nvidia --net="host" -e NVIDIA_VISIBLE_DEVICES=0,1,2,3 --shm-size 8g -it huangzc/reid:v1 /bin/bash
錯誤2.RuntimeError: DataLoader worker (pid 53617) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.
啟動容器的時候增加交換內存 --shm-size 8g