硬件配置
顯卡：Tesla V100
問題呈現
- 某次重啟后，發現服務器nvidia-docker無法
start已有的部分容器
$ docker start ******
Error response from daemon: failed to create shim: OCI runtime create failed
: container_linux.go:380: starting container process caused
: process_linux.go:545: container init caused
: Running hook #0:: error running hook: exit status 1, stdout: , stderr
: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown
Error: failed to start containers: ******
- 為排查問題的可能原因，嘗試另行啟動容器，發現啟動時無法使用GPU
$ docker run -it --gpus all [Image_Name]:[Image_Tag]
docker:
Error response from daemon: failed to create shim: OCI runtime create failed
: container_linux.go:380: starting container process caused
: process_linux.go:545: container init caused
: Running hook #0:: error running hook: exit status 1, stdout: , stderr
: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
- 為了進一步排查問題的可能原因，使用
nvidia-smi命令查看顯卡情況，發現異常
$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.
問題定位
docker報錯中包含nvml error: driver not loaded: unknown，因此問題產生原因可能是顯卡驅動被擠掉了
解決方案
- 查驗現狀
用nvcc -V確認顯卡驅動存在
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0
- 查看已安裝驅動的版本信息
$ ls /usr/src | grep nvidia
nvidia-450.51.06
- 安裝一個不知道什么東西，再安一個不知道什么東西（注意需要sudo權限，無權限聯系管理員處理）
$ sudo apt-get install dkms
$ sudo dkms install -m nvidia -v 450.51.06 # 注意這個版本號和上一步查到的相對應
- 完成操作后，docker和Nvidia-SMI均恢復正常