解決RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:70&NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the late

解決兩個問題：

（1）RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:70

（2）NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

　　

之前還好好的代碼 時隔一個月就出現了問題

 

haiou@gpu4:/home/gpu4/haiou/newdata/tqa$ cat f2
WARNING:root:WARN: Duplicate embedding found for ;
/home/gpu4/haiou/newdata/tqa/src_0826/layers.py:227: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  alpha_flat = F.softmax(scores.view(-1, y.size(1)))
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=70 error=30 : unknown error
Traceback (most recent call last):
  File "src_0826/main.py", line 129, in <module>
    b_info,b_dev_acc, b_test_acc = run_epoch(args,model,train_data,dev_data,test_data,checkpoint_path)
  File "src_0826/main.py", line 68, in run_epoch
    model.train(cur_train_data)
  File "/home/gpu4/haiou/newdata/tqa/src_0826/model.py", line 64, in train
    pred_proba = self.network(*feed_input)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/gpu4/haiou/newdata/tqa/src_0826/tmn_v21_update.py", line 101, in forward
    self.mfunction(p_hiddens, q_hiddens, c_hiddens,p_mask, q_mask, c_mask)
  File "/home/gpu4/haiou/newdata/tqa/src_0826/tmn_v21_update.py", line 222, in NA_TriMatching
    x_mask = p_mask)
  File "/home/gpu4/haiou/newdata/tqa/src_0826/tmn_v21_update.py", line 185, in tri_matching
    init_mem = torch.zeros(dim[0],dim[1],dim[2]).float().cuda()  #zero mem
RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:70

 詢問了實驗室的小伙，then發現問題：

gpu4@gpu4:~$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
gpu4@gpu4:~$ nvidia-
nvidia-bug-report.sh           nvidia-container-runtime-hook  nvidia-debugdump               nvidia-installer               nvidia-settings                nvidia-xconfig                 
nvidia-container-cli           nvidia-cuda-mps-control        nvidia-detector                nvidia-modprobe                nvidia-smi                     
nvidia-container-runtime       nvidia-cuda-mps-server         nvidia-docker                  nvidia-persistenced            nvidia-uninstall

然后解決問題：

好吧，兜兜轉轉后續兒重裝，不是安裝cuda就相對簡單了:

安裝NVIDIA驅動程序（如果事先有裝但失敗的，先卸載，所以要卸載。。）

$ sudo apt remove --purge nvidia*

1）禁用nouveau驅動 
終端中運行：

$ lsmod | grep nouveau

2）關閉圖形化界面

$ sudo service lightdm stop

3）安裝驅動
NVIDIA驅動安裝也有多種方式，runfile文件形式的安裝和apt安裝

apt安裝：

a.添加Graphic Drivers PPA

$ sudo add-apt-repository ppa:graphics-drivers/ppa
$ sudo apt-get update

尋找合適的驅動版本

$ ubuntu-drivers devices

c.我的顯示是415

$ sudo apt-get install nvidia-415

d.安裝完成后，重啟

$ sudo reboot

e.這時，執行下面的命令，查看驅動的安裝狀態

$ sudo nvidia-smi

 

查看安裝版本：

nvidia-settings --v

參考鏈接：https://blog.csdn.net/Jonms/article/details/79318566?utm_source=blogxgwz2