Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered

1. 當一個層的bottom和top維度一致時，取同樣的名字沒問題(比如說relu)，但是當兩者維度不一致時，就不能取同一個名字。建議查看log中bottom 和top名字一樣的層，數據維度是否一直。
2. I just encountered this issue, in my case this seems due to the use of rectangular kernels into conv layers.
   Reshaping the kernels into square ones resolved the issue.
   Note that I also used rectangular kernels for the pooling layers and those work properly

3. I have the similar problem:
   F0309 11:30:48.307298 892 syncedmem.hpp:19] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
   *** Check failure stack trace: ***
   Aborted (core dumped)

   When I run demo.py with gpu_id = 0, it is OK. But when I set gpu_id = 1 or 2 or 3(I have 4 gpu), the problem arises.

4. if using cuda-8, the problem seems missing from my side.
5. 在我這里是這樣，因為我的環境數據其實都沒有問題。 原本做2D Object detetction 的網絡去掉了RPN部分，在roi pooling層中gt_boxes代替了rois，一來是要注意gt_boxes（xmin,ymin,xmax,ymax,label）五元組的順序和rois(label,xmin,ymin,xmax,ymax)有所不同,二來要注意rois的label在proposal target 這一層是直接設置為0的，需要注意，如果訓練的txt中是5類（0-4）,那么就出錯了。要確保數據在每一個環節都沒有問題。