docker不穩定 short running containers with -rm failed to destroy

正常運行以下命令

sudo docker run --rm busybox echo helloworld

 

/var/log/upstart/docker.log 日志如下：

2014/08/07 00:12:02 POST /v1.13/containers/create
[339dd1d9] +job create()
[339dd1d9] -job create() = OK (0)
2014/08/07 00:12:02 POST /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/attach?stderr=1&stdout=1&stream=1
[339dd1d9] +job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
[339dd1d9] +job attach(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
2014/08/07 00:12:02 POST /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/start
[339dd1d9] +job start(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] +job allocate_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job allocate_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
[339dd1d9] -job start(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
[339dd1d9] +job release_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job release_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
[339dd1d9] -job attach(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
2014/08/07 00:12:03 POST /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/wait
[339dd1d9] +job wait(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job wait(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
2014/08/07 00:12:03 GET /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/json
[339dd1d9] +job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
2014/08/07 00:12:03 DELETE /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0?v=1
[339dd1d9] +job container_delete(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job container_delete(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)

 

我把這個寫入crontab中，每分鍾執行一次，有幾率出現如下錯誤（如果更加頻繁執行docker run那出現錯誤的幾率更大，目前一分鍾一次的出錯概率大概是2天一個error）

Error response from daemon: Cannot destroy container b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e: Driver aufs failed to remove root filesystem b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e: rename /var/lib/docker/aufs/mnt/b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e /var/lib/docker/aufs/mnt/b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e-removing: device or resource busy

Error response from daemon: Cannot destroy container b103443f2f2d74862e4aa37132043b9ca4aa1ade7bbac37a514c051f859d0473: Unable to remove filesystem for b103443f2f2d74862e4aa37132043b9ca4aa1ade7bbac37a514c051f859d0473: remove /var/lib/docker/containers/b103443f2f2d74862e4aa37132043b9ca4aa1ade7bbac37a514c051f859d0473: directory not empty

 

原因：基本還是docker不穩定造成的，並行快速的運行container的時候，有幾率出現如如上錯誤

與之有關的issue如下：

https://github.com/docker/docker/issues/4095

https://github.com/docker/docker/issues/3968

https://github.com/docker/docker/issues/3823

 

我機器的信息如下：

root@localhost:~# docker info
Containers: 4
Images: 12
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 20
Execution Driver: native-0.2
Kernel Version: 3.8.0-44-generic
WARNING: No swap limit support
root@localhost:~# docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070

root@hadoop-yarn-three:~# uname -a

Linux localhost 3.8.0-44-generic #66~precise1-Ubuntu SMP Tue Jul 15 04:01:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

 

如果哪位發現docker解決了這個issue了，請帖出docker version和機器信息，多謝~

@Tom哥，說 rhel6.5 內核版本2.6.32，docker version 1.1.2, 沒有這個問題

@IT人， 說ubuntu 14.04  centos7.0  7.1，沒有這個問題

我還沒有驗證過。