背景：平時運行的好好的k8s集群，早上過來一看發現好多紅色指標，馬上查 下集群狀況，發現一個節點跪了。
1
可以確定work01是真的跪了了，那么如何查找問題呢？ 我們用它：journalctl工具是CentOS-7才有的工具
Systemd 統一管理所有 Unit 的啟動日志。帶來的好處就是 ，可以只用journalctl一個命令，查看所有日志（內核日志和 應用日志）。日志的配置文件/etc/systemd/journald.conf
journalctl用法：
查看所有日志（默認情況下 ，只保存本次啟動的日志）： journalctl
查看內核日志（不顯示應用日志）： journalctl -k
查看系統本次啟動的日志： journalctl -b
查看上一次啟動的日志（需更改設置）：
在該[Journal]部分下，將該Storage=選項設置為“persistent”以啟用持久記錄：
vim /etc/systemd/journald.conf
. . .
[Journal]
Storage=persistent
查看httpd服務的日志信息：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
我們使用的是：
追蹤日志
要主動追蹤當前正在編寫的日志，大家可以使用-f標記。同樣功能類似為tail -f，只要不終止，會一直監控
journalctl -f
也許最有用的過濾方式是你感興趣的單位。我們可以使用這個-u選項來過濾我們可以使用這個-u選項來過濾
journalctl -u
所以我們最終使用的命令是：
journalctl -f -u kubelet
1
2
3
4
5
6
7
8
9
[root@***-***-work01 ~]# journalctl -f -u kubelet
-- Logs begin at Wed 2018-09-05 11:16:12 CST. --
Sep 05 11:43:30 ***-***-work01 kubelet[5764]: E0905 11:43:30.325784 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:32 ***-***-work01 kubelet[5764]: E0905 11:43:32.048633 5764 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": path does not exist, ignoring
Sep 05 11:43:32 ***-***-work01 kubelet[5764]: E0905 11:43:32.304730 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:34 ***-***-work01 kubelet[5764]: E0905 11:43:34.325829 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:36 ***-***-work01 kubelet[5764]: E0905 11:43:36.293212 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:38 ***-***-work01 kubelet[5764]: E0905 11:43:38.301224 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:40 ***-***-work01 kubelet[5764]: E0905 11:43:40.330581 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:42 ***-***-work01 kubelet[5764]: E0905 11:43:42.289427 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:44 ***-***-work01 kubelet[5764]: E0905 11:43:44.311478 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:46 ***-***-work01 kubelet[5764]: E0905 11:43:46.296680 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:48 ***-***-work01 kubelet[5764]: E0905 11:43:48.320159 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:50 ***-***-work01 kubelet[5764]: E0905 11:43:50.295335 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:52 ***-***-work01 kubelet[5764]: E0905 11:43:52.049269 5764 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": path does not exist, ignoring
Sep 05 11:43:52 ***-***-work01 kubelet[5764]: E0905 11:43:52.332253 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:54 ***-***-work01 kubelet[5764]: E0905 11:43:54.319333 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:56 ***-***-work01 kubelet[5764]: E0905 11:43:56.298336 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
NOTE:因為我的原因，很明顯，work01節點是機器直接跪了，
最后重新啟動就OK了。
systemctl restart docker
————————————————
版權聲明：本文為CSDN博主「common_util」的原創文章，遵循 CC 4.0 BY-SA 版權協議，轉載請附上原文出處鏈接及本聲明。
原文鏈接：https://blog.csdn.net/shenhonglei1234/article/details/82421742