一、heapster簡介
什么是Heapster？
Heapster是容器集群監控和性能分析工具，天然的支持Kubernetes和CoreOS。
Kubernetes有個出名的監控agent---cAdvisor。在每個kubernetes Node上都會運行cAdvisor，它會收集本機以及容器的監控數據(cpu,memory,filesystem,network,uptime)。
在較新的版本中，K8S已經將cAdvisor功能集成到kubelet組件中。每個Node節點可以直接進行web訪問。
cAdvisor web界面訪問： http://< Node-IP >:4194
cAdvisor也提供Restful API: https://github.com/google/cadv ... pi.md
Heapster是一個收集者，將每個Node上的cAdvisor的數據進行匯總，然后導到第三方工具(如InfluxDB)。
二、heapster調用kubelet源碼分析
1、整體源碼
heapster整體的源碼分析可以參考文章：https://segmentfault.com/a/1190000008863353#articleHeader2，
上面鏈接中的文章中會對heapster從啟動開始進行分析，主要講述了下面內容：
- main（）
- 創建數據源
- 創建后端服務
- 創建數據processors
- 獲取源數據並存儲
- heapster API創建
給上述文章的作者點個贊，思路真的非常清晰。不過筆者最關注的點是heapster如何進行的kubelet調用。
2、調用kubelet源碼分析
從上述文章提到的 NewKubeletProvider 函數中，我們找到了創建kubeClient和kubeletClient的地方，見下面代碼的飄黃部分：
func NewKubeletProvider(uri *url.URL) (MetricsSourceProvider, error) { // 創建kubernetes master及kubelet client相關的配置 kubeConfig, kubeletConfig, err := GetKubeConfigs(uri) if err != nil { return nil, err } // 創建kubeClient及kubeletClient kubeClient := kube_client.NewOrDie(kubeConfig) kubeletClient, err := NewKubeletClient(kubeletConfig) if err != nil { return nil, err } // 獲取下所有的Nodes，測試下創建的client是否能正常通訊 if _, err := kubeClient.Nodes().List(kube_api.ListOptions{ LabelSelector: labels.Everything(), FieldSelector: fields.Everything()}); err != nil { glog.Errorf("Failed to load nodes: %v", err) } // 監控k8s的nodes變更 // 這里會創建協程進行watch，便於后面調用nodeLister.List()列出所有的nodes。 // 該Watch的實現，需要看下apiServer中的實現，后面會進行講解 lw := cache.NewListWatchFromClient(kubeClient, "nodes", kube_api.NamespaceAll, fields.Everything()) nodeLister := &cache.StoreToNodeLister{Store: cache.NewStore(cache.MetaNamespaceKeyFunc)} reflector := cache.NewReflector(lw, &kube_api.Node{}, nodeLister.Store, time.Hour) reflector.Run() // 結構在前面介紹過 return &kubeletProvider{ nodeLister: nodeLister, reflector: reflector, kubeletClient: kubeletClient, }, nil }
1）heapster源碼目錄
我們發現這個函數所在的文件為kubelet.go，所在的包為metrics/sources/kubelet
我們可以看到，kubelet調用相關的代碼都在這個里面：
2）heapster調用kubelet的API
我們進入到NewKubeletClient函數看一下：
// 傳入的是KubeletClientConfig，通過讀取相關配置，來初始化client
func NewKubeletClient(kubeletConfig *kubelet_client.KubeletClientConfig) (*KubeletClient, error) {
transport, err := kubelet_client.MakeTransport(kubeletConfig) if err != nil { return nil, err } c := &http.Client{ // 此處可以看到，是http調用 Transport: transport, Timeout: kubeletConfig.HTTPTimeout, } return &KubeletClient{ config: kubeletConfig, client: c, }, nil }
接下來我們看一下kubelet_client.go中KubeletClient有哪些方法：
(i)獲取所有的containersInfo
第一個，獲取所有的container統計信息，如果對cAdvisor比較了解的話，可以進入到該方法的返回值的ContainerInfo里面看下，是非常全面的容器監控信息（后續會有cAdvisor的源碼分析，敬請期待）
func (self *KubeletClient) getAllContainers(url string, start, end time.Time) ([]cadvisor.ContainerInfo, error) { // Request data from all subcontainers.此處是構造requestBody request := statsRequest{ ContainerName: "/", NumStats: 1, Start: start, // 2017-11-10T06:46:17Z 這種utc格式的時間戳 End: end, Subcontainers: true, } body, err := json.Marshal(request) if err != nil { return nil, err } req, err := http.NewRequest("POST", url, bytes.NewBuffer(body)) if err != nil { return nil, err }
// 設置請求頭 req.Header.Set("Content-Type", "application/json") var containers map[string]cadvisor.ContainerInfo client := self.client if client == nil { client = http.DefaultClient } err = self.postRequestAndGetValue(client, req, &containers) if err != nil { return nil, fmt.Errorf("failed to get all container stats from Kubelet URL %q: %v", url, err) } result := make([]cadvisor.ContainerInfo, 0, len(containers)) for _, containerInfo := range containers { cont := self.parseStat(&containerInfo) if cont != nil { result = append(result, *cont) } } return result, nil }
我們可以看到上面的源碼其實是構造了一個http的post請求。那么我們在本地通過restClient模擬一下：
http://ip:10255/stats/container/（為什么用10255端口呢？在上面的源碼中初始化NewKubeletClient的地方，我們進入到GetKubeClientConfig看一下就會發現，用的是kubelet的默認開放端口，在configs.go文件中）
返回值部分摘錄如下（內容太多，折疊一下）：
"/": { "name": "/", "subcontainers": [ { "name": "/docker" }, { "name": "/init.scope" }, { "name": "/kube-proxy" }, { "name": "/kubepods" }, { "name": "/system.slice" }, { "name": "/user.slice" } ], "spec": { "creation_time": "2017-10-14T17:16:59.086488213+08:00", "has_cpu": true, "cpu": { "limit": 1024, "max_limit": 0, "mask": "0-3", "period": 100000 }, "has_memory": true, "memory": { "limit": 33604153344, "reservation": 9223372036854771712 }, "has_network": true, "has_filesystem": true, "has_diskio": true, "has_custom_metrics": false }, "stats": [ { "timestamp": "2017-11-10T14:29:25.105117933+08:00", "cpu": { "usage": { "total": 916886132160774, "per_cpu_usage": [ 225713500943530, 230866928282148, 229896967301966, 230408735633130 ], "user": 443962220000000, "system": 438380440000000 }, "cfs": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 }, "load_average": 0 }, "diskio": { "io_service_bytes": [ { "major": 7, "minor": 4, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 15, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 14, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 12, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 0, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 7, "minor": 0, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 11, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 4, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 253, "minor": 0, "stats": { "Async": 61989669888, "Read": 23649280, "Sync": 17028616192, "Total": 79018286080, "Write": 78994636800 } }, { "major": 7, "minor": 7, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 9, "minor": 0, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 13, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 6, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 7, "minor": 6, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 5, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 5, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 3, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 7, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 253, "minor": 16, "stats": { "Async": 258048, "Read": 21504, "Sync": 4773888, "Total": 5031936, "Write": 5010432 } }, { "major": 7, "minor": 3, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 2, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 1, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 10, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 9, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 8, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 2, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 1, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } } ], "io_serviced": [ { "major": 7, "minor": 4, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 9, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 8, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 9, "minor": 0, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 1, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 15, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 7, "minor": 7, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 1, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 4, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 0, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 253, "minor": 0, "stats": { "Async": 2738092, "Read": 2965, "Sync": 4157380, "Total": 6895472, "Write": 6892507 } }, { "major": 1, "minor": 13, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 7, "minor": 5, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 14, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 5, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 253, "minor": 16, "stats": { "Async": 57, "Read": 7, "Sync": 32, "Total": 89, "Write": 82 } }, { "major": 1, "minor": 3, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 2, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 7, "minor": 6, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 2, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 11, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 10, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 6, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 12, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 7, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 7, "minor": 3, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 0, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } } ] }, "memory": { "usage": 12189630464, "cache": 288108544, "rss": 3631652864, "swap": 0, "working_set": 7756312576, "failcnt": 0, "container_data": { "pgfault": 8947531, "pgmajfault": 87 }, "hierarchical_data": { "pgfault": 8947531, "pgmajfault": 87 } }, "network": { "name": "cbr0", "rx_bytes": 1885776219, "rx_packets": 8199780, "rx_errors": 0, "rx_dropped": 0, "tx_bytes": 1854690043, "tx_packets": 8765796, "tx_errors": 0, "tx_dropped": 0, "interfaces": [ { "name": "cbr0", "rx_bytes": 1885776219, "rx_packets": 8199780, "rx_errors": 0, "rx_dropped": 0, "tx_bytes": 1854690043, "tx_packets": 8765796, "tx_errors": 0, "tx_dropped": 0 }, { "name": "ens3", "rx_bytes": 25108832715, "rx_packets": 74352449, "rx_errors": 0, "rx_dropped": 0, "tx_bytes": 93157559917, "tx_packets": 75983196, "tx_errors": 0, "tx_dropped": 0 } ], "tcp": { "Established": 0, "SynSent": 0, "SynRecv": 0, "FinWait1": 0, "FinWait2": 0, "TimeWait": 0, "Close": 0, "CloseWait": 0, "LastAck": 0, "Listen": 0, "Closing": 0 }, "tcp6": { "Established": 0, "SynSent": 0, "SynRecv": 0, "FinWait1": 0, "FinWait2": 0, "TimeWait": 0, "Close": 0, "CloseWait": 0, "LastAck": 0, "Listen": 0, "Closing": 0 }, "udp": { "Listen": 0, "Dropped": 0, "RxQueued": 0, "TxQueued": 0 }, "udp6": { "Listen": 0, "Dropped": 0, "RxQueued": 0, "TxQueued": 0 } }, "filesystem": [ { "device": "/dev/root", "type": "vfs", "capacity": 20749852672, "usage": 9800732672, "base_usage": 0, "available": 10932342784, "has_inodes": true, "inodes": 2560000, "inodes_free": 2340243, "reads_completed": 0, "reads_merged": 0, "sectors_read": 0, "read_time": 0, "writes_completed": 0, "writes_merged": 0, "sectors_written": 0, "write_time": 0, "io_in_progress": 0, "io_time": 0, "weighted_io_time": 0 }, { "device": "tmpfs", "type": "vfs", "capacity": 16802074624, "usage": 2224128, "base_usage": 0, "available": 16799850496, "has_inodes": true, "inodes": 4102069, "inodes_free": 4101999, "reads_completed": 0, "reads_merged": 0, "sectors_read": 0, "read_time": 0, "writes_completed": 0, "writes_merged": 0, "sectors_written": 0, "write_time": 0, "io_in_progress": 0, "io_time": 0, "weighted_io_time": 0 }, { "device": "shm", "type": "vfs", "capacity": 67108864, "usage": 0, "base_usage": 0, "available": 67108864, "has_inodes": true, "inodes": 4102069, "inodes_free": 4102068, "reads_completed": 0, "reads_merged": 0, "sectors_read": 0, "read_time": 0, "writes_completed": 0, "writes_merged": 0, "sectors_written": 0, "write_time": 0, "io_in_progress": 0, "io_time": 0, "weighted_io_time": 0 }, { "device": "/dev/vdb", "type": "vfs", "capacity": 53660876800, "usage": 35852288, "base_usage": 0, "available": 53625024512, "has_inodes": true, "inodes": 26214400, "inodes_free": 26214393, "reads_completed": 457, "reads_merged": 0, "sectors_read": 13162, "read_time": 3428, "writes_completed": 231, "writes_merged": 20, "sectors_written": 72963, "write_time": 1756, "io_in_progress": 0, "io_time": 3584, "weighted_io_time": 5184 } ], "task_stats": { "nr_sleeping": 0, "nr_running": 0, "nr_stopped": 0, "nr_uninterruptible": 0, "nr_io_wait": 0 } }, { "timestamp": "2017-11-10T14:29:36.487311961+08:00", "cpu": { "usage": { "total": 916888682985895, "per_cpu_usage": [ 225714036142385, 230867670611465, 229897633996290, 230409342235755 ], "user": 443963990000000, "system": 438381050000000 }, "cfs": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 }, "load_average": 0 }, "diskio": { "io_service_bytes": [ { "major": 9, "minor": 0, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 253, "minor": 0, "stats": { "Async": 61989710848, "Read": 23649280, "Sync": 17028677632, "Total": 79018388480, "Write": 78994739200 } }, { "major": 7, "minor": 6, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 1, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 10, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 8, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 4, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 2, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 13, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 1, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 7, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 0, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 253, "minor": 16, "stats": { "Async": 258048, "Read": 21504, "Sync": 4773888, "Total": 5031936, "Write": 5010432 } }, { "major": 1, "minor": 14, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 12, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 9, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 6, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 7, "minor": 7, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 15, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 1, "minor": 5, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 7, "minor": 5, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 4, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 3, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 2, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 3, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } }, { "major": 7, "minor": 0, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 11, "stats": { "Async": 12288, "Read": 12288, "Sync": 0, "Total": 12288, "Write": 0 } } ], "io_serviced": [ { "major": 1, "minor": 11, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 9, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 7, "minor": 4, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 3, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 14, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 10, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 5, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 2, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 1, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 0, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 9, "minor": 0, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 253, "minor": 0, "stats": { "Async": 2738102, "Read": 2965, "Sync": 4157395, "Total": 6895497, "Write": 6892532 } }, { "major": 7, "minor": 1, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 15, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 3, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 7, "minor": 5, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 13, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 12, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 6, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 1, "minor": 4, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 253, "minor": 16, "stats": { "Async": 57, "Read": 7, "Sync": 32, "Total": 89, "Write": 82 } }, { "major": 7, "minor": 6, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 2, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 7, "minor": 0, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 7, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } }, { "major": 7, "minor": 7, "stats": { "Async": 0, "Read": 0, "Sync": 0, "Total": 0, "Write": 0 } }, { "major": 1, "minor": 8, "stats": { "Async": 3, "Read": 3, "Sync": 0, "Total": 3, "Write": 0 } } ] }, "memory": { "usage": 12189343744, "cache": 288108544, "rss": 3631366144, "swap": 0, "working_set": 7756025856, "failcnt": 0, "container_data": { "pgfault": 8947531, "pgmajfault": 87 }, "hierarchical_data": { "pgfault": 8947531, "pgmajfault": 87 } }, "network": { "name": "cbr0", "rx_bytes": 1885783888, "rx_packets": 8199819, "rx_errors": 0, "rx_dropped": 0, "tx_bytes": 1854698778, "tx_packets": 8765835, "tx_errors": 0, "tx_dropped": 0, "interfaces": [ { "name": "cbr0", "rx_bytes": 1885783888, "rx_packets": 8199819, "rx_errors": 0, "rx_dropped": 0, "tx_bytes": 1854698778, "tx_packets": 8765835, "tx_errors": 0, "tx_dropped": 0 }, { "name": "ens3", "rx_bytes": 25108971486, "rx_packets": 74353959, "rx_errors": 0, "rx_dropped": 0, "tx_bytes": 93160391886, "tx_packets": 75985129, "tx_errors": 0, "tx_dropped": 0 } ], "tcp": { "Established": 0, "SynSent": 0, "SynRecv": 0, "FinWait1": 0, "FinWait2": 0, "TimeWait": 0, "Close": 0, "CloseWait": 0, "LastAck": 0, "Listen": 0, "Closing": 0 }, "tcp6": { "Established": 0, "SynSent": 0, "SynRecv": 0, "FinWait1": 0, "FinWait2": 0, "TimeWait": 0, "Close": 0, "CloseWait": 0, "LastAck": 0, "Listen": 0, "Closing": 0 }, "udp": { "Listen": 0, "Dropped": 0, "RxQueued": 0, "TxQueued": 0 }, "udp6": { "Listen": 0, "Dropped": 0, "RxQueued": 0, "TxQueued": 0 } }, "filesystem": [ { "device": "/dev/root", "type": "vfs", "capacity": 20749852672, "usage": 9800732672, "base_usage": 0, "available": 10932342784, "has_inodes": true, "inodes": 2560000, "inodes_free": 2340243, "reads_completed": 0, "reads_merged": 0, "sectors_read": 0, "read_time": 0, "writes_completed": 0, "writes_merged": 0, "sectors_written": 0, "write_time": 0, "io_in_progress": 0, "io_time": 0, "weighted_io_time": 0 }, { "device": "tmpfs", "type": "vfs", "capacity": 16802074624, "usage": 2224128, "base_usage": 0, "available": 16799850496, "has_inodes": true, "inodes": 4102069, "inodes_free": 4101999, "reads_completed": 0, "reads_merged": 0, "sectors_read": 0, "read_time": 0, "writes_completed": 0, "writes_merged": 0, "sectors_written": 0, "write_time": 0, "io_in_progress": 0, "io_time": 0, "weighted_io_time": 0 }, { "device": "shm", "type": "vfs", "capacity": 67108864, "usage": 0, "base_usage": 0, "available": 67108864, "has_inodes": true, "inodes": 4102069, "inodes_free": 4102068, "reads_completed": 0, "reads_merged": 0, "sectors_read": 0, "read_time": 0, "writes_completed": 0, "writes_merged": 0, "sectors_written": 0, "write_time": 0, "io_in_progress": 0, "io_time": 0, "weighted_io_time": 0 }, { "device": "/dev/vdb", "type": "vfs", "capacity": 53660876800, "usage": 35852288, "base_usage": 0, "available": 53625024512, "has_inodes": true, "inodes": 26214400, "inodes_free": 26214393, "reads_completed": 457, "reads_merged": 0, "sectors_read": 13162, "read_time": 3428, "writes_completed": 231, "writes_merged": 20, "sectors_written": 72963, "write_time": 1756, "io_in_progress": 0, "io_time": 3584, "weighted_io_time": 5184 } ], "task_stats": { "nr_sleeping": 0, "nr_running": 0, "nr_stopped": 0, "nr_uninterruptible": 0, "nr_io_wait": 0 } } ] }
我們可以看到這一段json數據主要分了4部分：
- reference——container本身的引用
- subcontainers——子容器的信息
- spec——說明信息
- stats——統計信息
與cAdvisor提供的統計數據一致：
type ContainerInfo struct { ContainerReference // The direct subcontainers of the current container. Subcontainers []ContainerReference `json:"subcontainers,omitempty"` // The isolation used in the container. Spec ContainerSpec `json:"spec,omitempty"` // Historical statistics gathered from the container. Stats []*ContainerStats `json:"stats,omitempty"` }
太棒了。
那么我就可以知道，獲取allcontainers的邏輯，其實就是向kubelet發送了一個http的post請求，然后獲取到返回的json數據。
(ii) 獲取pod相關的summary
在kubelet_client.go文件中還有一個獲取summary的方法：
func (self *KubeletClient) GetSummary(host Host) (*stats.Summary, error) { url := url.URL{ Scheme: "http", Host: host.String(), Path: "/stats/summary/", } if self.config != nil && self.config.EnableHttps { url.Scheme = "https" } req, err := http.NewRequest("GET", url.String(), nil) if err != nil { return nil, err } summary := &stats.Summary{} client := self.client if client == nil { client = http.DefaultClient } err = self.postRequestAndGetValue(client, req, summary) return summary, err }
恩，分析一下，很簡單，就是一個http get方法的調用。那我們再來模擬一下：
直接調用這個url即可，返回值太多，就不一一分析了。返回的數據結構如下：
type Summary struct { // Overall node stats. Node NodeStats `json:"node"` // Per-pod stats. Pods []PodStats `json:"pods"` }
主要就是當前node的統計信息，還有pod的聚合信息。可以自己追進NodeStats和PodeStats的源碼中看一下結構，在此就不列了。
值得注意的是，summary里面的stats和ContainerInfo是有些不一樣的。比如ContainerInfo里面會有DiskInfo相關的信息。
3）小結
heapster調用kubelet，其實就是發送了兩個http請求，一個是get方法，獲取所有pod的summary信息，另一個是post方法，獲取cAdvisor提供的所有ContainerInfo。然后在pod_aggregator.go方法中進行的數據相關的聚合。那我們對kubelet調用的源碼分析就到此為止了。接下來是筆者自己做的一個小實驗。
三、實驗
1、禁掉節點的4194端口
禁掉node的4194端口后，就無法通過ip：4194獲取到cAdvisor的stats數據了。那我們調用一下kubelet的api：調用kubelet的api：http://ip:10255/stats/container/
我們會發現，返回的數據項里缺少了stats這一項，而這一項恰恰是cAdvisor提供的最重要的監控統計數據。
2、重新開啟節點的4194端口
重新開啟node的4194端口，再重新調用一下kubelet的api，這下返回的結果里面又重新出現了stats這一項。
3、總結
結合kubelet的源碼可以發現，在kubelet啟動的時候會去啟動cAdvisor的4194端口，獲取基礎的統計數據。