[k8s]elk架構設計-k8s集群里搭建

elasticsearch和mysql的對比

https://blog.csdn.net/qq_21383435/article/details/79323383

Mapping ~ Schema

master: 負責在集群范圍內創建/刪除索引,將分片分配給這些節點.
data:   用來保存數據和倒排索引,node.data=ture
client: 將節點配置為客戶端節點，並充當負載平衡器，將傳入的請求路由到集群中的不同節點。node.master和node.data設置為false

https://blog.csdn.net/sdksdk0/article/details/78469190

Index templates

索引可使用預定義的模板進行創建,這個模板稱作Index templates。模板設置包括settings和mappings，通過模式匹配的方式使得多個索引重用一個模板，例如：

curl -XPUT localhost:9200/_template/template_1 -d '
{
    "template" : "te*",
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "type1" : {
            "_source" : {"enabled" : false }
        }
    }
}

elk設計架構

參考
k8sgithub上 https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

Elasticsearch最佳實踐建議將這些節點分成三個角色：

Master 節點 - 僅用於集群管理，沒有數據，沒有HTTP API
Client 節點 - 用於客戶端使用，無數據，  使用HTTP API
Data   節點 - 旨在存儲和索引數據，      沒有HTTP API

分為 管理區 存儲區 讀寫區

定制鏡像

- es
0. 我把鏡像推到了dockerhub: lanny/quay.io_pires_docker-elasticsearch-kubernetes:5.6.0
1. 鏡像里用到的yaml: https://github.com/lannyMa/docker-elasticsearch-kubernetes
2. 可以安裝自定義插件后重新build

- kibana
0. 用到這個人的這個鏡像 cfontes/kibana-xpack-less:5.5.0 https://hub.docker.com/r/cfontes/kibana-xpack-less/
   這個人的git(包含了elk的k8s yaml): https://github.com/cfontes/kubernetes-elasticsearch-cluster


- k8s的elk集群的yaml
https://github.com/pires/kubernetes-elasticsearch-cluster
鏡像到這個地址: https://github.com/lannyMa/kubernetes-elasticsearch-cluster

內含elk 普羅 harbor的yaml,比較實用
https://github.com/cnych/k8s-repo

集群yaml里搜HTTP_ENABLE,統一打開head訪問權限

es-master.yaml

        - name: HTTP_ENABLE
          value: "true"
        - name: HTTP_CORS_ALLOW_ORIGIN
          value: "*"

創建集群

創建pv

$ cat es-data_claim.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: opspvc
  namespace: kube-ops
  annotations:
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

先創建svc類型

后依次創建 data master client kibana

訪問es api

訪問kibana

打開kibana,插入示例數據

PUT _bulk
{"index":{"_index":"lagou","_type":"job","_id":"1"}}
{"title":"python分布式爬蟲開發","salary_min":15000,"city":"深圳","company":{"name":"騰訊","company_addr":"深圳市軟件園"},"publish_date":"2017-11-11","comments":15}
{"index":{"_index":"lagou","_type":"job","_id":"2"}}
{"title":"django開發","salary_min":15000,"city":"上海","company":{"name":"阿里","company_addr":"廣州市軟件園"},"publish_date":"2017-11-12","comments":20}

打開head查看數據

 docker run -d -v /etc/localtime:/etc/localtime --restart=always -p 9100:9100 mobz/elasticsearch-head:5

查看數據

todo:Curator(館長) 管理索引的工具

Curator的主要用途:

• 獲取索引列表

• 過濾索引,刪除/快照

• 可以對索引做這些操作

name	用途
Alias	別號
Allocation	分配
Close	關
Cluster Routing	群集路由
Create Index	創建索引
Delete Indices	刪除索引
Delete Snapshots	刪除快照
forceMerge	forceMerge
Index Settings	索引設置
Open	打開
Reindex	重新編制
Replicas	副本
Restore	恢復
Rollover	滾下
Shrink	收縮
Snapshot	快照

elk整體鏈條架構

es版本兼容

    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"

curl http://10.100.32.137:9200


{
  "name" : "es-data-5c5969967-wb2b8",
  "cluster_name" : "myesdb",
  "cluster_uuid" : "qSps-b9dRI2ngGHBguJ44Q",
  "version" : {
    "number" : "6.3.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "424e937",
    "build_date" : "2018-06-11T23:38:03.357887Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}