kubernetes中通過static pod部署elasticsearch生產集群

系統：centos 7.6
環境：x.x.x.x x.x.x. x.x.x.x

資源配置：

內存：16G

CPU：8核

硬盤：500G

目前部署的es磁盤/內存比在30:1，如果預算充足這個比例越小越好

1.創建和配置保存目錄,因為es官方鏡像啟動用戶為uid為1000的elasticsearch，所以需要在宿主機上創建一個uid為1000的用戶

groupadd -g 1000 elasticsearch && useradd -g 1000 -u 1000 -s /sbin/nologin && mkdir -pv /data/k8s/volumn_data/{es_config,es_data} && chown -R 777 /data/k8s/volumn_data/{es_config,es_data}

2.調整系統配置

echo "* soft memlock unlimited" >>/etc/security/limits.conf 

echo "* hard memlock unlimited" >>/etc/security/limits.conf

echo "vm.max_map_count=655360" >> /etc/sysctl.conf    

 sysctl ‐p

            

3.創建configmap

apiVersion: v1
data:
    cluster.name: 'cdp-prd-cluster'
    node.name: 'prd-cdp-es-147'
    path.data: '/data'
    bootstrap.memory_lock: 'true'
    discovery.seed_hosts: '["x.x.x.x", "x.x.x.x","x.x.x.x"]'
    cluster.initial_master_nodes: '["x.x.x.x", "x.x.x.x","x.x.x.x"]'
    ELASTIC_PASSWORD: 'Aa111111'
kind: ConfigMap
metadata:
    name: es-config
    namespace: default

4.部署elasticsearch

apiVersion: v1
kind: Pod
metadata:
    labels:
    app: cdp-elasticsearch
    name: cdp-elasticsearch
    namespace: default
spec:
    containers:
    - image: hub.docker.cn/es:v7.1.1
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 3
        httpGet:
        path: /
        port: 9200
        scheme: HTTP
        httpHeaders:
        - name: Authorization
          value: "xxx"
        initialDelaySeconds: 30
        periodSeconds: 15
        successThreshold: 1
        timeoutSeconds: 5
    name: cdp-elasticsearch
    ports:
    - containerPort: 9200
      name: db
      protocol: TCP
      hostPort: 9200
    - containerPort: 9300   
      name: transport
      protocol: TCP
      hostPort: 9300
    volumeMounts:
    - mountPath: /data
      name: elasticsearch-data
    - mountPath: /usr/share/elasticsearch/config
      name: elasticsearch-config
  hostNetwork: true
  volumes:
  - name: elasticsearch-data
    hostPath:
        path: /data/k8s/volumn_data/es_data
  - name : elasticsearch-config
    hostPath:
        path: /data/k8s/volumn_data/es_config/config
  initContainers:
  - image: alpine:3.6
    command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"]
    name: elasticsearch-logging-init
    securityContext:
      privileged: true

以上httpHeaders中的 value是更加之后生成的es賬號密碼經過base64加密而來，所在在集群部署完成之前先不要httpHeaders字段，待完全部署完成之后再加上認證

5.配置TLS
elasticsearch集群正常部署之后，進入到其中一個es節點，執行一下命令生成證書

./bin/elasticsearch‐certutil ca ‐‐days 3660
# 兩次回車
./bin/elasticsearch‐certutil cert ‐‐ca elastic‐stack‐ca.p12
#三次回車
mkdir config/certs
mv elastic‐*.p12 config/certs/  

再把證書文件 elastic-certificates.p12 復制到其他master節點並賦予權限(/data/k8s/volumn_data/es_config/config)

6.修改所有節點配置文件

vim /data/k8s/volumn_data/es_config/config/elasticsearch.yml


cluster.name: "cdp-prd-cluster"
network.host: 0.0.0.0
node.name: xxx
path.data: /data
#bootstrap.memory_lock: true
discovery.seed_hosts: ["x.x.x.x","x.x.x.x","x.x.x.x"]
cluster.initial_master_nodes: ["x.x.x.x","x.x.x.x","x.x.x.x"]
#-----
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

7.重啟所有節點es

8.生成客戶端證書
進入任意一個es節點

./bin/elasticsearch‐certutil cert ‐‐ca config/certs/elastic‐stack‐ca.p12 ‐name "CN=esuser,OU=prd,DC=ddd,DC=com"
# 回車
client.p12
回車

9.拆分證書

mv client.p12 config/certs/

cd config/certs/ 

openssl pkcs12 ‐in client.p12 ‐nocerts ‐nodes > client‐key.pem

openssl pkcs12 ‐in client.p12 ‐clcerts ‐nokeys >11.集群驗證 client.crt

openssl pkcs12 ‐in client.p12 ‐cacerts ‐nokeys ‐chain > client‐ca.crt

 chown -R elasticsearch.elasticsearch config/

10.配置密碼

./bin/elasticsearch‐setup‐passwords interactive #手動設置各個賬號的密碼
./bin/elasticsearch‐setup‐passwords auto #隨機密碼

11.集群驗證

curl ‐‐user elastic:xxxxx ‐XGET '127.0.0.1:9200/_cat/health?v&pretty'

12.elasticsearch用戶權限創建

#創建所有index讀寫權限：
curl -XPOST --user XXX:XXX'http://127.0.0.1:9200/_xpack/security/role/readwriterole' -H "Content-Type: application/json" -d '{"indices":[{"names":["*"],"privileges":["read","write"]}]}'

#查詢權限：
curl -XPOST --user XXX:XXX'http://127.0.0.1:9200/_xpack/security/role?pretty
#創建用戶並授權：
curl -XPOST --user elastic:xxx 'http://127.0.0.1:9200/_xpack/security/user/rwuser' -H "Content-Type: application/json" -d '{
           "password" : "xxx",
           "full_name" : "read write user",
           "email" : "",
           "roles" : [ "readwriterole" ]
            }'
#查詢用戶：
curl -XPOST --user XXX:XXX'http://127.0.0.1:9200/_xpack/security/user?pretty