Kubernetes：通過 minikube 安裝單機測試環境

官網

https://kubernetes.io/docs/tasks/tools/install-minikube/
https://kubernetes.io/docs/tasks/tools/install-kubectl/
https://kubernetes.io/docs/setup/learning-environment/minikube/

Kubernetes 組件

• kubeadm：CLI 命令，用於安裝初始化 Kubernetes 集群，添加新節點（minikube 會用到 kubeadm）
• kubectl：CLI 命令，用於操作集群，kubectl create/apply/scale/get/delete/exec/logs/etc
  
  
• Master：控制節點，包括 API Server、Scheduler、Controller Manager、etcd 等
• Node：計算節點，包括 Kubelet、Kube-Proxy、Container Runtime 等
• Add-On：除 Master、Node 的核心組件外還有一些推薦的可選組件，如 DNS、Dashboard、CNI 等等
  
  
• API Server：資源操作的唯一入口，REST API，無論外部如 kubectl 還是內部如 scheduler、controller、etcd、kubelet 都是通過 API Server 通信，支持 authentication/authorization
• Scheduler：負責資源的調度，按照預定的調度策略將 Pod 調度到相應的機器上
• Controller Manager：控制管理 Namespace、ServiceAccount、ResourceQuota、Replication、Volume、NodeLifecycle、Service、Endpoint、DaemonSet、Job、Cronjob、ReplicaSet、Deployment 等等
• etcd：high-available distributed key-value store，用於保存集群的狀態，提供 watch for changes 功能
  
  
• Kubelet：負責維護容器的生命周期，通過 API Server 把狀態告知 Master，同時負責 Volume (CVI) 和網絡 (CNI) 的管理
• Kube-Proxy：網絡代理和負載均衡，主要用於幫助 service 實現虛擬 IP
• Container Runtime：管理鏡像，運行 Pod 和容器，最常用的是 Docker
  
  
• Registry：存儲鏡像
  
  
• DNS：為集群提供 DNS 服務（Kube-DNS、CoreDNS）
• Dashboard：提供 Web UI
• Container Resource Monitoring：cAdvisor + Kubelet、Prometheus、Google Cloud Monitoring
• Cluster-level Logging：Fluentd
• CNI：Container Network Interface（Flannel、Calico）
• CSI：Container Storage Interface
• Ingress Controller：為服務提供外網入口

Kubernetes 是 Go 語言編寫的

Kubernetes 概念

• Pod：可以創建、調度、部署、管理的最小單位，Pod 包含一個或多個緊耦合的 container，並共享 hostname、IPC、network、storage 等
• ReplicaSet：確保 Pod 以指定的副本個數運行，和 Replication 的唯一區別是對選擇器的支持不一樣
• Deployment：用於管理 Pod、ReplicaSet，可以實現 Deploy/Scaling/Upgrade/Rollback
• Service：對外提供一個虛擬 IP，后端是一組有相同 Label 的 Pod，並在這些 Pod 之間做負載均衡（ClusterIP、NodePort、LoadBalancer 等幾種配置），即負責轉發數據到 Pod
• Ingress：鏈接多個 Service 作為統一入口，根據不同的路徑，將數據轉發到不同的 Service
• ConfigMap：用於解耦部署與配置的關系，即 Deployment、Pod 等不需要定義好配置，只需要指定對應的 ConfigMap，具體的內容由 ConfigMap 決定
• Secrets：ConfigMap 存儲不敏感的信息，而 Secrets 用於存儲敏感信息，Secrets 以加密的形式存儲（可能是存在 etcd 中），當 Secrets 掛載到 Pod 時，會被解密，並且實質上是存在內存中，可以以文件的形式掛載

以上這些都需要通過 yaml 文件定義（這些文件也被稱為 Manifest），然后通過 kubectl create -f xxx.yaml 啟動

Minikube Features

Minikube 用於創建單機版的 Kubernetes

• DNS
• NodePorts
• ConfigMaps and Secrets
• Dashboards
• Container Runtime: Docker, CRI-O, and containerd
• Enabling CNI (Container Network Interface)
• Ingress
  
  

Install kubectl

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl

chmod +x ./kubectl

sudo cp ./kubectl /usr/local/bin/kubectl

sudo kubectl version --client

sudo kubectl --help

實現 kubectl 的命令補全功能

# make sure bash-completion is installed
sudo apt-get install bash-completion

# make sure bash-completion is sourced in ~/.bashrc (root and other users)
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi

# or
if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
    . /etc/bash_completion
fi

# make sure kubectl completion is sourced in ~/.bashrc (root and other users)
echo 'source <(kubectl completion bash)' >>~/.bashrc

# generate kubectl completion file
sudo bash -c "./kubectl completion bash >/etc/bash_completion.d/kubectl"

kubectl 是 Go 語言實現的

Install a Hypervisor

這是可選項，通過安裝 KVM 或 VirtualBox 等工具，Minikube 可以創建 VM 並在上面安裝運行程序，如果不安裝 Hypervisor 那 Minikube 就在本機上安裝運行程序

Install Minikube

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

chmod +x minikube

sudo cp ./minikube /usr/local/bin/minikube

sudo minikube start --driver=<driver_name>

如果沒有安裝 Hypervisor，需要將 driver 指定為 none

sudo minikube start --driver=none

通過 none driver 啟動，結果報錯

😄  Ubuntu 16.04 (vbox/amd64) 上的 minikube v1.11.0
✨  根據用戶配置使用 none 驅動程序
💣  Sorry, Kubernetes 1.18.3 requires conntrack to be installed in root's path

提示要安裝 conntrack

sudo apt-get install conntrack

重新啟動 Minikube

😄  Ubuntu 16.04 (vbox/amd64) 上的 minikube v1.11.0
✨  根據用戶配置使用 none 驅動程序
👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=4, Memory=7976MB, Disk=18014MB) ...
ℹ️  OS release is Ubuntu 16.04.6 LTS
🐳  正在 Docker 19.03.8 中准備 Kubernetes v1.18.3…
❗  This bare metal machine is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
    > kubeadm.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubectl.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubelet.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubectl: 41.99 MiB / 41.99 MiB [----------------] 100.00% 6.79 MiB p/s 6s
    > kubeadm: 37.97 MiB / 37.97 MiB [----------------] 100.00% 4.44 MiB p/s 9s
    > kubelet: 108.04 MiB / 108.04 MiB [-------------] 100.00% 6.35 MiB p/s 18s

💥  initialization failed, will try again: run: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.18.3:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap": exit status 1
stdout:
[init] Using Kubernetes version: v1.18.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

stderr:
W0609 16:35:49.251770   29943 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING Swap]: running with swap on is not supported. Please disable swap
        [WARNING FileExisting-ebtables]: ebtables not found in system path
        [WARNING FileExisting-socat]: socat not found in system path
        [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.18.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.18.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.18.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.18.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.4.3-0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.6.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

還是報錯，無法訪問 https://k8s.gcr.io，使用國內的源

sudo minikube start --driver=none --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

成功了

😄  Ubuntu 16.04 (vbox/amd64) 上的 minikube v1.11.0
✨  根據現有的配置文件使用 none 驅動程序
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing none bare metal machine for "minikube" ...
ℹ️  OS release is Ubuntu 16.04.6 LTS
🐳  正在 Docker 19.03.8 中准備 Kubernetes v1.18.3…
🤹  開始配置本地主機環境...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❗  kubectl 和 minikube 配置將存儲在 /home/lin 中
❗  如需以您自己的用戶身份使用 kubectl 或 minikube 命令，您可能需要重新定位該命令。例如，如需覆蓋您的自定義設置，請運行：

    ▪ sudo mv /home/lin/.kube /home/lin/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  此操作還可通過設置環境變量 CHANGE_MINIKUBE_NONE_USER=true 自動完成
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  完成！kubectl 已經配置至 "minikube"
💡  為獲得最佳結果，請安裝 kubectl：https://kubernetes.io/docs/tasks/tools/install-kubectl/

Minikube 默認至少要雙核，如果只有單核，需要指定配置

sudo minikube start --driver=none \
                    --extra-config=kubeadm.ignore-preflight-errors=NumCPU \
                    --force --cpus 1 \
                    --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

檢查 Minikube 的狀態

sudo minikube status

正常返回如下

minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

停止集群

sudo minikube stop

刪除集群

sudo minikube delete

驗證 kubectl

sudo kubectl version --client
sudo kubectl cluster-info

返回

lin@lin-VirtualBox:~/K8S$ sudo kubectl version --client
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
lin@lin-VirtualBox:~/K8S$
lin@lin-VirtualBox:~/K8S$
lin@lin-VirtualBox:~/K8S$ sudo kubectl cluster-info
Kubernetes master is running at https://xxx.xxx.xxx.xxx:8443
KubeDNS is running at https://xxx.xxx.xxx.xxx:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Minikube 是 Go 語言實現的

例子：echoserver

echoserver 鏡像是一個簡單的 HTTP 服務器，將請求的 body 攜待的參數返回


這里沒有定義 manifest 文件，而是直接指定 image 做 deploy，這一步會啟動一個 deployment 和對應的 pod

sudo kubectl create deployment hello-minikube \
     --image=registry.cn-hangzhou.aliyuncs.com/google_containers/echoserver:1.10

暴露端口，這一步會啟動一個 service

sudo kubectl expose deployment hello-minikube --type=NodePort --port=8080

查看 pod 的狀態

sudo kubectl get pod
sudo kubectl get pods
sudo kubectl get pods -o wide

get pod 的返回

NAME                              READY   STATUS    RESTARTS   AGE
hello-minikube-7df785b6bb-v2phl   1/1     Running   0          5m51s

查看 pod 的信息

sudo kubectl describe pod hello-minikube

describe pod 的返回

Name:         hello-minikube-7df785b6bb-mw6kv
Namespace:    default
Priority:     0
Node:         lin-virtualbox/100.98.137.196
Start Time:   Wed, 10 Jun 2020 16:30:18 +0800
Labels:       app=hello-minikube
              pod-template-hash=7df785b6bb
Annotations:  <none>
Status:       Running
IP:           172.17.0.6
IPs:
  IP:           172.17.0.6
Controlled By:  ReplicaSet/hello-minikube-7df785b6bb
Containers:
  echoserver:
    Container ID:   docker://ca6c7070ef7afc260f6fe6538da49e91bc60ba914b623d6080b03bd2886343b3
    Image:          registry.cn-hangzhou.aliyuncs.com/google_containers/echoserver:1.10
    Image ID:       docker-pullable://registry.cn-hangzhou.aliyuncs.com/google_containers/echoserver@sha256:56bec57144bd3610abd4a1637465ff491dd78a5e2ae523161569fa02cfe679a8
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 10 Jun 2020 16:30:21 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-znf6q (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-znf6q:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-znf6q
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

查看 deployment 的狀態

sudo kubectl get deployment

get deployment 的返回

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
hello-minikube   1/1     1            1           80m

查看 service 的狀態

sudo minikube service hello-minikube --url

# or

sudo minikube service hello-minikube

返回

http://100.98.137.196:31526

# or

|-----------|----------------|-------------|-----------------------------|
| NAMESPACE |      NAME      | TARGET PORT |             URL             |
|-----------|----------------|-------------|-----------------------------|
| default   | hello-minikube |        8080 | http://100.98.137.196:31526 |
|-----------|----------------|-------------|-----------------------------|

向 echoserver 發送請求

curl -X POST -d '{"abc":123}' http://100.98.137.196:31526/api/v1/hello

返回

Hostname: hello-minikube-7df785b6bb-v2phl

Pod Information:
        -no pod information available-

Server values:
        server_version=nginx: 1.13.3 - lua: 10008

Request Information:
        client_address=172.17.0.1
        method=POST
        real path=/api/v1/hello
        query=
        request_version=1.1
        request_scheme=http
        request_uri=http://100.98.137.196:8080/api/v1/hello

Request Headers:
        accept=*/*
        content-length=11
        content-type=application/x-www-form-urlencoded
        host=100.98.137.196:31384
        user-agent=curl/7.47.0

Request Body:
{"abc":123}

刪除 service

sudo kubectl delete services hello-minikube

刪除 service 后 Pod 不受影響還在 running

刪除 deployment 后 Pod 才會被刪除

sudo kubectl delete deployment hello-minikube

啟動 Dashboard

sudo minikube dashboard

返回

🔌  正在開啟 dashboard ...
🤔  正在驗證 dashboard 運行情況 ...
🚀  Launching proxy ...
🤔  正在驗證 proxy 運行狀況 ...

http://127.0.0.1:42155/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/

登陸 URL 可以看到

命名空間選擇 "全部 namespaces"，可以看到，K8S 的組件如 apiserver、controller、etcd、scheduler 等等也是容器化的

實際上這些鏡像和啟動的容器通過 docker 命令也可以看到

lin@lin-VirtualBox:~$ docker images
REPOSITORY                                                                    TAG                 IMAGE ID            CREATED             SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.18.3             3439b7546f29        2 weeks ago         117MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.18.3             76216c34ed0c        2 weeks ago         95.3MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.18.3             7e28efa976bd        2 weeks ago         173MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.18.3             da26705ccb4b        2 weeks ago         162MB
kubernetesui/dashboard                                                        v2.0.0              8b32422733b3        7 weeks ago         222MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.2                 80d28bedfe5d        3 months ago        683kB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   1.6.7               67da37a9a360        4 months ago        43.8MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.4.3-0             303ce5db0e90        7 months ago        288MB
kubernetesui/metrics-scraper                                                  v1.0.2              3b08661dc379        7 months ago        40.1MB
registry.cn-hangzhou.aliyuncs.com/google_containers/echoserver                1.10                365ec60129c5        2 years ago         95.4MB
registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner       v1.8.1              4689081edb10        2 years ago         80.8MB

docker ps -a

可以用 docker 命令操作這些 K8S 啟動的容器，比如

docker exec
docker logs

kubectl 也有相應的命令做操作，比如

kubectl exec
kubectl logs

另外繞過 K8S 部署的容器 K8S 無法管理

Wordaround if coredns fail

從 Dashboard 可以看到，有個 coredns 的服務出錯了，有的服務會受到影響，比如后面要講的 Flink on K8S

通過 kubectl get pod 查看狀態

lin@lin-VirtualBox:~/K8S$ sudo kubectl get pod -n kube-system
NAME                                     READY   STATUS             RESTARTS   AGE
coredns-546565776c-5fq7p                 0/1     CrashLoopBackOff   3          7h21m
coredns-546565776c-zx72j                 0/1     CrashLoopBackOff   3          7h21m
etcd-lin-virtualbox                      1/1     Running            0          7h21m
kube-apiserver-lin-virtualbox            1/1     Running            0          7h21m
kube-controller-manager-lin-virtualbox   1/1     Running            0          7h21m
kube-proxy-rgsgg                         1/1     Running            0          7h21m
kube-scheduler-lin-virtualbox            1/1     Running            0          7h21m
storage-provisioner                      1/1     Running            0          7h21m

通過 kubectl logs 查看日志

lin@lin-VirtualBox:~/K8S$ sudo kubectl logs -n kube-system coredns-546565776c-5fq7p
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b
[FATAL] plugin/loop: Loop (127.0.0.1:58992 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 5310754532638830744.3332451342029566297."

臨時方案如下

# 編輯 coredns 的 ConfigMap，有一行 loop，將其刪除 
sudo kubectl edit cm coredns -n kube-system

# 重啟服務
sudo kubectl delete pod coredns-546565776c-5fq7p -n kube-system
sudo kubectl delete pod coredns-546565776c-zx72j -n kube-system

重啟后可以看到 coredns 變成 running 了

例子：Flink on K8S

定義 manifest 文件
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html

flink-configuration-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: flink-config
  labels:
    app: flink
data:
  flink-conf.yaml: |+
    jobmanager.rpc.address: flink-jobmanager
    taskmanager.numberOfTaskSlots: 1
    blob.server.port: 6124
    jobmanager.rpc.port: 6123
    taskmanager.rpc.port: 6122
    jobmanager.heap.size: 1024m
    taskmanager.memory.process.size: 1024m
  log4j.properties: |+
    log4j.rootLogger=INFO, file
    log4j.logger.akka=INFO
    log4j.logger.org.apache.kafka=INFO
    log4j.logger.org.apache.hadoop=INFO
    log4j.logger.org.apache.zookeeper=INFO
    log4j.appender.file=org.apache.log4j.FileAppender
    log4j.appender.file.file=${log.file}
    log4j.appender.file.layout=org.apache.log4j.PatternLayout
    log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
    log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

jobmanager-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flink-jobmanager
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flink
      component: jobmanager
  template:
    metadata:
      labels:
        app: flink
        component: jobmanager
    spec:
      containers:
      - name: jobmanager
        image: flink:latest
        workingDir: /opt/flink
        command: ["/bin/bash", "-c", "$FLINK_HOME/bin/jobmanager.sh start;\
          while :;
          do
            if [[ -f $(find log -name '*jobmanager*.log' -print -quit) ]];
              then tail -f -n +1 log/*jobmanager*.log;
            fi;
          done"]
        ports:
        - containerPort: 6123
          name: rpc
        - containerPort: 6124
          name: blob
        - containerPort: 8081
          name: ui
        livenessProbe:
          tcpSocket:
            port: 6123
          initialDelaySeconds: 30
          periodSeconds: 60
        volumeMounts:
        - name: flink-config-volume
          mountPath: /opt/flink/conf
        securityContext:
          runAsUser: 9999  # refers to user _flink_ from official flink image, change if necessary
      volumes:
      - name: flink-config-volume
        configMap:
          name: flink-config
          items:
          - key: flink-conf.yaml
            path: flink-conf.yaml
          - key: log4j.properties
            path: log4j.properties

taskmanager-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flink-taskmanager
spec:
  replicas: 2
  selector:
    matchLabels:
      app: flink
      component: taskmanager
  template:
    metadata:
      labels:
        app: flink
        component: taskmanager
    spec:
      containers:
      - name: taskmanager
        image: flink:latest
        workingDir: /opt/flink
        command: ["/bin/bash", "-c", "$FLINK_HOME/bin/taskmanager.sh start; \
          while :;
          do
            if [[ -f $(find log -name '*taskmanager*.log' -print -quit) ]];
              then tail -f -n +1 log/*taskmanager*.log;
            fi;
          done"]
        ports:
        - containerPort: 6122
          name: rpc
        livenessProbe:
          tcpSocket:
            port: 6122
          initialDelaySeconds: 30
          periodSeconds: 60
        volumeMounts:
        - name: flink-config-volume
          mountPath: /opt/flink/conf/
        securityContext:
          runAsUser: 9999  # refers to user _flink_ from official flink image, change if necessary
      volumes:
      - name: flink-config-volume
        configMap:
          name: flink-config
          items:
          - key: flink-conf.yaml
            path: flink-conf.yaml
          - key: log4j.properties
            path: log4j.properties

jobmanager-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: flink-jobmanager
spec:
  type: ClusterIP
  ports:
  - name: rpc
    port: 6123
  - name: blob
    port: 6124
  - name: ui
    port: 8081
  selector:
    app: flink
    component: jobmanager

jobmanager-rest-service.yaml. Optional service, that exposes the jobmanager rest port as public Kubernetes node’s port.

apiVersion: v1
kind: Service
metadata:
  name: flink-jobmanager-rest
spec:
  type: NodePort
  ports:
  - name: rest
    port: 8081
    targetPort: 8081
  selector:
    app: flink
    component: jobmanager

通過 manifest 文件啟動，注意有先后順序

sudo kubectl create -f flink-configuration-configmap.yaml
sudo kubectl create -f jobmanager-service.yaml
sudo kubectl create -f jobmanager-deployment.yaml
sudo kubectl create -f taskmanager-deployment.yaml

查看配置的 ConfigMap

lin@lin-VirtualBox:~/K8S$ sudo kubectl get configmap
NAME           DATA   AGE
flink-config   2      4h28m

查看啟動的 Pod

lin@lin-VirtualBox:~/K8S$ sudo kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
flink-jobmanager-574676d5d5-g75gh    1/1     Running   0          5m24s
flink-taskmanager-5bdb4857bc-vvn2j   1/1     Running   0          5m23s
flink-taskmanager-5bdb4857bc-wn5c2   1/1     Running   0          5m23s
hello-minikube-7df785b6bb-j9g6g      1/1     Running   0          55m

查看啟動的 Deployment

lin@lin-VirtualBox:~/K8S$ sudo kubectl get deployment
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
flink-jobmanager    1/1     1            1           4h28m
flink-taskmanager   1/2     2            1           4h28m
hello-minikube      1/1     1            1           5h18m

查看啟動的 Service

lin@lin-VirtualBox:~/K8S$ sudo kubectl get service
NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
flink-jobmanager   ClusterIP   10.96.132.16     <none>        6123/TCP,6124/TCP,8081/TCP   4h28m
hello-minikube     NodePort    10.104.137.240   <none>        8080:30041/TCP               5h18m
kubernetes         ClusterIP   10.96.0.1        <none>        443/TCP                      5h25m

登陸 Flink UI 以及提交 Flink Job 的幾種方式
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html#deploy-flink-session-cluster-on-kubernetes

1. proxy 方式

命令

kubectl proxy

登陸 URL

http://localhost:8001/api/v1/namespaces/default/services/flink-jobmanager:ui/proxy

（這種方式沒講到怎么 run job）

2. NodePort service 方式

命令

sudo kubectl create -f jobmanager-rest-service.yaml

sudo kubectl get svc flink-jobmanager-rest

lin@lin-VirtualBox:~/K8S$ sudo kubectl get svc flink-jobmanager-rest
NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
flink-jobmanager-rest   NodePort   10.96.150.145   <none>        8081:32598/TCP   12s

登陸 URL

http://10.96.150.145:8081

提交 Job

./bin/flink run -m 10.96.150.145:8081 ./examples/streaming/WordCount.jar

在 UI 上可以看到提交的 Job

3. port-forward

宿主機安裝 socat

sudo apt-get install socat

命令

sudo kubectl port-forward flink-jobmanager-574676d5d5-xd9kx 8081:8081

登陸 URL

http://localhost:8081

提交 Job

./bin/flink run -m localhost:8081 ./examples/streaming/WordCount.jar

在 UI 上可以看到提交的 Job


刪除 Flink

sudo kubectl delete -f jobmanager-deployment.yaml
sudo kubectl delete -f taskmanager-deployment.yaml
sudo kubectl delete -f jobmanager-service.yaml
sudo kubectl delete -f flink-configuration-configmap.yaml