寫在前面：
公司簡單的業務場景：
1. 基於域名進行訪問后端不同的容器服務
2. 后端服務需要根據負載情況進行動態伸縮
3. 暴露創建和刪除服務的接口
1. Ingress簡介
引用官方關於ingress的介紹我們可以得知，ingress是一種通過http協議暴露kubernetes內部服務的api對象，即充當Edge Router邊界路由器的角色對外基於七層的負載均衡調度機制，能夠提供以下幾個功能：
- 負載均衡，將請求自動負載均衡到后端的Pod上；
- SSL加密，客戶端到Ingress Controller為https加密，到后端Pod為明文的http；
- 基於名稱的虛擬主機，提供基於域名或URI更靈活的路由方式
實現Ingress包含的組件有：
- Ingress，客戶端，負責定義ingress配置，將請求轉發給Ingress Controller；
- Ingress Controller，Ingress控制器，實現七層轉發的Edge Router，通過調用k8s的api動態感知集群中Pod的變化而動態更新配置文件並重載， Controller需要部署在k8s集群中以實現和集群中的pod通信，通常以DaemonSets或Deployments的形式部署，並對外暴露80和443端口，對於DaemonSets來說，一般是以hostNetwork或者hostPort的形式暴露，Deployments則以NodePort的方式暴露，控制器的多個節點則借助外部負載均衡ExternalLB以實現統一接入；
- Ingress配置規則，Controller控制器通過service服務發現機制動態實現后端Pod路由轉發規則的實現；
- Service，kuberntes中四層的負載均衡調度機制，Ingress借助service的服務發現機制實現集群中Pod資源的動態感知；
- Pod，后端實際負責響應請求容器，由控制器如Deployment創建，通過標簽Labels和service關聯，服務發現。
簡而言之，ingress控制器借助service的服務發現機制實現配置的動態更新以實現Pod的負載均衡機制實現，由於涉及到Ingress Controller的動態更新，目前社區Ingress Controller大體包含兩種類型的控制器：
- 傳統的七層負載均衡如Nginx，HAproxy，開發了適應微服務應用的插件，具有成熟，高性能等優點；
- 新型微服務負載均衡如Traefik，Envoy，Istio，專門適用於微服務+容器化應用場景，具有動態更新特點；
| 類型 | 常見類型 | 優點 | 缺點 |
|---|---|---|---|
| 傳統負載均衡 | nginx，haproxy | 成熟，穩定，高性能 | 動態更新需reload配置文件 |
| 微服務負載均衡 | Traefik，Envoy，Istio | 天生為微服務而生，動態更新 | 性能還有待提升 |
2. Nginx Ingress
2.1 Nginx ingress介紹
Nginx Ingress Controller是實現ingress的具體實現，包含有兩個版本：Ngnix OSS和Nginx Plus版，后者是商業化增強版，支持更多的功能，詳情參考官方文檔介紹https://www.nginx.com/products/nginx/kubernetes-ingress-controller#compare-versions
2.2 Nginx ingress安裝
首先需要安裝Nginx Ingress Controller控制器，控制器安裝方式包含兩種：DaemonSets和Deployments（我們這里采用DaemonSets）。
- DaemonSets通過hostPort的方式暴露80和443端口，可通過Node的調度由專門的節點實現部署；用於確保k8s集群每個work節點上擁有唯一一個pod節點，支持work節點動態增加而動態創建對應的pod節點。
- Deployments則通過NodePort的方式實現控制器端口的暴露，借助外部負載均衡實現高可用負載均衡；管理無狀態應用（關注的是群體性行為）的pod控制器/pod內應用守護進程運行的
除此之外，還需要部署Namespace，ServiceAccount，RBAC，Secrets，Custom Resource Definitions等資源，如下開始部署。
2.2.1 基礎依賴環境准備
1、github中下載源碼包,安裝部署文件在kubernetes-ingress/deployments/目錄下
shell>git clone https://github.com/nginxinc/kubernetes-ingress.git
shell>tree kubernetes-ingress/deployments/
kubernetes-ingress/deployments/
├── common
│ ├── default-server-secret.yaml
│ ├── nginx-config.yaml
│ ├── ns-and-sa.yaml
│ ├── vs-definition.yaml
│ └── vsr-definition.yaml
├── daemon-set
│ ├── nginx-ingress.yaml
│ └── nginx-plus-ingress.yaml
├── deployment
│ ├── nginx-ingress.yaml
│ └── nginx-plus-ingress.yaml
├── helm-chart
│ ├── chart-icon.png
│ ├── Chart.yaml
│ ├── README.md
│ ├── templates
│ │ ├── controller-configmap.yaml
│ │ ├── controller-daemonset.yaml
│ │ ├── controller-deployment.yaml
│ │ ├── controller-leader-election-configmap.yaml
│ │ ├── controller-secret.yaml
│ │ ├── controller-serviceaccount.yaml
│ │ ├── controller-service.yaml
│ │ ├── controller-vs-definition.yaml
│ │ ├── controller-vsr-definition.yaml
│ │ ├── controller-wildcard-secret.yaml
│ │ ├── _helpers.tpl
│ │ ├── NOTES.txt
│ │ └── rbac.yaml
│ ├── values-icp.yaml
│ ├── values-plus.yaml
│ └── values.yaml
├── rbac
│ └── rbac.yaml
├── README.md
└── service
├── loadbalancer-aws-elb.yaml
├── loadbalancer.yaml
└── nodeport.yaml
備注：這里作者對各個文件進行了整合，最終用5個配置文件實現場景配置
2、創建Namespace、ServiceAccount、Secret（證書為默認證書，可自定義修改）、ConfigMap
執行生效命令： kubectl apply -f 1.ns-sa-secret.yaml
#創建nginx-ingress命名空間 apiVersion: v1 kind: Namespace metadata: name: nginx-ingress --- #創建serviceAccount apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress namespace: nginx-ingress --- #創建https證書 apiVersion: v1 kind: Secret metadata: name: clouddev.apicloud.saas.com-server-secret namespace: nginx-ingress type: Opaque data: tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN2akNDQWFZQ0NRREFPRjl0THNhWFhEQU5CZ2txaGtpRzl3MEJBUXNGQURBaE1SOHdIUVlEVlFRRERCWk8KUjBsT1dFbHVaM0psYzNORGIyNTBjbTlzYkdWeU1CNFhEVEU0TURreE1qRTRNRE16TlZvWERUSXpNRGt4TVRFNApNRE16TlZvd0lURWZNQjBHQTFVRUF3d1dUa2RKVGxoSmJtZHlaWE56UTI5dWRISnZiR3hsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUwvN2hIUEtFWGRMdjNyaUM3QlBrMTNpWkt5eTlyQ08KR2xZUXYyK2EzUDF0azIrS3YwVGF5aGRCbDRrcnNUcTZzZm8vWUk1Y2Vhbkw4WGM3U1pyQkVRYm9EN2REbWs1Qgo4eDZLS2xHWU5IWlg0Rm5UZ0VPaStlM2ptTFFxRlBSY1kzVnNPazFFeUZBL0JnWlJVbkNHZUtGeERSN0tQdGhyCmtqSXVuektURXUyaDU4Tlp0S21ScUJHdDEwcTNRYzhZT3ExM2FnbmovUWRjc0ZYYTJnMjB1K1lYZDdoZ3krZksKWk4vVUkxQUQ0YzZyM1lma1ZWUmVHd1lxQVp1WXN2V0RKbW1GNWRwdEMzN011cDBPRUxVTExSakZJOTZXNXIwSAo1TmdPc25NWFJNV1hYVlpiNWRxT3R0SmRtS3FhZ25TZ1JQQVpQN2MwQjFQU2FqYzZjNGZRVXpNQ0F3RUFBVEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQWpLb2tRdGRPcEsrTzhibWVPc3lySmdJSXJycVFVY2ZOUitjb0hZVUoKdGhrYnhITFMzR3VBTWI5dm15VExPY2xxeC9aYzJPblEwMEJCLzlTb0swcitFZ1U2UlVrRWtWcitTTFA3NTdUWgozZWI4dmdPdEduMS9ienM3bzNBaS9kclkrcUI5Q2k1S3lPc3FHTG1US2xFaUtOYkcyR1ZyTWxjS0ZYQU80YTY3Cklnc1hzYktNbTQwV1U3cG9mcGltU1ZmaXFSdkV5YmN3N0NYODF6cFErUyt1eHRYK2VBZ3V0NHh3VlI5d2IyVXYKelhuZk9HbWhWNThDd1dIQnNKa0kxNXhaa2VUWXdSN0diaEFMSkZUUkk3dkhvQXprTWIzbjAxQjQyWjNrN3RXNQpJUDFmTlpIOFUvOWxiUHNoT21FRFZkdjF5ZytVRVJxbStGSis2R0oxeFJGcGZnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdi91RWM4b1JkMHUvZXVJTHNFK1RYZUprckxMMnNJNGFWaEMvYjVyYy9XMlRiNHEvClJOcktGMEdYaVN1eE9ycXgrajlnamx4NXFjdnhkenRKbXNFUkJ1Z1B0ME9hVGtIekhvb3FVWmcwZGxmZ1dkT0EKUTZMNTdlT1l0Q29VOUZ4amRXdzZUVVRJVUQ4R0JsRlNjSVo0b1hFTkhzbysyR3VTTWk2Zk1wTVM3YUhudzFtMApxWkdvRWEzWFNyZEJ6eGc2clhkcUNlUDlCMXl3VmRyYURiUzc1aGQzdUdETDU4cGszOVFqVUFQaHpxdmRoK1JWClZGNGJCaW9CbTVpeTlZTW1hWVhsMm0wTGZzeTZuUTRRdFFzdEdNVWozcGJtdlFmazJBNnljeGRFeFpkZFZsdmwKMm82MjBsMllxcHFDZEtCRThCay90elFIVTlKcU56cHpoOUJUTXdJREFRQUJBb0lCQVFDZklHbXowOHhRVmorNwpLZnZJUXQwQ0YzR2MxNld6eDhVNml4MHg4Mm15d1kxUUNlL3BzWE9LZlRxT1h1SENyUlp5TnUvZ2IvUUQ4bUFOCmxOMjRZTWl0TWRJODg5TEZoTkp3QU5OODJDeTczckM5bzVvUDlkazAvYzRIbjAzSkVYNzZ5QjgzQm9rR1FvYksKMjhMNk0rdHUzUmFqNjd6Vmc2d2szaEhrU0pXSzBwV1YrSjdrUkRWYmhDYUZhNk5nMUZNRWxhTlozVDhhUUtyQgpDUDNDeEFTdjYxWTk5TEI4KzNXWVFIK3NYaTVGM01pYVNBZ1BkQUk3WEh1dXFET1lvMU5PL0JoSGt1aVg2QnRtCnorNTZud2pZMy8yUytSRmNBc3JMTnIwMDJZZi9oY0IraVlDNzVWYmcydVd6WTY3TWdOTGQ5VW9RU3BDRkYrVm4KM0cyUnhybnhBb0dCQU40U3M0ZVlPU2huMVpQQjdhTUZsY0k2RHR2S2ErTGZTTXFyY2pOZjJlSEpZNnhubmxKdgpGenpGL2RiVWVTbWxSekR0WkdlcXZXaHFISy9iTjIyeWJhOU1WMDlRQ0JFTk5jNmtWajJTVHpUWkJVbEx4QzYrCk93Z0wyZHhKendWelU0VC84ajdHalRUN05BZVpFS2FvRHFyRG5BYWkyaW5oZU1JVWZHRXFGKzJyQW9HQkFOMVAKK0tZL0lsS3RWRzRKSklQNzBjUis3RmpyeXJpY05iWCtQVzUvOXFHaWxnY2grZ3l4b25BWlBpd2NpeDN3QVpGdwpaZC96ZFB2aTBkWEppc1BSZjRMazg5b2pCUmpiRmRmc2l5UmJYbyt3TFU4NUhRU2NGMnN5aUFPaTVBRHdVU0FkCm45YWFweUNweEFkREtERHdObit3ZFhtaTZ0OHRpSFRkK3RoVDhkaVpBb0dCQUt6Wis1bG9OOTBtYlF4VVh5YUwKMjFSUm9tMGJjcndsTmVCaWNFSmlzaEhYa2xpSVVxZ3hSZklNM2hhUVRUcklKZENFaHFsV01aV0xPb2I2NTNyZgo3aFlMSXM1ZUtka3o0aFRVdnpldm9TMHVXcm9CV2xOVHlGanIrSWhKZnZUc0hpOGdsU3FkbXgySkJhZUFVWUNXCndNdlQ4NmNLclNyNkQrZG8wS05FZzFsL0FvR0FlMkFVdHVFbFNqLzBmRzgrV3hHc1RFV1JqclRNUzRSUjhRWXQKeXdjdFA4aDZxTGxKUTRCWGxQU05rMXZLTmtOUkxIb2pZT2pCQTViYjhibXNVU1BlV09NNENoaFJ4QnlHbmR2eAphYkJDRkFwY0IvbEg4d1R0alVZYlN5T294ZGt5OEp0ek90ajJhS0FiZHd6NlArWDZDODhjZmxYVFo5MWpYL3RMCjF3TmRKS2tDZ1lCbyt0UzB5TzJ2SWFmK2UwSkN5TGhzVDQ5cTN3Zis2QWVqWGx2WDJ1VnRYejN5QTZnbXo5aCsKcDNlK2JMRUxwb3B0WFhNdUFRR0xhUkcrYlNNcjR5dERYbE5ZSndUeThXczNKY3dlSTdqZVp2b0ZpbmNvVlVIMwphdmxoTUVCRGYxSjltSDB5cDBwWUNaS2ROdHNvZEZtQktzVEtQMjJhTmtsVVhCS3gyZzR6cFE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo= --- #創建ConfigMap，用於匹配后端虛機使用 kind: ConfigMap apiVersion: v1 metadata: name: nginx-config namespace: nginx-ingress data:
3. 為虛擬雲主機和虛擬雲主機路由定義自定義資源，支持自定義虛擬主機和虛擬路由
執行生效命令： kubectl apply -f 2.1.custom-resource-vsr.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: virtualserverroutes.clouddev.apicloud.saas.com
spec:
group: clouddev.apicloud.saas.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
kind: VirtualServerRoute
plural: virtualserverroutes
singular: virtualserverroute
shortNames:
- vsr
preserveUnknownFields: false
validation:
openAPIV3Schema:
type: object
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
type: object
properties:
host:
type: string
subroutes:
type: array
items:
description: Route defines a route.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
errorPages:
type: array
items:
description: ErrorPage defines an ErrorPage in a Route.
type: object
properties:
codes:
type: array
items:
type: integer
redirect:
description: ErrorPageRedirect defines a redirect for an
ErrorPage.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ErrorPageReturn defines a return for an ErrorPage.
type: object
properties:
body:
type: string
code:
type: integer
headers:
type: array
items:
description: Header defines an HTTP Header.
type: object
properties:
name:
type: string
value:
type: string
type:
type: string
matches:
type: array
items:
description: Match defines a match.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an
Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
conditions:
type: array
items:
description: Condition defines a condition in a MatchRule.
type: object
properties:
argument:
type: string
cookie:
type: string
header:
type: string
value:
type: string
variable:
type: string
splits:
type: array
items:
description: Split defines a split.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect
in an Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in
an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
weight:
type: integer
path:
type: string
route:
type: string
splits:
type: array
items:
description: Split defines a split.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an
Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
weight:
type: integer
upstreams:
type: array
items:
description: Upstream defines an upstream.
type: object
properties:
buffer-size:
type: string
buffering:
type: boolean
buffers:
description: UpstreamBuffers defines Buffer Configuration for
an Upstream.
type: object
properties:
number:
type: integer
size:
type: string
client-max-body-size:
type: string
connect-timeout:
type: string
fail-timeout:
type: string
healthCheck:
description: HealthCheck defines the parameters for active Upstream
HealthChecks.
type: object
properties:
connect-timeout:
type: string
enable:
type: boolean
fails:
type: integer
headers:
type: array
items:
description: Header defines an HTTP Header.
type: object
properties:
name:
type: string
value:
type: string
interval:
type: string
jitter:
type: string
passes:
type: integer
path:
type: string
port:
type: integer
read-timeout:
type: string
send-timeout:
type: string
statusMatch:
type: string
tls:
description: UpstreamTLS defines a TLS configuration for an
Upstream.
type: object
properties:
enable:
type: boolean
keepalive:
type: integer
lb-method:
type: string
max-conns:
type: integer
max-fails:
type: integer
name:
type: string
next-upstream:
type: string
next-upstream-timeout:
type: string
next-upstream-tries:
type: integer
port:
type: integer
queue:
description: UpstreamQueue defines Queue Configuration for an
Upstream.
type: object
properties:
size:
type: integer
timeout:
type: string
read-timeout:
type: string
send-timeout:
type: string
service:
type: string
sessionCookie:
description: SessionCookie defines the parameters for session
persistence.
type: object
properties:
domain:
type: string
enable:
type: boolean
expires:
type: string
httpOnly:
type: boolean
name:
type: string
path:
type: string
secure:
type: boolean
slow-start:
type: string
subselector:
type: object
additionalProperties:
type: string
tls:
description: UpstreamTLS defines a TLS configuration for an Upstream.
type: object
properties:
enable:
type: boolean
執行生效命令： kubectl apply -f 2.2.custom-resource-vs.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: virtualservers.clouddev.apicloud.saas.com
spec:
group: clouddev.apicloud.saas.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
kind: VirtualServer
plural: virtualservers
singular: virtualserver
shortNames:
- vs
preserveUnknownFields: false
validation:
openAPIV3Schema:
description: VirtualServer defines the VirtualServer resource.
type: object
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: VirtualServerSpec is the spec of the VirtualServer resource.
type: object
properties:
host:
type: string
routes:
type: array
items:
description: Route defines a route.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
errorPages:
type: array
items:
description: ErrorPage defines an ErrorPage in a Route.
type: object
properties:
codes:
type: array
items:
type: integer
redirect:
description: ErrorPageRedirect defines a redirect for an
ErrorPage.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ErrorPageReturn defines a return for an ErrorPage.
type: object
properties:
body:
type: string
code:
type: integer
headers:
type: array
items:
description: Header defines an HTTP Header.
type: object
properties:
name:
type: string
value:
type: string
type:
type: string
matches:
type: array
items:
description: Match defines a match.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an
Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
conditions:
type: array
items:
description: Condition defines a condition in a MatchRule.
type: object
properties:
argument:
type: string
cookie:
type: string
header:
type: string
value:
type: string
variable:
type: string
splits:
type: array
items:
description: Split defines a split.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect
in an Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in
an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
weight:
type: integer
path:
type: string
route:
type: string
splits:
type: array
items:
description: Split defines a split.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an
Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
weight:
type: integer
tls:
description: TLS defines TLS configuration for a VirtualServer.
type: object
properties:
redirect:
description: TLSRedirect defines a redirect for a TLS.
type: object
properties:
basedOn:
type: string
code:
type: integer
enable:
type: boolean
secret:
type: string
upstreams:
type: array
items:
description: Upstream defines an upstream.
type: object
properties:
buffer-size:
type: string
buffering:
type: boolean
buffers:
description: UpstreamBuffers defines Buffer Configuration for
an Upstream.
type: object
properties:
number:
type: integer
size:
type: string
client-max-body-size:
type: string
connect-timeout:
type: string
fail-timeout:
type: string
healthCheck:
description: HealthCheck defines the parameters for active Upstream
HealthChecks.
type: object
properties:
connect-timeout:
type: string
enable:
type: boolean
fails:
type: integer
headers:
type: array
items:
description: Header defines an HTTP Header.
type: object
properties:
name:
type: string
value:
type: string
interval:
type: string
jitter:
type: string
passes:
type: integer
path:
type: string
port:
type: integer
read-timeout:
type: string
send-timeout:
type: string
statusMatch:
type: string
tls:
description: UpstreamTLS defines a TLS configuration for an
Upstream.
type: object
properties:
enable:
type: boolean
keepalive:
type: integer
lb-method:
type: string
max-conns:
type: integer
max-fails:
type: integer
name:
type: string
next-upstream:
type: string
next-upstream-timeout:
type: string
next-upstream-tries:
type: integer
port:
type: integer
queue:
description: UpstreamQueue defines Queue Configuration for an
Upstream.
type: object
properties:
size:
type: integer
timeout:
type: string
read-timeout:
type: string
send-timeout:
type: string
service:
type: string
sessionCookie:
description: SessionCookie defines the parameters for session
persistence.
type: object
properties:
domain:
type: string
enable:
type: boolean
expires:
type: string
httpOnly:
type: boolean
name:
type: string
path:
type: string
secure:
type: boolean
slow-start:
type: string
subselector:
type: object
additionalProperties:
type: string
tls:
description: UpstreamTLS defines a TLS configuration for an Upstream.
type: object
properties:
enable:
type: boolean
4. 配置RBAC認證授權，實現ingress控制器訪問集群中的其他資源
執行下面命令可以生效： kubectl apply -f 3.rbac.yaml
#創建集群規則 kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: nginx-ingress rules: - apiGroups: - "" resources: - services - endpoints verbs: - get - list - watch - apiGroups: - "" resources: - secrets verbs: - get - list - watch - apiGroups: - "" resources: - configmaps verbs: - get - list - watch - update - create - apiGroups: - "" resources: - pods verbs: - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - extensions resources: - ingresses verbs: - list - watch - get - apiGroups: - "extensions" resources: - ingresses/status verbs: - update - apiGroups: - clouddev.apicloud.saas.com resources: - virtualservers - virtualserverroutes verbs: - list - watch - get --- #綁定認證規則 kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: nginx-ingress subjects: - kind: ServiceAccount name: nginx-ingress namespace: nginx-ingress roleRef: kind: ClusterRole name: nginx-ingress apiGroup: rbac.authorization.k8s.io
2.2.2 部署Ingress控制器
1、 部署控制器，以DaemonSets的形式部署
執行下面命令可實現創建 kubectl apply -f 4.nginx-ingress.yaml
#創建Nginx Ingress Controller控制器,使用DaemonSet模式將使用nginx部署監聽在每個node節點，用於四層、七層代理后端的service
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
#annotations:
#prometheus.io/scrape: "true"
#prometheus.io/port: "9113"
spec:
serviceAccountName: nginx-ingress
containers:
- image: nginx/nginx-ingress:edge
imagePullPolicy: IfNotPresent #kubectl explain DaemonSet.spec.template.spec.containers查看該參數Always, Never, IfNotPresent. Defaults to Always
name: nginx-ingress
ports:
- name: http
containerPort: 80
hostPort: 80
- name: https
containerPort: 443
hostPort: 443
#- name: prometheus #開發自定義端口轉發服務
#containerPort: 9113
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/clouddev.apicloud.saas.com-server-secret #指定默認的https證書，如果是默認創建的證書可以不修改，否則會報證書找不到的錯誤
#- -v=3 # Enables extensive logging. Useful for troubleshooting.
#- -report-ingress-status
#- -external-service=nginx-ingress
#- -enable-leader-election
#- -enable-prometheus-metrics
2. 我們以DaemonSets的方式部署，DaemonSet部署集群中各個節點都是對等
shell>kubectl get daemonsets -n nginx-ingress NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE nginx-ingress 3 3 3 3 3 <none> 15s shell> kubectl get pods -n nginx-ingress -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ingress-7mpfc 1/1 Running 0 2m44s 10.244.0.50 node-1 <none> <none> nginx-ingress-l2rtj 1/1 Running 0 2m44s 10.244.1.144 node-2 <none> <none> nginx-ingress-tgf6r 1/1 Running 0 2m44s 10.244.2.160 node-3 <none> <none>
3、校驗Nginx Ingress安裝情況，此時三個節點均是對等，即訪問任意一個節點均能實現相同的效果。
從外部訪問，我們這里采用的是阿里雲的SLB，四層負載到3個node節點的80/443端口，泛域名解析到SLB公網IP。
3. Ingress資源定義
上面的章節已安裝了一個Nginx Ingress Controller控制器，有了Ingress控制器后，我們就可以定義Ingress資源來實現七層負載轉發了，大體上Ingress支持三種使用方式：
1. 基於虛擬主機轉發
2. 基於虛擬機主機URI轉發
3. 支持TLS加密轉發
3.1 Ingress定義 （這里以配置nginx demo為例進行配置）
待完善
3.3 線上測試環境yaml文件
經過多方面的資料整合、學習、測試，終於完成大部分的yaml文件編寫。
該文件實現了deployment創建無狀態資源用於運行后端service，通過容器內部與node節點宿主機本地目錄做一一映射（多個node節點采用共享存儲的方式，共享文件路徑）；
實現service自動匹配pod資源；
實現自動注冊到nginx實現域名虛機訪問；
實現https支持；
待實現：HPA自動伸縮功能 （默認副本為0，監測cpu增加后彈性伸縮pod副本數）
待實現：容器資源限制（目前限制100M內存和cpu運行不成功）
具體yaml文件見下面：
apiVersion: apps/v1
kind: Deployment
metadata:
name: nodejs-a10000000002-deployment #Deployment名稱
spec:
replicas: 1 #目標副本數量
minReadySeconds: 10 # 最小准備時間,這里需要估一個比較合理的值，從容器啟動到應用正常提供服務
strategy:
rollingUpdate:
maxSurge: 1 #滾動升級時最大同時升級1個pod
maxUnavailable: 1 #滾動升級時最大允許不可用的pod個數
selector:
matchLabels:
app: nodejs-a10000000002 #匹配模板名稱
template:
metadata:
labels:
app: nodejs-a10000000002 #模板名稱
spec: #定義容器模板，該模板可以包含多個容器
containers:
- name: nodejs-a10000000002 #containers名稱
image: node:latest #容器啟動鏡像版本
#args: [""] #給ENTRYPOINT命令的傳參
imagePullPolicy: IfNotPresent #kubectl explain DaemonSet.spec.template.spec.containers查看該參數Always, Never, IfNotPresent. Defaults to Always
ports:
- containerPort: 3000 #容器內監聽端口
protocol: TCP
#resources: #容器內資源限制
# requests:
# cpu: 100m
# memory: 30Mi
# limits:
# cpu: 100m
# memory: 30Mi
#terminationMessagePath: /dev/termination-log
#terminationMessagePolicy: File
volumeMounts: #容器內掛載點
- name: nodejs-code #必須有名稱
mountPath: /root
- name: nodejs-nodemodules
mountPath: /root/node_modules
workingDir: /root
command: ["/usr/local/bin/node","server/server.js"]
#command: ["bash","-c","while true;do date;sleep 1;done"]
#nodeSelector: #節點選擇器
# type: volume-data #節點的label,將根據這個label去選擇節點
volumes:
- name: nodejs-code #跟上面的名稱對應
hostPath:
path: /home/clouddev/data/A10000000002 #宿主機掛載點
- name: nodejs-nodemodules #跟上面的名稱對應
hostPath:
path: /home/clouddev/node_modules #宿主機掛載點
#dnsPolicy: ClusterFirst
#restartPolicy: Always
#schedulerName: default-scheduler
#securityContext: {}
#terminationGracePeriodSeconds: 30
---
#創建service資源
apiVersion: v1
kind: Service
metadata:
labels:
run: nodejs-a10000000002-service
name: nodejs-a10000000002-service
namespace: default
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 3000
#- name: https
# port: 443
# protocol: TCP
# targetPort: 3000
selector:
app: nodejs-a10000000002 #匹配帶有app=nodejs-a10000000002標簽的pod資源
sessionAffinity: None
type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: a10000000002.clouddev.apicloud-saas.com #用於注冊到nginx負載系統，保持唯一
# namespace: service
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
spec:
rules:
- host: a10000000002.clouddev.apicloud-saas.com #指定該域名流量到指定的service，然后由service負載調度流量到各個pod
http:
paths:
- path: /
backend:
serviceName: nodejs-a10000000002-service #service名稱
servicePort: 80 #service監聽端口
tls:
- hosts:
- a10000000002.clouddev.apicloud-saas.com
secretName: nginx-test-secret #創建命令kubectl create secret tls nginx-test-secret --cert=1_ngrok.apicloud-saas.com_bundle.crt --key=2_ngrok.apicloud-saas.com.key
4. 問題總結
1. HPA獲取不到cpu參數，需要部署Message-server，但是通過原生安裝的k8s，手動配置Message-server一直有問題，待后解。
官網推薦 Message-server部署用的項目路徑，但是始終創建不成功 https://github.com/kubernetes-sigs/metrics-server/tree/master/deploy/kubernetes
網上看到的鏈接：
http://blog.ljmict.com/?p=98 部署METRICS-SERVER時遇到的問題
https://blog.51cto.com/13740724/2368066?source=dra 基於metrics-server的HPA
https://cloud.tencent.com/developer/article/1492864 Kubernetes1.15.1的Pod 自動擴縮容（23）
https://zhuanlan.zhihu.com/p/34722886 Kubernetes 集群狀態異常排錯
https://www.jianshu.com/p/c1866c0a98cb Kubernetes metrics-server 組件的安裝配置
https://blog.51cto.com/14320361/2474234 k8s的HPA自動擴容與縮容