写在前面：
公司简单的业务场景：
1. 基于域名进行访问后端不同的容器服务
2. 后端服务需要根据负载情况进行动态伸缩
3. 暴露创建和删除服务的接口
1. Ingress简介
引用官方关于ingress的介绍我们可以得知，ingress是一种通过http协议暴露kubernetes内部服务的api对象，即充当Edge Router边界路由器的角色对外基于七层的负载均衡调度机制，能够提供以下几个功能：
- 负载均衡，将请求自动负载均衡到后端的Pod上；
- SSL加密，客户端到Ingress Controller为https加密，到后端Pod为明文的http；
- 基于名称的虚拟主机，提供基于域名或URI更灵活的路由方式
实现Ingress包含的组件有：
- Ingress，客户端，负责定义ingress配置，将请求转发给Ingress Controller；
- Ingress Controller，Ingress控制器，实现七层转发的Edge Router，通过调用k8s的api动态感知集群中Pod的变化而动态更新配置文件并重载， Controller需要部署在k8s集群中以实现和集群中的pod通信，通常以DaemonSets或Deployments的形式部署，并对外暴露80和443端口，对于DaemonSets来说，一般是以hostNetwork或者hostPort的形式暴露，Deployments则以NodePort的方式暴露，控制器的多个节点则借助外部负载均衡ExternalLB以实现统一接入；
- Ingress配置规则，Controller控制器通过service服务发现机制动态实现后端Pod路由转发规则的实现；
- Service，kuberntes中四层的负载均衡调度机制，Ingress借助service的服务发现机制实现集群中Pod资源的动态感知；
- Pod，后端实际负责响应请求容器，由控制器如Deployment创建，通过标签Labels和service关联，服务发现。
简而言之，ingress控制器借助service的服务发现机制实现配置的动态更新以实现Pod的负载均衡机制实现，由于涉及到Ingress Controller的动态更新，目前社区Ingress Controller大体包含两种类型的控制器：
- 传统的七层负载均衡如Nginx，HAproxy，开发了适应微服务应用的插件，具有成熟，高性能等优点；
- 新型微服务负载均衡如Traefik，Envoy，Istio，专门适用于微服务+容器化应用场景，具有动态更新特点；
| 类型 | 常见类型 | 优点 | 缺点 |
|---|---|---|---|
| 传统负载均衡 | nginx，haproxy | 成熟，稳定，高性能 | 动态更新需reload配置文件 |
| 微服务负载均衡 | Traefik，Envoy，Istio | 天生为微服务而生，动态更新 | 性能还有待提升 |
2. Nginx Ingress
2.1 Nginx ingress介绍
Nginx Ingress Controller是实现ingress的具体实现，包含有两个版本：Ngnix OSS和Nginx Plus版，后者是商业化增强版，支持更多的功能，详情参考官方文档介绍https://www.nginx.com/products/nginx/kubernetes-ingress-controller#compare-versions
2.2 Nginx ingress安装
首先需要安装Nginx Ingress Controller控制器，控制器安装方式包含两种：DaemonSets和Deployments（我们这里采用DaemonSets）。
- DaemonSets通过hostPort的方式暴露80和443端口，可通过Node的调度由专门的节点实现部署；用于确保k8s集群每个work节点上拥有唯一一个pod节点，支持work节点动态增加而动态创建对应的pod节点。
- Deployments则通过NodePort的方式实现控制器端口的暴露，借助外部负载均衡实现高可用负载均衡；管理无状态应用（关注的是群体性行为）的pod控制器/pod内应用守护进程运行的
除此之外，还需要部署Namespace，ServiceAccount，RBAC，Secrets，Custom Resource Definitions等资源，如下开始部署。
2.2.1 基础依赖环境准备
1、github中下载源码包,安装部署文件在kubernetes-ingress/deployments/目录下
shell>git clone https://github.com/nginxinc/kubernetes-ingress.git
shell>tree kubernetes-ingress/deployments/
kubernetes-ingress/deployments/
├── common
│ ├── default-server-secret.yaml
│ ├── nginx-config.yaml
│ ├── ns-and-sa.yaml
│ ├── vs-definition.yaml
│ └── vsr-definition.yaml
├── daemon-set
│ ├── nginx-ingress.yaml
│ └── nginx-plus-ingress.yaml
├── deployment
│ ├── nginx-ingress.yaml
│ └── nginx-plus-ingress.yaml
├── helm-chart
│ ├── chart-icon.png
│ ├── Chart.yaml
│ ├── README.md
│ ├── templates
│ │ ├── controller-configmap.yaml
│ │ ├── controller-daemonset.yaml
│ │ ├── controller-deployment.yaml
│ │ ├── controller-leader-election-configmap.yaml
│ │ ├── controller-secret.yaml
│ │ ├── controller-serviceaccount.yaml
│ │ ├── controller-service.yaml
│ │ ├── controller-vs-definition.yaml
│ │ ├── controller-vsr-definition.yaml
│ │ ├── controller-wildcard-secret.yaml
│ │ ├── _helpers.tpl
│ │ ├── NOTES.txt
│ │ └── rbac.yaml
│ ├── values-icp.yaml
│ ├── values-plus.yaml
│ └── values.yaml
├── rbac
│ └── rbac.yaml
├── README.md
└── service
├── loadbalancer-aws-elb.yaml
├── loadbalancer.yaml
└── nodeport.yaml
备注：这里作者对各个文件进行了整合，最终用5个配置文件实现场景配置
2、创建Namespace、ServiceAccount、Secret（证书为默认证书，可自定义修改）、ConfigMap
执行生效命令： kubectl apply -f 1.ns-sa-secret.yaml
#创建nginx-ingress命名空间 apiVersion: v1 kind: Namespace metadata: name: nginx-ingress --- #创建serviceAccount apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress namespace: nginx-ingress --- #创建https证书 apiVersion: v1 kind: Secret metadata: name: clouddev.apicloud.saas.com-server-secret namespace: nginx-ingress type: Opaque data: tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN2akNDQWFZQ0NRREFPRjl0THNhWFhEQU5CZ2txaGtpRzl3MEJBUXNGQURBaE1SOHdIUVlEVlFRRERCWk8KUjBsT1dFbHVaM0psYzNORGIyNTBjbTlzYkdWeU1CNFhEVEU0TURreE1qRTRNRE16TlZvWERUSXpNRGt4TVRFNApNRE16TlZvd0lURWZNQjBHQTFVRUF3d1dUa2RKVGxoSmJtZHlaWE56UTI5dWRISnZiR3hsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUwvN2hIUEtFWGRMdjNyaUM3QlBrMTNpWkt5eTlyQ08KR2xZUXYyK2EzUDF0azIrS3YwVGF5aGRCbDRrcnNUcTZzZm8vWUk1Y2Vhbkw4WGM3U1pyQkVRYm9EN2REbWs1Qgo4eDZLS2xHWU5IWlg0Rm5UZ0VPaStlM2ptTFFxRlBSY1kzVnNPazFFeUZBL0JnWlJVbkNHZUtGeERSN0tQdGhyCmtqSXVuektURXUyaDU4Tlp0S21ScUJHdDEwcTNRYzhZT3ExM2FnbmovUWRjc0ZYYTJnMjB1K1lYZDdoZ3krZksKWk4vVUkxQUQ0YzZyM1lma1ZWUmVHd1lxQVp1WXN2V0RKbW1GNWRwdEMzN011cDBPRUxVTExSakZJOTZXNXIwSAo1TmdPc25NWFJNV1hYVlpiNWRxT3R0SmRtS3FhZ25TZ1JQQVpQN2MwQjFQU2FqYzZjNGZRVXpNQ0F3RUFBVEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQWpLb2tRdGRPcEsrTzhibWVPc3lySmdJSXJycVFVY2ZOUitjb0hZVUoKdGhrYnhITFMzR3VBTWI5dm15VExPY2xxeC9aYzJPblEwMEJCLzlTb0swcitFZ1U2UlVrRWtWcitTTFA3NTdUWgozZWI4dmdPdEduMS9ienM3bzNBaS9kclkrcUI5Q2k1S3lPc3FHTG1US2xFaUtOYkcyR1ZyTWxjS0ZYQU80YTY3Cklnc1hzYktNbTQwV1U3cG9mcGltU1ZmaXFSdkV5YmN3N0NYODF6cFErUyt1eHRYK2VBZ3V0NHh3VlI5d2IyVXYKelhuZk9HbWhWNThDd1dIQnNKa0kxNXhaa2VUWXdSN0diaEFMSkZUUkk3dkhvQXprTWIzbjAxQjQyWjNrN3RXNQpJUDFmTlpIOFUvOWxiUHNoT21FRFZkdjF5ZytVRVJxbStGSis2R0oxeFJGcGZnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdi91RWM4b1JkMHUvZXVJTHNFK1RYZUprckxMMnNJNGFWaEMvYjVyYy9XMlRiNHEvClJOcktGMEdYaVN1eE9ycXgrajlnamx4NXFjdnhkenRKbXNFUkJ1Z1B0ME9hVGtIekhvb3FVWmcwZGxmZ1dkT0EKUTZMNTdlT1l0Q29VOUZ4amRXdzZUVVRJVUQ4R0JsRlNjSVo0b1hFTkhzbysyR3VTTWk2Zk1wTVM3YUhudzFtMApxWkdvRWEzWFNyZEJ6eGc2clhkcUNlUDlCMXl3VmRyYURiUzc1aGQzdUdETDU4cGszOVFqVUFQaHpxdmRoK1JWClZGNGJCaW9CbTVpeTlZTW1hWVhsMm0wTGZzeTZuUTRRdFFzdEdNVWozcGJtdlFmazJBNnljeGRFeFpkZFZsdmwKMm82MjBsMllxcHFDZEtCRThCay90elFIVTlKcU56cHpoOUJUTXdJREFRQUJBb0lCQVFDZklHbXowOHhRVmorNwpLZnZJUXQwQ0YzR2MxNld6eDhVNml4MHg4Mm15d1kxUUNlL3BzWE9LZlRxT1h1SENyUlp5TnUvZ2IvUUQ4bUFOCmxOMjRZTWl0TWRJODg5TEZoTkp3QU5OODJDeTczckM5bzVvUDlkazAvYzRIbjAzSkVYNzZ5QjgzQm9rR1FvYksKMjhMNk0rdHUzUmFqNjd6Vmc2d2szaEhrU0pXSzBwV1YrSjdrUkRWYmhDYUZhNk5nMUZNRWxhTlozVDhhUUtyQgpDUDNDeEFTdjYxWTk5TEI4KzNXWVFIK3NYaTVGM01pYVNBZ1BkQUk3WEh1dXFET1lvMU5PL0JoSGt1aVg2QnRtCnorNTZud2pZMy8yUytSRmNBc3JMTnIwMDJZZi9oY0IraVlDNzVWYmcydVd6WTY3TWdOTGQ5VW9RU3BDRkYrVm4KM0cyUnhybnhBb0dCQU40U3M0ZVlPU2huMVpQQjdhTUZsY0k2RHR2S2ErTGZTTXFyY2pOZjJlSEpZNnhubmxKdgpGenpGL2RiVWVTbWxSekR0WkdlcXZXaHFISy9iTjIyeWJhOU1WMDlRQ0JFTk5jNmtWajJTVHpUWkJVbEx4QzYrCk93Z0wyZHhKendWelU0VC84ajdHalRUN05BZVpFS2FvRHFyRG5BYWkyaW5oZU1JVWZHRXFGKzJyQW9HQkFOMVAKK0tZL0lsS3RWRzRKSklQNzBjUis3RmpyeXJpY05iWCtQVzUvOXFHaWxnY2grZ3l4b25BWlBpd2NpeDN3QVpGdwpaZC96ZFB2aTBkWEppc1BSZjRMazg5b2pCUmpiRmRmc2l5UmJYbyt3TFU4NUhRU2NGMnN5aUFPaTVBRHdVU0FkCm45YWFweUNweEFkREtERHdObit3ZFhtaTZ0OHRpSFRkK3RoVDhkaVpBb0dCQUt6Wis1bG9OOTBtYlF4VVh5YUwKMjFSUm9tMGJjcndsTmVCaWNFSmlzaEhYa2xpSVVxZ3hSZklNM2hhUVRUcklKZENFaHFsV01aV0xPb2I2NTNyZgo3aFlMSXM1ZUtka3o0aFRVdnpldm9TMHVXcm9CV2xOVHlGanIrSWhKZnZUc0hpOGdsU3FkbXgySkJhZUFVWUNXCndNdlQ4NmNLclNyNkQrZG8wS05FZzFsL0FvR0FlMkFVdHVFbFNqLzBmRzgrV3hHc1RFV1JqclRNUzRSUjhRWXQKeXdjdFA4aDZxTGxKUTRCWGxQU05rMXZLTmtOUkxIb2pZT2pCQTViYjhibXNVU1BlV09NNENoaFJ4QnlHbmR2eAphYkJDRkFwY0IvbEg4d1R0alVZYlN5T294ZGt5OEp0ek90ajJhS0FiZHd6NlArWDZDODhjZmxYVFo5MWpYL3RMCjF3TmRKS2tDZ1lCbyt0UzB5TzJ2SWFmK2UwSkN5TGhzVDQ5cTN3Zis2QWVqWGx2WDJ1VnRYejN5QTZnbXo5aCsKcDNlK2JMRUxwb3B0WFhNdUFRR0xhUkcrYlNNcjR5dERYbE5ZSndUeThXczNKY3dlSTdqZVp2b0ZpbmNvVlVIMwphdmxoTUVCRGYxSjltSDB5cDBwWUNaS2ROdHNvZEZtQktzVEtQMjJhTmtsVVhCS3gyZzR6cFE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo= --- #创建ConfigMap，用于匹配后端虚机使用 kind: ConfigMap apiVersion: v1 metadata: name: nginx-config namespace: nginx-ingress data:
3. 为虚拟云主机和虚拟云主机路由定义自定义资源，支持自定义虚拟主机和虚拟路由
执行生效命令： kubectl apply -f 2.1.custom-resource-vsr.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: virtualserverroutes.clouddev.apicloud.saas.com
spec:
group: clouddev.apicloud.saas.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
kind: VirtualServerRoute
plural: virtualserverroutes
singular: virtualserverroute
shortNames:
- vsr
preserveUnknownFields: false
validation:
openAPIV3Schema:
type: object
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
type: object
properties:
host:
type: string
subroutes:
type: array
items:
description: Route defines a route.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
errorPages:
type: array
items:
description: ErrorPage defines an ErrorPage in a Route.
type: object
properties:
codes:
type: array
items:
type: integer
redirect:
description: ErrorPageRedirect defines a redirect for an
ErrorPage.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ErrorPageReturn defines a return for an ErrorPage.
type: object
properties:
body:
type: string
code:
type: integer
headers:
type: array
items:
description: Header defines an HTTP Header.
type: object
properties:
name:
type: string
value:
type: string
type:
type: string
matches:
type: array
items:
description: Match defines a match.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an
Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
conditions:
type: array
items:
description: Condition defines a condition in a MatchRule.
type: object
properties:
argument:
type: string
cookie:
type: string
header:
type: string
value:
type: string
variable:
type: string
splits:
type: array
items:
description: Split defines a split.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect
in an Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in
an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
weight:
type: integer
path:
type: string
route:
type: string
splits:
type: array
items:
description: Split defines a split.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an
Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
weight:
type: integer
upstreams:
type: array
items:
description: Upstream defines an upstream.
type: object
properties:
buffer-size:
type: string
buffering:
type: boolean
buffers:
description: UpstreamBuffers defines Buffer Configuration for
an Upstream.
type: object
properties:
number:
type: integer
size:
type: string
client-max-body-size:
type: string
connect-timeout:
type: string
fail-timeout:
type: string
healthCheck:
description: HealthCheck defines the parameters for active Upstream
HealthChecks.
type: object
properties:
connect-timeout:
type: string
enable:
type: boolean
fails:
type: integer
headers:
type: array
items:
description: Header defines an HTTP Header.
type: object
properties:
name:
type: string
value:
type: string
interval:
type: string
jitter:
type: string
passes:
type: integer
path:
type: string
port:
type: integer
read-timeout:
type: string
send-timeout:
type: string
statusMatch:
type: string
tls:
description: UpstreamTLS defines a TLS configuration for an
Upstream.
type: object
properties:
enable:
type: boolean
keepalive:
type: integer
lb-method:
type: string
max-conns:
type: integer
max-fails:
type: integer
name:
type: string
next-upstream:
type: string
next-upstream-timeout:
type: string
next-upstream-tries:
type: integer
port:
type: integer
queue:
description: UpstreamQueue defines Queue Configuration for an
Upstream.
type: object
properties:
size:
type: integer
timeout:
type: string
read-timeout:
type: string
send-timeout:
type: string
service:
type: string
sessionCookie:
description: SessionCookie defines the parameters for session
persistence.
type: object
properties:
domain:
type: string
enable:
type: boolean
expires:
type: string
httpOnly:
type: boolean
name:
type: string
path:
type: string
secure:
type: boolean
slow-start:
type: string
subselector:
type: object
additionalProperties:
type: string
tls:
description: UpstreamTLS defines a TLS configuration for an Upstream.
type: object
properties:
enable:
type: boolean
执行生效命令： kubectl apply -f 2.2.custom-resource-vs.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: virtualservers.clouddev.apicloud.saas.com
spec:
group: clouddev.apicloud.saas.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
kind: VirtualServer
plural: virtualservers
singular: virtualserver
shortNames:
- vs
preserveUnknownFields: false
validation:
openAPIV3Schema:
description: VirtualServer defines the VirtualServer resource.
type: object
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: VirtualServerSpec is the spec of the VirtualServer resource.
type: object
properties:
host:
type: string
routes:
type: array
items:
description: Route defines a route.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
errorPages:
type: array
items:
description: ErrorPage defines an ErrorPage in a Route.
type: object
properties:
codes:
type: array
items:
type: integer
redirect:
description: ErrorPageRedirect defines a redirect for an
ErrorPage.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ErrorPageReturn defines a return for an ErrorPage.
type: object
properties:
body:
type: string
code:
type: integer
headers:
type: array
items:
description: Header defines an HTTP Header.
type: object
properties:
name:
type: string
value:
type: string
type:
type: string
matches:
type: array
items:
description: Match defines a match.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an
Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
conditions:
type: array
items:
description: Condition defines a condition in a MatchRule.
type: object
properties:
argument:
type: string
cookie:
type: string
header:
type: string
value:
type: string
variable:
type: string
splits:
type: array
items:
description: Split defines a split.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect
in an Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in
an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
weight:
type: integer
path:
type: string
route:
type: string
splits:
type: array
items:
description: Split defines a split.
type: object
properties:
action:
description: Action defines an action.
type: object
properties:
pass:
type: string
redirect:
description: ActionRedirect defines a redirect in an
Action.
type: object
properties:
code:
type: integer
url:
type: string
return:
description: ActionReturn defines a return in an Action.
type: object
properties:
body:
type: string
code:
type: integer
type:
type: string
weight:
type: integer
tls:
description: TLS defines TLS configuration for a VirtualServer.
type: object
properties:
redirect:
description: TLSRedirect defines a redirect for a TLS.
type: object
properties:
basedOn:
type: string
code:
type: integer
enable:
type: boolean
secret:
type: string
upstreams:
type: array
items:
description: Upstream defines an upstream.
type: object
properties:
buffer-size:
type: string
buffering:
type: boolean
buffers:
description: UpstreamBuffers defines Buffer Configuration for
an Upstream.
type: object
properties:
number:
type: integer
size:
type: string
client-max-body-size:
type: string
connect-timeout:
type: string
fail-timeout:
type: string
healthCheck:
description: HealthCheck defines the parameters for active Upstream
HealthChecks.
type: object
properties:
connect-timeout:
type: string
enable:
type: boolean
fails:
type: integer
headers:
type: array
items:
description: Header defines an HTTP Header.
type: object
properties:
name:
type: string
value:
type: string
interval:
type: string
jitter:
type: string
passes:
type: integer
path:
type: string
port:
type: integer
read-timeout:
type: string
send-timeout:
type: string
statusMatch:
type: string
tls:
description: UpstreamTLS defines a TLS configuration for an
Upstream.
type: object
properties:
enable:
type: boolean
keepalive:
type: integer
lb-method:
type: string
max-conns:
type: integer
max-fails:
type: integer
name:
type: string
next-upstream:
type: string
next-upstream-timeout:
type: string
next-upstream-tries:
type: integer
port:
type: integer
queue:
description: UpstreamQueue defines Queue Configuration for an
Upstream.
type: object
properties:
size:
type: integer
timeout:
type: string
read-timeout:
type: string
send-timeout:
type: string
service:
type: string
sessionCookie:
description: SessionCookie defines the parameters for session
persistence.
type: object
properties:
domain:
type: string
enable:
type: boolean
expires:
type: string
httpOnly:
type: boolean
name:
type: string
path:
type: string
secure:
type: boolean
slow-start:
type: string
subselector:
type: object
additionalProperties:
type: string
tls:
description: UpstreamTLS defines a TLS configuration for an Upstream.
type: object
properties:
enable:
type: boolean
4. 配置RBAC认证授权，实现ingress控制器访问集群中的其他资源
执行下面命令可以生效： kubectl apply -f 3.rbac.yaml
#创建集群规则 kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: nginx-ingress rules: - apiGroups: - "" resources: - services - endpoints verbs: - get - list - watch - apiGroups: - "" resources: - secrets verbs: - get - list - watch - apiGroups: - "" resources: - configmaps verbs: - get - list - watch - update - create - apiGroups: - "" resources: - pods verbs: - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - extensions resources: - ingresses verbs: - list - watch - get - apiGroups: - "extensions" resources: - ingresses/status verbs: - update - apiGroups: - clouddev.apicloud.saas.com resources: - virtualservers - virtualserverroutes verbs: - list - watch - get --- #绑定认证规则 kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: nginx-ingress subjects: - kind: ServiceAccount name: nginx-ingress namespace: nginx-ingress roleRef: kind: ClusterRole name: nginx-ingress apiGroup: rbac.authorization.k8s.io
2.2.2 部署Ingress控制器
1、 部署控制器，以DaemonSets的形式部署
执行下面命令可实现创建 kubectl apply -f 4.nginx-ingress.yaml
#创建Nginx Ingress Controller控制器,使用DaemonSet模式将使用nginx部署监听在每个node节点，用于四层、七层代理后端的service
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
#annotations:
#prometheus.io/scrape: "true"
#prometheus.io/port: "9113"
spec:
serviceAccountName: nginx-ingress
containers:
- image: nginx/nginx-ingress:edge
imagePullPolicy: IfNotPresent #kubectl explain DaemonSet.spec.template.spec.containers查看该参数Always, Never, IfNotPresent. Defaults to Always
name: nginx-ingress
ports:
- name: http
containerPort: 80
hostPort: 80
- name: https
containerPort: 443
hostPort: 443
#- name: prometheus #开发自定义端口转发服务
#containerPort: 9113
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/clouddev.apicloud.saas.com-server-secret #指定默认的https证书，如果是默认创建的证书可以不修改，否则会报证书找不到的错误
#- -v=3 # Enables extensive logging. Useful for troubleshooting.
#- -report-ingress-status
#- -external-service=nginx-ingress
#- -enable-leader-election
#- -enable-prometheus-metrics
2. 我们以DaemonSets的方式部署，DaemonSet部署集群中各个节点都是对等
shell>kubectl get daemonsets -n nginx-ingress NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE nginx-ingress 3 3 3 3 3 <none> 15s shell> kubectl get pods -n nginx-ingress -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ingress-7mpfc 1/1 Running 0 2m44s 10.244.0.50 node-1 <none> <none> nginx-ingress-l2rtj 1/1 Running 0 2m44s 10.244.1.144 node-2 <none> <none> nginx-ingress-tgf6r 1/1 Running 0 2m44s 10.244.2.160 node-3 <none> <none>
3、校验Nginx Ingress安装情况，此时三个节点均是对等，即访问任意一个节点均能实现相同的效果。
从外部访问，我们这里采用的是阿里云的SLB，四层负载到3个node节点的80/443端口，泛域名解析到SLB公网IP。
3. Ingress资源定义
上面的章节已安装了一个Nginx Ingress Controller控制器，有了Ingress控制器后，我们就可以定义Ingress资源来实现七层负载转发了，大体上Ingress支持三种使用方式：
1. 基于虚拟主机转发
2. 基于虚拟机主机URI转发
3. 支持TLS加密转发
3.1 Ingress定义 （这里以配置nginx demo为例进行配置）
待完善
3.3 线上测试环境yaml文件
经过多方面的资料整合、学习、测试，终于完成大部分的yaml文件编写。
该文件实现了deployment创建无状态资源用于运行后端service，通过容器内部与node节点宿主机本地目录做一一映射（多个node节点采用共享存储的方式，共享文件路径）；
实现service自动匹配pod资源；
实现自动注册到nginx实现域名虚机访问；
实现https支持；
待实现：HPA自动伸缩功能 （默认副本为0，监测cpu增加后弹性伸缩pod副本数）
待实现：容器资源限制（目前限制100M内存和cpu运行不成功）
具体yaml文件见下面：
apiVersion: apps/v1
kind: Deployment
metadata:
name: nodejs-a10000000002-deployment #Deployment名称
spec:
replicas: 1 #目标副本数量
minReadySeconds: 10 # 最小准备时间,这里需要估一个比较合理的值，从容器启动到应用正常提供服务
strategy:
rollingUpdate:
maxSurge: 1 #滚动升级时最大同时升级1个pod
maxUnavailable: 1 #滚动升级时最大允许不可用的pod个数
selector:
matchLabels:
app: nodejs-a10000000002 #匹配模板名称
template:
metadata:
labels:
app: nodejs-a10000000002 #模板名称
spec: #定义容器模板，该模板可以包含多个容器
containers:
- name: nodejs-a10000000002 #containers名称
image: node:latest #容器启动镜像版本
#args: [""] #给ENTRYPOINT命令的传参
imagePullPolicy: IfNotPresent #kubectl explain DaemonSet.spec.template.spec.containers查看该参数Always, Never, IfNotPresent. Defaults to Always
ports:
- containerPort: 3000 #容器内监听端口
protocol: TCP
#resources: #容器内资源限制
# requests:
# cpu: 100m
# memory: 30Mi
# limits:
# cpu: 100m
# memory: 30Mi
#terminationMessagePath: /dev/termination-log
#terminationMessagePolicy: File
volumeMounts: #容器内挂载点
- name: nodejs-code #必须有名称
mountPath: /root
- name: nodejs-nodemodules
mountPath: /root/node_modules
workingDir: /root
command: ["/usr/local/bin/node","server/server.js"]
#command: ["bash","-c","while true;do date;sleep 1;done"]
#nodeSelector: #节点选择器
# type: volume-data #节点的label,将根据这个label去选择节点
volumes:
- name: nodejs-code #跟上面的名称对应
hostPath:
path: /home/clouddev/data/A10000000002 #宿主机挂载点
- name: nodejs-nodemodules #跟上面的名称对应
hostPath:
path: /home/clouddev/node_modules #宿主机挂载点
#dnsPolicy: ClusterFirst
#restartPolicy: Always
#schedulerName: default-scheduler
#securityContext: {}
#terminationGracePeriodSeconds: 30
---
#创建service资源
apiVersion: v1
kind: Service
metadata:
labels:
run: nodejs-a10000000002-service
name: nodejs-a10000000002-service
namespace: default
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 3000
#- name: https
# port: 443
# protocol: TCP
# targetPort: 3000
selector:
app: nodejs-a10000000002 #匹配带有app=nodejs-a10000000002标签的pod资源
sessionAffinity: None
type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: a10000000002.clouddev.apicloud-saas.com #用于注册到nginx负载系统，保持唯一
# namespace: service
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
spec:
rules:
- host: a10000000002.clouddev.apicloud-saas.com #指定该域名流量到指定的service，然后由service负载调度流量到各个pod
http:
paths:
- path: /
backend:
serviceName: nodejs-a10000000002-service #service名称
servicePort: 80 #service监听端口
tls:
- hosts:
- a10000000002.clouddev.apicloud-saas.com
secretName: nginx-test-secret #创建命令kubectl create secret tls nginx-test-secret --cert=1_ngrok.apicloud-saas.com_bundle.crt --key=2_ngrok.apicloud-saas.com.key
4. 问题总结
1. HPA获取不到cpu参数，需要部署Message-server，但是通过原生安装的k8s，手动配置Message-server一直有问题，待后解。
官网推荐 Message-server部署用的项目路径，但是始终创建不成功 https://github.com/kubernetes-sigs/metrics-server/tree/master/deploy/kubernetes
网上看到的链接：
http://blog.ljmict.com/?p=98 部署METRICS-SERVER时遇到的问题
https://blog.51cto.com/13740724/2368066?source=dra 基于metrics-server的HPA
https://cloud.tencent.com/developer/article/1492864 Kubernetes1.15.1的Pod 自动扩缩容（23）
https://zhuanlan.zhihu.com/p/34722886 Kubernetes 集群状态异常排错
https://www.jianshu.com/p/c1866c0a98cb Kubernetes metrics-server 组件的安装配置
https://blog.51cto.com/14320361/2474234 k8s的HPA自动扩容与缩容