1 Ingress 簡介
1.1 kubernetes訪問方式
在Kubernetes中，服務和Pod的IP地址僅可以在集群網絡內部使用，對於集群外的應用是不可見的。為了使外部的應用能夠訪問集群內的服務，在Kubernetes 目前 提供了以下幾種方案：
NodePort
LoadBalancer
Ingress
1.2 Ingress 組成
ingress controller
將新加入的Ingress轉化成Nginx的配置文件並使之生效
ingress服務
將Nginx的配置抽象成一個Ingress對象，每添加一個新的服務只需寫一個新的Ingress的yaml文件即可
1.3 Ingress 工作原理
1.ingress controller通過和kubernetes api交互，動態的去感知集群中ingress規則變化，
2.然后讀取它，按照自定義的規則，規則就是寫明了哪個域名對應哪個service，生成一段nginx配置，
3.再寫到nginx-ingress-control的pod里，這個Ingress controller的pod里運行着一個Nginx服務，控制器會把生成的nginx配置寫入/etc/nginx.conf文件中，
4.然后reload一下使配置生效。以此達到域名分配置和動態更新的問題。
1.4 Ingress可以解決什么問題
1.動態配置服務
如果按照傳統方式, 當新增加一個服務時, 我們可能需要在流量入口加一個反向代理指向我們新的k8s服務. 而如果用了Ingress, 只需要配置好這個服務, 當服務啟動時, 會自動注冊到Ingress的中, 不需要而外的操作.
2.減少不必要的端口暴露
配置過k8s的都清楚, 第一步是要關閉防火牆的, 主要原因是k8s的很多服務會以NodePort方式映射出去, 這樣就相當於給宿主機打了很多孔, 既不安全也不優雅. 而Ingress可以避免這個問題, 除了Ingress自身服務可能需要映射出去, 其他服務都不要用NodePort方式
2 部署配置Ingress
2.1 部署文件介紹、准備
獲取配置文件位置： https://github.com/kubernetes/ingress-nginx/tree/nginx-0.20.0/deploy
2.2下載部署文件
提供了兩種方式 ：
默認下載最新的yaml：
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
指定版本號下載對應的yaml
2.3部署文件介紹
1.namespace.yaml
創建一個獨立的命名空間 ingress-nginx
2.configmap.yaml
ConfigMap是存儲通用的配置變量的，類似於配置文件，使用戶可以將分布式系統中用於不同模塊的環境變量統一到一個對象中管理；而它與配置文件的區別在於它是存在集群的“環境”中的，並且支持K8S集群中所有通用的操作調用方式。
從數據角度來看，ConfigMap的類型只是鍵值組，用於存儲被Pod或者其他資源對象（如RC）訪問的信息。這與secret的設計理念有異曲同工之妙，主要區別在於ConfigMap通常不用於存儲敏感信息，而只存儲簡單的文本信息。
ConfigMap可以保存環境變量的屬性，也可以保存配置文件。
創建pod時，對configmap進行綁定，pod內的應用可以直接引用ConfigMap的配置。相當於configmap為應用/運行環境封裝配置。
pod使用ConfigMap，通常用於：設置環境變量的值、設置命令行參數、創建配置文件。
3.default-backend.yaml
如果外界訪問的域名不存在的話，則默認轉發到default-http-backend這個Service，其會直接返回404：
4.rbac.yaml
負責Ingress的RBAC授權的控制，其創建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding
5.with-rbac.yaml
是Ingress的核心，用於創建ingress-controller。前面提到過，ingress-controller的作用是將新加入的Ingress進行轉化為Nginx的配置
2.4部署ingress
2.4.1 准備鏡像
從這里mandatory.yaml查看需要哪些鏡像
注意：如果k8s版本比較低，則需要k8s.gcr.io/defaultbackend-amd64鏡像，nginx-ingress-controller鏡像也可以選擇最新版本。如果版本比較高，則不需要k8s.gcr.io/defaultbackend-amd64鏡像。
| 鏡像名稱 |
版本 |
下載地址 |
| k8s.gcr.io/defaultbackend-amd64 |
1.5 |
registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64 |
| quay.io/kubernetes-ingress-controller/nginx-ingress-controller |
0.20.0 |
registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0 |
在每一個節點(Node)上拉取鏡像:
[root@k8s-node01 ~]# docker pull registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5
1.5: Pulling from kubernetes_xingej/defaultbackend-amd64
26de8f6c1f4b: Pull complete
Digest: sha256:2cdff48ab9b20ca5f9b0ee48bf3c139c51d6fb1a15245966583bc371c121c238
Status: Downloaded newer image for registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5
registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5
[root@k8s-node01 ~]# docker pull registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0
0.20.0: Pulling from kubernetes_xingej/nginx-ingress-controller
8e41b996a802: Pull complete
f8e7d603ef88: Pull complete
610da4f3123f: Pull complete
62702a85e6c9: Pull complete
7cedf5b2083f: Pull complete
755f7fa719b6: Pull complete
6adbdd0c8aaf: Pull complete
2389d8b0d2a2: Pull complete
1ea794448393: Pull complete
bb0c388ee432: Pull complete
9626641c5a97: Pull complete
bd0bebb5ba38: Pull complete
Digest: sha256:3f06079f7727b2fb7ad5c97d8152eb622ae504674395dfa71fda7ce315aaaf30
Status: Downloaded newer image for registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0
registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0
[root@k8s-node01 ~]# docker images #檢查鏡像是否下載成功
REPOSITORY TAG IMAGE ID CREATED SIZE
kubernetesui/dashboard v2.0.3 503bc4b7440b 6 months ago 225MB
kubernetesui/metrics-scraper v1.0.4 86262685d9ab 8 months ago 36.9MB
registry.aliyuncs.com/google_containers/kube-proxy v1.18.0 43940c34f24f 8 months ago 117MB
registry.aliyuncs.com/google_containers/pause 3.2 80d28bedfe5d 10 months ago 683kB
registry.aliyuncs.com/google_containers/coredns 1.6.7 67da37a9a360 10 months ago 43.8MB
tomcat 7-alpine 81cd9536b1e6 19 months ago 147MB
lizhenliang/flannel v0.11.0-amd64 ff281650a721 23 months ago 52.6MB
registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64 1.5 d8f37b8cdaf4 2 years ago 5.13MB
registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller 0.20.0 3cc332ecde4f 2 years ago 513MB
ikubernetes/myapp v2 54202d3f0f35 2 years ago 15.5MB
2.4.2 下載yaml文件並更新mandatory.yaml中的鏡像地址（master上）
[root@k8s-master ~]# mkdir ingress-nginx
[root@k8s-master ingress-nginx]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.20.0/deploy/mandatory.yaml
[root@k8s-master ingress-nginx]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml #對外提供服務，如果不需要可以不下載
[root@k8s-master ingress-nginx]# sed -i 's#k8s.gcr.io/defaultbackend-amd64#registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64#g' mandatory.yaml #替換defaultbackend-amd64鏡像地址
sed -i 's#quay.io/kubernetes-ingress-controller/nginx-ingress-controller#registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller#g' mandatory.yaml #替換nginx-ingress-controller鏡像地址
[root@k8s-master ingress-nginx]# grep image mandatory.yaml #檢查替換結果
# Any image is permissible as long as:
image: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5
image: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0
2.4.3 修改service-nodeport.yaml文件，添加NodePort端口，默認為隨機端口
[root@k8s-master ingress-nginx]# cat service-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
nodePort: 32080 #http
- name: https
port: 443
targetPort: 443
protocol: TCP
nodePort: 32443 #https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
2.4.4 部署nginx-ingress-controller
[root@k8s-master ingress-nginx]# kubectl apply -f mandatory.yaml
namespace "ingress-nginx" created
deployment.extensions "default-http-backend" created
service "default-http-backend" created
configmap "nginx-configuration" created
configmap "tcp-services" created
configmap "udp-services" created
serviceaccount "nginx-ingress-serviceaccount" created
clusterrole.rbac.authorization.k8s.io "nginx-ingress-clusterrole" created
role.rbac.authorization.k8s.io "nginx-ingress-role" created
rolebinding.rbac.authorization.k8s.io "nginx-ingress-role-nisa-binding" created
clusterrolebinding.rbac.authorization.k8s.io "nginx-ingress-clusterrole-nisa-binding" created
deployment.extensions "nginx-ingress-controller" created
[root@k8s-master ingress-nginx]# kubectl apply -f service-nodeport.yaml
service "ingress-nginx" created
注意：如果k8s版本比較新，會出現以下錯誤
nginx version: nginx/1.15.5
W1219 14:25:27.879996 7 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1219 14:25:27.880178 7 main.go:196] Creating API client for https://10.96.0.1:443
I1219 14:25:27.892515 7 main.go:240] Running in Kubernetes cluster version v1.18 (v1.18.0) - git (clean) commit 9e991415386e4cf155a24b1da15becaa390438d8 - platform linux/amd64
I1219 14:25:27.895518 7 main.go:101] Validated ingress-nginx/default-http-backend as the default backend.
F1219 14:25:28.183697 7 main.go:115] Error generating self-signed certificate: could not create temp pem file /etc/ingress-controller/ssl/default-fake-certificate.pem: open /etc/ingress-controller/ssl/default-fake-certificate.pem423715344: permission denied
原因是因為新版本的權限比老版本多許多導致
解決方法就是復制以下配置到mandatory.yaml重新創建一下資源 kubectl create -f mandatory.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
# wait up to five minutes for the drain of connections
terminationGracePeriodSeconds: 300
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: registry.aliyuncs.com/google_containers/nginx-ingress-controller:0.26.1
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
查看ingress-nginx組件狀態：
[root@k8s-master ingress-nginx]# kubectl get pods -n ingress-nginx #pod狀態
NAME READY STATUS RESTARTS AGE
default-http-backend-66c4fbf5b4-x2n8w 1/1 Running 0 58s
nginx-ingress-controller-64bcff8657-5gdrd 1/1 Running 0 58s
[root@k8s-master ingress-nginx]# kubectl get svc -n ingress-nginx #service狀態及暴露端口
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend ClusterIP 10.96.87.65 <none> 80/TCP 1m
ingress-nginx NodePort 10.100.48.237 <none> 80:32080/TCP,443:32443/TCP 1m
訪問ingress-nginx服務，查看是否配置成功
可以看到，提示404，這個因為當前ingress-nginx服務現在還沒有后端服務，這是正常的。如果k8s版本較新，使用上面提供的mandatory.yaml配置，只能看到404提示
3 創建ingress-nginx后端服務
3.1 創建一個Service及后端Deployment(以nginx為例)
[root@k8s-master ingress]# cat deploy-demon.yaml
apiVersion: v1
kind: Service
metadata:
name: myapp
namespace: default
spec:
selector:
app: myapp
release: canary
ports:
- name: http
port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-deploy
spec:
replicas: 5
selector:
matchLabels:
app: myapp
release: canary
template:
metadata:
labels:
app: myapp
release: canary
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v2
ports:
- name: httpd
containerPort: 80
創建相關服務及檢查狀態是否就緒
[root@k8s-master ingress-nginx]# kubectl apply -f deploy-demon.yaml
service "myapp" created
deployment.apps "myapp-deploy" created
[root@k8s-master nginx-ingress-controller]# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-deploy-77c5c86fdb-l7c4f 1/1 Running 0 35s
myapp-deploy-77c5c86fdb-m5lnl 1/1 Running 0 35s
myapp-deploy-77c5c86fdb-nq6vl 1/1 Running 0 35s
myapp-deploy-77c5c86fdb-v2vpp 1/1 Running 0 35s
myapp-deploy-77c5c86fdb-w2d59 1/1 Running 0 35s
[root@k8s-master nginx-ingress-controller]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 50d
myapp ClusterIP 10.98.49.121 <none> 80/TCP 43s
將myapp添加至ingress-nginx中
[root@k8s-master ingress-nginx]# cat ingress-myapp.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-myapp
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: myapp.magedu.com #生產中該域名應當可以被公網解析
http:
paths:
- path:
backend:
serviceName: myapp
servicePort: 80
[root@k8s-master ingress-nginx]# kubectl apply -f ingress-myapp.yaml
ingress.extensions "ingress-myapp" created
配置域名解析，當前測試環境我們使用hosts文件進行解析
192.168.10.77 myapp.magedu.com
使用域名進行訪問
3.2 再創建一個Service及后端Deployment(以tomcat為例)
[root@k8s-master ingress-nginx]# cat tomcat-deploy.yaml
apiVersion: v1
kind: Service
metadata:
name: tomcat
namespace: default
spec:
selector:
app: tomcat
release: canary
ports:
- name: http
port: 8080
targetPort: 8080
- name: ajp
port: 8009
targetPort: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-deploy
spec:
replicas: 3
selector:
matchLabels:
app: tomcat
release: canary
template:
metadata:
labels:
app: tomcat
release: canary
spec:
containers:
- name: tomcat
image: tomcat:7-alpine
ports:
- name: httpd
containerPort: 8080
- name: ajp
containerPort: 8009
[root@k8s-master ingress-nginx]# kubectl apply -f tomcat-deploy.yaml
service "tomcat" created
deployment.apps "tomcat-deploy" created
[root@k8s-master ingress-nginx]# kubectl get pod #等待pod狀態就緒
將tomcat添加至ingress-nginx中
[root@k8s-master ingress-nginx]# cat ingress-tomcat.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-tomcat
namespace: default
annotations:
kubernets.io/ingress.class: "nginx"
spec:
rules:
- host: tomcat.magedu.com
http:
paths:
- path:
backend:
serviceName: tomcat
servicePort: 8080
[root@k8s-master ingress-nginx]# kubectl apply -f ingress-tomcat.yaml
ingress.extensions "ingress-tomcat" created
添加域名解析及訪問服務
3.3下面我們對tomcat服務添加httpds服務
創建私有證書及secret
[root@k8s-master ingress-nginx]# openssl genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
.......+++
..............................+++
e is 65537 (0x10001)
[root@k8s-master ingress-nginx]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.magedu.com #注意域名要和服務的域名一致
[root@k8s-master ingress-nginx]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key #創建secret
secret "tomcat-ingress-secret" created
[root@k8s-master ingress-nginx]# kubectl get secret
NAME TYPE DATA AGE
default-token-bf52l kubernetes.io/service-account-token 3 9d
tomcat-ingress-secret kubernetes.io/tls 2 7s
[root@k8s-master ingress-nginx]# kubectl describe secret tomcat-ingress-secret
Name: tomcat-ingress-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.crt: 1294 bytes #base64加密
tls.key: 1679 bytes
將證書應用至tomcat服務中
[root@k8s-master01 ingress]# cat ingress-tomcat-tls.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-tomcat-tls
namespace: default
annotations:
kubernets.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- tomcat.magedu.com #與secret證書的域名需要保持一致
secretName: tomcat-ingress-secret #secret證書的名稱
rules:
- host: tomcat.magedu.com
http:
paths:
- path:
backend:
serviceName: tomcat
servicePort: 8080
[root@k8s-master01 ingress]# kubectl apply -f ingress-tomcat-tls.yaml
訪問服務
參考文檔：
https://www.cnblogs.com/panwenbin-logs/p/9915927.html
https://blog.csdn.net/zhang_zhang__/article/details/104697421
https://blog.csdn.net/ht1032279753/article/details/103867138