Ingress+ hostNetwork方式實現nodeIP:80訪問

http://blog.itpub.net/28916011/viewspace-2214747/

https://www.cnblogs.com/xuxinkun/p/11052646.html

https://www.cnblogs.com/zhangb8042/p/10149429.html

node ip + 非80端口，訪問k8s集群內部的服務。可是，我們實際生產中更希望的是node ip + 80端口的方式，訪問k8s集群內的服務

 

隨機方式很麻煩，我是真不知道該怎么去解析

80端口，可以通過雲平台/F5直接解析到邊緣節點80去實現訪問。

 

相比較起來，nodePort部署模式中需要部署的ingress-controller容器較少。一個集群可以部署幾個就可以了。而hostNetwork模式需要在每個節點部署一個ingress-controller容器，因此總起來消耗資源較多。另外一個比較直觀的區別，nodePort模式主要占用的是svc的nodePort端口。而hostNetwork則需要占用物理機的80和443端口。

從網絡流轉來說，通過nodePort訪問時，該node節點不一定部署了ingress-controller容器。因此還需要iptables將其轉發到部署有ingress-controller的節點上去，多了一層流轉。

另外，通過nodePort訪問時，nginx接收到的http請求中的source ip將會被轉換為接受該請求的node節點的ip，而非真正的client端ip。

而使用hostNetwork的方式，ingress-controller將會使用的是物理機的DNS域名解析(即物理機的/etc/resolv.conf)。而無法使用內部的比如coredns的域名解析。

我們重新部署一下，如果已經部署完成，修改配置文件，重新加載即可。

 

1  下載yaml文件

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

#這個文件，實際上是ingress-nginx/deploy/static 下  config.yaml rbac.yaml… 那一堆文件集合

 

 

 

2 修改文件

Deployment 部署的副本 Pod 會分布在各個 Node 上，每個 Node 都可能運行好幾個副本。DaemonSet 的不同之處在於：每個 Node 上最多只能運行一個副本。

 

• kind: DaemonSet：官方原始文件使用的是deployment，replicate 為 1，這樣將會在某一台節點上啟動對應的nginx-ingress-controller pod。外部流量訪問至該節點，由該節點負載分擔至內部的service。測試環境考慮防止單點故障，改為DaemonSet然后刪掉replicate ，配合親和性部署在制定節點上啟動nginx-ingress-controller pod，確保有多個節點啟動nginx-ingress-controller pod，后續將這些節點加入到外部硬件負載均衡組實現高可用性。
• hostNetwork: true：添加該字段，暴露nginx-ingress-controller pod的服務端口（80）
• nodeSelector: 增加親和性部署，有custom/ingress-controller-ready 標簽的節點才會部署該DaemonSet

為需要部署nginx-ingress-controller的節點設置lable

kubectl label nodes node2 custom/ingress-controller-ready=true
kubectl label nodes node3 custom/ingress-controller-ready=true
kubectl label nodes node4 custom/ingress-controller-ready=true

 

如果你想除了master節點其他的都部署，那直接對master設置污點就行了

kubectl taint nodes k8s-master node-role.kubernetes.io/master=true:NoSchedule

 

我的是禁止master執行

 

3 執行yaml文件

[root@k8s-master tmp]# kubectl apply -f mandatory.yaml
namespace/ingress-nginx unchanged
configmap/nginx-configuration unchanged
configmap/tcp-services unchanged
configmap/udp-services unchanged
serviceaccount/nginx-ingress-serviceaccount unchanged
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole unchanged
role.rbac.authorization.k8s.io/nginx-ingress-role unchanged
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding unchanged
daemonset.apps/nginx-ingress-controller created
[root@k8s-master tmp]#

 

 

4 查看ingress服務

[root@k8s-master tmp]# kubectl get pods   -n ingress-nginx  -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
nginx-ingress-controller-8n6fz   1/1     Running   0          51s   10.6.76.23   k8s-node-1   <none>           <none>
nginx-ingress-controller-jt82z   1/1     Running   0          51s   10.6.76.24   k8s-node-2   <none>           <none>
[root@k8s-master tmp]#

 

 

 

5 創建一個Nginx測試服務svc和deployment

 

apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  selector: #標簽選擇
    name: nginx
  ports:
  - port: 80 #服務器端口
    name: http #名稱
    targetPort: 80 #容器端口
    protocol: TCP #協議,默認TCP
 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx-deploy
spec:
  replicas: 3
  selector: 
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx 
    spec:
      containers: 
      - name: nginx
        image: nginx:alpine #鏡像
        ports:
        - name: http
          containerPort: 80 #容器端口

[root@k8s-master tmp]# vim nginx-test.yaml
[root@k8s-master tmp]#
[root@k8s-master tmp]# kubectl apply -f nginx-test.yaml
service/nginx-svc created
deployment.apps/my-nginx-deploy created
[root@k8s-master tmp]#
[root@k8s-master tmp]# kubectl get pod,svc -o wide | grep nginx
pod/my-nginx-deploy-b97f5f447-65xh7           1/1     Running   0          7m43s   10.254.1.46    k8s-node-1   <none>           <none>

pod/my-nginx-deploy-b97f5f447-rmsqc           1/1     Running   0          7m43s   10.254.2.77    k8s-node-2   <none>           <none>
pod/my-nginx-deploy-b97f5f447-z57f8           1/1     Running   0          7m43s   10.254.2.78    k8s-node-2   <none>           <none>
service/nginx-svc    ClusterIP   10.105.38.50    <none>        80/TCP           7m43s   name=nginx
[root@k8s-master tmp]#

 

 

6 創建一個基於Nginx-test的 ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-nginx
spec:
  rules:
  - host: nginx.wangxu.com
    http:
      paths:
      - backend:
          serviceName: nginx-svc
          servicePort: 80

 

[root@k8s-master tmp]# kubectl apply -f nginx-test-ingress.yaml
ingress.extensions/ingress-nginx created
[root@k8s-master tmp]#
[root@k8s-master tmp]# kubectl get ingresses
NAME            HOSTS              ADDRESS   PORTS   AGE
ingress-nginx   nginx.wangxu.com             80      22s
[root@k8s-master tmp]#
[root@k8s-master tmp]# kubectl describe ingresses ingress-nginx
Name:             ingress-nginx
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host              Path  Backends
  ----              ----  --------
  nginx.wangxu.com
                       nginx-svc:80 (10.254.1.46:80,10.254.2.77:80,10.254.2.78:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"ingress-nginx","namespace":"default"},"spec":{"rules":[{"host":"nginx.wangxu.com","http":{"paths":[{"backend":{"serviceName":"nginx-svc","servicePort":80}}]}}]}}

Events:
  Type    Reason  Age    From                      Message
  ----    ------  ----   ----                      -------
  Normal  CREATE  5m11s  nginx-ingress-controller  Ingress default/ingress-nginx
  Normal  CREATE  5m11s  nginx-ingress-controller  Ingress default/ingress-nginx
[root@k8s-master tmp]#

 

 

7  測試Nginx-test服務

對nodeIP（邊緣節點）進行host解析

 

10.6.76.23 nginx.wangxu.com

10.6.76.24 nginx.wangxu.com

 

[root@k8s-master tmp]# curl -I  nginx.wangxu.com
HTTP/1.1 200 OK
Server: openresty/1.15.8.2
Date: Mon, 14 Oct 2019 03:31:02 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 24 Sep 2019 16:01:13 GMT
ETag: "5d8a3dc9-264"
Accept-Ranges: bytes

[root@k8s-master tmp]#

 

 

 

8 實現HTTPS

 

8.1  創建nginx-ssl的service和deployment

apiVersion: v1
kind: Service
metadata:
  name: myapp-ssl
  namespace: default
spec:
  selector:
    app: myhttps
  ports:
  - name: http
    targetPort: 80
    port: 80
  - name: https
    targetPort: 443
    port: 443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deploy
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myhttps
  template:
    metadata:
      labels:
        app: myhttps
    spec:
      containers:
      - name: myhttps
        image: nginx:alpine
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443

 

[root@k8s-master tmp]# kubectl apply -f www.yaml
service/myapp-ssl created
deployment.apps/myapp-deploy created
[root@k8s-master tmp]# kubectl get pod,svc -o wide | grep myapp
pod/myapp-deploy-f4b7cc99-8mnzc               1/1     Running   0          7s    10.254.2.81    k8s-node-2   <none>           <none>

pod/myapp-deploy-f4b7cc99-9g2dp               1/1     Running   0          7s    10.254.1.48    k8s-node-1   <none>           <none>
pod/myapp-deploy-f4b7cc99-p9m75               1/1     Running   0          7s    10.254.2.82    k8s-node-2   <none>           <none>
service/myapp-ssl    ClusterIP   10.102.105.57    <none>        80/TCP,443/TCP   7s      app=myhttps
[root@k8s-master tmp]#

 

 

8.2  創建ssl的證書和secret

# 創建一個基於自身域名的證書
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout nginxssl.wangxu.com.key -out nginxssl.wangxu.com.pem -subj "/CN=nginxssl.wangxu.com"

# 導入 域名的證書到secret 中
kubectl create secret tls nginxssl-secret  --cert nginxssl.wangxu.com.pem --key nginxssl.wangxu.com.key
#查看
kubectl  get secret | grep nginxssl

 

 

[root@k8s-master tmp]# kubectl  get secret | grep nginxssl
nginxssl-secret                      kubernetes.io/tls                     2      39s
[root@k8s-master tmp]#

 

 

8.3  配置https的ingress

[root@k8s-master tmp]# cat www-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: www-https
spec:
  tls:
  - hosts:
    - nginxssl.wangxu.com
    secretName: nginxssl-secret
  rules:
  - host: nginxssl.wangxu.com
    http:
      paths:
      - path: /
        backend:
          serviceName:
          servicePort: 443
        backend:
          serviceName: myapp-ssl
          servicePort: 80

[root@k8s-master tmp]#
[root@k8s-master tmp]# kubectl  get ingresses
NAME            HOSTS                 ADDRESS   PORTS     AGE
ingress-nginx   nginx.wangxu.com                80        43m
www-https       nginxssl.wangxu.com             80, 443   9s

 

 

9 測試NginxSSL服務

對nodeIP（邊緣節點）進行host解析

 

10.6.76.23 nginxssl.wangxu.com

10.6.76.24 nginxssl.wangxu.com

 

[root@k8s-master tmp]# curl -k -I https://nginxssl.wangxu.com
HTTP/1.1 200 OK
Server: openresty/1.15.8.2
Date: Mon, 14 Oct 2019 04:00:52 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Strict-Transport-Security: max-age=15724800; includeSubDomains
Last-Modified: Tue, 24 Sep 2019 16:01:13 GMT
ETag: "5d8a3dc9-264"
Accept-Ranges: bytes

[root@k8s-master tmp]#

 

 

 

 

 

10 公網發布配置

將雲平台的負載均衡/F5 映射到node節點ip上，再把slb/f5公網地址解析出去，外網就能通過域名訪問k8s服務了，生產中，證書一般是花錢的拉，並非我們測試的自簽名證書。

域名—slb/f5外網ip—nodeIP

 

 

 

ngi