kube-proxy ipvs calico

1. 關閉防火牆

#/etc/selinux/config

SELINUX=disabled

#sestatus

SELinux status: disabled

#systemctl stop firewalld
#systemctl disable firewalld

　　

2. 修改系統參數

# /etc/sysctl.conf

net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1

# sysctl -p

　　

3. 安裝Ipvsadm

yum -y install ipvsadm ipset

# lsmod | grep ip_vs

ip_vs_sh               12688  0
ip_vs_wrr              12697  0
ip_vs_rr               12600  27
ip_vs                 141092  33 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          133387  9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c              12644  4 xfs,ip_vs,nf_nat,nf_conntrack

 

安裝所需模塊

 

modprobe -a ip_set ip_tables  ip6_tables ipt_REJECT ipt_rpfilter ipt_set nf_conntrack_netlink  nf_conntrack_proto_sctp sctp xt_addrtype xt_comment xt_conntrack xt_icmp  xt_icmp6 xt_ipvs xt_mark xt_multiport xt_rpfilter xt_sctp xt_set xt_u32 ipip

 

其中有幾個不存在

modprobe: WARNING: Module nf_conntrack_proto_sctp not found.
modprobe: WARNING: Module xt_icmp not found.
modprobe: WARNING: Module xt_icmp6 not found.
modprobe: WARNING: Module xt_rpfilter not found.

當前環境使用缺失以上mod可以運行，先不管了 

 

如果使用ipip 還要確認ipip模塊

modprobe ipip
ip link set dev tunl0 up

啟動calico后查看tunl0網卡的ip地址
如果沒有分配地址，需要重新刪除calico,清理calico存儲尤其是ippool
直到分配地址

tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN qlen 1
link/ipip 0.0.0.0 brd 0.0.0.0
inet 172.18.39.0/32 scope global tunl0
valid_lft forever preferred_lft forever

 

如果模塊沒有加載

cat > /etc/sysconfig/modules/ipvs.modules <<EOF

#!/bin/bash

modprobe -- ip_vs

modprobe -- ip_vs_rr

modprobe -- ip_vs_wrr

modprobe -- ip_vs_sh

modprobe -- nf_conntrack_ipv4

EOF

# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

上面腳本創建了的/etc/sysconfig/modules/ipvs.modules文件，保證在節點重啟后能自動加載所需模塊。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已經正確加載所需的內核模塊。

 

4. 修改kubelet配置

(新版已經移除了相關參數，使用默認的kube-proxy鏡像提交daemonset即可)

KUBE_PROXY_ARGS="--bind-address=10.10.1.8 \
                 --hostname-override=docker4.node \
                 --masquerade-all \
                 --feature-gates=SupportIPVSProxyMode=true \
                 --proxy-mode=ipvs \
                 --ipvs-min-sync-period=5s \
                 --ipvs-sync-period=5s \
                 --ipvs-scheduler=rr \
                 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
                 --cluster-cidr=10.254.0.0/16"

　

5. calico 配置 port range (沒有指定的可以跳過)

  calico.yaml
          
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              value: "172.18.0.0/16"
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            # Set Felix logging to "info"
            - name: FELIX_LOGSEVERITYSCREEN
              value: "info"
            - name: FELIX_HEALTHENABLED
              value: "true"
            # Node port range while kube-proxy in ipvs mode
            - name: FELIX_KUBENODEPORTRANGES
              value: "10000:11000"

　　

　注意事項 ：

1.calico會自動選擇網卡，這里要看calico-node的日志確定是否是需要的網卡。

2.calico的 CIDR不要與clusterip和hostip重疊，否則網絡不通。

3. namespaceSelector,podSelector 與ingress-nginx 的namespace不可用的情況，考慮是不是用了hostNetwork 

     https://superuser.com/questions/1481901/networkpolicy-has-no-effect-on-nginx-ingress-namespace

 

參考資料

https://docs.projectcalico.org/networking/use-ipvs

https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises

https://blog.51cto.com/1000682/2362853

https://www.kubernetes.org.cn/3025.html