Calico作為k8s的一個流行網絡插件，它依賴BGP路由協議實現集群節點上的POD路由互通；而路由互通的前提是節點間建立 BGP Peer 連接。BGP 路由反射器（Route Reflectors，簡稱 RR）可以簡化集群BGP Peer的連接方式，它是解決BGP擴展性問題的有效方式；具體來說：
- 沒有 RR 時，所有節點之間需要兩兩建立連接（IBGP全互聯），節點數量增加將導致連接數劇增、資源占用劇增
- 引入 RR 后，其他 BGP 路由器只需要與它建立連接並交換路由信息，節點數量增加連接數只是線性增加，節省系統資源
calico-node 版本 v3.3 開始支持內建路由反射器，非常方便，因此使用 calico 作為網絡插件可以支持大規模節點數的K8S集群。
本文檔主要講解配置 BGP Route Reflectors，建議首先閱讀基礎calico文檔。
前提條件
實驗環境為按照kubeasz安裝的2主2從集群，calico 版本 v3.3.2
$ kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.1.1 Ready,SchedulingDisabled master 178m v1.13.1
192.168.1.2 Ready,SchedulingDisabled master 178m v1.13.1
192.168.1.3 Ready node 178m v1.13.1
192.168.1.4 Ready node 178m v1.13.1
$ kubectl get pod -n kube-system -o wide | grep calico
calico-kube-controllers-77487546bd-jqrlc 1/1 Running 0 179m 192.168.1.3 192.168.1.3 <none> <none>
calico-node-67t5m 2/2 Running 0 179m 192.168.1.1 192.168.1.1 <none> <none>
calico-node-drmhq 2/2 Running 0 179m 192.168.1.2 192.168.1.2 <none> <none>
calico-node-rjtkv 2/2 Running 0 179m 192.168.1.4 192.168.1.4 <none> <none>
calico-node-xtspl 2/2 Running 0 179m 192.168.1.3 192.168.1.3 <none> <none>
查看當前集群中BGP連接情況：可以看到集群中4個節點兩兩建立了 BGP 連接
$ ansible all -m shell -a '/opt/kube/bin/calicoctl node status'
192.168.1.3 | SUCCESS | rc=0 >>
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+-------------+
| 192.168.1.1 | node-to-node mesh | up | 03:08:20 | Established |
| 192.168.1.2 | node-to-node mesh | up | 03:08:18 | Established |
| 192.168.1.4 | node-to-node mesh | up | 03:08:19 | Established |
+--------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
192.168.1.2 | SUCCESS | rc=0 >>
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+-------------+
| 192.168.1.4 | node-to-node mesh | up | 03:08:17 | Established |
| 192.168.1.3 | node-to-node mesh | up | 03:08:18 | Established |
| 192.168.1.1 | node-to-node mesh | up | 03:08:20 | Established |
+--------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
192.168.1.1 | SUCCESS | rc=0 >>
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+-------------+
| 192.168.1.2 | node-to-node mesh | up | 03:08:21 | Established |
| 192.168.1.3 | node-to-node mesh | up | 03:08:21 | Established |
| 192.168.1.4 | node-to-node mesh | up | 03:08:21 | Established |
+--------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
192.168.1.4 | SUCCESS | rc=0 >>
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+-------------+
| 192.168.1.2 | node-to-node mesh | up | 03:08:17 | Established |
| 192.168.1.3 | node-to-node mesh | up | 03:08:19 | Established |
| 192.168.1.1 | node-to-node mesh | up | 03:08:20 | Established |
+--------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
配置全局禁用全連接（BGP full mesh）
$ cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Info
nodeToNodeMeshEnabled: false
asNumber: 64512
EOF
上述命令配置完成后，再次使用命令ansible all -m shell -a '/opt/kube/bin/calicoctl node status'查看，可以看到之前所有的bgp連接都消失了。
配置 BGP node 與 Route Reflector 的連接建立規則
$ cat << EOF | calicoctl create -f - kind: BGPPeer apiVersion: projectcalico.org/v3 metadata: name: peer-to-rrs spec: # 規則1：普通 bgp node 與 rr 建立連接 nodeSelector: "!has(i-am-a-route-reflector)" peerSelector: has(i-am-a-route-reflector) --- kind: BGPPeer apiVersion: projectcalico.org/v3 metadata: name: rr-mesh spec: # 規則2：route reflectors 之間也建立連接 nodeSelector: has(i-am-a-route-reflector) peerSelector: has(i-am-a-route-reflector) EOF
上述命令配置完成后，使用命令：calicoctl get bgppeer calicoctl get bgppeer rr-mesh -o yaml 檢查配置是否正確。
選擇並配置 Route Reflector 節點
首先查看當前集群中的節點：
$ calicoctl get node -o wide
NAME ASN IPV4 IPV6
k8s401 (64512) 192.168.1.1/24
k8s402 (64512) 192.168.1.2/24
k8s403 (64512) 192.168.1.3/24
k8s404 (64512) 192.168.1.4/24
可以在集群中選擇1個或多個節點作為 rr 節點，這里先選擇節點：k8s401
# 1.先導出 node k8s401 的配置，准備修改 $ calicoctl get node k8s401 --export -o yaml |tee rr01.yml apiVersion: projectcalico.org/v3 kind: Node metadata: creationTimestamp: null name: k8s401 spec: bgp: ipv4Address: 192.168.1.1/24 ipv4IPIPTunnelAddr: 172.20.7.128 orchRefs: - nodeName: 192.168.1.1 orchestrator: k8s # 2.修改上述 rr01.yml 的配置如下 apiVersion: projectcalico.org/v3 kind: Node metadata: creationTimestamp: null name: k8s401 labels: # 設置標簽 i-am-a-route-reflector: true spec: bgp: ipv4Address: 192.168.1.1/24 ipv4IPIPTunnelAddr: 172.20.7.128 # 設置集群ID routeReflectorClusterID: 224.0.0.1 orchRefs: - nodeName: 192.168.1.1 orchestrator: k8s # 3.應用修改后的 rr node 配置 $ calicoctl apply -f rr01.yml
查看增加 rr 之后的bgp 連接情況
$ ansible all -m shell -a '/opt/kube/bin/calicoctl node status'
192.168.1.4 | SUCCESS | rc=0 >>
Calico process is running.
IPv4 BGP status
+--------------+-----------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-----------+-------+----------+-------------+
| 192.168.1.1 | node specific | up | 11:02:55 | Established |
+--------------+-----------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
192.168.1.3 | SUCCESS | rc=0 >>
Calico process is running.
IPv4 BGP status
+--------------+-----------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-----------+-------+----------+-------------+
| 192.168.1.1 | node specific | up | 11:02:55 | Established |
+--------------+-----------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
192.168.1.1 | SUCCESS | rc=0 >>
Calico process is running.
IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+---------------+-------+----------+-------------+
| 192.168.1.2 | node specific | up | 11:02:55 | Established |
| 192.168.1.3 | node specific | up | 11:02:55 | Established |
| 192.168.1.4 | node specific | up | 11:02:55 | Established |
+--------------+---------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
192.168.1.2 | SUCCESS | rc=0 >>
Calico process is running.
IPv4 BGP status
+--------------+-----------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-----------+-------+----------+-------------+
| 192.168.1.1 | node specific | up | 11:02:55 | Established |
+--------------+-----------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
可以看到所有其他節點都與所選rr節點建立bgp連接。
再增加一個 rr 節點
步驟同上述選擇第1個 rr 節點，這里省略；添加成功后可以看到所有其他節點都與兩個rr節點建立bgp連接，兩個rr節點之間也建立bgp連接。
- 對於節點數較多的
K8S集群建議配置3-4個 RR 節點