使用CNI網(wǎng)絡(luò)插件(calico)實(shí)現(xiàn)docker容器跨主機(jī)互聯(lián)

這篇具有很好參考價(jià)值的文章主要介紹了使用CNI網(wǎng)絡(luò)插件(calico)實(shí)現(xiàn)docker容器跨主機(jī)互聯(lián)。希望對(duì)大家有所幫助。如果存在錯(cuò)誤或未考慮完全的地方，請(qǐng)大家不吝賜教，您也可以點(diǎn)擊"舉報(bào)違法"按鈕提交疑問。

一.系統(tǒng)環(huán)境
二.前言
三.CNI網(wǎng)絡(luò)插件簡(jiǎn)介
四.常見的幾種CNI網(wǎng)絡(luò)插件對(duì)比
五.Calico網(wǎng)絡(luò)之間是如何通信的
六.配置calico讓物理機(jī)A上的docker容器c1可以訪問物理機(jī)B上的docker容器c2
- 6.1 安裝部署etcd集群
- 6.2 安裝部署docker
- 6.3 配置calico
- 6.4 使用Calico實(shí)現(xiàn)Docker容器跨主機(jī)互聯(lián)
七.Kubernetes(k8s)環(huán)境里的calico
八.總結(jié)

一.系統(tǒng)環(huán)境

本文主要基于Docker 20.10.12和Linux操作系統(tǒng)CentOS7.4。

服務(wù)器版本	calico版本	docker軟件版本	Kubernetes(k8s)集群版本	CPU架構(gòu)
CentOS Linux release 7.4.1708 (Core)	v2.6.12	Docker version 20.10.12	v1.21.9	x86_64

etcd集群架構(gòu)：etcd1為leader，etcd2為follower，etcd3為follower。

服務(wù)器	操作系統(tǒng)版本	CPU架構(gòu)	進(jìn)程	功能描述
etcd1/192.168.110.133	CentOS Linux release 7.4.1708 (Core)	x86_64	etcd	leader
etcd2/192.168.110.131	CentOS Linux release 7.4.1708 (Core)	x86_64	etcd	follower
etcd3/192.168.110.132	CentOS Linux release 7.4.1708 (Core)	x86_64	etcd	follower

Kubernetes集群架構(gòu)：k8scloude1作為master節(jié)點(diǎn)，k8scloude2，k8scloude3作為worker節(jié)點(diǎn)。

服務(wù)器	操作系統(tǒng)版本	CPU架構(gòu)	進(jìn)程	功能描述
k8scloude1/192.168.110.130	CentOS Linux release 7.4.1708 (Core)	x86_64	docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico	k8s master節(jié)點(diǎn)
k8scloude2/192.168.110.129	CentOS Linux release 7.4.1708 (Core)	x86_64	docker，kubelet，kube-proxy，calico	k8s worker節(jié)點(diǎn)
k8scloude3/192.168.110.128	CentOS Linux release 7.4.1708 (Core)	x86_64	docker，kubelet，kube-proxy，calico	k8s worker節(jié)點(diǎn)

二.前言

在Kubernetes集群中，容器之間的通信是非常重要的一部分。為了實(shí)現(xiàn)容器之間的跨主機(jī)互聯(lián)，需要使用CNI網(wǎng)絡(luò)插件。本文將介紹CNI網(wǎng)絡(luò)插件的概念和常見的幾種插件對(duì)比，并詳細(xì)講解如何使用Calico實(shí)現(xiàn)Docker容器跨主機(jī)互聯(lián)。

calico的信息保存在etcd里，所以需要一套etcd集群，關(guān)于etcd集群的安裝部署，可以查看博客《Kubernetes后臺(tái)數(shù)據(jù)庫etcd：安裝部署etcd集群，數(shù)據(jù)備份與恢復(fù)》。

查看Kubernetes(k8s)環(huán)境里的calico的前提是已經(jīng)有一套可以正常運(yùn)行的Kubernetes集群，關(guān)于Kubernetes(k8s)集群的安裝部署，可以查看博客《Centos7 安裝部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/16686769.html。

三.CNI網(wǎng)絡(luò)插件簡(jiǎn)介

CNI(Containernetworking Interface)網(wǎng)絡(luò)插件是一個(gè)由Linux基金會(huì)維護(hù)的開源項(xiàng)目，它可以為容器提供網(wǎng)絡(luò)連接。在Kubernetes中，可以通過CNI網(wǎng)絡(luò)插件來為Pod提供網(wǎng)絡(luò)連接。

目前市面上主流的CNI網(wǎng)絡(luò)插件有以下幾種：

Flannel：使用VXLAN技術(shù)實(shí)現(xiàn)網(wǎng)絡(luò)隔離和扁平化IP；
Calico：采用BGP協(xié)議實(shí)現(xiàn)高效的容器網(wǎng)絡(luò)互連；
Weave Net：使用虛擬機(jī)間通信(VXLAN)技術(shù)，在容器之間創(chuàng)建多層網(wǎng)絡(luò)；
Canal：結(jié)合Flannel和Calico兩種CNI網(wǎng)絡(luò)插件的優(yōu)點(diǎn)，實(shí)現(xiàn)網(wǎng)絡(luò)隔離和BGP路由。

四.常見的幾種CNI網(wǎng)絡(luò)插件對(duì)比

下面我們來對(duì)比這幾種CNI網(wǎng)絡(luò)插件。

CNI網(wǎng)絡(luò)插件	優(yōu)點(diǎn)	缺點(diǎn)	是否支持網(wǎng)絡(luò)策略
Flannel	部署簡(jiǎn)單，性能優(yōu)秀	網(wǎng)絡(luò)層延遲高	否
Calico	性能最好，支持容器內(nèi)BGP協(xié)議，支持網(wǎng)絡(luò)策略	配置復(fù)雜	是
Weave Net	功能強(qiáng)大，跨平臺(tái)支持	性能低下，容易出現(xiàn)網(wǎng)絡(luò)死鎖	是
Canal	結(jié)合了Flannel和Calico兩種插件的優(yōu)點(diǎn)，支持多種網(wǎng)絡(luò)模式，可以滿足不同的需求	部署和配置較為繁瑣	是

綜上所述，每種CNI網(wǎng)絡(luò)插件都有其獨(dú)特的優(yōu)勢(shì)和局限性，需要根據(jù)實(shí)際情況進(jìn)行選擇。

五.Calico網(wǎng)絡(luò)之間是如何通信的

Calico是一種基于IP路由技術(shù)的CNI網(wǎng)絡(luò)插件，它利用BGP協(xié)議來實(shí)現(xiàn)高效的容器網(wǎng)絡(luò)互連。在Calico中，每個(gè)容器都被賦予了一個(gè)唯一的IP地址，這些IP地址在網(wǎng)絡(luò)層面上是可達(dá)的，并且是通過數(shù)據(jù)包路由直接到達(dá)目標(biāo)容器的。

Calico使用路由表來管理容器網(wǎng)絡(luò)，每個(gè)主機(jī)上都會(huì)存在一個(gè)Calico Agent，它會(huì)監(jiān)聽Kubernetes API服務(wù)器，從而了解集群中所有容器的IP地址和狀態(tài)。當(dāng)某個(gè)容器需要向其他容器發(fā)起請(qǐng)求時(shí)，Calico會(huì)根據(jù)路由表信息進(jìn)行查找，找到合適的路徑，并將數(shù)據(jù)包轉(zhuǎn)發(fā)給目標(biāo)容器。

六.配置calico讓物理機(jī)A上的docker容器c1可以訪問物理機(jī)B上的docker容器c2

現(xiàn)在要解決的問題是：讓物理機(jī)A上的docker容器c1可以訪問物理機(jī)B上的docker容器c2！

方法一：物理機(jī)A上的容器c1想和物理機(jī)B上的容器c2通信，可以通過容器c1在物理機(jī)上映射一個(gè)端口，容器c2在物理機(jī)上映射一個(gè)端口，訪問物理機(jī)的端口達(dá)到訪問容器的目的，但是這樣過于麻煩，有沒有更好的方法呢？

方法二：可以通過網(wǎng)絡(luò)插件來實(shí)現(xiàn)這個(gè)需求，這里使用calico網(wǎng)絡(luò)插件。

6.1 安裝部署etcd集群

因?yàn)閏alico的信息保存在etcd里，所以需要一套etcd集群。

查看etcd集群的健康狀態(tài)。

[root@etcd1 ~]# etcdctl cluster-health
member 341a3c460c1c993a is healthy: got healthy result from http://192.168.110.131:2379
member 4679fe0fcb37326d is healthy: got healthy result from http://192.168.110.132:2379
member ab23bcc86cf3190b is healthy: got healthy result from http://192.168.110.133:2379
cluster is healthy

查看etcd集群的成員，可以看到etcd133是Leader。

[root@etcd1 ~]# etcdctl member list
341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false
4679fe0fcb37326d: name=etcd132 peerURLs=http://192.168.110.132:2380 clientURLs=http://192.168.110.132:2379,http://localhost:2379 isLeader=false
ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true

etcd現(xiàn)在什么數(shù)據(jù)也沒有

[root@etcd1 ~]# etcdctl ls /

6.2 安裝部署docker

三個(gè)節(jié)點(diǎn)安裝docker，用來啟動(dòng)容器。

[root@etcd1 ~]# yum -y install docker-ce

[root@etcd2 ~]# yum -y install docker-ce

[root@etcd3 ~]# yum -y install docker-ce

修改docker的啟動(dòng)參數(shù)，設(shè)置docker使用etcd來存儲(chǔ)數(shù)據(jù)，可以看到docker的啟動(dòng)腳本在/usr/lib/systemd/system/docker.service。

[root@etcd1 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: https://docs.docker.com
[root@etcd1 ~]# etcdctl member list
341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false
4679fe0fcb37326d: name=etcd132 peerURLs=http://192.168.110.132:2380 clientURLs=http://192.168.110.132:2379,http://localhost:2379 isLeader=false
ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true

添加啟動(dòng)參數(shù)：--cluster-store=etcd://192.168.110.133:2379。

[root@etcd1 ~]# vim /usr/lib/systemd/system/docker.service

[root@etcd1 ~]# grep ExecStart /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --cluster-store=etcd://192.168.110.133:2379 -H fd:// --containerd=/run/containerd/containerd.sock

重新加載配置，啟動(dòng)docker。

[root@etcd1 ~]# systemctl daemon-reload ;systemctl restart docker

可以看到參數(shù)添加成功：/usr/bin/dockerd --cluster-store=etcd://192.168.110.133:2379 -H fd:// --containerd=/run/containerd/containerd.sock。這樣設(shè)置之后，etcd就可以存儲(chǔ)docker的后端數(shù)據(jù)了。

[root@etcd1 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since 三 2022-02-16 15:39:50 CST; 39s ago
     Docs: https://docs.docker.com
 Main PID: 1390 (dockerd)
   Memory: 30.8M
   CGroup: /system.slice/docker.service
           └─1390 /usr/bin/dockerd --cluster-store=etcd://192.168.110.133:2379 -H fd:// --containerd=/run/containerd/containerd.sock

其他兩個(gè)節(jié)點(diǎn)進(jìn)行相同操作，但是etcd的IP要修改為本機(jī)的地址。

[root@etcd2 ~]# vim /usr/lib/systemd/system/docker.service

[root@etcd2 ~]# grep ExecStart /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --cluster-store=etcd://192.168.110.131:2379 -H fd:// --containerd=/run/containerd/containerd.sock

[root@etcd2 ~]# systemctl daemon-reload ;systemctl restart docker

[root@etcd2 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since 三 2022-02-16 15:39:57 CST; 41s ago
     Docs: https://docs.docker.com
 Main PID: 1348 (dockerd)
   Memory: 32.4M
   CGroup: /system.slice/docker.service
           └─1348 /usr/bin/dockerd --cluster-store=etcd://192.168.110.131:2379 -H fd:// --containerd=/run/containerd/containerd.sock


[root@etcd3 ~]# vim /usr/lib/systemd/system/docker.service

[root@etcd3 ~]# grep ExecStart /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --cluster-store=etcd://192.168.110.132:2379 -H fd:// --containerd=/run/containerd/containerd.sock

[root@etcd3 ~]# systemctl daemon-reload ;systemctl restart docker

[root@etcd3 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since 三 2022-02-16 15:39:59 CST; 41s ago
     Docs: https://docs.docker.com
 Main PID: 1355 (dockerd)
   Memory: 34.7M
   CGroup: /system.slice/docker.service
           └─1355 /usr/bin/dockerd --cluster-store=etcd://192.168.110.132:2379 -H fd:// --containerd=/run/containerd/containerd.sock

6.3 配置calico

創(chuàng)建calico目錄，并創(chuàng)建配置文件，三個(gè)節(jié)點(diǎn)都需要。

[root@etcd1 ~]# mkdir /etc/calico

[root@etcd1 ~]# cat > /etc/calico/calicoctl.cfg <<EOF
> apiVersion: v1
> kind: calicoApiConfig
> metadata:
> spec:
>   datastoreType: "etcdv2"
>   etcdEndpoints: "http://192.168.110.133:2379"
> EOF

#calico的配置文件已經(jīng)配置好了
[root@etcd1 ~]# cat /etc/calico/calicoctl.cfg
apiVersion: v1
kind: calicoApiConfig
metadata:
spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://192.168.110.133:2379"

[root@etcd2 ~]# mkdir /etc/calico

[root@etcd2 ~]# cat > /etc/calico/calicoctl.cfg <<EOF
> apiVersion: v1
> kind: calicoApiConfig
> metadata:
> spec:
>   datastoreType: "etcdv2"
>   etcdEndpoints: "http://192.168.110.131:2379"
> EOF

[root@etcd2 ~]# cat /etc/calico/calicoctl.cfg
apiVersion: v1
kind: calicoApiConfig
metadata:
spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://192.168.110.131:2379"

[root@etcd3 ~]# mkdir /etc/calico

[root@etcd3 ~]# cat > /etc/calico/calicoctl.cfg <<EOF
> apiVersion: v1
> kind: calicoApiConfig
> metadata:
> spec:
>   datastoreType: "etcdv2"
>   etcdEndpoints: "http://192.168.110.132:2379"
> EOF

[root@etcd3 ~]# cat /etc/calico/calicoctl.cfg
apiVersion: v1
kind: calicoApiConfig
metadata:
spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://192.168.110.132:2379"

創(chuàng)建目錄放置calico鏡像和工具。

[root@etcd1 ~]# mkdir etcd-calico

[root@etcd1 ~]# cd etcd-calico/

calicoctl是calico命令行工具，calico-node-v2.tar是calico-node鏡像包。

[root@etcd1 etcd-calico]# ls
calicoctl  calico-node-v2.tar

其他兩個(gè)節(jié)點(diǎn)也需要這兩個(gè)文件

[root@etcd1 etcd-calico]# scp ./* etcd2:/root/etcd-calico/
root@etcd2's password: 
calicoctl                                                                                                                                                                       100%   31MB  98.1MB/s   00:00    
calico-node-v2.tar                                                                                                                                                              100%  269MB  29.9MB/s   00:09    

[root@etcd1 etcd-calico]# scp ./* etcd3:/root/etcd-calico/
root@etcd3's password: 
calicoctl                                                                                                                                                                       100%   31MB  96.3MB/s   00:00    
calico-node-v2.tar                                                                                                                                                              100%  269MB  67.3MB/s   00:04

給calicoctl賦予可執(zhí)行權(quán)限

[root@etcd1 etcd-calico]# chmod +x calicoctl 

[root@etcd1 etcd-calico]# mv calicoctl /bin/

加載鏡像

[root@etcd1 etcd-calico]# docker load -i calico-node-v2.tar
df64d3292fd6: Loading layer [==================================================>]  4.672MB/4.672MB
d6f0e85be2d0: Loading layer [==================================================>]  8.676MB/8.676MB
c9818c503193: Loading layer [==================================================>]  250.9kB/250.9kB
1f748fca5871: Loading layer [==================================================>]  4.666MB/4.666MB
714c5990d9e8: Loading layer [==================================================>]  263.9MB/263.9MB
Loaded image: quay.io/calico/node:v2.6.12

另外兩個(gè)節(jié)點(diǎn)也是相同的操作。

[root@etcd2 ~]# mkdir etcd-calico

[root@etcd2 ~]# cd etcd-calico/

[root@etcd2 etcd-calico]# pwd
/root/etcd-calico

[root@etcd2 etcd-calico]# ls
calicoctl  calico-node-v2.tar

[root@etcd2 etcd-calico]# chmod +x calicoctl
[root@etcd2 etcd-calico]# mv calicoctl /bin/

[root@etcd2 etcd-calico]# docker load -i calico-node-v2.tar

[root@etcd3 ~]# mkdir etcd-calico

[root@etcd3 ~]# cd etcd-calico/

[root@etcd3 etcd-calico]# ls
calicoctl  calico-node-v2.tar

[root@etcd3 etcd-calico]# chmod +x calicoctl

[root@etcd3 etcd-calico]# mv calicoctl /bin/

[root@etcd3 etcd-calico]# docker load -i calico-node-v2.tar

三個(gè)節(jié)點(diǎn)上都啟動(dòng)Calico node

[root@etcd1 etcd-calico]# calicoctl node run --node-image=quay.io/calico/node:v2.6.12 -c /etc/calico/calicoctl.cfg
Running command to load modules: modprobe -a xt_set ip6_tables
......
Running the following command to start calico-node:

docker run --net=host --privileged --name=calico-node -d --restart=always -e NODENAME=etcd1 -e CALICO_NETWORKING_BACKEND=bird -e CALICO_LIBNETWORK_ENABLED=true -e ETCD_ENDPOINTS=http://192.168.110.133:2379 -v /var/log/calico:/var/log/calico -v /var/run/calico:/var/run/calico -v /lib/modules:/lib/modules -v /run:/run -v /run/docker/plugins:/run/docker/plugins -v /var/run/docker.sock:/var/run/docker.sock quay.io/calico/node:v2.6.12

Image may take a short time to download if it is not available locally.
Container started, checking progress logs.

2022-02-16 08:00:06.363 [INFO][9] startup.go 173: Early log level set to info
......
2022-02-16 08:00:06.536 [INFO][14] client.go 202: Loading config from environment
Starting libnetwork service
Calico node started successfully

每個(gè)節(jié)點(diǎn)都建立了一個(gè)calico-node容器

[root@etcd1 etcd-calico]# docker ps
CONTAINER ID   IMAGE                         COMMAND         CREATED          STATUS          PORTS     NAMES
ac7d48a378b6   quay.io/calico/node:v2.6.12   "start_runit"   57 seconds ago   Up 56 seconds             calico-node

另外兩個(gè)節(jié)點(diǎn)也啟動(dòng)Calico node

[root@etcd2 etcd-calico]# calicoctl node run --node-image=quay.io/calico/node:v2.6.12 -c /etc/calico/calicoctl.cfg

[root@etcd2 etcd-calico]# docker ps
CONTAINER ID   IMAGE                         COMMAND         CREATED              STATUS              PORTS     NAMES
bc99f286802f   quay.io/calico/node:v2.6.12   "start_runit"   About a minute ago   Up About a minute             calico-node

[root@etcd3 etcd-calico]# calicoctl node run --node-image=quay.io/calico/node:v2.6.12 -c /etc/calico/calicoctl.cfg

[root@etcd3 etcd-calico]# docker ps
CONTAINER ID   IMAGE                         COMMAND         CREATED              STATUS              PORTS     NAMES
07ba9ccdcd4d   quay.io/calico/node:v2.6.12   "start_runit"   About a minute ago   Up About a minute             calico-node

因?yàn)槲覀兪鞘褂胑tcd保存數(shù)據(jù)的，可以看到對(duì)方的主機(jī)的信息。

[root@etcd1 etcd-calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+-----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS   |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+-----------------+-------------------+-------+----------+-------------+
| 192.168.110.131 | node-to-node mesh | up    | 08:00:13 | Established |
| 192.168.110.132 | node-to-node mesh | up    | 08:00:14 | Established |
+-----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

[root@etcd2 etcd-calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+-----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS   |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+-----------------+-------------------+-------+----------+-------------+
| 192.168.110.133 | node-to-node mesh | up    | 08:00:13 | Established |
| 192.168.110.132 | node-to-node mesh | up    | 08:00:14 | Established |
+-----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.


[root@etcd3 etcd-calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+-----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS   |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+-----------------+-------------------+-------+----------+-------------+
| 192.168.110.133 | node-to-node mesh | up    | 08:00:15 | Established |
| 192.168.110.131 | node-to-node mesh | up    | 08:00:15 | Established |
+-----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

查看docker網(wǎng)絡(luò)類型

[root@etcd1 etcd-calico]# docker network list
NETWORK ID     NAME      DRIVER    SCOPE
2db83772936d   bridge    bridge    local
3c0a5a224b09   host      host      local
422becf3aa3b   none      null      local

創(chuàng)建calico類型的網(wǎng)絡(luò)，--driver calico 指定使用 calico 的 libnetwork CNM driver。 --ipam-driver calico-ipam 指定使用 calico 的 IPAM driver 管理 IP。 calico 為 global 網(wǎng)絡(luò)，etcd 會(huì)將 calnet1 同步到所有主機(jī)。

[root@etcd1 etcd-calico]# docker network create --driver calico --ipam-driver calico-ipam calnet1
735f15b514db3a7310a7f3ef0734a6cd6b966753dc8cf0f7847305e0ba9fe51f

calico 為 global 網(wǎng)絡(luò)，etcd 會(huì)將 calnet1 同步到所有主機(jī)。

[root@etcd1 etcd-calico]# docker network list
NETWORK ID     NAME      DRIVER    SCOPE
2db83772936d   bridge    bridge    local
735f15b514db   calnet1   calico    global
3c0a5a224b09   host      host      local
422becf3aa3b   none      null      local

[root@etcd2 etcd-calico]# docker network list
NETWORK ID     NAME      DRIVER    SCOPE
df0044c9f6f6   bridge    bridge    local
735f15b514db   calnet1   calico    global
03b08fa135f8   host      host      local
c19501b7ea7b   none      null      local

[root@etcd3 etcd-calico]# docker network list
NETWORK ID     NAME      DRIVER    SCOPE
331a6b638487   bridge    bridge    local
735f15b514db   calnet1   calico    global
08f90f4840c1   host      host      local
0d2160ce7298   none      null      local

6.4 使用Calico實(shí)現(xiàn)Docker容器跨主機(jī)互聯(lián)

三個(gè)節(jié)點(diǎn)拉取busybox鏡像用來創(chuàng)建容器

[root@etcd1 etcd-calico]# docker pull busybox

[root@etcd2 etcd-calico]# docker pull busybox

[root@etcd3 etcd-calico]# docker pull busybox

三個(gè)節(jié)點(diǎn)上都創(chuàng)建一個(gè)容器，指定網(wǎng)絡(luò)類型為calnet1

[root@etcd1 etcd-calico]# docker run --name c1 --net calnet1 -itd busybox
73359e36becf9859e073ebce9370b83ac36754f40356e53b82a1e2a8cd7b0066

[root@etcd2 etcd-calico]# docker run --name c2 --net calnet1 -itd busybox
28d27f3effb0ea15e6f5e6cca9e8982c68d24f459978098967842242478b6d8b

[root@etcd3 etcd-calico]# docker run --name c3 --net calnet1 -itd busybox
995241af841f2da4f69c7c3cfa2ce0766de49e7b43ec327f5dc8d57ff7838b62

進(jìn)入容器c1，查看網(wǎng)卡信息

[root@etcd1 etcd-calico]# docker exec -it c1 sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
4: cali0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.36.192/32 scope global cali0
       valid_lft forever preferred_lft forever
/ # exit

每在主機(jī)上創(chuàng)建一個(gè)容器，則會(huì)在物理機(jī)上創(chuàng)建一張?zhí)摂M網(wǎng)卡出來，注意：cali5aa980fa781@if4里的if4是容器里網(wǎng)卡的4，cali0@if5里的5是物理機(jī)網(wǎng)卡的5。從這里可以看到容器里的虛擬網(wǎng)卡 cali0 和物理機(jī)的 cali5aa980fa781 是 veth pair 關(guān)系。

[root@etcd1 etcd-calico]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:1e:33:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.110.133/24 brd 192.168.110.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe1e:333e/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:8b:19:bc:63 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: cali5aa980fa781@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 9a:3d:aa:d2:bc:a2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::983d:aaff:fed2:bca2/64 scope link 
       valid_lft forever preferred_lft forever

另外兩個(gè)節(jié)點(diǎn)也是類似的

[root@etcd2 etcd-calico]# docker exec -it c2 sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    ......
4: cali0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.57.64/32 scope global cali0
       valid_lft forever preferred_lft forever
/ # exit
[root@etcd2 etcd-calico]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
   ......
       valid_lft forever preferred_lft forever
5: cali2e3a79a8486@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether ce:2a:7a:5f:4e:83 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::cc2a:7aff:fe5f:4e83/64 scope link 
       valid_lft forever preferred_lft forever

[root@etcd3 etcd-calico]# docker exec -it c3 sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
   ......
4: cali0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.175.64/32 scope global cali0
       valid_lft forever preferred_lft forever
/ # exit
[root@etcd3 etcd-calico]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
   ......
5: califd96a41066a@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 2e:ca:96:03:96:83 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::2cca:96ff:fe03:9683/64 scope link 
       valid_lft forever preferred_lft forever

使用route -n查看路由信息：

192.168.57.64 192.168.110.131 255.255.255.192 UG 0 0 0 ens32，表示在容器里ping 192.168.57.64這個(gè)地址，都會(huì)轉(zhuǎn)發(fā)到192.168.110.131這臺(tái)機(jī)器；

192.168.57.64 0.0.0.0 255.255.255.255 UH 0 0 0 cali2e3a79a8486，表示目的地址是192.168.57.64的數(shù)據(jù)包，轉(zhuǎn)發(fā)到cali2e3a79a8486這張網(wǎng)卡。

cali2e3a79a8486和容器里的網(wǎng)卡cali0是veth pair 關(guān)系，所以就可以從容器c1訪問到容器c2，其他以此類推，calico相當(dāng)于建立了一個(gè)隧道，可以在物理機(jī)A的c1容器訪問物理機(jī)B的c2容器。

[root@etcd1 etcd-calico]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.110.2   0.0.0.0         UG    0      0        0 ens32
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 ens32
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.36.192  0.0.0.0         255.255.255.255 UH    0      0        0 cali5aa980fa781
192.168.36.192  0.0.0.0         255.255.255.192 U     0      0        0 *
192.168.57.64   192.168.110.131 255.255.255.192 UG    0      0        0 ens32
192.168.110.0   0.0.0.0         255.255.255.0   U     0      0        0 ens32
192.168.175.64  192.168.110.132 255.255.255.192 UG    0      0        0 ens32

在容器c1里，ping c2容器可以ping通。

[root@etcd1 etcd-calico]# docker exec -it c1 sh

/ # ping 192.168.57.64
PING 192.168.57.64 (192.168.57.64): 56 data bytes
64 bytes from 192.168.57.64: seq=0 ttl=62 time=0.578 ms
64 bytes from 192.168.57.64: seq=1 ttl=62 time=0.641 ms
64 bytes from 192.168.57.64: seq=2 ttl=62 time=0.543 ms
^C
--- 192.168.57.64 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.543/0.587/0.641 ms

/ # exit

在物理機(jī)上ping不通容器c2。

[root@etcd1 etcd-calico]# ping 192.168.57.64
PING 192.168.57.64 (192.168.57.64) 56(84) bytes of data.
^C
--- 192.168.57.64 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5000ms

看下路由的關(guān)系:不管目的地是哪里都走 cali0。

[root@etcd1 etcd-calico]# docker exec c1 ip route
default via 169.254.1.1 dev cali0 
169.254.1.1 dev cali0 scope link

看下etcd1 的路由，目的地址到 192.168.36.192 的數(shù)據(jù)包都從 cali5aa980fa781（etcd1 新產(chǎn)生的虛擬網(wǎng)卡）走，目的地址到 192.168.57.64/26 網(wǎng)段的數(shù)據(jù)包都從 ens32 發(fā)到 192.168.110.131 上去，每臺(tái)主機(jī)都知道不同的容器在哪臺(tái)主機(jī)上，所以會(huì)動(dòng)態(tài)的設(shè)置路由。

[root@etcd1 etcd-calico]# ip route
default via 192.168.110.2 dev ens32 
169.254.0.0/16 dev ens32 scope link metric 1002 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
192.168.36.192 dev cali5aa980fa781 scope link 
blackhole 192.168.36.192/26 proto bird 
192.168.57.64/26 via 192.168.110.131 dev ens32 proto bird 
192.168.110.0/24 dev ens32 proto kernel scope link src 192.168.110.133 
192.168.175.64/26 via 192.168.110.132 dev ens32 proto bird

七.Kubernetes(k8s)環(huán)境里的calico

在k8s環(huán)境里，每個(gè)節(jié)點(diǎn)上都有calico-node,calico數(shù)據(jù)存在etcd里。

[root@k8scloude1 ~]# kubectl get pod -o wide -n kube-system 
NAME                                       READY   STATUS    RESTARTS   AGE   IP                NODE         NOMINATED NODE   READINESS GATES
calico-kube-controllers-6b9fbfff44-4jzkj   1/1     Running   55         38d   10.244.251.210    k8scloude3   <none>           <none>
calico-node-bdlgm                          1/1     Running   27         38d   192.168.110.130   k8scloude1   <none>           <none>
calico-node-hx8bk                          1/1     Running   27         38d   192.168.110.128   k8scloude3   <none>           <none>
calico-node-nsbfs                          1/1     Running   27         38d   192.168.110.129   k8scloude2   <none>           <none>
coredns-545d6fc579-7wm95                   1/1     Running   27         38d   10.244.158.121    k8scloude1   <none>           <none>
coredns-545d6fc579-87q8j                   1/1     Running   27         38d   10.244.158.122    k8scloude1   <none>           <none>
etcd-k8scloude1                            1/1     Running   27         38d   192.168.110.130   k8scloude1   <none>           <none>
kube-apiserver-k8scloude1                  1/1     Running   18         27d   192.168.110.130   k8scloude1   <none>           <none>
kube-controller-manager-k8scloude1         1/1     Running   29         38d   192.168.110.130   k8scloude1   <none>           <none>
kube-proxy-599xh                           1/1     Running   27         38d   192.168.110.128   k8scloude3   <none>           <none>
kube-proxy-lpj8z                           1/1     Running   27         38d   192.168.110.129   k8scloude2   <none>           <none>
kube-proxy-zxlk9                           1/1     Running   27         38d   192.168.110.130   k8scloude1   <none>           <none>
kube-scheduler-k8scloude1                  1/1     Running   29         38d   192.168.110.130   k8scloude1   <none>           <none>
metrics-server-bcfb98c76-n4fnb             1/1     Running   26         30d   10.244.251.196    k8scloude3   <none>           <none>

八.總結(jié)

本文介紹了CNI網(wǎng)絡(luò)插件的概念和常見的幾種插件對(duì)比，詳細(xì)講解了如何使用Calico實(shí)現(xiàn)Docker容器跨主機(jī)互聯(lián)。通過使用Calico，我們可以輕松地在Kubernetes集群中實(shí)現(xiàn)高效的容器網(wǎng)絡(luò)互連，提升應(yīng)用程序的可靠性和可擴(kuò)展性。文章來源地址http://www.zghlxwxcb.cn/news/detail-476909.html

到了這里，關(guān)于使用CNI網(wǎng)絡(luò)插件(calico)實(shí)現(xiàn)docker容器跨主機(jī)互聯(lián)的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！