3節(jié)點(diǎn)測(cè)試集群 k8s 1.17 docker 19.03

每個(gè)節(jié)點(diǎn)2個(gè)網(wǎng)卡 ：

• enp0s3? 用來(lái)橋接無(wú)線網(wǎng)卡用來(lái)連接外網(wǎng)，因此IP不固定。
• enp0s8? 192.168.56.0/24 用來(lái)和其它節(jié)點(diǎn)互通

某一天開機(jī) 突然發(fā)現(xiàn)大量pod異常，異常的Pod 全部沒(méi)有正常獲取到IP, service也都打不開了。

檢查控制平面

除了 kube-system下的api-server, etcd-admin, scheduler, controller manager, 以及各個(gè)3個(gè)節(jié)點(diǎn)的kube-proxy 處于running狀態(tài)。這說(shuō)明集群健康狀態(tài)是正常的，節(jié)點(diǎn)OS和系統(tǒng)資源也沒(méi)問(wèn)題，POD都被正常調(diào)度到node了。kube-proxy和calico-node都是ds, 使用Hostnetwork，因此IP就是所在節(jié)點(diǎn)IP。

Coredns 處于completed ：

[root@admin ~ ]$k describe po  coredns-9d85f5447-sjs2j  -n kube-system

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 15 Feb 2024 09:25:15 +0800
      Finished:     Thu, 15 Feb 2024 19:58:02 +0800
    Ready:          False
    Restart Count:  36
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-j84s8 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
---
Events:
  Type    Reason          Age                    From            Message
  ----    ------          ----                   ----            -------
  Normal  SandboxChanged  4m3s (x575 over 129m)  kubelet, admin  Pod sandbox changed, it will be killed and re-created.

顯示sandbox 退出Kill了，看日志

[root@admin ~ ]$k logs?? coredns-9d85f5447-sjs2j? -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.5
linux/amd64, go1.13.4, c2fd1b2
[INFO] SIGTERM: Shutting down servers then terminating
[INFO] plugin/health: Going into lameduck mode for 5s

CNI組件calico-node 處于 Crashloopbackoff, 查看events 和Logs

Events:
? Type???? Reason???? Age??????????????????? From??????????? Message
? ----???? ------???? ----?????????????????? ----??????????? -------
? Warning? Unhealthy? 11m (x304 over 176m)?? kubelet, node2? Readiness probe failed: calico/node is not ready: felix is not ready: Get http://localhost:9099/readiness: dial tcp [::1]:9099: connect: connection refused
? Warning? BackOff??? 105s (x574 over 172m)? kubelet, node2? Back-off restarting failed container

[root@admin ~ ]$k logs? calico-node-7kvkf? -n kube-system
2024-02-16 04:58:08.483 [INFO][8] startup.go 259: Early log level set to info
2024-02-16 04:58:08.483 [INFO][8] startup.go 275: Using NODENAME environment for node name
2024-02-16 04:58:08.483 [INFO][8] startup.go 287: Determined node name: node2
2024-02-16 04:58:08.484 [INFO][8] k8s.go 228: Using Calico IPAM
2024-02-16 04:58:08.484 [INFO][8] startup.go 319: Checking datastore connection
2024-02-16 04:58:08.485 [INFO][8] startup.go 334: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: connect: network is unreachable
2024-02-16 04:58:09.486 [INFO][8] startup.go 334: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: connect: network is unreachable
2024-02-16 04:58:10.489 [INFO][8] startup.go 334: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: connect: network is unreachable
2024-02-16 04:58:11.499 [INFO][8] startup.go 334: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: connect: network is unreachable
2024-02-16 04:58:12.570 [INFO][8] startup.go 334: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: connect: network is unreachable
2024-02-16 04:58:13.571 [INFO][8] startup.go 334: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: connect: network is unreachable
2024-02-16 04:58:14.572 [INFO][8] startup.go 334: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: connect: network is unreachable
2024-02-16 04:58:15.578 [INFO][8] startup.go 334: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: connect: network is unreachable
2024-02-16 04:58:16.580 [INFO][8] startup.go 334: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: connect: network is unreachable
2024-02-16 04:58:17.581 [INFO][8] startup.go 334: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: connect: network is unreachable

查看calico contorller 日志，沒(méi)發(fā)現(xiàn)有用信息

[root@admin ~ ]$k logs? calico-kube-controllers-7489ff5b7c-6nl5p? -n kube-system
2024-02-15 01:25:31.218 [INFO][1] main.go 87: Loaded configuration from environment config=&config.Config{LogLevel:"info", ReconcilerPeriod:"5m", CompactionPeriod:"10m", EnabledControllers:"node", WorkloadEndpointWorkers:1, ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:"", HealthEnabled:true, SyncNodeLabels:true, DatastoreType:"kubernetes"}
2024-02-15 01:25:31.222 [INFO][1] k8s.go 228: Using Calico IPAM
W0215 01:25:31.222664?????? 1 client_config.go:541] Neither --kubeconfig nor --master was specified.? Using the inClusterConfig.? This might not work.
2024-02-15 01:25:31.223 [INFO][1] main.go 108: Ensuring Calico datastore is initialized
2024-02-15 01:25:31.228 [INFO][1] main.go 182: Starting status report routine
2024-02-15 01:25:31.228 [INFO][1] main.go 364: Starting controller ControllerType="Node"
2024-02-15 01:25:31.228 [INFO][1] node_controller.go 133: Starting Node controller
2024-02-15 01:25:31.329 [INFO][1] node_controller.go 146: Node controller is now running
2024-02-15 01:25:31.345 [INFO][1] kdd.go 167: Node and IPAM data is in sync

calico-node 日志發(fā)現(xiàn)報(bào)錯(cuò)連接datastore tcp 10.96.0.1:443 失敗， 那么這里是etcd嗎 這個(gè)IP是誰(shuí)呢

檢查api-server 配置 發(fā)現(xiàn)這是svc ip range的第一個(gè)地址

[root@admin ~ ]$ps -ef|grep apiserver
root????? 1121 17490? 0 13:34 pts/0??? 00:00:00 grep --color=auto apiserver
root????? 2939? 2885? 1 09:59 ???????? 00:04:00 kube-apiserver --advertise-address=192.168.56.3 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

?檢查SVC, 發(fā)現(xiàn)他是k8s本身這個(gè)服務(wù)

[root@admin ~ ]$k describe svc? kubernetes ?
Name:????????????? kubernetes
Namespace:???????? default
Labels:??????????? component=apiserver
?????????????????? provider=kubernetes
Annotations:?????? <none>
Selector:????????? <none>
Type:????????????? ClusterIP
IP:??????????????? 10.96.0.1
Port:????????????? https? 443/TCP
TargetPort:??????? 6443/TCP
Endpoints:???????? 192.168.56.3:6443
Session Affinity:? None
Events:??????????? <none>?

它是一個(gè)clusterIP類型服務(wù)，指向的EP是 192.168.56.3:6443, 那么6443是誰(shuí)暴露的呢

[root@admin ~ ]$lsof -i:6443
COMMAND??? PID USER?? FD?? TYPE DEVICE SIZE/OFF NODE NAME
kubelet?? 1355 root?? 25u? IPv4? 37928????? 0t0? TCP admin:40182->admin:sun-sr-https (ESTABLISHED)
kube-apis 2939 root??? 5u? IPv6? 37530????? 0t0? TCP *:sun-sr-https (LISTEN)

確定它就是k8s kube-apiserver-admin這個(gè)pod所單獨(dú)暴露出來(lái)的svc, 是單例的pod，不屬于任何rs/ds/deployment/sts。

pod正常，svc故障導(dǎo)致calico無(wú)法訪問(wèn)apiserver，也更無(wú)法為pod分配IP以及在每個(gè)節(jié)點(diǎn)配置iptables規(guī)則。那為什么網(wǎng)絡(luò)不可達(dá)呢，這就要從路由層面查了。

一番排查有

因?yàn)榻裉靉dmin節(jié)點(diǎn)連接了有線網(wǎng)絡(luò)，沒(méi)連WIFI，導(dǎo)致用來(lái)橋接的wifi網(wǎng)卡沒(méi)有通過(guò)DHCP獲取到IP、默認(rèn)網(wǎng)關(guān)路由信息，所以admin節(jié)點(diǎn)就會(huì)出現(xiàn)10.96.0.1 網(wǎng)絡(luò)不可達(dá)

[root@admin /etc/kubernetes ]$ansible k8s-1? -m shell? -a 'traceroute 10.96.0.1'
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
192.168.56.3 | FAILED | rc=1 >>
traceroute to 10.96.0.1 (10.96.0.1), 30 hops max, 60 byte packets
connect: Network is unreachablenon-zero return code
192.168.56.4 | CHANGED | rc=0 >>
traceroute to 10.96.0.1 (10.96.0.1), 30 hops max, 60 byte packets
?1? * * *
?2? * * *
?3? * * *
?4? * * *
?5? * * *
?6? * * *
?7? * * *
?8? * * *
?9? * * *
10? * * *
11? * * node1 (192.168.31.226)? 3008.015 ms !H
192.168.56.5 | CHANGED | rc=0 >>
traceroute to 10.96.0.1 (10.96.0.1), 30 hops max, 60 byte packets
?1? * * *
?2? * * *
?3? * * *
?4? * * *
?5? * * *
?6? * * *
?7? * * *
?8? * * *
?9? * * *
10? * * *
11? * * node2 (192.168.31.20)? 3005.846 ms !H

?[root@admin /etc/kubernetes ]$ansible k8s-1? -m shell? -a 'ip route'
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
192.168.56.4 | CHANGED | rc=0 >>
default via 192.168.31.1 dev enp0s3 proto dhcp metric 102
10.10.0.0/26 via 192.168.56.5 dev tunl0 proto bird onlink
10.10.0.128/26 via 192.168.56.3 dev tunl0 proto bird onlink
blackhole 10.10.0.192/26 proto bird
10.10.0.196 dev cali2473e8d3fe5 scope link
10.10.0.199 dev cali931cf856fe5 scope link
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.31.0/24 dev enp0s3 proto kernel scope link src 192.168.31.226 metric 102
192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.4 metric 101
192.168.56.3 | CHANGED | rc=0 >>? ## 默認(rèn)路由缺失！！
10.10.0.0/26 via 192.168.56.5 dev tunl0 proto bird onlink
blackhole 10.10.0.128/26 proto bird
10.10.0.129 dev calib35f38918a6 scope link
10.10.0.130 dev cali3d6a8137e9b scope link
10.10.0.131 dev calief752050065 scope link
10.10.0.189 dev cali9cd0964c823 scope link
10.10.0.192/26 via 192.168.56.4 dev tunl0 proto bird onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.3 metric 101
192.168.56.5 | CHANGED | rc=0 >>
default via 192.168.31.1 dev enp0s3 proto dhcp metric 102
blackhole 10.10.0.0/26 proto bird
10.10.0.37 dev caliae17495c610 scope link
10.10.0.38 dev cali7c21225184f scope link
10.10.0.128/26 via 192.168.56.3 dev tunl0 proto bird onlink
10.10.0.192/26 via 192.168.56.4 dev tunl0 proto bird onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.31.0/24 dev enp0s3 proto kernel scope link src 192.168.31.20 metric 102
192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.5 metric 101
[root@admin /etc/kubernetes ]$

此時(shí)去檢查admin節(jié)點(diǎn)的路由表和iptables-nat表會(huì)發(fā)現(xiàn)，雖然針對(duì)這個(gè)svc的dnat規(guī)則已經(jīng)配置，但是從網(wǎng)絡(luò)流量的處理順序來(lái)看，經(jīng)過(guò)PREROUTING鏈后接著檢查路由表，路由表匹配不到任何條目，也沒(méi)有default gw的話，就直接unreachable了

Chain KUBE-SEP-G5V522HWZT6RKRAC (1 references)
?pkts bytes target???? prot opt in???? out???? source?????????????? destination??????? ?
??? 0???? 0 KUBE-MARK-MASQ? all? --? *????? *?????? 192.168.56.3???????? 0.0.0.0/0????????? ?
??? 7?? 420 DNAT?????? tcp? --? *????? *?????? 0.0.0.0/0??????????? 0.0.0.0/0??????????? tcp to:192.168.56.3:6443

那么已知我的無(wú)線路由器地址為192.168.31.1, 它已經(jīng)是node1, node2的默認(rèn)網(wǎng)關(guān)，顯然，無(wú)線路由器的路由表里肯定沒(méi)有我K8s集群內(nèi)部這個(gè)svc ip的相關(guān)條目并不知道向哪轉(zhuǎn)發(fā)，ip數(shù)據(jù)包從admin host傳輸?shù)?0.96.0.1后面的podIP 到底跟網(wǎng)關(guān)地址有沒(méi)有關(guān)系呢？

很顯然? 沒(méi)有，只需要有一個(gè)路由條目能匹配就行，手動(dòng)把默認(rèn)網(wǎng)關(guān)配置到任何一個(gè)可達(dá)IP 都可以讓數(shù)據(jù)繼續(xù)往下流動(dòng)，路由決策后從enp0s3網(wǎng)卡out, 繼續(xù)經(jīng)過(guò)FORWARD--->POSTROUTING鏈， 然后從enp0s8 in ，自下而上流經(jīng)協(xié)議棧，以太網(wǎng)-->IP-->socket 進(jìn)入apiserver這個(gè)Pod。

PS：這里因?yàn)橛昧薶ostnetwork, pod跟主機(jī)在一個(gè)netns下面。

實(shí)現(xiàn)10.96.0.1-->dnat to 192.168.56.3:6443 成功

?Pod 日志里也能看到，在我手動(dòng)配置默認(rèn)gw成功的瞬間，訪問(wèn)恢復(fù)正常。

此后，所有資源對(duì)k8s svc的訪問(wèn)正常，scheduler 和各個(gè)controller開始負(fù)責(zé)實(shí)現(xiàn)目標(biāo)狀態(tài)，集群恢復(fù)正常。

當(dāng)然，這個(gè)問(wèn)題可能只有用vm測(cè)試會(huì)遇到吧，生產(chǎn)不會(huì)有這個(gè)問(wèn)題。

但是對(duì)于理解k8s CNI, SVC以及iptables 工作過(guò)程還有所幫助的，所以記錄一下。文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-834496.html

到了這里，關(guān)于記錄一次K8s 集群故障（路由&Calico）的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！