Error: ImagePullBackOff 故障排除
1. 起因
起因是要在一組k8s環(huán)境下做個(gè)Prometheus的測(cè)試,當(dāng)時(shí)虛擬機(jī)用完直接暫停了.
啟動(dòng)完master和node節(jié)點(diǎn)后重啟了這些節(jié)點(diǎn).
當(dāng)檢查dashboard時(shí)候發(fā)現(xiàn)Pod處于ImagePullBackOff狀態(tài),使用命令查看詳細(xì)情況
kubectl describe pods -n kubernetes-dashboard kubernetes-dashboard-6948fdc5fd-7szc9
發(fā)現(xiàn)鏡像拉取失敗
1.1 報(bào)錯(cuò)信息如下:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 25s default-scheduler Successfully assigned kubernetes-dashboard/kubernetes-dashboard-6948fdc5fd-7szc9 to 192.168.31.112
Normal SandboxChanged 23s kubelet Pod sandbox changed, it will be killed and re-created.
Normal BackOff 20s (x3 over 22s) kubelet Back-off pulling image "harbor.intra.com/baseimages/kubernetesui/dashboard:v2.4.0"
Warning Failed 20s (x3 over 22s) kubelet Error: ImagePullBackOff
Normal Pulling 9s (x2 over 24s) kubelet Pulling image "harbor.intra.com/baseimages/kubernetesui/dashboard:v2.4.0"
Warning Failed 9s (x2 over 24s) kubelet Failed to pull image "harbor.intra.com/baseimages/kubernetesui/dashboard:v2.4.0": rpc error: code = Unknown desc = Error response from daemon: Get "https://harbor.intra.com/v2/": x509: certificate signed by unknown authority
Warning Failed 9s (x2 over 24s) kubelet Error: ErrImagePull
2. 排查思路
這個(gè)故障明顯是由于node節(jié)點(diǎn)拉取harbor中鏡像失敗造成.那么可能的原因就是以下幾點(diǎn),我們對(duì)此一一排查
- harbor.intra.com解析錯(cuò)誤或者服務(wù)器沒(méi)有啟動(dòng),使用ping命令排查
- harbor.intra.com上harbor服務(wù)異常造成,使用網(wǎng)頁(yè)瀏覽或者curl命令排查
- node節(jié)點(diǎn)到harbor身份驗(yàn)證異常,docker login檢查.檢查daemon.json和config.json
2.1 ping harbor
直接到node2節(jié)點(diǎn)ping
root@k8s-node-2:~# ping harbor.intra.com -c 3
PING harbor.intra.com (192.168.31.189) 56(84) bytes of data.
64 bytes from harbor.intra.com (192.168.31.189): icmp_seq=1 ttl=64 time=0.249 ms
64 bytes from harbor.intra.com (192.168.31.189): icmp_seq=2 ttl=64 time=1.36 ms
64 bytes from harbor.intra.com (192.168.31.189): icmp_seq=3 ttl=64 time=0.108 ms
現(xiàn)在確定ping是正常的,那么至少服務(wù)器我們開(kāi)了
2.2 檢查harbor服務(wù)是否正常
用curl測(cè)試下harbor是否通
root@k8s-node-2:~# curl https://harbor.intra.com/harbor -k
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>Harbor</title>
<base href="/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="icon" type="image/x-icon" href="favicon.ico?v=2">
<link rel="preload" as="style" href="./light-theme.css?buildTimestamp=1639627836207">
<link rel="preload" as="style" href="./dark-theme.css?buildTimestamp=1639627836207">
<link rel="stylesheet" href="styles.e71e5822ddf4adf262c4.css"></head>
<body>
<harbor-app>
<div class="spinner spinner-lg app-loading app-loading-fixed">
Loading...
</div>
</harbor-app>
<script src="runtime.5ed5a3869dd69991407a.js" defer></script><script src="polyfills.a5e9bc0ea6dbbbdc0878.js" defer></script><script src="scripts.fc1928a0f22676249790.js" defer></script><script src="main.8b949aee92f43fe7c3ab.js" defer></script></body>
這里我們確認(rèn)了harbor服務(wù)是正常的,通過(guò)網(wǎng)頁(yè)也能訪問(wèn)到
2.3 docker login harbor
此時(shí)發(fā)生了報(bào)錯(cuò),明顯是驗(yàn)證失敗了.且失敗原因是沒(méi)有授權(quán)的倉(cāng)庫(kù)造成的.
root@k8s-node-2:~# docker login https://harbor.intra.com
Password: ting with existing credentials...
Error: Password Requiredrror: Error response from daemon: Get "https://harbor.intra.com/v2/": x509: certificate signed by unknown authority
那么我們到node1上試下是否登錄harbor成功
root@k8s-node-1:~# docker login https://harbor.intra.com
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
那么我們將node1的/etc/docker/daemon.json同步到node2上
root@k8s-node-1:~# scp /etc/docker/daemon.json 192.168.31.112:/etc/docker/daemon.json
root@192.168.31.112's password:
daemon.json
再到node2上重啟docker服務(wù),此時(shí)可以看到授權(quán)的鏡像倉(cāng)庫(kù)里有了https://harbor.intra.com/
root@k8s-node-2:~# systemctl restart docker
root@k8s-node-2:~# docker info |tail -10
WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
192.168.31.0/24
Registry Mirrors:
https://docker.mirrors.ustc.edu.cn/
http://hub-mirror.c.163.com/
https://harbor.intra.com/
https://192.168.31.189/
Live Restore Enabled: true
3. 解決
嘗試再次登錄harbor并拉取鏡像文章來(lái)源：http://www.zghlxwxcb.cn/news/detail-822885.html
root@k8s-node-2:~# docker login https://harbor.intra.com
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
root@k8s-node-2:~# docker pull harbor.intra.com/baseimages/kubernetesui/dashboard:v2.4.0
v2.4.0: Pulling from baseimages/kubernetesui/dashboard
Digest: sha256:2d2ac5c357a97715ee42b2186fda39527b826fdd7df9f7ade56b9328efc92041
Status: Image is up to date for harbor.intra.com/baseimages/kubernetesui/dashboard:v2.4.0
harbor.intra.com/baseimages/kubernetesui/dashboard:v2.4.0
此時(shí)dashboard pod狀態(tài)也變成了Running文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-822885.html
root@k8s-master-01:~# kubectl get pod -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
dashboard-metrics-scraper-6848d4dd7d-g7k6b 1/1 Running 4 (49m ago) 226d
kubernetes-dashboard-6948fdc5fd-7szc9 1/1 Running 0 6m2s
到了這里，關(guān)于K8s Error: ImagePullBackOff 故障排除的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
