集群環(huán)境準(zhǔn)備

主機(jī)規(guī)劃

軟件版本

網(wǎng)絡(luò)分配

參考：25.基于二進(jìn)制包安裝kubernetes v1.21 --集群部署(二) - 掘金 (juejin.cn)

安裝Rocky 8

Vmware 安裝

Node網(wǎng)絡(luò)用的是NAT模式，然后又加了一個(gè)網(wǎng)卡，用于僅主機(jī)模式

系統(tǒng)初始化

升級(jí)系統(tǒng)

yum update -y --exclude=kernel*

更新內(nèi)核

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum install  -y https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
yum install -y https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm

yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
yum --enablerepo=elrepo-kernel install kernel-lt -y

查看目前可用內(nèi)核

awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg

使用序號(hào)為0的內(nèi)核，序號(hào)0是前面查出來的可用內(nèi)核編號(hào)

grub2-set-default 0

生成 grub 配置文件并重啟

grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

修改網(wǎng)絡(luò)IP

保證每臺(tái)機(jī)器的網(wǎng)絡(luò)UUID不同, 去掉UUID

vim /etc/sysconfig/network-scripts/ifcfg-ens192

... 

BOOTPROTO=static
...
ONBOOT=yes
IPADDR=10.58.32.31
NETMASK=255.255.255.0

nmcli connection reload

基礎(chǔ)配置

# 關(guān)閉防火墻
systemctl stop firewalld
systemctl disable firewalld

# 安裝vim
yum install -y vim lrzsz

# 關(guān)閉selinux
sed -i 's/SELINUX=.*$/SELINUX=disabled/g'  /etc/selinux/config

# 關(guān)閉swap
swapoff -a 
sed -ri 's/.*swap.*/#&/' /etc/fstab

# 設(shè)置 open file
cat >> /etc/security/limits.conf << EOF

* hard noproc 65536
* soft noproc 65536
* hard nofile 65536
* soft nofile 65536
EOF

編輯hosts文件

cat >>  /etc/hosts <<EOF

10.58.32.30 k8s-master-lb
10.58.32.31 k8s-master01
10.58.32.32 k8s-master02
10.58.32.33 k8s-master03
10.58.32.34 k8s-node01
10.58.32.35 k8s-node02
EOF

ntp配置

yum install epel-release -y
yum -ivh https://mirrors.wlnmp.com/rockylinux/wlnmp-release-rocky-8.noarch.rpm
yum install wntp -y

ntpdate time2.aliyun.com

crontab -e
*/5 * * * * ntpdate time2.aliyun.com

cat >> /etc/rc.local << EOF

ntpdate time2.aliyun.com
EOF

chmod +x /etc/rc.local

系統(tǒng)優(yōu)化

cat > /etc/sysctl.d/k8s_better.conf << EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

sysctl -p /etc/sysctl.d/k8s_better.conf

modprobe br_netfilter
lsmod |grep conntrack
modprobe ip_conntrack

安裝IPVS

yum install -y conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git

## 開啟ipvs 轉(zhuǎn)發(fā)
modprobe br_netfilter 

cat > /etc/sysconfig/modules/ipvs.modules << EOF 
#!/bin/bash 
modprobe -- ip_vs 
modprobe -- ip_vs_rr 
modprobe -- ip_vs_lc 
modprobe -- ip_vs_lblc 
modprobe -- ip_vs_dh
modprobe -- ip_vs_fo 
modprobe -- ip_vs_nq 
modprobe -- ip_vs_wrr 
modprobe -- ip_vs_sh 
modprobe -- ip_vs_sed 
modprobe -- ip_vs_ftp
modprobe -- nf_conntrack
modprobe -- ip_tables
modprobe -- ip_set
modprobe -- xt_set
modprobe -- ipt_set
modprobe -- ipt_rpfilter
modprobe -- ipt_REJECT
modprobe -- ipip
EOF 

chmod 755 /etc/sysconfig/modules/ipvs.modules 
bash /etc/sysconfig/modules/ipvs.modules 
lsmod | grep -e ip_vs -e nf_conntrack

特別注意，如果內(nèi)核是4.19以下的，nf_conntrack 要改為nf_conntrack_ipv4

安裝docker

yum install -y yum-utils device-mapper-persistent-data lvm2
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo 

yum list docker-ce --showduplicates
yum install docker-ce-19.03.13-3.el8

修改從cgroup方式

cat > /etc/docker/daemon.json <<EOF
{
    "registry-mirrors":[
        "https://registry.docker-cn.com",
        "https://hub-mirrors.c.163.com",
        "https://docker.mirrors.utsc.edu.cn"
    ],
    "exec-opts":["native.cgroupdriver=systemd"],
    "max-concurrent-downloads": 10,
    "max-concurrent-uploads": 5,
    "log-opts":{
        "max-size": "300m",
        "max-file":"2"
    },
    "live-restore": true
}
EOF

然后

systemctl daemon-reload && systemctl restart docker && systemctl enable docker

master01打通ssh到其他節(jié)點(diǎn)

master01節(jié)點(diǎn)執(zhí)行

ssh-keygen

# 執(zhí)行所有節(jié)點(diǎn)
for node in 10.58.32.{31..35};do
    ssh-copy-id root@${node}
done

下載k8s-server

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md #server-binaries

下載kubernetes-server-linux-amd64.tar.gz 并上傳到所有節(jié)點(diǎn)

tar -xvf kubernetes-server-linux-amd64.tar.gz
mv kubernetes /

vim /etc/profile.d/k8s.sh

export K8S_HOME=/kubernetes
export K8S_SERVER=$K8S_HOME/server
export PATH=$PATH:$K8S_SERVER/bin

chmod +x /etc/profile.d/k8s.sh
ssource /etc/profile.d/k8s.sh

生成證書

master01節(jié)點(diǎn)下載cfssl

創(chuàng)建工作目錄
mkdir /data/k8s-work
cd /data/k8s-work

下載地址： https://github.com/cloudflare/cfssl/releases?page=1
cfssl_1.6.3_linux_amd64
cfssljson_1.6.3_linux_amd64

移動(dòng)到指定目錄

chmod +x cfssl_1.6.3_linux_amd64 cfssljson_1.6.3_linux_amd64
cp cfssl_1.6.3_linux_amd64 /usr/local/bin/cfssl
cp cfssljson_1.6.3_linux_amd64 /usr/local/bin/cfssljson

創(chuàng)建證書保存目錄

# 所有節(jié)點(diǎn)
mkdir -p /etc/etcd/ssl
mkdir -p /etc/kubernetes/pki

下載開源工具包

git clone https://github.com/dotbalo/k8s-ha-install.git

cd k8s-ha-install
git branch -a
git checkout remotes/origin/manual-installation-v1.20.x

生成etcd證書

cd /root/k8s-ha-install/pki

[root@k8s-master01 pki]# ls
admin-csr.json ca-csr.json  front-proxy-ca-csr.json  kube-proxy-csr.json apiserver-csr.json  etcd-ca-csr.json  front-proxy-client-csr.json  manager-csr.json  ca-config.json etcd-csr.json  kubelet-csr.json  scheduler-csr.json

etcd的證書和k8s的證書是完全獨(dú)立的，沒有任何關(guān)系

CSR文件：證書簽名請(qǐng)求文件

# ca證書
cfssl gencert -initca etcd-ca-csr.json|cfssljson -bare /etc/etcd/ssl/etcd-ca
# 客戶端證書
cfssl gencert \
-ca=/etc/etcd/ssl/etcd-ca.pem \
-ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
-config=ca-config.json \
-hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,10.58.32.31,10.58.32.32,10.58.32.33 \
-profile=kubernetes \
etcd-csr.json|cfssljson -bare /etc/etcd/ssl/etcd

生成kubernetes證書

# ca證書
cfssl gencert -initca ca-csr.json|cfssljson -bare /etc/kubernetes/pki/ca

# apiserver需要證書
# 10.96.0是k8s service的網(wǎng)段，如果需要更改k8s service的網(wǎng)段，就需要更改10.96.0.1
# 10.58.32.30 vip網(wǎng)段
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-hostname=10.96.0.1,10.58.32.30,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,10.58.32.31,10.58.32.32,10.58.32.33 \
-profile=kubernetes \
apiserver-csr.json|cfssljson -bare /etc/kubernetes/pki/apiserver


# 生成controller-manage的證書
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
manager-csr.json|cfssljson -bare /etc/kubernetes/pki/controller-manager

# 生成scheduler證書
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
scheduler-csr.json|cfssljson -bare /etc/kubernetes/pki/scheduler


# 生成admin的證書
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
admin-csr.json|cfssljson -bare /etc/kubernetes/pki/admin

生成apiserver聚合證書

# ca證書
cfssl gencert -initca front-proxy-ca-csr.json|cfssljson -bare /etc/kubernetes/pki/front-proxy-ca

# 生成聚合證書
# Requestheader-client-xxx    requestheader-allowed-xxx:aggerator
cfssl gencert \
-ca=/etc/kubernetes/pki/front-proxy-ca.pem \
-ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
front-proxy-client-csr.json|cfssljson -bare /etc/kubernetes/pki/front-proxy-client

生成kubeconfig文件

生成controller的kubeconfig

# 設(shè)置集群項(xiàng)
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 設(shè)置用戶項(xiàng)
kubectl config set-credentials system:kube-contoller-manager \
--client-certificate=/etc/kubernetes/pki/controller-manager.pem \
--client-key=/etc/kubernetes/pki/controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 設(shè)置環(huán)境項(xiàng)，上下文
kubectl config set-context system:kube-contoller-manager@kubernetes \
--cluster=kubernetes \
--user=system:kube-contoller-manager \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 將上面的環(huán)境設(shè)置為默認(rèn)環(huán)境
kubectl config use-context system:kube-contoller-manager@kubernetes \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

生成schedule的kubeconfig

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 設(shè)置用戶項(xiàng)
kubectl config set-credentials system:kube-scheduler \
--client-certificate=/etc/kubernetes/pki/scheduler.pem \
--client-key=/etc/kubernetes/pki/scheduler-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 設(shè)置環(huán)境項(xiàng)，上下文
kubectl config set-context system:kube-scheduler@kubernetes \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 將上面的環(huán)境設(shè)置為默認(rèn)環(huán)境
kubectl config use-context system:kube-scheduler@kubernetes \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

生成admin的kubeconfig

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

# 設(shè)置用戶項(xiàng)
kubectl config set-credentials kubernetes-admin \
--client-certificate=/etc/kubernetes/pki/admin.pem \
--client-key=/etc/kubernetes/pki/admin-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

# 設(shè)置環(huán)境項(xiàng)，上下文
kubectl config set-context kubernetes-admin@kubernetes \
--cluster=kubernetes \
--user=kubernetes-admin \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

# 將上面的環(huán)境設(shè)置為默認(rèn)環(huán)境
kubectl config use-context kubernetes-admin@kubernetes \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

生成ServiceAccount key

openssl genrsa -out /etc/kubernetes/pki/sa.key 2048

openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

拷貝證書到其他master節(jié)點(diǎn)

cd /etc/kubernetes/pki
scp -r ./* root@10.58.32.33:/etc/kubernetes/

配置etcd

一定要配置奇數(shù)個(gè)，否則會(huì)產(chǎn)生腦裂

一下操作master節(jié)點(diǎn)都要做

下載etcd

wget https://github.com/etcd-io/etcd/releases/download/v3.4.24/etcd-v3.4.24-linux-amd64.tar.gz

tar -xvf etcd-v3.4.24-linux-amd64.tar.gz
mv etcd-v3.4.24-linux-amd64 etcd-v3.4.24
mv etcd-v3.4.24 /kubernetes/

添加到PATH

vim /etc/profile.d/k8s.sh

export K8S_HOME=/kubernetes
export K8S_SERVER=$K8S_HOME/server
export PATH=$PATH:$K8S_SERVER/bin:$K8S_HOME/etcd-v3.4.24

source /etc/profile.d/k8s.sh

配置文件

配置文件每個(gè)節(jié)點(diǎn)多少有點(diǎn)不同，/etc/etcd/etcd.config.yaml

k8s-master01

name: 'k8s-master01'
data-dir: /kubernetes/etcd-v3.4.24/data
wal-dir: /kubernetes/etcd-v3.4.24/data/wal
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: https://10.58.32.31:2380
listen-client-urls: https://10.58.32.31:2379,http://127.0.0.1:2379
max-snapshots: 5
max-wals: 5
cors:
initial-advertise-peer-urls: https://10.58.32.31:2380
advertise-client-urls: https://10.58.32.31:2379
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.58.32.31:2380,k8s-master02=https://10.58.32.32:2380,k8s-master03=https://10.58.32.33:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
debug: false
logger: zap
log-outputs: [default]
force-new-cluster: false
auto-compaction-mode: periodic
auto-compaction-retention: "1"

k8s-master02

name: 'k8s-master02'
data-dir: /kubernetes/etcd-v3.4.24/data
wal-dir: /kubernetes/etcd-v3.4.24/data/wal
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: https://10.58.32.32:2380
listen-client-urls: https://10.58.32.32:2379,http://127.0.0.1:2379
max-snapshots: 5
max-wals: 5
cors:
initial-advertise-peer-urls: https://10.58.32.32:2380
advertise-client-urls: https://10.58.32.32:2379
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.58.32.31:2380,k8s-master02=https://10.58.32.32:2380,k8s-master03=https://10.58.32.33:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
debug: false
logger: zap
log-outputs: [default]
force-new-cluster: false
auto-compaction-mode: periodic
auto-compaction-retention: "1"

k8s-master03

name: 'k8s-master03'
data-dir: /kubernetes/etcd-v3.4.24/data
wal-dir: /kubernetes/etcd-v3.4.24/data/wal
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: https://10.58.32.33:2380
listen-client-urls: https://10.58.32.33:2379,http://127.0.0.1:2379
max-snapshots: 5
max-wals: 5
cors:
initial-advertise-peer-urls: https://10.58.32.33:2380
advertise-client-urls: https://10.58.32.33:2379
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.58.32.31:2380,k8s-master02=https://10.58.32.32:2380,k8s-master03=https://10.58.32.33:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
debug: false
logger: zap
log-outputs: [default]
force-new-cluster: false
auto-compaction-mode: periodic
auto-compaction-retention: "1"

啟動(dòng)system文件

cat > /etc/systemd/system/etcd.service <<"EOF"
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/kubernetes/etcd-v3.4.24/etcd --config-file=/etc/etcd/etcd.config.yaml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF

啟動(dòng)etcd集群

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd

驗(yàn)證集群狀態(tài)

ETCDCTL_API=3 /kubernetes/etcd-v3.4.24/etcdctl --write-out=table --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://10.58.32.31:2379,https://10.58.32.32:2379,https://10.58.32.33:2379 endpoint health

負(fù)載均衡器準(zhǔn)備

安裝haproxy與keepalived

yum -y install haproxy keepalived

HAProxy配置

cat >/etc/haproxy/haproxy.cfg<<"EOF"
global
 maxconn 2000
 ulimit-n 16384
 log 127.0.0.1 local0 err
 stats timeout 30s

defaults
 log global
 mode http
 option httplog
 timeout connect 5000
 timeout client 50000
 timeout server 50000
 timeout http-request 15s
 timeout http-keep-alive 15s

frontend monitor-in
 bind *:33305
 mode http
 option httplog
 monitor-uri /monitor

frontend k8s-master
 bind 0.0.0.0:6443
 bind 127.0.0.1:6443
 mode tcp
 option tcplog
 tcp-request inspect-delay 5s
 default_backend k8s-master

backend k8s-master
 mode tcp
 option tcplog
 option tcp-check
 balance roundrobin
 default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
 server  k8s-master01  10.58.32.31:6443 check
 server  k8s-master02  10.58.32.32:6443 check
 server  k8s-master03  10.58.32.33:6443 check
EOF

Keepalived

主從配置不一致，需要注意。

云上不支持keepalived

k8s-master01

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state MASTER
   interface ens192
   mcast_src_ip 10.58.32.31
   virtual_router_id 51
   priority 100
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       10.58.32.30
   }
   track_script {
      chk_apiserver
   }
}
EOF

k8s-master02，k8s-master02

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
  interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state BACKUP
   interface ens192
   mcast_src_ip 10.58.32.32,33 # 注意32,33是兩個(gè)IP后綴
   virtual_router_id 51
   priority 99
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       10.58.32.30
   }
   track_script {
      chk_apiserver
   }
}
EOF

健康檢查腳本

master節(jié)點(diǎn)均要配置

cat > /etc/keepalived/check_apiserver.sh <<"EOF"
#!/bin/bash
err=0
for k in $(seq 1 3)
do
   check_code=$(pgrep haproxy)
   if [[ $check_code == "" ]]; then
       err=$(expr $err + 1)
       sleep 1
       continue
   else
       err=0
       break
   fi
done

if [[ $err != "0" ]]; then
   echo "systemctl stop keepalived"
   /usr/bin/systemctl stop keepalived
   exit 1
else
   exit 0
fi
EOF

chmod +x /etc/keepalived/check_apiserver.sh

啟動(dòng)服務(wù)并驗(yàn)證

systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

ip address show

k8s組件安裝

創(chuàng)建相關(guān)目錄

mkdir -p /etc/kubernetes/manifests  /var/lib/kubelet /var/log/kubernetes

apiserver部署

該部署所有master節(jié)點(diǎn)都要執(zhí)行

配置啟動(dòng)腳本

注意master節(jié)點(diǎn)的advertise-address不同，需要修改

vim /etc/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.service

[Service]
ExecStart=/kubernetes/server/bin/kube-apiserver \
  --v=2 \
  --logtostderr=true \
  --anonymous-auth=false \
  --bind-address=0.0.0.0 \
  --secure-port=6443 \
  --insecure-port=0 \
  --advertise-address=10.58.32.31 \
  --allow-privileged=true \
  --service-cluster-ip-range=10.96.0.0/16 \
  --service-node-port-range=30000-32767 \
  --etcd-servers=https://10.58.32.31:2379,https://10.58.32.32:2379,https://10.58.32.33:2379 \
  --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --client-ca-file=/etc/kubernetes/pki/ca.pem \
  --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
  --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/pki/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/pki/ca-key.pem  \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --authorization-mode=Node,RBAC \
  --enable-bootstrap-token-auth \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --requestheader-allowed-names=aggregator \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
  --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem


Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

啟動(dòng)并驗(yàn)證

systemctl daemon-reload
systemctl enable --now kube-apiserver

systemctl status kube-apiserver

controller-manager部署

配置啟動(dòng)腳本

注意master節(jié)點(diǎn)的advertise-address不同，需要修改

vim /etc/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.service

[Service]
ExecStart=/kubernetes/server/bin/kube-controller-manager \
  --v=2 \
  --logtostderr=true \
  --bind-address=127.0.0.1 \
  --root-ca-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
  --service-account-private-key-file=/etc/kubernetes/pki/ca-key.pem \
  --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
  --leader-elect=true \
  --use-service-account-credentials=true \
  --node-monitor-grace-period=40s \
  --node-monitor-period=5s \
  --pod-eviction-timeout=2m0s \
  --controllers=*,bootstrapsigner,tokencleaner \
  --allocate-node-cidrs=true \
  --cluster-cidr=172.16.0.0/12 \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --node-cidr-mask-size=24 \
  --tls-cert-file=/etc/kubernetes/pki/controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/pki/controller-manager-key.pem


Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

systemctl daemon-reload 
systemctl enable --now kube-controller-manager
systemctl status kube-controller-manager

schedule部署

cat >/etc/systemd/system/kube-scheduler.service<<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.service


[Service]
ExecStart=/kubernetes/server/bin/kube-scheduler \
	--v=2 \
	--address=127.0.0.1 \
	--leader-elect=true \
	--logtostderr=true \
	--kubeconfig=/etc/kubernetes/scheduler.kubeconfig


Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload 
systemctl enable --now kube-scheduler
systemctl status kube-scheduler

TLS bootstrap自動(dòng)頒發(fā)證書

注意： 如果不是高可用集群，vip要改成master01的IP，8443端口，改成默認(rèn)的6443端口

要特別注意--token參數(shù)，是token_id.token_secret

自動(dòng)給kubectl頒發(fā)的證書有效期是5年，到期會(huì)自動(dòng)申請(qǐng)

cd /root/k8s-ha-install/bootstrap

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap \
--token=c8ad9c.2e4d610cf3e7426e \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

kubectl config set-context tls-bootstrap-token-user@kubernetes \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

kubectl config use-context tls-bootstrap-token-user@kubernetes \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

如果要修改bootstrap.secret.yaml的token-id和token-secret，要保證下面中的字符串保持一致，并且位數(shù)一樣，還要保證上面命令中token信息和下面一致：

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-c8ad9c
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  description: "The default bootstrap token generated by 'kubelet '."
  token-id: c8ad9c  # 注意這個(gè)
  token-secret: 2e4d610cf3e7426e  # 注意這個(gè)
  usage-bootstrap-authentication: "true"
... ...

然后配置kubectl的admin權(quán)限

mkdir -p /root/.kube
cp /etc/kubernetes/admin.kubeconfig /root/.kube/config

cd /root/k8s-ha-install/bootstrap
kubectl apply -f bootstrap.secret.yaml

secret/bootstrap-token-c8ad9c created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-certificate-rotation created
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created

Node節(jié)點(diǎn)配置

拷貝證書

# node節(jié)點(diǎn)上
$ mkdir -p /etc/kubernetes/pki
$ mkdir -p /etc/etcd/ssl

# master01節(jié)點(diǎn)上
cd /etc/etcd/ssl
scp ./* root@10.58.32.34:/etc/etcd/ssl/
scp ./* root@10.58.32.35:/etc/etcd/ssl/
# master01節(jié)點(diǎn)上
for i in 10.58.32.{32..35};then do
  scp pki/ca.pem root@$i:/etc/kubernetes/pki/
  scp pki/ca-key.pem root@$i:/etc/kubernetes/pki
  scp pki/front-proxy-ca.pem root@$i:/etc/kubernetes/pki/
  scp kubelet-bootstrap.kubeconfig root@$i:/etc/kubernetes/
done

所有節(jié)點(diǎn)執(zhí)行

mkdir -p /etc/kubernetes/manifests /var/lib/kubelet  /etc/systemd/system/kubelet.service.d /var/log/kubernetes

部署kubelet

vim /etc/kubernetes/kubelet-conf.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
  - 10.96.0.10
clusterDomain: cluster.local
containerLogMaxSize: 20Mi
containerLogMaxFiles: 5
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableDebuggingHandlers: true
enableControllerAttachDetach: true
enforceNodeAllocatable: 
- pods
eventRecordQPS: 5
eventBurst: 20
evictionHard:
  memory.available:  100Mi
  imagefs.available: 15%
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
fileCheckFrequency: 20s
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIQPS: 5
kubeAPIBurst: 10
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryPullQPS: 5
registryBurst: 10
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 10m
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionldleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

添加kubelet啟動(dòng)腳本

vim /etc/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/kubernetes/server/bin/kubelet

Restart=always
StartLimitInterval=0
RestartSec=5


[Install]
WantedBy=multi-user.target

vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig "
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin "
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yaml --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.4.1 "
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/kubernetes/server/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

然后

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet

出現(xiàn)如下報(bào)錯(cuò)，并且有如下報(bào)錯(cuò)內(nèi)容

Unable to register node "k8s-node02" with API server: Unauthorized

說明是認(rèn)證的問題，重置TSL bootstrap有效

部署kube-proxy

只在master01節(jié)點(diǎn)執(zhí)行就OK

注意：如果不是高可用集群，10.58.32.30:8443 的IP改成master01的IP，端口改為6443端口

kubectl -n kube-system create serviceaccount kube-proxy

kubectl create clusterrolebinding system:kube-proxy   \
--clusterrole system:node-proxier  \
--serviceaccount kube-system:kube-proxy

SECRET=$(kubectl -n kube-system get sa/kube-proxy \
    --output=jsonpath='{.secrets[0].name}')
JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET \
--output=jsonpath='{.data.token}' | base64 -d)
PKI_DIR=/etc/kubernetes/pki
K8S_DIR=/etc/kubernetes


kubectl config set-cluster kubernetes   \
--certificate-authority=/etc/kubernetes/pki/ca.pem  \
--embed-certs=true   \
--server=https://10.58.32.30:8443   \
--kubeconfig=${K8S_DIR}/kube-proxy.kubeconfig

kubectl config set-credentials kubernetes  \
--token=${JWT_TOKEN}   \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig


kubectl config set-context kubernetes  \
--cluster=kubernetes  \
--user=kubernetes \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig


kubectl config use-context kubernetes  \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

然后切換目錄

cd /root/k8s-ha-install/kube-proxy
ll 
-rw-r--r-- 1 root root  813 Mar 14 11:15 kube-proxy.conf
-rw-r--r-- 1 root root  288 Mar 14 11:15 kube-proxy.service
-rw-r--r-- 1 root root 3677 Mar 14 11:15 kube-proxy.yml

如果更改了集群Pod的網(wǎng)段，需要更改kube-proxy/kube-proxy.conf的clusterCIDR: 192.168.0.0/12參數(shù)為pod的網(wǎng)段（我們之前設(shè)置的）。

vim kube-proxy.conf

apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/12
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms

然后是kube-proxy.service，修改可執(zhí)行文件地址為我們的地址

vim kube-proxy.service

[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/kubernetes/server/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.conf \
  --v=2

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

修改完畢，把他們發(fā)送到所有節(jié)點(diǎn)

for node in 10.58.32.{31..35};do
scp kube-proxy.conf root@$node:/etc/kubernetes/
scp kube-proxy.service root@$node:/etc/systemd/system/
scp /etc/kubernetes/kube-proxy.kubeconfig root@$node:/etc/kubernetes/
done

然后所有節(jié)點(diǎn)啟動(dòng)kube-proxy

systemctl daemon-reload && systemctl enable --now kube-proxy
systemctl status kube-proxy

部署calico

以下步驟只在master執(zhí)行一次

cd /root/k8s-ha-install/calico

cp calico-etcd.yaml calico-etcd.yaml_backup

ETCD_CA=`cat /etc/etcd/ssl/etcd-ca.pem | base64 | tr -d '\n'`
ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n'`
ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n'`

sed -i "s#.*etcd-ca:.*#  etcd-ca: ${ETCD_CA}#g" calico-etcd.yaml
sed -i "s#.*etcd-cert:.*#  etcd-cert: ${ETCD_CERT}#g" calico-etcd.yaml
sed -i "s#.*etcd-key:.*#  etcd-key: ${ETCD_KEY}#g" calico-etcd.yaml

sed -i 's#.*etcd_ca:.*#  etcd_ca: "/calico-secrets/etcd-ca"#g' calico-etcd.yaml
sed -i 's#.*etcd_cert:.*#  etcd_cert: "/calico-secrets/etcd-cert"#g' calico-etcd.yaml
sed -i 's#.*etcd_key:.*#  etcd_key: "/calico-secrets/etcd-key"#g' calico-etcd.yaml

sed -i "s#__ETCD_CA_CERT_FILE__#/etc/etcd/ssl/etcd-ca.pem#g" calico-etcd.yaml
sed -i "s#__ETCD_CERT_FILE__#/etc/etcd/ssl/etcd.pem#g" calico-etcd.yaml
sed -i "s#__ETCD_KEY_FILE__#/etc/etcd/ssl/etcd-key.pem#g" calico-etcd.yaml
sed -i "s#__KUBECONFIG_FILEPATH__#/etc/cni/net.d/calico-kubeconfig#g" calico-etcd.yaml


$ vim calico-etcd.yaml 

注意修改如下部分：

kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  etcd_endpoints: "https://10.58.32.31:2379,https://10.58.32.32:2379,https://10.58.32.33:2379"
  etcd_ca: ""
  etcd_cert: ""
  etcd_key: ""
   ... ...
  
  - name: CALICO_IPV4POOL_CIDR
    value: "172.16.0.0/12"  # pod的IP

出現(xiàn)如下報(bào)錯(cuò)，可解決

參考該博客解決問題： 簡單部署3節(jié)點(diǎn)Kubernetes(k8s)集群之calico網(wǎng)絡(luò)組件 - 簡書 (jianshu.com)

部署CoreDNS

cd /root/k8s-ha-install/CoreDNS

把網(wǎng)關(guān)修改成我們自己的service的網(wǎng)關(guān)

vim coredns.yaml

spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.96.0.10
  ports:
  - name: dns

然后安裝CoreDNS

kubectl apply -f coredns.yaml

部署Metircs Server

cd  /root/k8s-ha-install/metrics-server-0.4.x
kubectl apply -f comp.yaml

然后查看狀態(tài)

[root@k8s-master01 metrics-server-0.4.x]# kubectl get po -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
... ...
kube-system   metrics-server-595f65d8d5-txkkl            1/1     Running   0          103s

安裝圖形化工具

使用kuboard且使用docker話安裝

Kuboard介紹 | Kuboard

demo命令(具體查看官方文檔)：

docker run -d \
  --restart=unless-stopped \
  --name=kuboard \
  -p 80:80/tcp \
  -p 10081:10081/tcp \
  -e KUBOARD_ENDPOINT="http://10.58.32.32:80" \
  -e KUBOARD_AGENT_SERVER_TCP_PORT="10081" \
  -v /kubernetes/kuboard-data:/data \
  eipwork/kuboard:v3

集群可用性測試

1. Pod 必須能解析 Servicee

2. Pod 必須能解析跨 namespace 的 Service

3. 每個(gè)節(jié)點(diǎn)都必須要能訪問 Kubernetes 的kubermetes svc 443 和 kube-dns的 service 53

4. Pod 和Pod之前要能通

• 同namespace 能通信
• 跨 namespace 能通信
• 跨機(jī)器能通信

主機(jī)IP一定要用NAT模式，這樣才能解析/etc/resolv.conf中的網(wǎng)關(guān)

問題可參考：

搭建Kubernetes集群時(shí)DNS無法解析問題的處理過程 - 掘金 (juejin.cn)

自定義 DNS 服務(wù) | Kubernetes

調(diào)試 DNS 問題 | Kubernetes

kubectl exec busybox  -- nslookup kubernetes.default   # pod必須能解析service
kubectl exec busybox  -- nslookup kube-dns.kube-system # pod必須能解析跨namespace的service

telnet 10.96.0.1 443  # 對(duì)應(yīng)說明3
telnet 10.96.0.10 53

# 對(duì)應(yīng)第三點(diǎn)，進(jìn)入一個(gè)pod，去ping其他機(jī)器或者pod
kubectl exec -it busybox -- sh   # 跨namespace/node
/ # ping 10.58.32.31

# 驗(yàn)證deployment部署
kubectl create deploy nginx-deployment --image=nginx --replicas=3
kubectl delete nginx-deployment

生產(chǎn)環(huán)境關(guān)鍵性配置

docker容器daemon.json

vim /etc/docker/daemon.json

{
    "registry-mirrors":[
        "https://registry.docker-cn.com",
        "https://hub-mirrors.c.163.com",
        "https://docker.mirrors.utsc.edu.cn"
    ],
    "exec-opts":["native.cgroupdriver=systemd"],
    "max-concurrent-downloads": 10, //下載最大線程數(shù)
    "max-concurrent-uploads": 5, // 上傳最大線程數(shù)
    "log-opts":{
        "max-size": "300m",
        "max-file":"2"
    },
    "live-restore": true    // 重啟docker，保證運(yùn)行中的容器不重啟
}

controller-manager證書有效期

增長證書簽發(fā)有效期

vim /etc/systemd/system/kube-controller-manager.service
# 添加
--experimental-cluster-signing-duration=87600h0m0s \

kubelet加密及資源配置

vim /etc/systemd/system/kube-kubelet.service
# 添加參數(shù)
# 增強(qiáng)kubelet加密強(qiáng)度
--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
# 延長鏡像下載時(shí)間（有些鏡像下載很慢）
--image-pull-progress-deadline=30m

對(duì)應(yīng)的配置文件：

allowdUnsafeSysctls 按需配置，會(huì)修改內(nèi)核參數(shù)，

kubeReserved，systemReserved是給k8s和系統(tǒng)預(yù)留的資源

vim /etc/kubernetes/kubelet-config.yaml

# 添加如下配置
allowdUnsafeSysctls:
- "net.core*"
- "net.ipv4.*"
kubeReserved:
 cpu: "10m"
 memory: 1Gi
 ephemeral-storage: 10Gi
systemReserved:
 cpu: "1"
 memory: 1Gi
 ephemeral-storage: 10Gi

額外的說明

給master和worker節(jié)點(diǎn)設(shè)置roler

[root@k8s-master01 ~]# kubectl label nodes k8s-master02 node-role.kubernetes.io/master=''
node/k8s-master02 labeled
[root@k8s-master01 ~]# kubectl label nodes k8s-master03 node-role.kubernetes.io/master=''
node/k8s-master03 labeled
[root@k8s-master01 ~]# kubectl label nodes k8s-node01 node-role.kubernetes.io/woker=''
node/k8s-node01 labeled
[root@k8s-master01 ~]# kubectl label nodes k8s-node02 node-role.kubernetes.io/woker=''
node/k8s-node02 labeled


$ kubectl get node --show-labels

openssl 查看證書有效期

$ openssl x509 -in ca.pem -noout dates

TLS bootstrapping頒發(fā)續(xù)簽/kubelet啟動(dòng)過程

官方文檔：TLS 啟動(dòng)引導(dǎo) | Kubernetes

1. Kubelet 啟動(dòng)

2. Kubelet 查看 kubelet.kubeconfig,文件，假設(shè)沒有這個(gè)文件

3. Kubelet會(huì)查看本地的 bootstrap.kubeconfig

4. Kubelet讀取 bootstrap.kubeconfig 文件，檢索 apiserver 的 ur和一個(gè) token

5. Kubelet鏈接 apiserver，使用這個(gè) token 進(jìn)行認(rèn)證

   • Apiserver會(huì)識(shí)別 tokenid， apiserver 會(huì)查看該 tokenid 對(duì)無的bootstrap 的一個(gè) secret
   • 找個(gè)這個(gè)secret 中的一個(gè)字段，apiserver ,把這個(gè) token 識(shí)別成一個(gè) username，名稱是system:bootstrap:<token-id>，屬于 system:bootstrappers 這個(gè)組，這個(gè)組具有申請(qǐng) csr的權(quán)限，該組的權(quán)限綁定在一個(gè)叫system:nodebootstrapper的 custerroles
     • clusterrole k8s 集群級(jí)別的權(quán)限控制，它作用整個(gè) k8s 集群
     • clusterrolebinding 集群權(quán)限的綁定，它可以幫某個(gè)clusterrole綁定到一個(gè)用戶組或者 seviceaccount
   • CSR:相當(dāng)于一個(gè)申請(qǐng)表，可以拿著這個(gè)申請(qǐng)表去申請(qǐng)我們的證書。

6. 經(jīng)過上面的認(rèn)證，kubelet 就有了一個(gè)創(chuàng)建和檢索 CSR 的權(quán)限

7. Kubelet 為自己創(chuàng)建一個(gè) CSR，名稱為 kubernetesio/kube-apiserver-client-kubelet

8. CSR 被允許有兩種方式:

   • K8s管理員使用 kubectl手動(dòng)的頒發(fā)證書
   • 如果配置了相關(guān)權(quán)限，kube-controller-manager 會(huì)自動(dòng)同意。
     • Controller-manager 有一個(gè)CSRApprovingContraller。他會(huì)校驗(yàn) kubelet發(fā)來的 csr的 username和 group 是否有創(chuàng)建 csr的權(quán)限，而且還要驗(yàn)證簽發(fā)著是否是kubernetes.io/kube-apiserver-client-kubelet
     • Controller-manager 同意 CSR 請(qǐng)求

9. CSR 被同意后，controller-manager 創(chuàng)建 kubelet 的證書文件

10. Controller-manager 將證書更新至 CSr的 status 字段

11. Kubelet 從apiserver 獲取證書

12. Kubelet 從獲取到的 key 和證書文件創(chuàng)建 kubelet.kubeconfig

13. Kubelet 啟動(dòng)完成并正常工作

14. 可選:如果配置了自動(dòng)續(xù)期，kubelet 會(huì)在證書文件過期的時(shí)候利用之前的 kubeconfig 文件去申請(qǐng)一個(gè)新的證書，相當(dāng)于續(xù)約。

15. 新的證書被同意或簽發(fā)，取決于我們的配置

    • Kubelet 創(chuàng)建的 CSR 是屬于一個(gè)O（orgian）: system:nodes
    • CN: system:nodes:主機(jī)名

添加默認(rèn)路由

chmod +x /etc/rc.local

vim /etc/rc.local
route add default gw 10.58.32.2 ens192

resolv.conf不斷被修改

解決方法
/etc/sysconfig/network-scripts/ifcfg-eth0或者自己網(wǎng)卡配置文件增加一行：PEERDNS=no，然后重新啟動(dòng)網(wǎng)絡(luò)即可。修改后，重啟主機(jī)等操作便不會(huì)使/etc/resolv.conf被dhclient修改

# Generated by NetworkManager
search localdomain
nameserver 10.58.32.2

關(guān)于service注冊(cè)服務(wù)

Linux配置service服務(wù) - 裸睡的豬 - 博客園 (cnblogs.com)

Linux下服務(wù)的管理_vendor preset_liuchonghua的博客-CSDN博客

Systemd 入門教程：命令篇 - 阮一峰的網(wǎng)絡(luò)日志 (ruanyifeng.com)

多網(wǎng)卡集群修改calico通信網(wǎng)卡

vim calico.yaml文章來源地址http://www.zghlxwxcb.cn/news/detail-604951.html

env:
  ... ...
  - name: IP_AUTODETECTION_METHOD
    value: "interface=ens33"
  ... ...

到了這里，關(guān)于k8s1.20二進(jìn)制包安裝的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！