概述

目前來說，kubernetes集群搭建的方式很多，選擇一個穩(wěn)定的適合自己的很重要。目前使用kubeadm方式搭建k8s集群還是很常見的，使用kubeadm搭建可以很簡單差不多兩條命令就行，也可以稍微復雜一點做一些基礎(chǔ)優(yōu)化，本文將分享一下使用kubeadm搭建集群并做了一定的優(yōu)化。

安裝

本環(huán)境將使用centos7.6.1810 的系統(tǒng)安裝kubernetes1.19.16版本集群，也可以選擇稍高版本的系統(tǒng)和k8s版本。但建議不要使用centos7.4以一下，要不然就自己升級一下內(nèi)核版本。容器技術(shù)依賴于內(nèi)核技術(shù)，低版本系統(tǒng)部署和運行后可能出去一些奇怪的問題。

環(huán)境規(guī)劃

說明：

• 所有操作都用root用戶進行，系統(tǒng)盤盡量大，要不然就單獨修改docker的數(shù)據(jù)盤，不然到時候鏡像多了例如超過85%會被gc回收鏡像
• 高可用一般建議大于等于3臺的奇數(shù)臺，本環(huán)境我使用的是3臺master來做高可用

事前準備(每臺機器)

系統(tǒng)層面設(shè)置：全新干凈系統(tǒng)，只做了網(wǎng)絡(luò)和dns的相關(guān)配置。

• 關(guān)閉 所有防火墻和SELinux，否則后續(xù) K8S 掛載目錄時可能報錯 Permission denied。

systemctl disable --now firewalld NetworkManager
setenforce 0
sed -ri '/^[^#]*SELINUX=/s#=.+$#=disabled#' /etc/selinux/config
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

• 關(guān)閉 dnsmasq (可選)
  linux 系統(tǒng)開啟了 dnsmasq 后(如 GUI 環(huán)境)，將系統(tǒng) DNS Server 設(shè)置為 127.0.0.1，這會導致 docker 容器無法解析域名，需要關(guān)閉它

systemctl disable --now dnsmasq

• Kubernetes 建議關(guān)閉系統(tǒng)Swap，在所有機器使用以下指令關(guān)閉swap并注釋掉/etc/fstab中swap的行，不想關(guān)閉可以不執(zhí)行，后面會有應對的配置選項：

swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s&^&#&' /etc/fstab

• 安裝一些基礎(chǔ)依賴包和工具

yum install epel-release -y
yum install -y \
    curl \
    wget \
    git \
    conntrack-tools \
    psmisc \
    nfs-utils \
    jq \
    socat \
    bash-completion \
    ipset \
    ipvsadm \
    conntrack \
    libseccomp \
    net-tools \
    crontabs \
    sysstat \
    unzip \
    bind-utils \
    tcpdump \
    telnet \
    lsof \
    htop

如果集群kube-proxy想使用ipvs模式的話（ipvs效率更高）需要加載以下模塊，按照規(guī)范使用systemd-modules-load來加載而不是在/etc/rc.local里寫modprobe

> /etc/modules-load.d/ipvs.conf
module=(
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
br_netfilter
  )
for kernel_module in ${module[@]};do
/sbin/modinfo -F filename $kernel_module |& grep -qv ERROR && echo $kernel_module >> /etc/modules-load.d/ipvs.conf || :
done
systemctl restart systemd-modules-load.service

上面如果systemctl restart 命令報錯可以使用systemctl status -l systemd-modules-load.service看看哪個內(nèi)核模塊不能正常加載，然后在/etc/modules-load.d/ipvs.conf里注釋掉它再restart試試

• 所有機器需要設(shè)定/etc/sysctl.d/k8s.conf的系統(tǒng)參數(shù)，目前對ipv6支持不怎么好，所以這里將ipv6也關(guān)閉了。

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.ip_forward = 1
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
# 要求iptables不對bridge的數(shù)據(jù)進行處理
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
net.netfilter.nf_conntrack_max = 2310720
fs.inotify.max_user_watches=89100
fs.may_detach_mounts = 1
fs.file-max = 52706963
fs.nr_open = 52706963
vm.overcommit_memory=1
vm.panic_on_oom=0
EOF
sysctl --system

• 如果選擇關(guān)閉swap也要在內(nèi)核里關(guān)閉，不關(guān)閉可以不執(zhí)行

echo 'vm.swappiness = 0' >> /etc/sysctl.d/k8s.conf

• 如果kube-proxy使用ipvs的話為了防止timeout需要設(shè)置下tcp參數(shù)

cat <<EOF >> /etc/sysctl.d/k8s.conf
# https://github.com/moby/moby/issues/31208 
# ipvsadm -l --timout
# 修復ipvs模式下長連接timeout問題 小于900即可
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 10
EOF
sysctl --system

• 修改systemctl啟動的最小文件打開數(shù)量。關(guān)閉ssh方向dns解析

sed -ri 's/^#(DefaultLimitCORE)=/\1=100000/' /etc/systemd/system.conf
sed -ri 's/^#(DefaultLimitNOFILE)=/\1=100000/' /etc/systemd/system.conf

sed -ri 's/^#(UseDNS )yes/\1no/' /etc/ssh/sshd_config

• 文件最大打開數(shù)，按照規(guī)范，在子配置文件添加

cat>/etc/security/limits.d/kubernetes.conf<<EOF
*       soft    nproc   131072
*       hard    nproc   131072
*       soft    nofile  131072
*       hard    nofile  131072
root    soft    nproc   131072
root    hard    nproc   131072
root    soft    nofile  131072
root    hard    nofile  131072
EOF

• 集群的HA依賴于時間一致性，安裝并配置chrony

yum install -y chrony
cat>/etc/chrony.conf<<EOF
server cn.pool.ntp.org iburst minpoll 4 maxpoll 10
server s1b.time.edu.cn iburst minpoll 4 maxpoll 10
# Ignor source level
stratumweight 0

# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/chrony.drift

# This directive enables kernel synchronisation (every 11 minutes) of the
# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.
rtcsync

# Allow the system clock to be stepped in the first three updates
# if its offset is larger than 1 second.
makestep 1.0 3

# Enable hardware timestamping on all interfaces that support it.
#hwtimestamp *

# Increase the minimum number of selectable sources required to adjust
# the system clock.
#minsources 2

bindcmdaddress 127.0.0.1

#bindcmdaddress ::1

# Specify file containing keys for NTP authentication.
keyfile /etc/chrony/chrony.keys

logdir /var/log/chrony

# adjust time big than 1 sec will log to file
logchange 1
EOF

systemctl enable --now chronyd

• 修改hostname
  kubelet和kube-proxy上報node信息默認是取hostname的，除非通過--hostname-override指定，這里自行設(shè)置hostname，并修改hosts文件。

#按規(guī)劃進行修改
hostnamectl set-hostname xxx
##所有主機都修改hosts文件
cat >>/etc/hosts << EOF
192.168.2.140 k8s-m1
192.168.2.141 k8s-m2
192.168.2.142 k8s-m3
EOF

• docker官方的內(nèi)核檢查腳本建議(RHEL7/CentOS7: User namespaces disabled; add ‘user_namespace.enable=1’ to boot command line)，使用下面命令開啟

grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

最后重啟系統(tǒng)

reboot

安裝docker

選擇官方建議的對應版本，可以通過對應版本地址進行查看https://github.com/kubernetes/kubernetes/blob/v1.19.16/build/dependencies.yaml，這里我們選擇docker 19.03版本，并使用docker官方的安裝腳本進行安裝(該腳本支持centos和ubuntu)。

export VERSION=19.03
curl -fsSL "https://get.docker.com/" | bash -s -- --mirror Aliyun

所有機器配置加速源并配置docker的啟動參數(shù)使用systemd，使用systemd是官方的建議，詳見 https://kubernetes.io/docs/setup/cri/

mkdir -p /etc/docker/
cat>/etc/docker/daemon.json<<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": [
      "https://5sssm2l6.mirror.aliyuncs.com",
      "https://docker.mirrors.ustc.edu.cn/",
  ],
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
  }
}
EOF

Live Restore Enabled這個注意別開，某些極端情況下容器Dead狀態(tài)之類的必須重啟docker daemon才能解決。

• 設(shè)置docker開機啟動-CentOS安裝完成后docker需要手動設(shè)置docker命令補全：

yum install -y epel-release bash-completion && \
cp /usr/share/bash-completion/completions/docker /etc/bash_completion.d/

• docker自1.13版起會自動設(shè)置iptables的FORWARD默認策略為DROP，這可能會影響Kubernetes集群依賴的報文轉(zhuǎn)發(fā)功能，防止FORWARD的DROP策略影響轉(zhuǎn)發(fā),給docker daemon添加下列參數(shù)修正，當然暴力點也可以iptables -P FORWARD ACCEPT

mkdir -p /etc/systemd/system/docker.service.d/
cat>/etc/systemd/system/docker.service.d/10-docker.conf<<EOF
[Service]
ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
ExecStopPost=/bin/bash -c '/sbin/iptables -D FORWARD -s 0.0.0.0/0 -j ACCEPT &> /dev/null || :'
EOF

• 啟動docker并看下信息是否正常

systemctl enable --now docker
docker info

如果enable docker的時候報錯開啟debug，如何開見

kubeadm部署

鏡像源準備
默認源在國外會無法安裝，我們使用國內(nèi)的鏡像源，所有機器都要操作

cat <<EOF >/etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
EOF

master部分安裝相關(guān)軟件
k8s的node就是kubelet+cri(一般是docker)，kubectl是一個agent讀取kubeconfig去訪問kube-apiserver來操作集群，kubeadm是部署，所以master節(jié)點需要安裝三個，node一般不需要kubectl但是yum安裝的時候還是會給你安裝最新版所以我這里node還是安裝了kubectl，不用就行。

yum install -y \
    kubeadm-1.19.16 \
    kubectl-1.19.16 \
    kubelet-1.19.16 \
    --disableexcludes=kubernetes && \
    systemctl enable kubelet

node部分安裝相關(guān)軟件

yum install -y \
    kubeadm-1.19.16 \
    kubectl-1.19.16 \
    kubelet-1.19.16 \
    --disableexcludes=kubernetes && \
    systemctl enable kubelet

配置kubelet的參數(shù)方法(有需要的話)
查看kubelet的systemd文件

$ systemctl cat kubelet
# /usr/lib/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

# /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

我們可以看到/etc/sysconfig/kubelet和/var/lib/kubelet/kubeadm-flags.env都是EnvironmentFile，通過可以通過注釋查看每個文件中如何設(shè)置運行參數(shù)。如/var/lib/kubelet/kubeadm-flags.env 中應該通過KUBELET_EXTRA_ARGS來給kubelet配置運行參數(shù)，如下。具體參數(shù)可以通過kubelet --help查看，參數(shù)很多，很多不了解的最好保持默認。

cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--aa=bb --xx=yy"
EOF
第二個EnvironmentFile 文件/var/lib/kubelet/kubeadm-flags.env也一樣

haproxy和keepalived的安裝

三臺master都需要安裝
haproxy配置文件
注意，三個節(jié)點上haproxy.conf的配置文件內(nèi)容其實是一樣的

cat <<EOF > /etc/haproxy/haproxy.cfg
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

listen stats
  bind    *:8006
  mode    http
  stats   enable
  stats   hide-version
  stats   uri       /stats
  stats   refresh   30s
  stats   realm     Haproxy\ Statistics
  stats   auth      admin:admin

frontend k8s-api
  bind 0.0.0.0:8443
  bind 127.0.0.1:8443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-api

backend k8s-api
  mode tcp
  option tcplog
  option httpchk GET /healthz
  http-check expect string ok
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server  api1  192.168.2.140:6443  check check-ssl verify none
    server  api2  192.168.2.141:6443  check check-ssl verify none
    server  api3  192.168.2.142:6443  check check-ssl verify none
EOF

keepalived配置文件
注意，keepalived.conf的配置文件內(nèi)容是有區(qū)別的。

cat <<EOF > /etc/keepalived/keepalived.conf
global_defs {
    enable_script_security
}

vrrp_script haproxy-check {
    user root
    script "/bin/bash /etc/keepalived/check_haproxy.sh"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance haproxy-vip {
    state MASTER  ##注意修改，其中主為MASTER，從為BACKUP
    priority 100 ##注意修改，數(shù)字越大，優(yōu)先級越高，主>從
    interface eth0   #注意實際環(huán)境中網(wǎng)卡名字，有的是ens*啥的
    virtual_router_id 48 #所有節(jié)點的id要一致
    advert_int 3
    unicast_src_ip 192.168.2.140           # 本機IP
    unicast_peer {
        192.168.2.141      # 對端IP
        192.168.2.142      # 對端IP
    }

    virtual_ipaddress {
        192.168.2.250/24   # VIP地址
    }

    track_script {
        haproxy-check
    }
}
EOF

說明：keeaplived這里需要注意，默認keepalived是采用的組播方式，加上unicast_peer參數(shù)后是單播方式，三臺keepalived配置文件不一樣unicast_src_ipc參數(shù)寫當前節(jié)點IP，unicast_peer參數(shù)寫另外兩個節(jié)點IP地址。其他按說明修改

keepalived 健康檢查腳本

 cat <<'EOF'> /etc/keepalived/check_haproxy.sh
#!/bin/bash
VIRTUAL_IP=192.168.2.250.250

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

if ip addr | grep -q $VIRTUAL_IP ; then
    curl -s --max-time 2 --insecure https://${VIRTUAL_IP}:8443/healthz  -o /dev/null || errorExit "Error GET https://${VIRTUAL_IP}:8443/healthz"
fi
EOF

部署外部Etcd服務(wù)

openssl 證書配置文件，注意IP地址書寫正確

mkdir -p /etc/kubernetes/pki/etcd
cd /etc/kubernetes/pki

cat <<EOF> /etc/kubernetes/pki/openssl.cnf
[ req ]
default_bits = 2048
default_md = sha256
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_ca ]
basicConstraints = critical, CA:TRUE
keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign

[ v3_req_server ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ v3_req_client ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth

[ v3_req_apiserver ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names_cluster

[ v3_req_etcd ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names_etcd

[ alt_names_cluster ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
DNS.5 = k8s-m1
DNS.6 = k8s-m2
DNS.7 = k8s-m3
DNS.8 = localhost
IP.1 = 10.96.0.1
IP.2 = 127.0.0.1
IP.3 = 192.168.2.140  
IP.4 = 192.168.2.141
IP.5 = 192.168.2.142
IP.6 = 192.168.2.250

[ alt_names_etcd ]
DNS.1 = localhost
DNS.2 = k8s-m1
DNS.3 = k8s-m2
DNS.4 = k8s-m3
IP.1 = 192.168.2.140
IP.2 = 192.168.2.141
IP.3 = 192.168.2.142
IP.4 = 127.0.0.1
EOF

根據(jù)上面的配置文件，生成各服務(wù)所需證書，并下發(fā)到其他節(jié)點。其他節(jié)點就不需要在進行證書生成
#生成證書，有效期10000d

openssl genrsa -out etcd/ca.key 2048
openssl req -x509 -new -nodes -key etcd/ca.key -config openssl.cnf -subj "/CN=etcd-ca" -extensions v3_ca -out etcd/ca.crt -days 10000
openssl genrsa -out apiserver-etcd-client.key 2048
openssl req -new -key apiserver-etcd-client.key -subj "/CN=apiserver-etcd-client/O=system:masters" -out apiserver-etcd-client.csr
openssl x509 -in apiserver-etcd-client.csr -req -CA etcd/ca.crt -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd -extfile openssl.cnf -out apiserver-etcd-client.crt -days 10000
openssl genrsa -out etcd/server.key 2048
openssl req -new -key etcd/server.key -subj "/CN=etcd-server" -out etcd/server.csr
openssl x509 -in etcd/server.csr -req -CA etcd/ca.crt -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd -extfile openssl.cnf -out etcd/server.crt -days 10000
openssl genrsa -out etcd/peer.key 2048
openssl req -new -key etcd/peer.key -subj "/CN=etcd-peer" -out etcd/peer.csr
openssl x509 -in etcd/peer.csr -req -CA etcd/ca.crt -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd -extfile openssl.cnf -out etcd/peer.crt -days 10000
openssl genrsa -out etcd/healthcheck-client.key 2048
openssl req -new -key etcd/healthcheck-client.key -subj "/CN=etcd-client" -out etcd/healthcheck-client.csr
openssl x509 -in etcd/healthcheck-client.csr -req -CA etcd/ca.crt -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd -extfile openssl.cnf -out etcd/healthcheck-client.crt -days 10000

scp -r /etc/kubernetes root@k8s-m2:/etc
scp -r /etc/kubernetes root@k8s-m3:/etc

下載部署，其他節(jié)點類似：
所需etcd對應版本可以通過https://github.com/kubernetes/kubernetes/blob/v1.19.16/build/dependencies.yaml查看

mkdir -p /var/lib/etcd
ETCD_VER=v3.4.13
wget https://storage.googleapis.com/etcd/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -O etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xf etcd-${ETCD_VER}-linux-amd64.tar.gz  --strip-components=1 -C /usr/local/bin etcd-${ETCD_VER}-linux-amd64/{etcd,etcdctl}

設(shè)置 unit file 并啟動 etcd,其他節(jié)點修改對應 ETCD_NAME 為 etcd1 和 etcd2，ip 改為節(jié)點 IP。

ETCD_NAME=etcd0
ETCD_IP="192.168.2.140"
ETCD_IPS=(192.168.2.140 192.168.2.141 192.168.2.142)

cat<<EOF> /usr/lib/systemd/system/etcd.service 
[Unit]
Description=etcd
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd
ExecStart=/usr/bin/etcd \\
    --name=${ETCD_NAME} \\
    --data-dir=/var/lib/etcd \\
    --listen-client-urls=https://127.0.0.1:2379,https://${ETCD_IP}:2379 \\
    --advertise-client-urls=https://${ETCD_IP}:2379 \\
    --listen-peer-urls=https://${ETCD_IP}:2380 \\
    --initial-advertise-peer-urls=https://${ETCD_IP}:2380 \\
    --cert-file=/etc/kubernetes/pki/etcd/server.crt \\
    --key-file=/etc/kubernetes/pki/etcd/server.key \\
    --client-cert-auth \\
    --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \\
    --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt \\
    --peer-key-file=/etc/kubernetes/pki/etcd/peer.key \\
    --peer-client-cert-auth \\
    --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \\
    --initial-cluster=etcd0=https://${ETCD_IPS[0]}:2380,etcd1=https://${ETCD_IPS[1]}:2380,etcd2=https://${ETCD_IPS[2]}:2380 \\
    --initial-cluster-token=my-etcd-token \\
    --initial-cluster-state=new \\
    --heartbeat-interval 1000 \\
    --election-timeout 5000

Restart=always
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl restart etcd
systemctl enable etcd

cat <<EOF > /etc/profile.d/etcd.sh
alias etcd_v2='etcdctl --cert-file /etc/kubernetes/pki/etcd/healthcheck-client.crt \
              --key-file /etc/kubernetes/pki/etcd/healthcheck-client.key  \
              --ca-file /etc/kubernetes/pki/etcd/ca.crt  \
              --endpoints https://192.168.2.140:2379,https://192.168.2.141:2379,https://192.168.2.142:2379'

alias etcd_v3='ETCDCTL_API=3 \
    etcdctl   \
   --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt \
   --key /etc/kubernetes/pki/etcd/healthcheck-client.key \
   --cacert /etc/kubernetes/pki/etcd/ca.crt \
   --endpoints https://192.168.2.140:2379,https://192.168.2.141:2379,https://192.168.2.142:2379'
EOF
source /etc/profile.d/etcd.sh
etcd_v3 version
etcdctl version: 3.4.13
API version: 3.4
etcd_v3 --write-out=table endpoint status

[root@k8s-m1 ~]# etcd_v3 --write-out=table endpoint status
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://192.168.2.140:2379 | 19df3a9852e0345a |  3.4.13 |   24 MB |     false |    267804 |  108091120 |
| https://192.168.2.141:2379 | 66d402f1ef2c996e |  3.4.13 |   24 MB |      true |    267804 |  108091120 |
| https://192.168.2.142:2379 | 3bb3629d60bef3f6 |  3.4.13 |   24 MB |     false |    267804 |  108091121 |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+```

k8s集群安裝(第一個master上配置)

打印默認init的配置信息

kubeadm config print init-defaults > initconfig.yaml

我們看下默認init的集群參數(shù)

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-m1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.19.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

我們主要關(guān)注和保留ClusterConfiguration的段，然后修改下，可以參考下列的v1beta2文檔，如果是低版本可能是v1beta1，某些字段和新的是不一樣的，自行查找godoc看
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#hdr-Basics
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#pkg-constants
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#ClusterConfiguration

controlPlaneEndpoint是規(guī)劃的vip地址，下面是最終的yaml

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
imageRepository: registry.aliyuncs.com/google_containers
kubernetesVersion: v1.19.16
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
networking: 
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16
controlPlaneEndpoint: 192.168.2.250:8443 # 單個master的話寫master的ip或者不寫，端口是haproxy運行的端口
apiServer: # https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3#APIServer
  timeoutForControlPlane: 4m0s
  extraArgs:
    authorization-mode: "Node,RBAC"
    enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeClaimResize,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,Priority"
    runtime-config: api/all=true
    storage-backend: etcd3
  certSANs:
  - 127.0.0.1 # 多個master的時候負載均衡出問題了能夠快速使用localhost調(diào)試
  - localhost
  - 192.168.2.140
  - 192.168.2.141
  - 192.168.2.142
  - k8s-m1
  - k8s-m2
  - k8s-m3
  extraVolumes:
  - hostPath: /etc/localtime
    mountPath: /etc/localtime
    name: localtime
    readOnly: true
controllerManager: 
  extraArgs:
    bind-address: "0.0.0.0"
  extraVolumes:
  - hostPath: /etc/localtime
    mountPath: /etc/localtime
    name: localtime
    readOnly: true
scheduler: 
  extraArgs:
    bind-address: "0.0.0.0"
  extraVolumes:
  - hostPath: /etc/localtime
    mountPath: /etc/localtime
    name: localtime
    readOnly: true
dns: {}
etcd: 
  external:
    endpoints:
    - https://192.168.2.140:2379
    - https://192.168.2.141:2379
    - https://192.168.2.142:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration 
mode: ipvs # or iptables
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: "rr" # 調(diào)度算法
  strictARP: false
  syncPeriod: 15s
iptables:
  masqueradeAll: true
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration 
cgroupDriver: systemd
failSwapOn: true # 如果開啟swap則設(shè)置為false

• swap的話看最后一行，單臺master的話把controlPlaneEndpoint的值改為第一個master的ip
• kubectl get cs 查看組件狀態(tài)發(fā)現(xiàn)controllerManager和scheduler 狀態(tài)Unhealthy 刪除/etc/kubernetes/manifests/目錄下對應文件中–port=0的配置即可，每個master節(jié)點都需要刪除
• 檢查文件是否錯誤，忽略warning，錯誤的話會拋出error，沒有錯誤則會輸出到包含字符串kubeadm join 等添加節(jié)點的信息

kubeadm init --config initconfig.yaml --dry-run

#檢查鏡像是否正確
kubeadm config images list --config initconfig.yaml
#預先拉取鏡像
kubeadm config images pull --config initconfig.yaml # 下面是輸出
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.19.16
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.19.16
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.19.16
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.19.16
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.2
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:1.7.0

kubeadm 初始化

下面init只在第一個master上面操作

kubeadm init --config initconfig.yaml 

初始化的時候可以添加參數(shù)–upload-certs， 作用為將相關(guān)的證書直接上傳到etcd中保存，這樣省去我們手動分發(fā)證書的過程。
初始化完成后記住init后打印的token，復制kubectl的kubeconfig，kubectl的kubeconfig路徑默認是~/.kube/config

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

init的yaml信息實際上會存在集群的configmap里，我們可以隨時查看，該yaml在其他node和master join的時候會使用到

kubectl -n kube-system get cm kubeadm-config -o yaml

配置其他master的k8s管理組件(某些低版本不支持上傳證書的時候操作需手動拷貝證書)

第一個master上拷貝ca證書到其他master節(jié)點上

scp -r /etc/kubernetes/pki root@k8s-m2:/etc/kubernetes/
scp -r /etc/kubernetes/pki root@k8s-m3:/etc/kubernetes/

其他master join進來

kubeadm join 192.168.2.250.250:8443 \
    --token xxx.zzzzzzzzz     \
    --discovery-token-ca-cert-hash sha256:xxxxxxxxxxx --control-plane

通過下列命令可以獲取sha256的值

openssl x509 -pubkey -in \
    /etc/kubernetes/pki/ca.crt | \
    openssl rsa -pubin -outform der 2>/dev/null | \
    openssl dgst -sha256 -hex | sed 's/^.* //'

• 如果集群在init時使用了 --upload-certs 參數(shù)將相關(guān)的證書直接上傳到etcd中保存，則其他master在加入時需要使用 --certificate-key 參數(shù)（某些低版本可能不支持）。
• token忘記的話可以kubeadm token list查看，沒有的話可以通過kubeadm token create創(chuàng)建。在高版本可以直接使用kubeadm token create --print-join-command來創(chuàng)建添加節(jié)點的命令，某些老版本可能不確定支持–print-join-command這個選項，不支持的話就不帶–print-join-command選項創(chuàng)建token。
• 通過將參數(shù) --upload-certs 添加到 kubeadm init，你可以將控制平面證書臨時上傳到集群中的 Secret 請注意此 Secret 將在 2小時后自動過期。證書使用 32 字節(jié)密鑰加密，可以使用 --certificate-key 指定。
  以下階段命令可用于證書到期后重新上傳證書：

kubeadm init phase upload-certs --upload-certs --certificate-key=SOME_VALUE

如果未將參數(shù) --certificate-key 傳遞給 kubeadm init 和 kubeadm init phase upload-certs， 則會自動生成一個新密鑰。

以下命令可用于按需生成新密鑰：

kubeadm alpha certs certificate-key

設(shè)置kubectl的補全腳本

kubectl completion bash > /etc/bash_completion.d/kubectl

獲取節(jié)點狀態(tài)信息

[root@k8s-m1 ~]# kubectl get nodes 
NAME     STATUS   ROLES    AGE    VERSION
k8s-m1   Ready    master   1d   v1.19.16
k8s-m2   Ready    master   1d   v1.19.16
k8s-m3   Ready    master   1d   v1.19.16

[root@k8s-m1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"} 

addon(此章開始到結(jié)尾選取任意一個master上執(zhí)行)

到此，集群還不能真正使用，因為集群的網(wǎng)絡(luò)組件是以插件的方式部署，在這里我選用了常用的flannel，后面將分享使用其他網(wǎng)絡(luò)組件，如calico。

#直接apply就行，或者先用wget先下載下來也可以，多嘗試幾次就能下載下來
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

驗證集群可用性
等kube-system空間下的pod都處于running狀態(tài)后再測試集群的可用性

[root@k8s-m1 k8s-total]# cat test-k8s.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      tier: frontend
  replicas: 1
  template:
    metadata:
      labels:
        tier: frontend
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
      tier: frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  containers:
  - name: busybox
    image: busybox:1.28.4
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
##注意busybox的版本要1.28.4之前的版本，不然解析有問題

[root@k8s-m1 k8s-total]# kubectl get po,svc
NAME                            READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
pod/busybox                     1/1     Running   0          5m23s   10.244.0.4   k8s-m2   <none>           <none>
pod/my-nginx-5b8555d6b8-vxcsj   1/1     Running   0          20m     10.244.2.3   k8s-m1   <none>           <none>

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE     SELECTOR
service/kubernetes             ClusterIP   10.96.0.1        <none>        443/TCP    436d    <none>service/nginx                  ClusterIP   10.107.34.204    <none>        80/TCP     16m
service/nginx                  ClusterIP   10.107.34.204    <none>        80/TCP     20m     tier=frontend

驗證集群dns，使用nslookup或者dig都可以

[root@k8s-m1 k8s-total]#  kubectl exec -ti busybox -- nslookup kubernetes
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

[root@k8s-m1 k8s-total]# dig -t A kubernetes.default.svc.cluster.local.  @10.96.0.10
#日常檢查其他服務(wù)能否正常解析時也可以使用此命令

在master上curl nginx的svc的ip出現(xiàn)nginx的index內(nèi)容即集群正常，例如我的nginx svc ip是10.107.34.204

[root@k8s-m1 k8s-total]# curl 10.107.34.204  -s
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

日志管理

以kube-apiserver為例將日志掛載出來方便管理（kube-controller-manager、kube-scheduler組件服務(wù)類似）
修改以下三處

#啟動參數(shù)
spec:
  containers:
  - command:
    - --logtostderr=false
    - --log-dir=/var/log/kubernetes/kube-apiserver
    - --v=2

#mount地址
volumeMounts:
- mountPath: /var/log/kubernetes/kube-apiserver
  name: k8s-logs

#被掛載券設(shè)置
volumes:
- hostPath:
    path: /var/log/kubernetes/kube-apiserver
    type: DirectoryOrCreate
  name: k8s-logs

kubelet日志（kubelet服務(wù)是用systemctl進行管理的非容器管理所以不用掛載），可以通過設(shè)置文件存放目錄直接進行修改。

[root@k8s-m1 manifests]# vim /etc/sysconfig/kubelet
--v=2 --logtostderr=false --log-dir=/var/log/kubernetes/kubelet

更多關(guān)于kubernetes的知識分享，請前往博客主頁。編寫過程中，難免出現(xiàn)差錯，敬請指出文章來源地址http://www.zghlxwxcb.cn/news/detail-606751.html

到了這里，關(guān)于【kubernetes系列】kubernetes之使用kubeadm搭建高可用集群的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！