一、集群規(guī)劃及架構(gòu)

官方文檔：

二進(jìn)制下載地址

環(huán)境規(guī)劃：

• pod網(wǎng)段：10.244.0.0/16

• service網(wǎng)段：10.10.0.0/16

• 注意： pod和service網(wǎng)段不可沖突，如果沖突會(huì)導(dǎo)致K8S集群安裝失敗。

本次實(shí)驗(yàn)架構(gòu)圖：

二、系統(tǒng)初始化準(zhǔn)備(所有節(jié)點(diǎn)同步操作)

1、關(guān)閉防火墻

systemctl disable firewalld --now
setenforce 0
sed  -i -r 's/SELINUX=[ep].*/SELINUX=disabled/g' /etc/selinux/config

2、配置域名解析

cat  >> /etc/hosts << EOF
16.32.15.200 master-1
16.32.15.201 master-2
16.32.15.202 node-1
EOF

在指定主機(jī)上面修改主機(jī)名

hostnamectl set-hostname master-1 && bash
hostnamectl set-hostname master-2 && bash
hostnamectl set-hostname node-1 && bash

3、配置服務(wù)器時(shí)間保持一致

yum -y install ntpdate
ntpdate ntp1.aliyun.com

添加定時(shí)同步 每天凌晨1點(diǎn)自動(dòng)同步時(shí)間

echo "0 1 * * * ntpdate ntp1.aliyun.com" >> /var/spool/cron/root
crontab -l

4、禁用swap交換分區(qū)（kubernetes強(qiáng)制要求禁用）

swapoff --all

禁止開(kāi)機(jī)自啟動(dòng)swap交換分區(qū)

sed -i -r '/swap/ s/^/#/' /etc/fstab

5、修改Linux內(nèi)核參數(shù)，添加網(wǎng)橋過(guò)濾器和地址轉(zhuǎn)發(fā)功能

cat >> /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sysctl -p /etc/sysctl.d/kubernetes.conf

加載網(wǎng)橋過(guò)濾器模塊

modprobe br_netfilter
lsmod | grep br_netfilter # 驗(yàn)證是否生效

6、配置ipvs功能

在kubernetes中Service有兩種代理模型，一種是基于iptables的，一種是基于ipvs，兩者對(duì)比ipvs的性能要高，如果想要使用ipvs模型，需要手動(dòng)載入ipvs模塊

yum -y install ipset ipvsadm

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4  
EOF

chmod +x /etc/sysconfig/modules/ipvs.modules 
# 執(zhí)行腳本
/etc/sysconfig/modules/ipvs.modules

# 驗(yàn)證ipvs模塊
lsmod | grep -e ip_vs -e nf_conntrack_ipv4

7、安裝Docker容器組件

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum makecache

# yum-utils軟件用于提供yum-config-manager程序
yum install -y yum-utils

# 使用yum-config-manager創(chuàng)建docker阿里存儲(chǔ)庫(kù)
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum install docker-ce-20.10.6 docker-ce-cli-20.10.6 -y

Docker默認(rèn)使用的Cgroup Driver為默認(rèn)文件驅(qū)動(dòng),而k8s默認(rèn)使用的文件驅(qū)動(dòng)為systemd,k8s要求驅(qū)動(dòng)類型必須要一致,所以需要將docker文件驅(qū)動(dòng)改成systemd,并且配置國(guó)內(nèi)加速器。

mkdir /etc/docker
cat <<EOF > /etc/docker/daemon.json
{
  "registry-mirrors": ["https://aoewjvel.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

# 啟動(dòng)docker并設(shè)置開(kāi)機(jī)自啟
systemctl enable docker --now
systemctl status docker

8、重啟服務(wù)器 可略過(guò)

reboot

三、安裝kubeadm(所有節(jié)點(diǎn)同步操作)

配置國(guó)內(nèi)yum源，一鍵安裝 kubeadm、kubelet、kubectl

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
EOF

yum -y install --setopt=obsoletes=0 kubeadm-1.23.0 kubelet-1.23.0 kubectl-1.23.0

kubeadm將使用kubelet服務(wù)以容器方式部署kubernetes的主要服務(wù)，所以需要先啟動(dòng)kubelet服務(wù)

systemctl enable kubelet.service --now

四、高可用組件安裝及配置

master-1、master-2主機(jī)上進(jìn)行操作

1、安裝Nginx及配置

在 master-1、 master-2主機(jī)同步執(zhí)行，Nginx配置是一致的

wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo
yum -y install nginx*

配置Nginx文件內(nèi)容如下：如果主機(jī)名稱一致，那么Nginx配置文件直接復(fù)制即可,如果主機(jī)名不一致則需要更改 upstream k8s-apiserver下面為對(duì)應(yīng)的主機(jī)名或者IP地址。

mv /etc/nginx/nginx.conf{,.bak}
cat >> /etc/nginx/nginx.conf << EOF

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
       server master-1:6443; 
       server master-2:6443;
    }

server {
       listen 16443; 
       proxy_pass k8s-apiserver;
    }
}


http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}
EOF

啟動(dòng)Nginx 并且加入開(kāi)機(jī)自啟動(dòng)

nginx -t
systemctl start nginx
systemctl enable nginx

我們代理apiservice的端口是16443， 驗(yàn)證端口是否啟動(dòng)

netstat -anput |grep 16443

2、安裝keepalived及配置

在 master-1、 master-2主機(jī)同步執(zhí)行，但是配置文件不一致！

yum -y install keepalived

添加腳本文件，實(shí)現(xiàn)當(dāng)Nginx宕機(jī)后停止keepalived，VIP地址會(huì)漂移到另一個(gè)節(jié)點(diǎn)，從而實(shí)現(xiàn)apiservice的高可用。

vim /etc/keepalived/checkNginx.sh

#!/bin/bash
# egrep -cv "grep|$$" 過(guò)濾掉包含grep 或者 $$ 表示的當(dāng)前Shell進(jìn)程ID
count=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$")
if [ $count -eq 0 ];then
    systemctl stop keepalived
fi

賦予可執(zhí)行權(quán)限

chmod +x /etc/keepalived/checkNginx.sh

Keepalived master-1主機(jī)配置文件內(nèi)容：

mv /etc/keepalived/keepalived.conf{,.default}

vim /etc/keepalived/keepalived.conf
vrrp_script checkNginx {
    script "/etc/keepalived/checkNginx.sh"      # 監(jiān)控Nginx狀態(tài)腳本
    interval 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33        # 本機(jī)網(wǎng)卡名稱
    virtual_router_id 51   # VRRP 路由 ID實(shí)例,每個(gè)實(shí)例是唯一的 
    priority 100           # 優(yōu)先級(jí)
    advert_int 1           # 指定VRRP 心跳包通告間隔時(shí)間,默認(rèn)1秒 
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
	16.32.15.100/24
    }

   track_script {
	checkNginx
    } 
}

Keepalived master-2備機(jī)配置文件內(nèi)容：

mv /etc/keepalived/keepalived.conf{,.default}
vim /etc/keepalived/keepalived.conf

vrrp_script checkNginx {
    script "/etc/keepalived/checkNginx.sh"      # 監(jiān)控Nginx狀態(tài)腳本
    interval 2
}

vrrp_instance VI_1 {
    state BACKUP           # 當(dāng)前角色
    interface ens33        
    virtual_router_id 52   # VRRP 路由 ID實(shí)例,每個(gè)實(shí)例是唯一的 
    priority 90            # 優(yōu)先級(jí)
    advert_int 1          
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
	16.32.15.100/24
    }

   track_script {
	checkNginx
    } 
}

重啟keepalived(兩臺(tái)master同步執(zhí)行)

systemctl restart keepalived

五、初始化Master集群

在master-1主機(jī)上進(jìn)行操作

1、創(chuàng)建初始化文件

vim kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.23.0
controlPlaneEndpoint: 16.32.15.100:16443
imageRepository: registry.aliyuncs.com/google_containers
apiServer:
 certSANs:
 - 16.32.15.200
 - 16.32.15.201
 - 16.32.15.202
 - 16.32.15.100
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.10.0.0/16
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind:  KubeProxyConfiguration
mode: ipvs

• controlPlaneEndpoint：將集群的控制平面連接到一個(gè)負(fù)載均衡器(填寫(xiě)VIP地址)
• imageRepository：鏡像下載地址，這里使用國(guó)內(nèi)阿里云的
• certSANs：配置DNS，把集群涉及到的IP全部寫(xiě)上即可,包括VIP地址
• podSubnet：Pod網(wǎng)絡(luò)段
• serviceSubnet：Service網(wǎng)絡(luò)段

2、進(jìn)行初始化

kubeadm init --config kubeadm-config.yaml --ignore-preflight-errors=SystemVerification

初始化成功后輸出如下內(nèi)容：

[init] Using Kubernetes version: v1.23.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master-1] and IPs [10.10.0.1 16.32.15.200 16.32.15.100 16.32.15.201 16.32.15.202]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master-1] and IPs [16.32.15.200 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master-1] and IPs [16.32.15.200 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 11.889660 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master-1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: giw3n1.8ys41tcqlvl9xhrk
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

# 將master節(jié)點(diǎn)加入集群中
  kubeadm join 16.32.15.100:16443 --token giw3n1.8ys41tcqlvl9xhrk \
	--discovery-token-ca-cert-hash sha256:2e97fe276dd9a52e91704fbd985f8c57c73c6ca750f07e9eeaf695f7639e0287 \
	--control-plane 

Then you can join any number of worker nodes by running the following on each as root:

# 將node節(jié)點(diǎn)加入集群中
kubeadm join 16.32.15.100:16443 --token giw3n1.8ys41tcqlvl9xhrk \
	--discovery-token-ca-cert-hash sha256:2e97fe276dd9a52e91704fbd985f8c57c73c6ca750f07e9eeaf695f7639e0287 

配置kubectl的配置文件config，相當(dāng)于對(duì)kubectl進(jìn)行授權(quán)，這樣kubectl命令可以使用這個(gè)證書(shū)對(duì)k8s集群進(jìn)行管理

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

驗(yàn)證使用可以使用 kubectl 命令

kubectl get nodes

六、擴(kuò)容K8S集群

1、擴(kuò)容master節(jié)點(diǎn)

把master-1主節(jié)點(diǎn)的證書(shū)文件拷貝到master-2主機(jī)

master-2主機(jī)執(zhí)行

cd /root && mkdir -p /etc/kubernetes/pki/etcd &&mkdir -p ~/.kube/

master-1主機(jī)執(zhí)行 拷貝證書(shū)等文件到master-2主機(jī)

scp /etc/kubernetes/pki/ca.key master-2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/ca.crt master-2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.key master-2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.pub master-2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.crt master-2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.key master-2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.crt master-2:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/etcd/ca.key master-2:/etc/kubernetes/pki/etcd/

master-2主機(jī)執(zhí)行 加入集群

kubeadm join 16.32.15.100:16443 --token giw3n1.8ys41tcqlvl9xhrk \
	--discovery-token-ca-cert-hash sha256:2e97fe276dd9a52e91704fbd985f8c57c73c6ca750f07e9eeaf695f7639e0287 \
	--control-plane

顯示如下圖表示成功加入集群中

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

master-1主機(jī)執(zhí)行 查看node狀態(tài)

kubectl get node

2、擴(kuò)容node節(jié)點(diǎn)

node-1主機(jī)上執(zhí)行 加入集群

kubeadm join 16.32.15.100:16443 --token giw3n1.8ys41tcqlvl9xhrk \
	--discovery-token-ca-cert-hash sha256:2e97fe276dd9a52e91704fbd985f8c57c73c6ca750f07e9eeaf695f7639e0287 

顯示如下圖表示成功加入集群中

master-1主機(jī)執(zhí)行 查看node狀態(tài)

kubectl get node

可以看到node-1的ROLES角色為空，就表示這個(gè)節(jié)點(diǎn)是工作節(jié)點(diǎn)。

可以把node-1的ROLES變成work，按照如下方法：

kubectl label node node-1 node-role.kubernetes.io/worker=worker

七、安裝網(wǎng)絡(luò)組件Calico

Calico在線文檔地址：

Calico.yaml下載地址：

1、查看自帶Pod狀態(tài)

kubectl get pods -n kube-system

當(dāng)我們查看自帶Pod狀態(tài)時(shí)，coredns是pending狀態(tài)，這是因?yàn)檫€沒(méi)有安裝網(wǎng)絡(luò)插件，等到下面安裝好網(wǎng)絡(luò)插件之后這個(gè)cordns就會(huì)變成running了

2、上傳calico.yaml文件到服務(wù)器中，下面提供calico.yaml文件內(nèi)容：

在master-1、master-2執(zhí)行

kubectl apply -f  calico.yaml

3、查看集群狀態(tài) && 查看自帶Pod狀態(tài)

kubectl get nodes

kubectl get pods -n kube-system

八、部署Tomcat測(cè)試集群可用性

1、創(chuàng)建tomcat的pod資源

vim tomcat.yaml 

apiVersion: v1  #pod屬于k8s核心組v1
kind: Pod  #創(chuàng)建的是一個(gè)Pod資源
metadata:  #元數(shù)據(jù)
  name: demo-pod  #pod名字
  namespace: default  #pod所屬的名稱空間
  labels:
    app: myapp  #pod具有的標(biāo)簽
    env: dev      #pod具有的標(biāo)簽
spec:
  containers:      #定義一個(gè)容器，容器是對(duì)象列表，下面可以有多個(gè)name
  - name:  tomcat-pod-java  #容器的名字
    ports:
    - containerPort: 8080
    image: tomcat:8.5-jre8-alpine   #容器使用的鏡像
    imagePullPolicy: IfNotPresent

執(zhí)行ymal文件

kubectl apply -f tomcat.yaml

查看pod狀態(tài)

kubectl get pod

2、創(chuàng)建tomcat的service資源

vim tomcat-service.yaml 

apiVersion: v1
kind: Service
metadata:
  name: tomcat
spec:
  type: NodePort
  ports:
    - port: 8080
      nodePort: 30080
  selector:
    app: myapp
    env: dev

執(zhí)行ymal文件

kubectl apply -f tomcat-service.yaml 

查看service狀態(tài)

kubectl get svc

3、瀏覽器訪問(wèn)測(cè)試(任意節(jié)點(diǎn)+30080端口)

4、測(cè)試coredns文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-421590.html

kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- sh

If you don't see a command prompt, try pressing enter.


/ # nslookup kubernetes.default.svc.cluster.local
Server:    10.10.0.10
Address 1: 10.10.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default.svc.cluster.local
Address 1: 10.10.0.1 kubernetes.default.svc.cluster.local

/ # nslookup tomcat.default.svc.cluster.local
Server:    10.10.0.10
Address 1: 10.10.0.10 kube-dns.kube-system.svc.cluster.local

Name:      tomcat.default.svc.cluster.local
Address 1: 10.10.164.81 tomcat.default.svc.cluster.local

• 注意：busybox要用指定的1.28版本，不能用最新版本，最新版本，nslookup會(huì)解析不到dns和ip

到了這里，關(guān)于Kubeadm方式搭建K8s高可用集群【1.23.0版本】的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！