只有一個問題，原來的httpGet存活、就緒檢測一直不通過，于是改為tcpSocket后pod正常。
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
修改后的yaml文件，鏡像修改為阿里云
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
verbs:
- get
- apiGroups:
- ""
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
# - --cert-dir=/tmp
# - --secure-port=4443
# - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
# - --kubelet-use-node-status-port
# - --metric-resolution=15s
- --cert-dir=/tmp
- --secure-port=4443
- --metric-resolution=30s
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --requestheader-username-headers=X-Remote-User
- --requestheader-group-headers=X-Remote-Group
- --requestheader-extra-headers-prefix=X-Remote-Extra-
image: registry.aliyuncs.com/google_containers/metrics-server:v0.6.4
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
tcpSocket:
port: 4443
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 4443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
tcpSocket:
port: 4443
initialDelaySeconds: 20
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 200Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp
name: tmp-dir
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
kubectl apply -f components.yaml
部署kube-prometheus
兼容1.27的為main分支
只克隆main分支
git clone --single-branch --branch main https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus
如果直接運行kubectl apply -f manifests/setup/，會有元數(shù)據(jù)注解太長的錯誤
Error from server (Invalid): error when creating "manifests/setup/0prometheusCustomResourceDefinition.yaml": CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
應(yīng)該使用以下方式
1. kubectl apply -f manifests/setup/
2. kubectl apply --server-side -f manifests/setup
CRD正常創(chuàng)建后然后再執(zhí)行:
kubectl apply -f manifests
鏡像下載使用這個項目的方法:https://github.com/DaoCloud/public-image-mirror,在鏡像前面加：m.daocloud.io
prometheus-k8s-0 pod報權(quán)限不足問題
ts=2023-08-21T03:14:34.582Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
處理：
修改prometheus-clusterRole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.26.0
name: prometheus-k8s
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
- services
- endpoints
- pods
verbs:
- get
- list
- watch
- nonResourceURLs:
- /metrics
verbs:
- get
跨節(jié)點pind不通monitoring下的pod ip，prometheus target異常問題處理，
刪除kube-prometheus/manifests下的netowrkprolic
alertmanager-networkPolicy.yaml
blackboxExporter-networkPolicy.yaml
grafana-networkPolicy.yaml
kubeStateMetrics-networkPolicy.yaml
nodeExporter-networkPolicy.yaml
prometheusAdapter-networkPolicy.yaml
prometheus-networkPolicy.yaml
prometheusOperator-networkPolicy.yaml
使用ServiceMonitor添加監(jiān)控：
以ingress-nginx為例
修改ingress-nginx.yaml的service
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/port: '10254' #新增內(nèi)容
prometheus.io/scrape: 'true' #新增內(nèi)容
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.1
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
externalTrafficPolicy: Local
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
- appProtocol: http #新增內(nèi)容
name: prometheus #新增內(nèi)容
port: 10254 #新增內(nèi)容
protocol: TCP #新增內(nèi)容
targetPort: 10254 #新增內(nèi)容
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: LoadBalancer
# ingress-nginx-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: ingress-nginx # ServiceMonitor名稱
namespace: monitoring # ServiceMonitor所在名稱空間
spec:
endpoints: # prometheus所采集Metrics地址配置，endpoints為一個數(shù)組，可以創(chuàng)建多個，但是每個endpoints包含三個字段interval、path、port
- interval: 15s # prometheus采集數(shù)據(jù)的周期，單位為秒
path: /metrics # prometheus采集數(shù)據(jù)的路徑
port: prometheus # prometheus采集數(shù)據(jù)的端口，這里為port的name，主要是通過spec.selector中選擇對應(yīng)的svc，在選中的svc中匹配該端口
namespaceSelector: # 需要發(fā)現(xiàn)svc的范圍
any: true # 有且僅有一個值true，當(dāng)該字段被設(shè)置時，表示監(jiān)聽所有符合selector所選擇的svc
selector:
matchLabels: # 選擇svc的標(biāo)簽
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.1
kubectl apply -f ingress-nginx-monitor.yaml
確認(rèn)ingress-nginx servicemonitors已添加
kubectl -n monitoring get servicemonitors.monitoring.coreos.com
NAME AGE
alertmanager-main 70m
blackbox-exporter 70m
coredns 70m
grafana 70m
ingress-nginx 24m
kube-apiserver 70m
kube-controller-manager 70m
kube-scheduler 70m
kube-state-metrics 70m
kubelet 70m
node-exporter 70m
prometheus-adapter 70m
prometheus-k8s 70m
prometheus-operator 70m
監(jiān)控外部ETCD：
etcd配置文件添加：ETCD_LISTEN_METRICS_URLS=“http://0.0.0.0:2381”
etcd-job.yaml
- job_name: "etcd-server"
scrape_interval: 30s
scrape_timeout: 10s
static_configs:
- targets: ["172.16.0.157:2381"]
labels:
instance: etcdserver
創(chuàng)建secret
kubectl create secret generic etcd-secret --from-file=etcd-job.yaml -n monitoring
kube-prometheus/manifests/prometheus-prometheus.yaml 末尾添加配置：
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.46.0
name: k8s
namespace: monitoring
spec:
alerting:
alertmanagers:
- apiVersion: v2
name: alertmanager-main
namespace: monitoring
port: web
enableFeatures: []
externalLabels: {}
image: quay.io/prometheus/prometheus:v2.46.0
nodeSelector:
kubernetes.io/os: linux
podMetadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.46.0
podMonitorNamespaceSelector: {}
podMonitorSelector: {}
probeNamespaceSelector: {}
probeSelector: {}
replicas: 2
resources:
requests:
memory: 400Mi
ruleNamespaceSelector: {}
ruleSelector: {}
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: 2.46.0
additionalScrapeConfigs: #新增
name: etcd-secret #新增
key: etcd-job.yaml #新增
kubectl apply -f kube-prometheus/manifests/prometheus-prometheus.yaml
grafana導(dǎo)入 3070模板
http存活檢測
kind: Probe
apiVersion: monitoring.coreos.com/v1
metadata:
name: example-com-website
namespace: monitoring
spec:
interval: 60s
module: http_2xx
prober:
url: blackbox-exporter.monitoring.svc.cluster.local:19115
targets:
staticConfig:
static:
- https://www.baidu.com
添加proxy監(jiān)控，默認(rèn)為http文章來源：http://www.zghlxwxcb.cn/news/detail-658236.html
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/name: kube-proxy
app.kubernetes.io/part-of: kube-prometheus
name: kube-proxy
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: http-metrics
scheme: http
tlsConfig:
insecureSkipVerify: true
jobLabel: app.kubernetes.io/name
namespaceSelector:
matchNames:
- kube-system
selector:
matchLabels:
app.kubernetes.io/name: kube-proxy
---
apiVersion: v1
kind: Endpoints
metadata:
name: kube-proxy
namespace: kube-system
labels:
app.kubernetes.io/name: kube-proxy
subsets:
- addresses:
- ip: 172.16.0.157
targetRef:
kind: Node
name: master1
- ip: 172.16.0.124
targetRef:
kind: Node
name: node1
- ip: 172.16.0.46
targetRef:
kind: Node
name: node2
ports:
- name: http-metrics
port: 10249
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: kube-proxy
namespace: kube-system
labels:
app.kubernetes.io/name: kube-proxy
spec:
type: ClusterIP
clusterIP: None
ports:
- name: http-metrics
port: 10249
targetPort: 10249
protocol: TCP
文章來源地址http://www.zghlxwxcb.cn/news/detail-658236.html
到了這里，關(guān)于k8s v1.27.4 部署metrics-serverv:0.6.4，kube-prometheus的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
