k8s配置Prometheus監(jiān)控時，可以通過servicemonitor的方式增加job，以此來增加監(jiān)控項，但這種方式進行監(jiān)控配置，只能手工一個一個的增加，如果k8s集群規(guī)模較大的情況下，這種方式會很麻煩。

一種方式是采用consul注冊中心的方式進行自動發(fā)現(xiàn)。

另外一種方式是基于kubernetes_sd_configs的自動發(fā)現(xiàn)的方式配置增加監(jiān)控項，本文主要講解此種配置方式。

基于consul的自動發(fā)現(xiàn)

在安裝consul后，可以通過指定consul讀取特定配置文件的方式發(fā)現(xiàn)并加載監(jiān)控項

?但這種方式和Prometheus基于文件的動態(tài)發(fā)現(xiàn)沒有本質區(qū)別，甚至還增加了系統(tǒng)的復雜度，并不可取。

可以通過調(diào)用consul的API的方式，讓程序自動向consul進行注冊，在Prometheus中配置consul的相關項，讓其自動增加監(jiān)控Target

#Prometheus主配置文件增加如下內(nèi)容
#以便可以從consul中自動獲取監(jiān)控信息
   - job_name: 'consul-prometheus'
     consul_sd_configs:
       - server: '10.0.12.8:8500'
         services: []

# 注冊服務
curl -X PUT -d '{"id": "consul-redis","name": "redis","address": "10.0.12.8","port": 6379,"tags": ["service"],"checks": [{"http": "http://10.0.12.8:6379/","interval": "5s"}]}' http://10.0.12.8:8500/v1/agent/service/register
 
# 查詢指定節(jié)點以及指定的服務信息
[root@iZ2zejaz33icbod2k4cvy6Z ~]# curl http://10.0.12.8:8500/v1/catalog/service/consul-redis
 
#刪除指定服務 redis為要刪除服務的id
curl -X PUT  http://10.0.12.8:8500/v1/agent/service/deregister/consul-redis

基于kubernetes_sd_configs的自動發(fā)現(xiàn)

準備Prometheus的自動發(fā)現(xiàn)的配置文件并加載

[root@VM-12-8-centos kube-prom]# cat prometheus-additional.yaml 
- job_name: 'blackbox'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - http://10.1.226.250:6000
      - http://10.1.38.97:3000/healthz/ready
      - http://10.1.116.84:5000
      - http://10.1.215.125:7000/healthz/ready
      - http://10.1.111.235:8000/healthz/ready
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:9115
- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: service
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: pod
    action: replace
- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod

運行生成secret文件

[root@VM-12-8-centos kube-prom]# kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run -oyaml > additional-scrape-configs.yaml

應用，配置進入Prometheus中

[root@VM-12-8-centos kube-prom]# kubectl apply -f additional-scrape-configs.yaml -n monitoring 
secret/additional-scrape-configs configured

運行curl -XPOST http://10.0.12.8:30090/-/reload熱加載一下，就可以在dashboard中看到增加的配置了

修改prometheus-k8s 的 ClusterRole權限?

?Prometheus 綁定了一個名為 prometheus-k8s 的 ServiceAccount 對象，而這個對象綁定的是一個名為 prometheus-k8s 的 ClusterRole，這個角色沒有對 Service 或者 Pod 的 list 權限，所以需要進行修改

[root@VM-12-8-centos manifests]# kubectl edit clusterrole prometheus-k8s -n monitoring -o yaml  
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2022-11-13T14:21:08Z"
  name: prometheus-k8s
  resourceVersion: "16164985"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/prometheus-k8s
  uid: 7a404fac-9462-486a-a109-65a1ef98e423
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

?pod配置自動發(fā)現(xiàn)

pod要自動發(fā)現(xiàn)，必須在annotations:增加prometheus.io/scrape: "true"

新建一個pod

[root@VM-12-8-centos k8s]# cat PODforheadlesssvr.yml
apiVersion: v1
kind: Pod
metadata:
  name: ex-podforheadlesssvr
  annotations:
    prometheus.io/scrape: "true"
spec:
  containers:
  - name: testcontainer
    image: docker.io/appropriate/curl
    imagePullPolicy: IfNotPresent
    command: ['sh', '-c']
    args: ['echo "test pod for headless service";sleep 96000']
[root@VM-12-8-centos k8s]# kubectl apply -f ex6_1_4PODforheadlesssvr.yml
pod/ex-podforheadlesssvr created
[root@VM-12-8-centos k8s]# kubectl get po
NAME                                     READY   STATUS      RESTARTS   AGE
ex-podforheadlesssvr                     1/1     Running     0          3s

過一會檢查dashboard，已經(jīng)在界面上了

?狀態(tài)為down，因為這個pod對應的鏡像并沒有相關的metrics接口，我們主要是用來進行自動發(fā)現(xiàn)測試的

在服務發(fā)現(xiàn)界面

在target labels部分

如上操作，就可以基于k8s自動發(fā)現(xiàn)?在Prometheus中增加監(jiān)控項了

?文章來源地址http://www.zghlxwxcb.cn/news/detail-454705.html

到了這里，關于Prometheus基于k8s的自動發(fā)現(xiàn)配置監(jiān)控的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關文章，希望大家以后多多支持TOY模板網(wǎng)！