ELK 日志系統(tǒng)收集K8s中日志

這篇具有很好參考價(jià)值的文章主要介紹了ELK 日志系統(tǒng)收集K8s中日志。希望對大家有所幫助。如果存在錯(cuò)誤或未考慮完全的地方，請大家不吝賜教，您也可以點(diǎn)擊"舉報(bào)違法"按鈕提交疑問。

容器特性給日志采集帶來的困難

? K8s彈性伸縮性：導(dǎo)致不能預(yù)先確定采集的目標(biāo)
? 容器隔離性：容器的文件系統(tǒng)與宿主機(jī)是隔離，導(dǎo)致日志采集器讀取日志文件受阻。

日志按體現(xiàn)方式分類

應(yīng)用程序日志記錄體現(xiàn)方式分為兩類：
? 標(biāo)準(zhǔn)輸出：輸出到控制臺，使用kubectl logs可以看到。
例如 nginx日志是將訪問日志輸出到標(biāo)準(zhǔn)輸出，可以用kubectl log 查看
elk收集k8s日志,kubernetes,elk

kubectl  logs ==>> apiserver ==>> kubecet ==>> docker api ==>><container-id>-json.log

elk收集k8s日志,kubernetes,elk

? 日志文件：寫到容器的文件系統(tǒng)的文件。

Kubernetes應(yīng)用日志收集

針對標(biāo)準(zhǔn)輸出：以DaemonSet方式在每個(gè)Node上部署一個(gè)日志收集程序，采集
/var/lib/docker/containers/目錄下所有容器日志。

針對容器中日志文件：在Pod中增加一個(gè)容器運(yùn)行日志采集器，使用emptyDir共享日志目錄讓日志采集器讀取到日志文件。

elk收集k8s日志,kubernetes,elk

ELK 日志系統(tǒng)

ELK 是三個(gè)開源軟件的縮寫，提供一套完整的企業(yè)級日志平臺解決方案。
分別是：
? Elasticsearch：搜索、分析和存儲數(shù)據(jù)
? Logstash ：采集日志、格式化、過濾，最后將數(shù)據(jù)
推送到Elasticsearch存儲
? Kibana：數(shù)據(jù)可視化
? Beats ：集合了多種單一用途數(shù)據(jù)采集器，用于實(shí)現(xiàn)從邊緣機(jī)器向 Logstash 和 Elasticsearch 發(fā)送數(shù)
據(jù)。里面應(yīng)用最多的是Filebeat，是一個(gè)輕量級日志采集器。

輕量級日志：graylog、grafana loki
elk收集k8s日志,kubernetes,elk
搭建日志系統(tǒng)：
? elasticsearch.yaml # ES數(shù)據(jù)庫
? kibana.yaml # 可視化展示
日志收集：
? filebeat-kubernetes.yaml # 采集所有容器標(biāo)準(zhǔn)輸出
? app-log-stdout.yaml # 標(biāo)準(zhǔn)輸出測試應(yīng)用
? app-log-logfile.yaml # 日志文件測試應(yīng)用

針對標(biāo)準(zhǔn)輸出：在每個(gè)節(jié)以deamsent方式部署filebeat，采集節(jié)點(diǎn)上所有的日志文件，目錄為/var/lib/docker/containers//容器id/xxxx-json.log

kubectl apply -f elasticsearch.yaml 
kubectl apply -f kibana.yaml 
kubectl apply -f filebeat-kubernetes.ya

 #elasticsearch.yaml
 apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: ops
  labels:
    k8s-app: elasticsearch
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: elasticsearch
  template:
    metadata:
      labels:
        k8s-app: elasticsearch
    spec:
      containers:
      - image: elasticsearch:7.9.2
        name: elasticsearch
        resources:
          limits:
            cpu: 2
            memory: 3Gi
          requests:
            cpu: 0.5
            memory: 500Mi
        env:
          - name: "discovery.type"
            value: "single-node"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx2g"
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: elasticsearch-data
        persistentVolumeClaim:
          claimName: es-pvc

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: es-pvc
  namespace: ops
spec:
  storageClassName: "managed-nfs-storage"
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

---

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: ops
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: 9200
  selector:
    k8s-app: elasticsearch

#kibana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: ops
  labels:
    k8s-app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana
  template:
    metadata:
      labels:
        k8s-app: kibana
    spec:
      containers:
      - name: kibana
        image: kibana:7.9.2
        resources:
          limits:
            cpu: 2
            memory: 2Gi
          requests:
            cpu: 0.5
            memory: 500Mi
        env:
          - name: ELASTICSEARCH_HOSTS
            value: http://10.244.169.139:9200
          - name: I18N_LOCALE
            value: zh-CN
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: ops
spec:
  type: NodePort
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
    nodePort: 30601
  selector:
    k8s-app: kibana


#filebeat-kubernetes.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: ops
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false

    output.elasticsearch:
      hosts: ['elasticsearch.ops:9200']
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: ops
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: ops
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: elastic/filebeat:7.9.2
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: inputs
          mountPath: /usr/share/filebeat/inputs.d
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: inputs
        configMap:
          defaultMode: 0600
          name: filebeat-inputs
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: ops
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: ops
  labels:
    k8s-app: filebeat

可視化展示日志：

查看索引（日志記錄集合）：Management -> Stack Management -> 索引管理
將索引關(guān)聯(lián)到Kibana：索引模式 -> 創(chuàng)建 -> 匹配模式 -> 選擇時(shí)間戳
在Discover選擇索引模式查看日志

將索引關(guān)聯(lián)到Kibana elk收集k8s日志,kubernetes,elk
Discover選擇索引模式查看日志

按條件查詢

二、針對容器中日志文件

#app-log-logfile.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-log-logfile
spec:
  replicas: 3
  selector:
    matchLabels:
      project: microservice
      app: nginx-logfile
  template:
    metadata:
      labels:
        project: microservice
        app: nginx-logfile
    spec:
      containers:
      # 應(yīng)用容器
      - name: nginx
        image: lizhenliang/nginx-php
        # 將數(shù)據(jù)卷掛載到日志目錄
        volumeMounts:
        - name: nginx-logs
          mountPath: /usr/local/nginx/logs
      # 日志采集器容器
      - name: filebeat
        image: elastic/filebeat:7.9.2
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            memory: 500Mi
        securityContext:
          runAsUser: 0
        volumeMounts:
        # 掛載filebeat配置文件
        - name: filebeat-config
          mountPath: /etc/filebeat.yml
          subPath: filebeat.yml
        # 將數(shù)據(jù)卷掛載到日志目錄
        - name: nginx-logs
          mountPath: /usr/local/nginx/logs
      # 數(shù)據(jù)卷共享日志目錄
      volumes:
      - name: nginx-logs
        emptyDir: {}
      - name: filebeat-config
        configMap:
          name: filebeat-nginx-config
---
apiVersion: v1
kind: Service
metadata:
  name: app-log-logfile
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    project: microservice
    app: nginx-logfile
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-nginx-config

data:
  # 配置文件保存在ConfigMap
  filebeat.yml: |-
    filebeat.inputs:
      - type: log
        paths:
          - /usr/local/nginx/logs/access.log
        # tags: ["access"]
        fields_under_root: true
        fields:
          project: microservice #項(xiàng)目名稱
          app: nginx            #pod名稱

    setup.ilm.enabled: false
    setup.template.name: "nginx-access"
    setup.template.pattern: "nginx-access-*"

    output.elasticsearch:
      hosts: ['elasticsearch.ops:9200']
      index: "nginx-access-%{+yyyy.MM.dd}" #索引名稱