国产 无码 综合区,色欲AV无码国产永久播放,无码天堂亚洲国产AV,国产日韩欧美女同一区二区

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息

這篇具有很好參考價值的文章主要介紹了docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息。希望對大家有所幫助。如果存在錯誤或未考慮完全的地方,請大家不吝賜教,您也可以點擊"舉報違法"按鈕提交疑問。

Centos7 隨筆記錄記錄 docker compose 統(tǒng)一管理 granfana + loki + vector? 監(jiān)控kafka 信息。

當(dāng)然如果僅僅是想通過 Grafana 監(jiān)控kafka,推薦使用?Grafana Prometheus 通過JMX監(jiān)控kafka文章來源地址http://www.zghlxwxcb.cn/news/detail-769261.html

目錄

1. 目錄結(jié)構(gòu)

2. 前提已安裝Docker-Compose

3. docker-compose 自定義服務(wù)相關(guān)配置文件

3.1??配置 docker-compose-grafana-loki-vector.yml 文件

3.2 創(chuàng)建 loki-config.yaml?

3.3 配置loki 服務(wù)

4. docker-compose? 單獨啟動指定服務(wù)

4.1 單獨啟動 loki?

4.1.1?啟動loki

4.1.2 查詢 log?

4.2 單獨啟動vector?

4.2.1?啟動 vector

4.2.2 查詢vector log?

4.2.2.1 錯誤log 問題定位?

4.2.2.2 解決方法

4.2.2.3 重啟vector 容器,查看無error log

4.3 單獨啟動grafana?

4.3.1?啟動 grafana?

4.3.2 查看?grafana log?

4.3.2.1 錯誤log 問題定位?

4.3.2.2 解決方法

4.3.2.3 重啟grafana 容器,查看無error log

4.3.3 訪問grafana?

4.4??查看loki metrics數(shù)據(jù)? 和loki 運行狀態(tài)

5.?docker-compose? 啟停所有服務(wù)

5.1 docker-compose 啟動所有自定義容器服務(wù)

5.2 docker-compose 停止所有容器服務(wù)

6. 配置grafana

6.1 添加loki 數(shù)據(jù)源

6.2 修改時區(qū)

6.3 添加query?

6.4 過濾展示kafka 消息

6.5 優(yōu)化后 dashboard, 展示kafka 指定字段?

6.5.1 添加添加dashboard

6.5.2 優(yōu)化dashboard, 顯示指定字段

6.5.2.1? 篩選kafka 原始消息

?6.5.2.2 優(yōu)化表格顯示指定字段

7. 其他異常問題

7.1? grafana 頁面有時跳轉(zhuǎn) 網(wǎng)址 http://localhost:3000/XXXX 錯誤

7.2 跳轉(zhuǎn)錯誤網(wǎng)址原因定位?

7.3 解決方法

7.3.1 Grafana 服務(wù)的環(huán)境變量 GF_SERVER_ROOT_URL 設(shè)置為正確的訪問地址

7.3.2 為了grafana 有數(shù)據(jù),必須修改vector.tml 配置參數(shù)

7.3.3 使配置生效

?7.4 驗證 跳轉(zhuǎn)網(wǎng)址正確


1. 目錄結(jié)構(gòu)

# 1 創(chuàng)建 4個文件夾:grafana loki vector wal 
# 2 為 wal 賦777 權(quán)限

[root@localhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1859 Dec 26 10:58 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root   78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# tree
.
├── docker-compose-grafana-loki-vector.yml
└── grafana_loki_vector                     # 創(chuàng)建 grafana folder 
    ├── grafana
    ├── loki                                # 創(chuàng)建 loki folder
    │?? └── loki-config.yaml
    ├── vector                              # 創(chuàng)建 vector folder
    │?? └── vector.toml
    └── wal                                 # 創(chuàng)建 wal folder,并賦 777 權(quán)限
     

8 directories, 5 files
[root@localhost grafana_loki_vector]#


# 為避免docker 中權(quán)限問題,為wal folder 賦權(quán)限

[root@localhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector/grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# chmod 777 wal
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# ll
total 0
drwxr-xr-x. 4 root root 41 Dec 26 10:04 grafana
drwxr-xr-x. 2 root root 44 Dec 26 10:47 loki
drwxr-xr-x. 2 root root 33 Dec 21 15:03 vector
drwxrwxrwx. 3 root root 59 Dec 26 11:19 wal
[root@localhost grafana_loki_vector]#

2. 前提已安裝Docker-Compose

如何安裝 Docker-Compose,請參考?安裝部署Docker-Compose

3. docker-compose 自定義服務(wù)相關(guān)配置文件

3.1??配置 docker-compose-grafana-loki-vector.yml 文件
# docker-compose 配置參考:https://github.com/grafana/loki/blob/master/production/docker-compose.yaml

[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1859 Dec 26 10:58 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root   78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# cat docker-compose-grafana-loki-vector.yml

version: "3"   # 這是Docker Compose的版本指定,該配置文件使用版本3的語法

# 網(wǎng)橋loki -> 方便相互通訊
networks:      # 定義了一個名為loki的網(wǎng)絡(luò),該網(wǎng)絡(luò)是為了Loki和其他服務(wù)之間相互通信而創(chuàng)建的網(wǎng)橋。
  loki:

services:      # 在這個部分定義了要運行的各個服務(wù)
  loki:                                                  # Loki服務(wù)的配置
    image: grafana/loki:latest                           # 使用了grafana/loki:latest鏡像
    container_name: grafana_loki_vector_demo_loki        # 定義 容器名
    restart: unless-stopped                              # 設(shè)置了重啟策略
    ports:
      - "3100:3100"   # 將主機(jī)的3100端口映射到Loki容器的3100端口,以便能夠從主機(jī)訪問Loki的API
    volumes:          # 將兩個主機(jī)目錄與容器內(nèi)的目錄進(jìn)行了卷映射
      - ./grafana_loki_vector/loki:/etc/loki  
      - ./grafana_loki_vector/wal:/wal       
    command: -config.file=/etc/loki/loki-config.yaml
    networks:
      - loki

 # vector  用于對接kafka 消息
  vector:                                                # vector 服務(wù)的配置
    # 指定使用 timberio/vector的Docker 鏡像,并使用0.10.0-alpine 版本
    image: timberio/vector:0.10.0-alpine       
    container_name: grafana_loki_vector_demo_vector      # 定義 容器名
    restart: unless-stopped                   # 指定容器在退出時重新啟動,除非顯式停止
    ports:
      - "80:80"                               # 容器的 80 端口映射到主機(jī)的 80 端口上
    volumes:
      # - ./grafana_loki_vector/vector/logs:/var/logs

      # 本地主機(jī)目錄./grafana_loki_vector/vector掛載到容器的/etc/vector目錄上
      - "./grafana_loki_vector/vector:/etc/vector"

    # 這是要在容器中運行的命令,通過-c /etc/vector/vector.toml指定了配置文件的路徑,并通過 -v 啟用了詳細(xì)日志輸出
    command: -c /etc/vector/vector.toml -v
    networks:
      - loki

 # 用于UI展示
  grafana:
    image: grafana/grafana:latest
    container_name: grafana_loki_vector_demo_grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - "./grafana_loki_vector/grafana/data:/var/lib/grafana"
      #      - "./grafana_promtail_loki/grafana/config:/etc/grafana"
      - "./grafana_loki_vector/grafana/log:/var/log/grafana"

    environment:
      # GF_EXPLORE_ENABLED: "true"

      # 避免有時Grafana 會將跳轉(zhuǎn) URL 設(shè)置為 "localhost"
      - GF_SERVER_ROOT_URL=http://192.168.2.247:3000  
    networks:
      - loki

 


=====================================================================
配置參數(shù)說明:

 volumes:          # 將兩個主機(jī)目錄與容器內(nèi)的目錄進(jìn)行了卷映射
      - ./grafana_loki_vector/loki:/etc/loki  
      # ./grafana_loki_vector/loki映射到了/etc/loki

      - ./grafana_loki_vector/wal_tmp1:/wal   
      # ./grafana_loki_vector/wal_tmp1映射到了/wal

command: -config.file=/etc/loki/loki-local-config.yaml
# -config.file=/etc/loki/loki-local-config.yaml 指定了Loki容器啟動時使用的配置文件路徑

environment:
      # 避免有時Grafana 會將跳轉(zhuǎn) URL 設(shè)置為 "localhost"
      - GF_SERVER_ROOT_URL=http://192.168.2.247:3000  
3.2 創(chuàng)建 loki-config.yaml?
創(chuàng)建loki-local-config.yaml 文件,配置loki 服務(wù)


[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root   78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# tree
.
├── docker-compose-grafana-loki-vector.yml
└── grafana_loki_vector
    ├── grafana
    │?? ├── data
    │?? └── log
    ├── loki
    │?? └── loki-config.yaml       # 創(chuàng)建loki-config.yaml文件,配置loki 服務(wù)
    ├── vector
    │?? └── vector.toml
    └── wal
3.3 配置loki 服務(wù)
# 這個是Loki的配置文件示例,其中包含了各個方面的配置,用于自定義和配置Loki日志系統(tǒng)的行為和存儲。

auth_enabled: false # 設(shè)置認(rèn)證功能是否啟用。設(shè)為false表示禁用認(rèn)證,即任何用戶都可以訪問Loki服務(wù)

server:
  http_listen_port: 3100  # 配置Loki服務(wù)監(jiān)聽的HTTP端口。在這個示例中,Loki將監(jiān)聽3100端口

ingester:                 # 配置Loki的ingester(攝取器)相關(guān)參數(shù)
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:                  # 使用內(nèi)存作為kvstore存儲方式
        store: inmemory
      replication_factor: 1     # 復(fù)制因子設(shè)置為 1
    final_sleep: 0s
  chunk_idle_period: 5m         # chunk的閑置期 5m
  chunk_retain_period: 30s      # chunk的保留期 30s
 
schema_config:                  # 配置存儲模式參數(shù)
  configs:
    - from: 2018-04-15          # 從特定日期開始使用的配置
      store: boltdb             # 數(shù)據(jù)存儲方式(boltdb,文件系統(tǒng))
      object_store: filesystem  # 數(shù)據(jù)存儲方式 (文件系統(tǒng))
      schema: v9                # schema版本
      index:                    # 索引相關(guān)參數(shù)
        prefix: index_
        period: 168h

storage_config:                 # 配置存儲配置,指定Loki數(shù)據(jù)的持久化存儲位置
  boltdb:
    directory: /tmp/loki/index  # 索引數(shù)據(jù)將存儲在/tmp/loki/index目錄

  filesystem:
    directory: /tmp/loki/chunks # chunk數(shù)據(jù)將存儲在/tmp/loki/chunks目錄

# 配置限制參數(shù),用于強(qiáng)制限制指標(biāo)名稱、拒絕舊樣本數(shù)據(jù),以及設(shè)置拒絕舊樣本數(shù)據(jù)的最大時間范圍
limits_config:     
  enforce_metric_name: false        # 強(qiáng)制限制指標(biāo)名稱
  reject_old_samples: true          # 拒絕舊樣本數(shù)據(jù)
  reject_old_samples_max_age: 168h  # 設(shè)置拒絕舊樣本數(shù)據(jù)的最大時間范圍

chunk_store_config:                 # 配置chunk存儲參數(shù),設(shè)置最大回溯時間范圍
  max_look_back_period: 0s          # 設(shè)置最大回溯時間范圍

table_manager:                         # 配置表管理器相關(guān)參數(shù)
  chunk_tables_provisioning:           # 設(shè)置chunk表的吞吐量
    inactive_read_throughput: 0   
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  index_tables_provisioning:           # 設(shè)置索引表的吞吐量
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  retention_deletes_enabled: false     # 保留刪除是否啟用
  retention_period: 0s                 # 保留期設(shè)置
    # retention_deletes_enabled: true
    # retention_period: 168h

4. docker-compose? 單獨啟動指定服務(wù)

建議docker-compose 自定義多個服務(wù)時,先單獨啟動每個服務(wù)調(diào)試,沒問題后,再統(tǒng)一啟動

4.1 單獨啟動 loki?
4.1.1?啟動loki

# 啟動指定自定義服務(wù) 

#  docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up <服務(wù)名> -d


# 啟動loki

[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root   78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#  docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up loki -d
[+] Running 7/7
 ? loki 6 layers [??????]      0B/0B      Pulled                                                                                                                        24.0s
   ? c926b61bad3b Pull complete                                                                                                                                         12.0s
   ? 21a0947e3c21 Pull complete                                                                                                                                          6.0s
   ? 19779106cfff Pull complete                                                                                                                                         17.5s
   ? 9c6ee7a30a41 Pull complete                                                                                                                                          7.4s
   ? 2f9c53357ffd Pull complete                                                                                                                                          8.7s
   ? d8cb9832c63f Pull complete                                                                                                                                         11.4s
[+] Running 1/1
 ? Container grafana_loki_vector_demo_loki  Started                                                                                                                      0.2s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps        # 查詢啟動容器
CONTAINER ID   IMAGE                    COMMAND                  CREATED         STATUS         PORTS                                                                   NAMES
58528f1b4782   grafana/loki:latest      "/usr/bin/loki -conf…"   9 seconds ago   Up 8 seconds   0.0.0.0:3100->3100/tcp, :::3100->3100/tcp                               grafana_loki_vector_demo_loki
c4f7b69648a8   wurstmeister/kafka       "start-kafka.sh"         18 hours ago    Up 18 hours    0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                               kafka
a893cce0d465   wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   18 hours ago    Up 18 hours    22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp   zookeeper
[root@localhost grafana_loki_vector]#

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

4.1.2 查詢 log?
# 查看 loki 容器 log
# docker logs <loki服務(wù)容器名>

[root@localhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root   78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_loki
level=warn ts=2023-12-26T03:39:33.891828641Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default per-tenant timeout (\"5m\")."

''''''
''''''
''''''
level=info ts=2023-12-26T03:44:33.988929326Z caller=checkpoint.go:502 msg="atomic checkpoint finished" old=wal/checkpoint.000006.tmp new=wal/checkpoint.000006
[root@localhost grafana_loki_vector]#
4.2 單獨啟動vector?
4.2.1?啟動 vector

# 啟動指定自定義服務(wù) 

#  docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up <服務(wù)名> -d


# 啟動 vector

[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root   78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up vector -d
[+] Running 6/6
 ? vector 5 layers [?????]      0B/0B      Pulled                                                                                                                       30.6s
   ? 21c83c524219 Pull complete                                                                                                                                         18.0s
   ? 88943f7483ee Pull complete                                                                                                                                          2.9s
   ? 24da72aaf03b Pull complete                                                                                                                                         23.4s
   ? 4145aaf02e3f Pull complete                                                                                                                                         10.7s
   ? b16d307dc057 Pull complete                                                                                                                                         12.0s
[+] Running 1/1
 ? Container grafana_loki_vector_demo_vector  Started                                                                                                                    0.3s
[root@localhost grafana_loki_vector]#

[root@localhost grafana_loki_vector]# docker ps        # 查詢啟動容器
CONTAINER ID   IMAGE                           COMMAND                  CREATED              STATUS              PORTS                                                                   NAMES
776ab907da32   timberio/vector:0.10.0-alpine   "/usr/local/bin/vect…"   About a minute ago   Up About a minute   0.0.0.0:80->80/tcp, :::80->80/tcp                                       grafana_loki_vector_demo_vector
2a5581eea2eb   grafana/loki:latest             "/usr/bin/loki -conf…"   2 minutes ago        Up 2 minutes        0.0.0.0:3100->3100/tcp, :::3100->3100/tcp                               grafana_loki_vector_demo_loki
c4f7b69648a8   wurstmeister/kafka              "start-kafka.sh"         21 hours ago         Up 21 hours         0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                               kafka
a893cce0d465   wurstmeister/zookeeper          "/bin/sh -c '/usr/sb…"   21 hours ago         Up 21 hours         22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp   zookeeper
[root@localhost grafana_loki_vector]#
4.2.2 查詢vector log?
# 查看 vector 容器log
# docker logs <vector 容器名>

[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_vector
Dec 26 06:02:01.842  INFO vector: Log level "debug" is enabled.
Dec 26 06:02:01.857  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Dec 26 06:02:01.862  INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-377-g0f0311a" released="Wed, 22 Jul 2020 20:01:29 +0000" arch="x86_64"
Dec 26 06:02:01.876 DEBUG vector::tls::settings: Fetching system root certs.
Dec 26 06:02:01.876  INFO vector::topology: Running healthchecks.
Dec 26 06:02:01.876  INFO vector::topology: Starting source "in"
Dec 26 06:02:01.876  INFO vector::topology: Starting sink "out"
Dec 26 06:02:01.876 DEBUG http: vector::sinks::util::http: sending request. uri=http://loki:3100/ready method=GET
Dec 26 06:02:01.884 DEBUG http: vector::sinks::util::http: response. status=503 version=HTTP/1.1
Dec 26 06:02:01.885 ERROR vector::topology::builder: Healthcheck: Failed Reason: A non-successful status returned: 503 Service Unavailable         # Error Log
Dec 26 06:02:05.978 DEBUG http: vector::sinks::util::http: sending request. uri=http://loki:3100/loki/api/v1/push method=POST
Dec 26 06:02:06.060 DEBUG http: vector::sinks::util::http: response. status=204 version=HTTP/1.1
[root@localhost grafana_loki_vector]#
4.2.2.1 錯誤log 問題定位?
# 確認(rèn) Vector 可以連接到 Loki 服務(wù)
在 Vector 配置文件中配置的 Loki 服務(wù)的地址應(yīng)該是可訪問的。
可以嘗試使用 curl 或類似的工具連接到 Loki 服務(wù)

# curl http://loki:3100/ready

[root@localhost grafana_loki_vector]# curl http://loki:3100/ready
curl: (6) Could not resolve host: loki; Unknown error
[root@localhost grafana_loki_vector]#


總結(jié): curl 請求無法解析 loki 主機(jī)名,并出現(xiàn)了 Unknown error 錯誤,vector 無法連接loki
4.2.2.2 解決方法
1) 檢查 DNS 配置:確認(rèn)您的 DNS 配置正確,可以解析主機(jī)名
# nslookup loki

[root@localhost grafana_loki_vector]# nslookup loki
Server:         114.114.114.114
Address:        114.114.114.114#53

** server can't find loki: NXDOMAIN   # loki無法解析為有效的 IP 地址,返回了 NXDOMAIN 錯誤。

[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#


2)添加主機(jī)名映射:如果 DNS 解析不起作用,您可以嘗試手動添加主機(jī)名到 IP 地址的映射

[root@localhost grafana_loki_vector]# cat /etc/hosts          # 添加前
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@localhost grafana_loki_vector]# 
[root@localhost grafana_loki_vector]# cat /etc/hosts          # 添加后
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.247 loki
[root@localhost grafana_loki_vector]#


3)檢查vector 是否可以連接loki
# curl http://loki:3100/ready

[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# curl http://loki:3100/ready
ready                                                # 可以連接loki
[root@localhost grafana_loki_vector]#

4.2.2.3 重啟vector 容器,查看無error log
# 重啟vector 容器 or 重新創(chuàng)建 vector 容器
# 無error log, vector 服務(wù)運行正常

[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_vector
Dec 26 06:20:58.569  INFO vector: Log level "debug" is enabled.
Dec 26 06:20:58.583  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Dec 26 06:20:58.588  INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-377-g0f0311a" released="Wed, 22 Jul 2020 20:01:29 +0000" arch="x86_64"
Dec 26 06:20:58.602 DEBUG vector::tls::settings: Fetching system root certs.
Dec 26 06:20:58.602  INFO vector::topology: Running healthchecks.
Dec 26 06:20:58.602  INFO vector::topology: Starting source "in"
Dec 26 06:20:58.602  INFO vector::topology: Starting sink "out"
Dec 26 06:20:58.602 DEBUG http: vector::sinks::util::http: sending request. uri=http://loki:3100/ready method=GET
Dec 26 06:20:58.606 DEBUG http: vector::sinks::util::http: response. status=200 version=HTTP/1.1
Dec 26 06:20:58.607  INFO vector::topology::builder: Healthcheck: Passed.
[root@localhost grafana_loki_vector]#
4.3 單獨啟動grafana?
4.3.1?啟動 grafana?

# 啟動指定自定義服務(wù) 

#  docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up <服務(wù)名> -d

[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up grafana -d
[+] Running 11/11
 ? grafana 10 layers [??????????]      0B/0B      Pulled                                                                                                               106.8s
   ? 96526aa774ef Pull complete                                                                                                                                          2.3s
   ? 8107a14e7703 Pull complete                                                                                                                                          1.4s
   ? 41aaa20cd87e Pull complete                                                                                                                                          2.7s
   ? e00aef65f013 Pull complete                                                                                                                                          5.1s
   ? 257ea9b0297f Pull complete                                                                                                                                          5.5s
   ? 599ebbeda3b8 Pull complete                                                                                                                                          6.1s
   ? ca6b21901a74 Pull complete                                                                                                                                         59.8s
   ? b5fb47d1c454 Pull complete                                                                                                                                         92.5s
   ? 4fc0f14a8243 Pull complete                                                                                                                                          8.0s
   ? f638e65e9fc0 Pull complete                                                                                                                                         13.7s
[+] Running 1/1
 ? Container grafana_loki_vector_demo_grafana  Started                                                                                                                   0.3s
[root@localhost grafana_loki_vector]#
4.3.2 查看?grafana log?
# 查看 vector 容器log
# docker logs <vector 容器名>

[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_grafana
GF_PATHS_DATA='/var/lib/grafana' is not writable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied
.....
....

[root@localhost grafana_loki_vector]#
4.3.2.1 錯誤log 問題定位?

這些錯誤通常表示在容器中的某個目錄或文件上沒有寫入權(quán)限。

4.3.2.2 解決方法
# 解決此問題的一種可能方法是確保您在運行容器時具有足夠的權(quán)限

# 確保您以具有適當(dāng)權(quán)限的用戶身份運行 Docker 命令
# sudo usermod -aG docker $USER

# 更改目標(biāo)目錄的所有者和權(quán)限
sudo chown -R 472:472 ./grafana_loki_vector/grafana/data
sudo chmod -R 755 ./grafana_loki_vector/grafana/data


# 檢查您的目標(biāo)目錄權(quán)限,并確保它具有足夠的寫入權(quán)限

[root@localhost grafana_loki_vector]# sudo chmod -R 777 ./grafana_loki_vector/grafana/data
[root@localhost grafana_loki_vector]#
4.3.2.3 重啟grafana 容器,查看無error log
?
# 重啟grafana 容器 or 重新創(chuàng)建 grafana 容器
# 無error log, grafana服務(wù)運行正常

# 此處每次都是:
1) 停止容器 # docker stop <container_ID or container_name>
2) 刪除容器 # docker rm <container_ID or container_name>
3) 刪除鏡像 # docker rmi <image_ID>
4)  創(chuàng)建 grafana 容器
5) 查看 log


[root@localhost grafana_loki_vector]# docker stop 839b53f097ee   # 停止容器
839b53f097ee
[root@localhost grafana_loki_vector]# docker rm 839b53f097ee     # 刪除容器
839b53f097ee
[root@localhost grafana_loki_vector]# docker images              # 查詢鏡像ID
REPOSITORY               TAG             IMAGE ID       CREATED         SIZE
grafana/grafana          latest          8387f19108f9   7 days ago      399MB
grafana/loki             latest          c2c89eb81f6e   13 days ago     67.7MB
hello-world              latest          9c7a54a9a43c   7 months ago    13.3kB
wurstmeister/kafka       latest          a692873757c0   19 months ago   468MB
timberio/vector          0.10.0-alpine   2ebd263245dd   3 years ago     80.7MB
wurstmeister/zookeeper   latest          3f43f72cb283   4 years ago     510MB
[root@localhost grafana_loki_vector]# docker rmi 8387f19108f9     # 刪除鏡像
Untagged: grafana/grafana:latest
Untagged: grafana/grafana@sha256:6b5b37eb35bbf30e7f64bd7f0fd41c0a5b7637f65d3bf93223b04a192b8bf3e2
Deleted: sha256:8387f19108f986254ecb7eeb1124445a9aeda16602114d98348ed5cd49151a7a
Deleted: sha256:06619c984d0071cd4b3a8fbfb377c1ca5b991497f4e1f6c2f280b9f33d402cb4
Deleted: sha256:5fe5dac0845a9eb44e2c5cfed4fb2754a4adf3224209f58c1a46cb7bd250f2e7
Deleted: sha256:91a3b38581f407bb488e03f002566138875732ba1da4a5e91c55b3dec08a2756
Deleted: sha256:18c0364cc25d61f2e9c8ba8582944e1fc6528ba44abc25ffc3e3bc5ed6a8dec9
Deleted: sha256:e335659dfae7632b652b2c796e742f11b2a9f0c0c7def8511b4df29158a50ee5
Deleted: sha256:5f7c41d4921ce413887919068764757d42744f4898c3f790953fee2e1b33b79b
Deleted: sha256:e8df864ddb32cefc11b6a549add118948327816e858570f6fcbb4b393643d196
Deleted: sha256:14b7f68d3d9094af53f80ebf90273ec864b731545874ee30cfb09cae5d097935
Deleted: sha256:baf151d12231a621fead4ca244823c83fb4ae6e01c36501e37c48a964138e724
Deleted: sha256:cc2447e1835a40530975ab80bb1f872fbab0f2a0faecf2ab16fbbb89b3589438
[root@localhost grafana_loki_vector]#


# 創(chuàng)建 grafana 容器 

[root@localhost grafana_loki_vector]# sudo docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up grafana -d
[+] Running 11/11
 ? grafana 10 layers [??????????]      0B/0B      Pulled                                                                                                               113.6s
   ? 96526aa774ef Pull complete                                                                                                                                         12.7s
   ? 8107a14e7703 Pull complete                                                                                                                                          2.3s
   ? 41aaa20cd87e Pull complete                                                                                                                                          2.2s
   ? e00aef65f013 Pull complete                                                                                                                                         10.4s
   ? 257ea9b0297f Pull complete                                                                                                                                          5.5s
   ? 599ebbeda3b8 Pull complete                                                                                                                                         10.5s
   ? ca6b21901a74 Pull complete                                                                                                                                         83.9s
   ? b5fb47d1c454 Pull complete                                                                                                                                         97.5s
   ? 4fc0f14a8243 Pull complete                                                                                                                                         17.4s
   ? f638e65e9fc0 Pull complete                                                                                                                                         28.6s
[+] Running 1/1
 ? Container grafana_loki_vector_demo_grafana  Started                                                                                                                   0.5s
[root@localhost grafana_loki_vector]# 


# 查看 log
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_grafana
logger=settings t=2023-12-26T07:25:30.701353225Z level=info msg="Starting Grafana" version=10.2.3 commit=1e84fede543acc892d2a2515187e545eb047f237 branch=HEAD compiled=2023-12-18T15:46:07Z
logger=settings t=2023-12-26T07:25:30.701604782Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
......
......
...... 

logger=ngalert.scheduler t=2023-12-26T07:26:18.891340858Z level=info msg="Starting scheduler" tickInterval=10s
logger=ticker t=2023-12-26T07:26:18.891454409Z level=info msg=starting first_tick=2023-12-26T07:26:20Z
logger=http.server t=2023-12-26T07:26:18.897364437Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=plugins.update.checker t=2023-12-26T07:26:19.502508394Z level=info msg="Update check succeeded" duration=611.312832ms
logger=grafana.update.checker t=2023-12-26T07:26:20.504658267Z level=info msg="Update check succeeded" duration=1.613658029s
logger=infra.usagestats t=2023-12-26T07:26:51.904796949Z level=info msg="Usage stats are ready to report"
[root@localhost grafana_loki_vector]# 
4.3.3 訪問grafana?

打開 http://<grafana 容器所在主機(jī)IP>:3000

默認(rèn) user/pwd: admin/admin

?docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

4.4??查看loki metrics數(shù)據(jù)? 和loki 運行狀態(tài)
??#   查看loki metrics數(shù)據(jù)?
  #  訪問:http://<loki 容器所在主機(jī)IP>:3100/metrics

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

#  查看loki運行狀態(tài)
#  訪問:http://<loki 容器所在主機(jī)IP>:3100/ready

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

到此已經(jīng)調(diào)試完成,可以通過docker-compose 統(tǒng)一啟動所有服務(wù)?

5.?docker-compose? 啟停所有服務(wù)

5.1 docker-compose 啟動所有自定義容器服務(wù)
# 啟動 定義服務(wù)
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d

[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
[+] Running 3/3
 ? Container grafana_loki_vector_demo_loki     Started                                                                                                                   0.0s
 ? Container grafana_loki_vector_demo_vector   Started                                                                                                                   0.0s
 ? Container grafana_loki_vector_demo_grafana  Started                                                                                                                   0.0s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps          # 查詢啟動容器
CONTAINER ID   IMAGE                           COMMAND                  CREATED        STATUS         PORTS                                                                   NAMES
252ca4e81d6f   grafana/grafana:latest          "/run.sh"                2 hours ago    Up 2 seconds   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                               grafana_loki_vector_demo_grafana
7068d0da2577   timberio/vector:0.10.0-alpine   "/usr/local/bin/vect…"   3 hours ago    Up 2 seconds   0.0.0.0:80->80/tcp, :::80->80/tcp                                       grafana_loki_vector_demo_vector
2a5581eea2eb   grafana/loki:latest             "/usr/bin/loki -conf…"   3 hours ago    Up 2 seconds   0.0.0.0:3100->3100/tcp, :::3100->3100/tcp                               grafana_loki_vector_demo_loki
c4f7b69648a8   wurstmeister/kafka              "start-kafka.sh"         24 hours ago   Up 24 hours    0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                               kafka
a893cce0d465   wurstmeister/zookeeper          "/bin/sh -c '/usr/sb…"   24 hours ago   Up 24 hours    22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp   zookeeper
[root@localhost grafana_loki_vector]#

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

?docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

5.2 docker-compose 停止所有容器服務(wù)
# docker-compose 停止所有容器服務(wù)

# docker-compose -f <docker-compose 自定義容器服務(wù)配置 yml 文件> -p <項目名> down


[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1909 Dec 28 17:27 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root   78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down
[+] Running 4/4
 ? Container grafana_loki_vector_demo_vector   Removed                                                                                                                   0.4s
 ? Container grafana_loki_vector_demo_loki     Removed                                                                                                                   2.0s
 ? Container grafana_loki_vector_demo_grafana  Removed                                                                                                                   0.0s
 ? Network grafana_loki_vector_loki            Removed                                                                                                                   0.2s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps -a
CONTAINER ID   IMAGE                    COMMAND                  CREATED       STATUS                   PORTS                                                                   NAMES
c4f7b69648a8   wurstmeister/kafka       "start-kafka.sh"         3 days ago    Up 3 days                0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                               kafka
a893cce0d465   wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   3 days ago    Up 3 days                22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp   zookeeper
b38e9b5b6a2e   hello-world              "/hello"                 3 weeks ago   Exited (0) 3 weeks ago                                                                           infallible_rosalind
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker images
REPOSITORY               TAG             IMAGE ID       CREATED         SIZE
grafana/grafana          latest          8387f19108f9   9 days ago      399MB
grafana/loki             latest          c2c89eb81f6e   2 weeks ago     67.7MB
hello-world              latest          9c7a54a9a43c   7 months ago    13.3kB
wurstmeister/kafka       latest          a692873757c0   19 months ago   468MB
timberio/vector          0.10.0-alpine   2ebd263245dd   3 years ago     80.7MB
wurstmeister/zookeeper   latest          3f43f72cb283   4 years ago     510MB
[root@localhost grafana_loki_vector]#

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

6. 配置grafana

6.1 添加loki 數(shù)據(jù)源

如果找不到data Source, 可以直接search 查找。

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eurekadocker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

6.2 修改時區(qū)

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eurekadocker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

6.3 添加query?
click 'Home' -> Explore -> change 'loki' option

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

6.4 過濾展示kafka 消息
原始kafka 消息數(shù)據(jù):

{
    "currentIndex":0,
    "header":{
        "headerMap":{
            "35":"381166"
        }
    },
    "mapList":[
        {
            "10007":"1"
        },
        {
            "37":"48",
            "38":"300000",
            "44":"15007900",
            "48":"688021",
            "1301":"1",
            "11001":"國盾量子",
            "11436":"20231227",
            "28000":"20231227",
            "28001":"1703642600915",
            "28005":"2",
            "28101":"893786271",
            "43220":"金額:[3000000,450416980],數(shù)量:[300000,300100],價格漲跌幅:[0.02,0.19934],市場成交量占比:[0.3,0.967752],區(qū)間內(nèi)最高賣出成交價: 1799.8,昨收盤價: 1500,下單數(shù)量: 300000,當(dāng)前交易價格: 1500.79,掛單數(shù)量: 0,掛單金額: 0,成交數(shù)量: 100,成交金額: 179980,市場成交數(shù)量: 10100",
            "64101":"2403",
            "64102":"1101",
            "64103":"2403-4096個賬戶-4096個證券",
            "65098":"1",
            "66002":"5",
            "66003":"1",
            "66004":"0",
            "66005":"2",
            "91001":"200000",
            "93020":"2"
        }
    ]
}

將以上kafka 原始消息數(shù)據(jù),篩選過濾出指定字段,語句如下:?

# 將以上kafka 原始消息數(shù)據(jù),篩選過濾出指定字段,語句如下:?

{key="mag1225"}|= ""|json| line_format "{{.message}}" | json mapList1="mapList[1]",ruleID="mapList[1][\"64101\"]",rcSysId="mapList[1][\"66002\"]",logDate="mapList[1][\"28000\"]",logTime="mapList[1][\"28001\"]",securityExchange="mapList[1][\"1301\"]",threshold="mapList[1][\"43220\"]",stockName="mapList[1][\"11001\"]",securityId="mapList[1][\"48\"]",orderId="mapList[1][\"37\"]",orderBs="mapList[1][\"93020\"]",tradeSysId="mapList[1][\"66003\"]",riskConCaption="mapList[1][\"64103\"]",riskConType="mapList[1][\"64102\"]"|line_format "{{.mapList1}}"| __error__=``

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

6.5 優(yōu)化后 dashboard, 展示kafka 指定字段?
{key="mag1225"}|= ""|json| line_format "{{.message}}" | json mapList1="mapList[1]"|line_format "{{.mapList1}}"| __error__=`` 
6.5.1 添加添加dashboard

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

Clieck 'Add visualizaion' Button? or click 'Add'-> Visualization

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eurekadocker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eurekadocker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

6.5.2 優(yōu)化dashboard, 顯示指定字段
6.5.2.1? 篩選kafka 原始消息
clieck Query 輸入 將 kafka 原始消息數(shù)據(jù),篩選過濾出指定字段語句?

過濾語句:
{key="mag1225"}|= ""|json| line_format "{{.message}}" | json mapList1="mapList[1]",ruleID="mapList[1][\"64101\"]",rcSysId="mapList[1][\"66002\"]",logDate="mapList[1][\"28000\"]",logTime="mapList[1][\"28001\"]",securityExchange="mapList[1][\"1301\"]",threshold="mapList[1][\"43220\"]",stockName="mapList[1][\"11001\"]",securityId="mapList[1][\"48\"]",orderId="mapList[1][\"37\"]",orderBs="mapList[1][\"93020\"]",tradeSysId="mapList[1][\"66003\"]",riskConCaption="mapList[1][\"64103\"]",riskConType="mapList[1][\"64102\"]"|line_format "{{.mapList1}}"| __error__=``


注意:
key=<vector.toml 中sinks.out中的自定義的labels.key>


[root@localhost vector]# cat vector.toml


#                                    __   __  __
#                                    \ \ / / / /
#                                     \ V / / /
#                                      \_/  \/
#
#                                    V E C T O R
#                                   Configuration
#
# ------------------------------------------------------------------------------
# Website: https://vector.dev
# Docs: https://vector.dev/docs/
# ------------------------------------------------------------------------------

# Note: A full config spec is located at ./vector.spec.toml and examples
#       in the ./examples folder.


# Vector config kafka + Loki

[ sources.in ]
    type = "kafka"
    bootstrap_servers = "192.168.2.247:9092"      # kafka 地址
    group_id = "kafka_mag_groupId1225"               # "<消費組id>"
    topics = ["alarm_warning","kafka_consumer"]                    # kafka_consumer
    auto_offset_reset = "smallest"                # 從最早未處理的消息開始消費
    # topics = ["^(prefix1|prefix2)-.+", "topic-1", "topic-2"]  # topic名字,支持正則
    #[sources.in.encoding]
    #codec = "json"
        #timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
        #json.fields = ["level", "message"]

[sinks.out]
    endpoint = "http://loki:3100"
    inputs = ["in"]                              # source.in
    type = "loki"
    labels.key = "mag1225"                       # 自定義key
#  labels.key = "{{ event_field }}"              # event的動態(tài)值
    #[sinks.out.encoding]
    #codec = "json"
        # timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
        # json.fields = ["level", "message"]
[root@localhost vector]#

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

?6.5.2.2 優(yōu)化表格顯示指定字段
# 優(yōu)化已表格顯示指定字段


1. Click 'Transform?data' --> Add Transformation --> Reformat -> Extract fields
2. 添加Extract files -->Source:Line -->Format:JSON?--> 打開 'Replacee all fields' --> 打開'Keep time'


重名列名->調(diào)整順序->顯示隱藏指定列,配置如下:
3. click 'Add another transformation' --> 'Reorder and rename' --> 'Organize fields by name' 

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eurekadocker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka?docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

7. 其他異常問題

7.1? grafana 頁面有時跳轉(zhuǎn) 網(wǎng)址 http://localhost:3000/XXXX 錯誤
grafana 容器所在主機(jī)IP:192.168.2.247
grafana web address: http://192.168.2.247:3000
grafana web 頁面有時操作跳轉(zhuǎn) 網(wǎng)址:http://localhost:3000/XXXX,導(dǎo)致頁面跳轉(zhuǎn)錯誤
7.2 跳轉(zhuǎn)錯誤網(wǎng)址原因定位?
原因定位:
 Grafana Explore 頁面上的 "Add to Dashboard" 后,跳轉(zhuǎn)網(wǎng)址為 "http://localhost:3000/..." 的問題可能是因為 Grafana 配置中使用了相對 URL 路徑導(dǎo)致的
7.3 解決方法
7.3.1 Grafana 服務(wù)的環(huán)境變量 GF_SERVER_ROOT_URL 設(shè)置為正確的訪問地址
1. 修改 docker-compose-grafana-loki-vector.yml 文件,
   將 Grafana 服務(wù)的環(huán)境變量 GF_SERVER_ROOT_URL 設(shè)置為正確的訪問地址 

environment:
  - GF_SERVER_ROOT_URL=http://192.168.2.247:3000


[root@localhost grafana_loki_vector]# cat docker-compose-grafana-loki-vector.yml
# docker-compose配置參考:https://github.com/grafana/loki/blob/master/production/docker-compose.yaml
version: "3"

# 網(wǎng)橋loki -> 方便相互通訊
networks:
  loki:

services:
  # Loki日志系統(tǒng)
  loki:
    image: grafana/loki:latest
    container_name: grafana_loki_vector_demo_loki
    restart: unless-stopped
    ports:
      - "3100:3100"
    volumes:
      - ./grafana_loki_vector/loki:/etc/loki
      - ./grafana_loki_vector/wal:/wal
    command: -config.file=/etc/loki/loki-config.yaml
    networks:
      - loki

  # 用于UI展示
  grafana:
    image: grafana/grafana:latest
    container_name: grafana_loki_vector_demo_grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - "./grafana_loki_vector/grafana/data:/var/lib/grafana"
      #      - "./grafana_promtail_loki/grafana/config:/etc/grafana"
      - "./grafana_loki_vector/grafana/log:/var/log/grafana"
    environment:
      # GF_EXPLORE_ENABLED: "true"
      - GF_SERVER_ROOT_URL=http://192.168.2.247:3000  # 修改正確網(wǎng)址 grafana 容器所在主機(jī)

    networks:
      - loki

  # vector  用于對接kafka 消息
  vector:
    image: timberio/vector:0.10.0-alpine
    container_name: grafana_loki_vector_demo_vector
    restart: unless-stopped
    ports:
      - "80:80"
    volumes:
      # - ./grafana_loki_vector/vector/logs:/var/logs
      - "./grafana_loki_vector/vector:/etc/vector"
    command: -c /etc/vector/vector.toml -v
    networks:
      - loki

[root@localhost grafana_loki_vector]#
7.3.2 為了grafana 有數(shù)據(jù),必須修改vector.tml 配置參數(shù)
# 為了grafana 展示kafka 數(shù)據(jù),必須修改vector.tml 文件中2個參數(shù)
  1. group_id = "kafka_mag_groupId1228"           # 任意值 從 *1225 改為 *1228
  2. labels.key = "mag1228"                       # 自定義key    從 *1225 改為 *1228


[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1909 Dec 28 17:27 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root   78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# cat grafana_loki_vector/vector/vector.toml


#                                    __   __  __
#                                    \ \ / / / /
#                                     \ V / / /
#                                      \_/  \/
#
#                                    V E C T O R
#                                   Configuration
#
# ------------------------------------------------------------------------------
# Website: https://vector.dev
# Docs: https://vector.dev/docs/
# ------------------------------------------------------------------------------

# Note: A full config spec is located at ./vector.spec.toml and examples
#       in the ./examples folder.


# Vector config kafka + Loki

[ sources.in ]
    type = "kafka"
    bootstrap_servers = "192.168.2.247:9092"      # kafka 地址
    group_id = "kafka_mag_groupId1228"               # "<消費組id>" 修改group_ID
    topics = ["alarm_warning","kafka_consumer"]                    # kafka_consumer
    auto_offset_reset = "smallest"                # 從最早未處理的消息開始消費
    # topics = ["^(prefix1|prefix2)-.+", "topic-1", "topic-2"]  # topic名字,支持正則
    #[sources.in.encoding]
    #codec = "json"
        #timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
        #json.fields = ["level", "message"]

[sinks.out]
    endpoint = "http://loki:3100"
    inputs = ["in"]                              # source.in
    type = "loki"
    labels.key = "mag1228"                       # 自定義key
#  labels.key = "{{ event_field }}"              # event的動態(tài)值
    #[sinks.out.encoding]
    #codec = "json"
        # timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
        # json.fields = ["level", "message"]
[root@localhost grafana_loki_vector]#
7.3.3 使配置生效
# 使配置生效
2. docker-compose 關(guān)閉項目
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down

3. docker-copose 啟動項目即可。
#  docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d



# docker-compose 關(guān)閉項目

[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down
[+] Running 4/4
 ? Container grafana_loki_vector_demo_vector   Removed                                                                                                                   0.5s
 ? Container grafana_loki_vector_demo_loki     Removed                                                                                                                   1.7s
 ? Container grafana_loki_vector_demo_grafana  Removed                                                                                                                   0.4s
 ? Network grafana_loki_vector_loki            Removed                                                                                                                   0.1s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#


# docker-copose 啟動項目

[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
[+] Running 4/4
 ? Network grafana_loki_vector_loki            Created                                                                                                                   0.1s
 ? Container grafana_loki_vector_demo_vector   Started                                                                                                                   0.1s
 ? Container grafana_loki_vector_demo_grafana  Started                                                                                                                   0.1s
 ? Container grafana_loki_vector_demo_loki     Started                                                                                                                   0.1s
[root@localhost grafana_loki_vector]#

?7.4 驗證 跳轉(zhuǎn)網(wǎng)址正確
在新頁面打開,跳轉(zhuǎn)網(wǎng)址正確  http://192.168.2.247:3000/XXXXX

docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

?docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息,docker,java,eureka

到此基本完成。

當(dāng)然如果僅僅是想通過 Grafana 監(jiān)控kafka,推薦使用?Grafana Prometheus 通過JMX監(jiān)控kafka

到了這里,關(guān)于docker compose 部署 grafana + loki + vector 監(jiān)控kafka消息的文章就介紹完了。如果您還想了解更多內(nèi)容,請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章,希望大家以后多多支持TOY模板網(wǎng)!

本文來自互聯(lián)網(wǎng)用戶投稿,該文觀點僅代表作者本人,不代表本站立場。本站僅提供信息存儲空間服務(wù),不擁有所有權(quán),不承擔(dān)相關(guān)法律責(zé)任。如若轉(zhuǎn)載,請注明出處: 如若內(nèi)容造成侵權(quán)/違法違規(guī)/事實不符,請點擊違法舉報進(jìn)行投訴反饋,一經(jīng)查實,立即刪除!

領(lǐng)支付寶紅包贊助服務(wù)器費用

相關(guān)文章

  • Linux搭建Promtail + Loki + Grafana 輕量日志監(jiān)控系統(tǒng)

    Linux搭建Promtail + Loki + Grafana 輕量日志監(jiān)控系統(tǒng)

    日志監(jiān)控告警系統(tǒng),較為主流的是 ELK(Elasticsearch 、 Logstash和Kibana核心套件構(gòu)成) ,雖然優(yōu)點是功能豐富,允許復(fù)雜的操作。但是,這些方案往往規(guī)模復(fù)雜,資源占用高,操作苦難。很多功能往往用不上,大多數(shù)查詢只關(guān)注一定時間范圍和一些簡單的參數(shù)(如host、service等)

    2024年02月15日
    瀏覽(27)
  • OpenLdap +PhpLdapAdmin + Grafana docker-compose部署安裝

    OpenLdap +PhpLdapAdmin + Grafana docker-compose部署安裝

    目錄 一、OpenLdap介紹 二、PhpLdapAdmin介紹 三、使用docker-compose進(jìn)行安裝 1. docker-compose.yml 2. grafana配置文件 3. provisioning 四、安裝openldap、phpldapadmin、grafana 五、配置OpenLDAP 1. 登陸PhpLdapAdmin web管理 2. 需要注意的細(xì)節(jié) 內(nèi)容介紹參考: ? OpenLDAP是輕型目錄訪問協(xié)議(Lightweight Director

    2024年02月09日
    瀏覽(25)
  • 【Ubuntu】從Graylog到Grafana Loki:構(gòu)建更強(qiáng)大的網(wǎng)絡(luò)設(shè)備管理和監(jiān)控系統(tǒng)

    【Ubuntu】從Graylog到Grafana Loki:構(gòu)建更強(qiáng)大的網(wǎng)絡(luò)設(shè)備管理和監(jiān)控系統(tǒng)

    在將Graylog部署到生產(chǎn)環(huán)境時,我們遇到了一些問題,其中最主要的是無法安裝MongoDB并且無法隨時重啟機(jī)器去修改BIOS設(shè)置來修復(fù)問題 【 WARNING: MongoDB 5.0+ requires a CPU with AVX support, and your current system does not appear to have that! 】。為了尋找一個更可行的解決方案,我們決定將注意力

    2024年02月12日
    瀏覽(20)
  • docker搭建Grafana+Loki+Promtail日志分析

    docker搭建Grafana+Loki+Promtail日志分析

    1. Docker 安裝 Docker 官網(wǎng)下載:Docker: Accelerated, Containerized Application Development Note: 本人個人原因,在WIndows OS 安裝Docker ?2. 安裝完成后,驗證Docker Service ? 3. 為了方便,推薦安裝Git Bash 請自行安裝Git Bash 4. 開始安裝grafana 1. 獲取grafana? image 方式: 方式一: cmd管理員窗口? ? ? ?

    2024年02月09日
    瀏覽(24)
  • 日志收集系統(tǒng)PLG(Promtail+Loki+Grafana)部署

    日志收集系統(tǒng)PLG(Promtail+Loki+Grafana)部署

    一、簡 介 Loki是受Prometheus啟發(fā)由Grafana Labs團(tuán)隊開源的水平可擴(kuò)展,高度可用的多租戶日志聚合系統(tǒng)。 開發(fā)語言: Google Go。它的設(shè)計具有很高的成本效益,并且易于操作。使用標(biāo)簽來作為索引,而不是對全文進(jìn)行檢索,也就是說,你通過這些標(biāo)簽既可以查詢?nèi)罩镜膬?nèi)容也可以查

    2024年04月11日
    瀏覽(20)
  • Grafana loki部署及使用及問題處理方法(超詳細(xì))

    Grafana loki部署及使用及問題處理方法(超詳細(xì))

    一、下載軟件 因為我是本地測試,所以用的windows版本的包,loki服務(wù)window版本的安裝包下載地址:下載地址,選擇 promtail-windows版本的安裝包下載地址:下載地址 Grafana服務(wù)的下載地址:下載地址 二、配置文件 在解壓后的根目錄下,新建loki-local-config.yaml配置文件,內(nèi)容如下:

    2024年02月08日
    瀏覽(18)
  • docker容器監(jiān)控:Cadvisor+InfluxDB+Grafana的安裝部署

    docker容器監(jiān)控:Cadvisor+InfluxDB+Grafana的安裝部署

    目錄 Cadvisor+InfluxDB+Grafan安裝部署 1、安裝docker-ce 2、阿里云鏡像加速器 3、下載組件鏡像 4、創(chuàng)建自定義網(wǎng)絡(luò) 5、創(chuàng)建influxdb容器 6、創(chuàng)建Cadvisor 容器 7、查看Cadvisor 容器: (1)準(zhǔn)備測試鏡像 (2)通過http://192.168.100.132:8080端口訪問測試 8、創(chuàng)建granafa容器 [root@huyang2 ~]# iptables -F

    2024年02月14日
    瀏覽(18)
  • docker容器監(jiān)控:Cadvisor +Prometheus+Grafana的安裝部署

    docker容器監(jiān)控:Cadvisor +Prometheus+Grafana的安裝部署

    目錄 Cadvisor +Prometheus+Grafana的安裝部署 一、安裝docker: 1、安裝docker-ce 2、阿里云鏡像加速器 3、下載組件鏡像 4、創(chuàng)建自定義網(wǎng)絡(luò) 二、部署Cadvisor 1、被監(jiān)控主機(jī)上部署Cadvisor容器 2、訪問cAdvisor頁面 三、安裝prometheus 1、部署Prometheus? 2、先準(zhǔn)備配置 3、訪問prometheus頁面 四、部

    2024年02月14日
    瀏覽(30)
  • [GitOps]微服務(wù)版本控制:使用ArgoCD 部署Grafana Loki

    [GitOps]微服務(wù)版本控制:使用ArgoCD 部署Grafana Loki

    請回答: 你們是如何保證線上部署的服務(wù),從服務(wù)版本到參數(shù)配置,都是和測試通過的版本是一致的呢? 本文將介紹GitOps的基本原理以及ArgoCD的使用:ArgoCD部署Grafana Loki 到k8s集群。 本文項目地址:郭麻花的Azure Devops argo-cd - Repos (azure.com) GitOps通常作為k8s集群中的一項基礎(chǔ)設(shè)

    2024年02月11日
    瀏覽(22)
  • docker compose 部署ELK 8.X及監(jiān)控

    docker compose 部署ELK 8.X及監(jiān)控

    安裝docker 安裝docker-compose 相關(guān)版本: ES:8.5.0 kibana:8.5.0 logstash:8.5.0 數(shù)據(jù)存儲映射路徑配置777權(quán)限 es配置文件 數(shù)據(jù)存儲映射路徑配置777權(quán)限 logstash配置文件 日志文件配置 pipelines配置文件 配置logstash數(shù)據(jù)處理流程,依據(jù)實際項目需求配置,此處以input為kafka,output為es舉例

    2024年02月07日
    瀏覽(19)

覺得文章有用就打賞一下文章作者

支付寶掃一掃打賞

博客贊助

微信掃一掃打賞

請作者喝杯咖啡吧~博客贊助

支付寶掃一掃領(lǐng)取紅包,優(yōu)惠每天領(lǐng)

二維碼1

領(lǐng)取紅包

二維碼2

領(lǐng)紅包