1、zookeeper 搭建

首先創(chuàng)建一個(gè)自定義網(wǎng)絡(luò)，后續(xù)的所有容器都放入同一個(gè)內(nèi)網(wǎng)中，容器之間還可以通過容器名稱進(jìn)行直接訪問，在后續(xù)的配置中只需要寫明容器名稱即可，會(huì)自動(dòng)找到對(duì)應(yīng)的IP地址，防止重啟容器后IP地址發(fā)生變化時(shí)，還要去修改配置文件的操作

docker network create kafka-net

1、zookeeper 單機(jī)搭建

創(chuàng)建目錄

mkdir -p /mnt/data/zookeeper/data # 數(shù)據(jù)掛載目錄
mkdir -p /mnt/data/zookeeper/conf # 配置掛載目錄
mkdir -p /mnt/data/zookeeper/logs # 日志掛載目錄

啟動(dòng)zookeeper

docker pull zookeeper:3.5.9

docker run -d --privileged=true --name zookeeper -p 2181:2181 --network kafka-net --network-alias zookeeper -v /mnt/data/zookeeper/data:/data -v /mnt/data/zookeeper/conf:/conf -v /mnt/data/zookeeper/logs:/datalog zookeeper:3.5.9

2888為組成zookeeper服務(wù)器之間的通信端口，3888為用來選舉leader的端口

進(jìn)入zookeeper容器

docker exec -it 容器id /bin/bash

找到配置文件位置

cat /conf/zoo.cfg

查看zookeeper狀態(tài)

zkServer.sh status

2、zookeeper 集群搭建

• 注意，此處是在一臺(tái)服務(wù)器上搭建的偽集群，以下是docker compose的部署方案
• 注意：安裝時(shí)遇到的問題，zookeeper 的版本必須是 3.5.0 以后，3.5.0 版本之前的配置與 3.5.0 版本之后是不同的，選錯(cuò)版本會(huì)導(dǎo)致 zookeeper 無法形成集群，最終導(dǎo)致使用 kafka 時(shí)報(bào)錯(cuò)?。。。。。?/li>

mkdir -p /mnt/data/zookeeper
cd /mnt/data/zookeeper
vim zookeeper-compose.yml

zookeeper-compose.yml 內(nèi)容如下

version: '3'

services:
  zoo1:
    image: zookeeper:3.5.9
    container_name: zoo1
    restart: always
    privileged: true
    ports:
      - 2182:2181
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo2:
    image: zookeeper:3.5.9
    container_name: zoo2
    restart: always
    privileged: true
    ports:
      - 2183:2181
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo3:
    image: zookeeper:3.5.9
    container_name: zoo3
    restart: always
    privileged: true
    ports:
      - 2184:2181
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

networks:
  default:
    external: true
    name: kafka-net

• 注意：docker-compose會(huì)默認(rèn)將當(dāng)前的容器名稱作為network的alias
• 執(zhí)行 docker-compose up，表示執(zhí)行當(dāng)前路徑下的yml文件。也可以指定執(zhí)行某一個(gè)yml文件
  • docker-compose -f zookeeper-compose.yml up -d
• 通過 docker inspect 容器ID 查看容器的網(wǎng)絡(luò)配置

2、kafka 搭建

1、docer搭建kafka集群

docker pull wurstmeister/kafka

docker run -d  --privileged=true \
--name kafka0 -p 9092:9092 \
--network kafka-net \
--network-alias kafka0 \
-e KAFKA_BROKER_ID=0 \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.231.10:9092 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
wurstmeister/kafka:latest

docker run -d  --privileged=true \
--name kafka1 -p 9093:9092 \
--network kafka-net \
--network-alias kafka1 \
-e KAFKA_BROKER_ID=1 \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.231.10:9093 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
wurstmeister/kafka:latest

docker run -d  --privileged=true \
--name kafka2 -p 9094:9092 \
--network kafka-net \
--network-alias kafka2 \
-e KAFKA_BROKER_ID=2 \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.231.10:9094 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
wurstmeister/kafka:latest

• KAFKA_LISTENERS 與 KAFKA_ADVERTISED_LISTENERS 是為了區(qū)分內(nèi)網(wǎng)和外網(wǎng)的
• 如果只有內(nèi)網(wǎng)訪問，就可以只配置 KAFKA_LISTENERS
• 如果涉及了外網(wǎng)訪問，比如要在云服務(wù)器上部署使用，就需要配置 KAFKA_ADVERTISED_LISTENERS 參數(shù)了，在本文開始時(shí)配置了一個(gè) kafka-net 的網(wǎng)絡(luò)，也就是處于這個(gè)網(wǎng)絡(luò)下的訪問才屬于內(nèi)網(wǎng)訪問，而 kafka 部署完畢后，需要在其他服務(wù)器上訪問 kafka，這就需要通過外網(wǎng)訪問 kafka，所以必須配置 KAFKA_ADVERTISED_LISTENERS ，且值為 PLAINTEXT://<服務(wù)器ip>:<暴露端口> ，例如我的服務(wù)器外網(wǎng)地址為 192.168.231.10，端口為容器暴露的端口
• KAFKA_BROKER_ID : broker的ID，這個(gè)ID是集群的標(biāo)識(shí)，不能重復(fù)
• KAFKA_ZOOKEEPER_CONNECT：zookeeper的連接地址
• KAFKA_LISTENERS：標(biāo)識(shí)kafka服務(wù)運(yùn)行在容器內(nèi)的9092端口，因?yàn)闆]有指定host，所以是0.0.0.0標(biāo)識(shí)所有的網(wǎng)絡(luò)接口
• KAFKA_ADVERTISED_LISTENERS：kafka發(fā)布到zookeeper供客戶端使用的服務(wù)地址

2、docker compose 搭建kafka集群

• 注意，此處是在一臺(tái)服務(wù)器上搭建的偽集群，以下是docker compose的部署方案
• 注意，需要等待 zookeeper 完全啟動(dòng)好后再啟動(dòng) kafka 集群

mkdir -p /mnt/data/kafka/node1/data # 數(shù)據(jù)掛載目錄
mkdir -p /mnt/data/kafka/node2/data # 數(shù)據(jù)掛載目錄
mkdir -p /mnt/data/kafka/node3/data # 數(shù)據(jù)掛載目錄

cd /mnt/data/kafka
vim kafka-compose.yml

version: '3'

services:
  broker1:
    image: wurstmeister/kafka:latest
    restart: always
    container_name: broker1
    privileged: true
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9092    ## 宿主機(jī)IP
      KAFKA_ADVERTISED_PORT: 9092
      ## zookeeper 集群，因?yàn)槎荚谕粋€(gè)內(nèi)網(wǎng)，所以可以通過容器名稱獲取到容器的ip
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181		
    volumes:
      - "/mnt/data/kafka/node1/data/:/kafka"

  broker2:
    image: wurstmeister/kafka:latest
    restart: always
    container_name: broker2
    privileged: true
    ports:
      - "9093:9092"
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9093    ## 宿主機(jī)IP
      KAFKA_ADVERTISED_PORT: 9093
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    volumes:
      - "/mnt/data/kafka/node2/data/:/kafka"

  broker3:
    image: wurstmeister/kafka:latest
    restart: always
    container_name: broker3
    privileged: true
    ports:
      - "9094:9092"
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9094   ## 宿主機(jī)IP
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    volumes:
      - "/mnt/data/kafka/node3/data/:/kafka"

networks:
  default:
    external: true
    name: kafka-net

• 執(zhí)行 docker-compose up，表示執(zhí)行當(dāng)前路徑下的yml文件。也可以指定執(zhí)行某一個(gè)yml文件
  • docker-compose -f kafka-compose.yml up -d
• 通過 docker inspect 容器ID 查看容器的網(wǎng)絡(luò)配置

3、docker compose 整體啟動(dòng)

前面可以省略，直接創(chuàng)建一個(gè)docker-compose.yml 文件即可啟動(dòng) 3臺(tái) zookeeper 與 3臺(tái) kafka

version: '3'

services:
  zoo1:
    image: zookeeper:3.5.9
    container_name: zoo1
    restart: always
    ports:
      - 2182:2181
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo2:
    image: zookeeper:3.5.9
    container_name: zoo2
    restart: always
    ports:
      - 2183:2181
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo3:
    image: zookeeper:3.5.9
    container_name: zoo3
    restart: always
    ports:
      - 2184:2181
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  broker1:
    image: wurstmeister/kafka:latest
    restart: always
    container_name: broker1
    ports:
      - "9092:9092"
    # depends_on：通過 depends_on 告訴 docker-compose 當(dāng)前服務(wù)啟動(dòng)之前先要把 depends_on 指定的服務(wù)啟動(dòng)起來才行
    # 例如：kafka 依賴 zookeeper
    depends_on:
      - zoo1
      - zoo2
      - zoo3
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9092    ## 宿主機(jī)IP
      KAFKA_ADVERTISED_HOST_NAME: broker1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    volumes:
      - "/mnt/data/kafka/node1/data/:/kafka"

  broker2:
    image: wurstmeister/kafka:latest
    restart: always
    container_name: broker2
    ports:
      - "9093:9092"
    depends_on:
      - zoo1
      - zoo2
      - zoo3
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9093    ## 宿主機(jī)IP
      KAFKA_ADVERTISED_HOST_NAME: broker2
      KAFKA_ADVERTISED_PORT: 9093
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    volumes:
      - "/mnt/data/kafka/node2/data/:/kafka"

  broker3:
    image: wurstmeister/kafka:latest
    restart: always
    container_name: broker3
    ports:
      - "9094:9092"
    depends_on:
      - zoo1
      - zoo2
      - zoo3
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9094   ## 宿主機(jī)IP
      KAFKA_ADVERTISED_HOST_NAME: broker3
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    volumes:
      - "/mnt/data/kafka/node3/data/:/kafka"

networks:
  default:
    external: true
    name: kafka-net

4、可視化工具 offsetexplorer

文章來源地址http://www.zghlxwxcb.cn/news/detail-695145.html

5、可視化工具 kafka-map

docker run -d \
-p 8080:8080 \
-v /opt/kafka-map/data:/usr/local/kafka-map/data \
-e DEFAULT_USERNAME=admin \
-e DEFAULT_PASSWORD=admin \
--name kafka-map \
--restart always dushixiang/kafka-map:latest

容器啟動(dòng)后，訪問：ip:8080 訪問

kafka-map官網(wǎng)：https://gitee.com/dushixiang/kafka-map

到了這里，關(guān)于Docker 搭建 zookeeper、kafka 集群的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！