Centos / RedHat 安裝 Redis 、MinIO、Zookeeper、Kafka

這篇具有很好參考價值的文章主要介紹了Centos / RedHat 安裝 Redis 、MinIO、Zookeeper、Kafka。希望對大家有所幫助。如果存在錯誤或未考慮完全的地方，請大家不吝賜教，您也可以點擊"舉報違法"按鈕提交疑問。

Centos 安裝Redis7.2

1、下載Redis

Redis 下載頁面：https://redis.io/download/

wget https://github.com/redis/redis/archive/7.2.4.tar.gz

2、解壓并編譯

# 1、進入目錄
cd /export/servers/

tar -xzvf redis-7.2.4

# 2、安裝依賴
yum install gcc make openssl-devel

# 3、安裝redis 本地依賴
cd deps

make hiredis jemalloc linenoise lua

# 4、執(zhí)行make
cd ../
make && make install

3、修改配置

daemonize yes
requirepass xxx
dir /export/backup/redis

4、修改內(nèi)存提交限制

vim /etc/sysctl.conf

# 啟用內(nèi)存可重復提交限制
vm.overcommit_memory=1

執(zhí)行：sysctl vm.overcommit_memory=1

5、自動啟動

Centos 安裝 MinIO

1、通過yum安裝

官網(wǎng)地址 min.io

# 1、下載rpm
wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio-20240226093348.0.0-1.x86_64.rpm -O minio.rpm

# 2、安裝
sudo dnf install minio.rpm

2. 創(chuàng)建文件存儲目錄

mkdir /export/data/minio

# 創(chuàng)建用戶組
groupadd -r minio-user

# 創(chuàng)建用戶
useradd -M -r -g minio-user minio-user

# 設(shè)置路勁訪問權(quán)限
chown minio-user:minio-user /export/data/minio


# 啟動minio(使用默認賬號啟動，非后臺進程)
minio server /export/data/minio --address 0.0.0.0:9000 --console-address 0.0.0.0:9001

3、MonIO做為服務后端系統(tǒng)

參考文檔： Create the systemd Service File

3.1、創(chuàng)建環(huán)境變量

# 創(chuàng)建并編輯服務環(huán)境變量
vim /etc/default/minio

# 設(shè)置路勁訪問權(quán)限
chown minio-user:minio-user /etc/default/minio

環(huán)境變量內(nèi)容

# MINIO_ROOT_USER and MINIO_ROOT_PASSWORD sets the root account for the MinIO server.
# This user has unrestricted permissions to perform S3 and administrative API operations on any resource in the deployment.
# Omit to use the default values 'minioadmin:minioadmin'.
# MinIO recommends setting non-default values as a best practice, regardless of environment.

MINIO_ROOT_USER=myminioadmin
MINIO_ROOT_PASSWORD=minio-secret-key-change-me

# MINIO_VOLUMES sets the storage volumes or paths to use for the MinIO server.
# The specified path uses MinIO expansion notation to denote a sequential series of drives between 1 and 4, inclusive.
# All drives or paths included in the expanded drive list must exist *and* be empty or freshly formatted for MinIO to start successfully.

MINIO_VOLUMES="/export/data/minio"

# MINIO_OPTS sets any additional commandline options to pass to the MinIO server.
# For example, `--console-address :9001` sets the MinIO Console listen port
MINIO_OPTS="--address :9000 --console-address :9001"

# MINIO_SERVER_URL sets the hostname of the local machine for use with the MinIO Server.
# MinIO assumes your network control plane can correctly resolve this hostname to the local machine.

# Uncomment the following line and replace the value with the correct hostname for the local machine.

#MINIO_SERVER_URL="http://minio.example.net"

3.2、創(chuàng)建systemctl 服務

vim /usr/lib/systemd/system/minio.service

輸入如下配置信息：

[Unit]
Description=MinIO
Documentation=https://min.io/docs/minio/linux/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local

User=minio-user
Group=minio-user
ProtectProc=invisible

EnvironmentFile=/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# MinIO RELEASE.2023-05-04T21-44-30Z adds support for Type=notify (https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=)
# This may improve systemctl setups where other services use `After=minio.server`
# Uncomment the line to enable the functionality
# Type=notify

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of threads this process can create
TasksMax=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name})

3.3、啟動服務


# 重新加載服務
systemctl daemon-reload

# 啟動服務
sudo systemctl start minio.service

# 服務狀態(tài)
systemctl status minio.service

# 服務自啟動
systemctl enable minio.service

4、MinIO 通過Nginx代理二級域名

參考文檔:https://min.io/docs/minio/linux/integrations/setup-nginx-proxy-with-minio.html

Centos 安裝Kafka

1、安裝Zookeeper

1.1 下載地址：https://downloads.apache.org/zookeeper/

wget  https://downloads.apache.org/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz

1.2 解壓安裝

tar -xzvf apache-zookeeper-3.8.4-bin.tar.gz
mv apache-zookeeper-3.8.4-bin zookeeper

1.3 添加到環(huán)境變量

vim /etc/profile
# 添加如下內(nèi)容
export ZOOKEEPER_HOME=/export/servers/zookeeper
export PATH=$PATH:$ZOOKEEPER_HOME/bin

1.4 修改zookeeper配置文件

cp /export/servers/zookeeper/conf/zoo_sample.cfg /export/servers/zookeeper/conf/zoo.cfg

vim /export/servers/zookeeper/conf/zoo.cfg


修改如下內(nèi)容：

# 數(shù)據(jù)存放路徑
dataDir=/export/data/zookeeper
#  the basic time unit in milliseconds used by ZooKeeper. It is used to do heartbeats and the minimum session timeout will be twice the tickTime.
tickTime=2000
# the port to listen for client connections
clientPort=2181

# 末尾追加(多節(jié)點需要配置)
# server.1=node2:2888:3888
# server.2=node3:2888:3888
# server.3=node4:2888:3888

1.5 創(chuàng)建節(jié)點ID

mkdir /export/data/zookeeper
echo "1" > /export/data/zookeeper/myid

1.6 啟動zookeeper

zkServer.sh start

# 使用jps 檢測是否啟動成功， QuorumPeerMain

1.7 開機自啟動

cat > /etc/systemd/system/zookeeper.service << EOF

[Unit]
Description=zookeeper
After=syslog.target network.target

[Service]
Type=forking
# 指定zookeeper 日志文件路徑，也可以在zkServer.sh 中定義
Environment=ZOO_LOG_DIR=/export/Logs/zookeeper
# 指定JDK路徑，也可以在zkServer.sh 中定義
Environment=JAVA_HOME=/export/servers/jdk1.8.0_401
ExecStart=/export/servers/zookeeper/bin/zkServer.sh start
ExecStop=/export/servers/zookeeper/bin/zkServer.sh stop
Restart=always
User=root
Group=root

[Install]
WantedBy=multi-user.target

EOF

重新加載服務

systemctl daemon-reload

開機自啟動

systemctl enable zookeeper

查看zookeeper狀態(tài)

systemctl status zookeeper

1.8 zookeeper 查看器

https://issues.apache.org/jira/secure/attachment/12436620/ZooInspector.zip

1.9 設(shè)置zookeeper SASL認證

1、編寫認證文件

vim /export/servers/zookeeper/conf/zk_server_jaas.conf

Server {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="user" password="user-password"
    user_kafka="kafka-password";
};


Client {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="kafka" password="kafka-password";
};

這里Server和Client 都使用kafka認證模式，需要導入kafka-clients-x.x.x.jar 到 zookeeper的lib目錄下。

2、編寫java.env 文件


vim /export/servers/zookeeper/conf/java.env

CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}  -Djava.security.auth.login.config=/export/servers/zookeeper/conf/zk_server_jaas.conf"
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/export/servers/zookeeper/conf/zk_server_jaas.conf"

zookeeper 自動時會使用該文件

3、修改配置文件


authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
zookeeper.sasl.client=true
allowSaslFailedClients=false
sessionRequireClientSASLAuth=true

啟動服務之后即可使用安全認證

4、導入 Kafka 客戶端

cp /export/servers/kafka/libs/kafka-clients-3.7.0.jar  /export/servers/zookeeper/lib

5、重啟Zookeeper

systemctl restart zookeeper

2、安裝Kafka

2.1 下載地址：https://downloads.apache.org/kafka/

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz

2.2 安裝

tar -xzvf kafka_2.13-3.7.0.tgz
mv kafka_2.13-3.7.0 kafka

2.3 添加到環(huán)境變量中

vim /etc/profile

JAVA_HOME=/opt/jdk
ZOOKEEPER_HOME=/opt/zookeeper
KAFKA_HOME=/opt/kafka
PATH=$PATH:$ZOOKEEPER_HOME/bin:$JAVA_HOME/bin:$KAFKA_HOME/bin
export JAVA_HOME ZOOKEEPER_HOME PATH

source /etc/profile

2.4 修改配置文件

cp /export/servers/kafka/config/server.properties /export/servers/kafka/config/server.properties.backup

cd /opt/kafka/config

vim /opt/kafka/config/server.properties

修改如下信息

#broker 的全局唯一編號，不能重復
broker.id=0
#刪除 topic 功能使能
delete.topic.enable=true
#kafka 運行日志存放的路徑
log.dirs=/export/Logs/kafka/logs
#配置連接 Zookeeper 集群地址
zookeeper.connect=hadoop102:2181,hadoop103:2181,hadoop104:2181

# -------------不需要修改------------------
#處理網(wǎng)絡請求的線程數(shù)量
num.network.threads=3
#用來處理磁盤 IO 的現(xiàn)成數(shù)量
num.io.threads=8
#發(fā)送套接字的緩沖區(qū)大小
socket.send.buffer.bytes=102400
#接收套接字的緩沖區(qū)大小
socket.receive.buffer.bytes=102400
#請求套接字的緩沖區(qū)大小
socket.request.max.bytes=104857600
#topic 在當前 broker 上的分區(qū)個數(shù)
num.partitions=1
#用來恢復和清理 data 下數(shù)據(jù)的線程數(shù)量
num.recovery.threads.per.data.dir=1
#segment 文件保留的最長時間，超時將被刪除
log.retention.hours=168

2.5 啟動集群

啟動集群

cd /export/servers/kafka
# 啟動
bin/kafka-server-start.sh -daemon config/server.properties

# 關(guān)閉
bin/kafka-server-stop.sh stop

kafka 群起腳本（部分）

for i in hadoop102 hadoop103 hadoop104
do
    echo "========== $i ==========" 
    ssh $i '/export/servers/kafka/bin/kafka-server-start.sh -daemon 
/export/servers/kafka/config/server.properties'
done

2.6 kafka 開機自啟動


cat > /etc/systemd/system/kafka.service << EOF
[Unit]
Description=kafka
After=syslog.target network.target zookeeper.service

[Service]
Type=simple
# 指定JDK路徑，也可以在zkServer.sh 中定義
Environment=JAVA_HOME=/export/servers/jdk1.8.0_401
ExecStart=/export/servers/kafka/bin/kafka-server-start.sh /export/servers/kafka/config/server.properties
ExecStop=/export/servers/kafka/bin/kafka-server-stop.sh stop
Restart=always
User=root
Group=root

[Install]
WantedBy=multi-user.target

EOF

刷新配置文件

systemctl daemon-reload

# 開機啟動
systemctl enable kafka

# 啟動服務
systemctl start kafka
# 關(guān)閉服務
systemctl stop kafka
# 查看狀態(tài)
systemctl status kafka

3. 啟動安全認證

3.1 添加認證文件

在config目錄下添加kafka_server_jaas.conf文件，內(nèi)容如下：

KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin" password="admin-pwd"
    user_admin="admin-pwd"
    user_producer="producer-pwd"
    user_consumer="customer-pwd";
};

Client {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="kafka"
    password="Jby@2024";
};

這里，KafkaServer中username和password 為 broker 內(nèi)部通信的用戶名密碼。user_producer和user_consumer分別是為生產(chǎn)者和消費者用戶設(shè)置的憑證。您可以根據(jù)需要創(chuàng)建更多的用戶和密碼。

其中 user_admin = “admin_pwd” 非常重要且必須與 username 和 password 一致。否則會出現(xiàn)如下錯誤：

[2024-03-06 10:42:59,070] INFO [Controller id=0, targetBrokerId=0] Node 0 disconnected. (org.apache.kafka.clients.NetworkClient)
[2024-03-06 10:42:59,070] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (iZwz94rqv754l5q4mca9nbZ/127.0.0.1:9092) failed authentication due to: Authentication failed: Invalid username or password (org.apache.kafka.clients.NetworkClient)

3.2 配置Kafka服務器屬性

編輯Kafka的server.properties文件，添加或修改以下配置以啟用SASL（Simple Authentication and Security Layer）和設(shè)置監(jiān)聽器：

listeners=SASL_PLAINTEXT://host.name:port
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN

3.3 修改啟動腳本

編輯Kafka的啟動腳本（通常是kafka-server-start.sh），找到export KAFKA_HEAP_OPTS行，并在其后添加JVM參數(shù)，指向您的JAAS配置文件：

export KAFKA_HEAP_OPTS="-Djava.security.auth.login.config=/export/servers/kafka/config $KAFKA_HEAP_OPTS"

同樣，確保將/path/to/kafka_server_jaas.conf替換為您實際的JAAS配置文件路徑。

3.4 修改客戶端配置

producer.properties or consumer.properties文章來源地址http://www.zghlxwxcb.cn/news/detail-855411.html

security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
   username="producer" \
   password="producer-pwd";

3.5 啟動kafka服務

systemctl restart kafka

到了這里，關(guān)于Centos / RedHat 安裝 Redis 、MinIO、Zookeeper、Kafka的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！