docker-compose部署elk
部署所需yml文件 —— docker-compose-elk.yml
version: '3.7'
services:
elasticsearch:
image: elasticsearch:8.9.0
container_name: elasticsearch
hostname: elasticsearch
restart: "no"
volumes:
- /opt/space/soft/docker/elasticsearch/data:/usr/share/elasticsearch/data:rw
- /opt/space/soft/docker/elasticsearch/config:/usr/share/elasticsearch/config:rw
- /opt/space/soft/docker/elasticsearch/logs:/usr/share/elasticsearch/logs:rw
- /opt/space/soft/docker/elasticsearch/plugins:/usr/share/elasticsearch/plugins:rw
ports:
- "21033:9200"
- "21133:9300"
networks:
elastic:
aliases:
- elasticsearch
kibana:
image: kibana:8.9.0
container_name: kibana
hostname: kibana
restart: "no"
volumes:
- /opt/space/soft/docker/kibana/data:/usr/share/kibana/data:rw
- /opt/space/soft/docker/kibana/config:/usr/share/kibana/config:rw
- /opt/space/soft/docker/kibana/logs:/usr/share/kibana/logs:rw
ports:
- 21041:5601
depends_on:
- elasticsearch
networks:
elastic:
aliases:
- kibana
logstash:
image: logstash:8.9.0
container_name: logstash
hostname: logstash
restart: "no"
volumes:
- /opt/space/soft/docker/logstash/config:/usr/share/logstash/config:rw
- /opt/space/soft/docker/logstash/logs:/usr/share/logstash/logs:rw
- /opt/space/soft/docker/logstash/pipeline:/usr/share/logstash/pipeline:rw
ports:
- 21066:5044
- 21067:9066
- 21068:21068
depends_on:
- elasticsearch
networks:
elastic:
aliases:
- logstash
networks:
elastic:
name: elastic
driver: bridge
這里做了端口映射以及掛載數(shù)據(jù)卷。
部署
1.使用docker-compose命令部署
docker-compose -f docker-compose-elk.yml up -d
部署前需要先將掛載的數(shù)據(jù)卷注釋掉，用于拷貝配置文件到本地磁盤
- 創(chuàng)建所需目錄，拷貝配置文件到本地，并設(shè)置讀寫權(quán)限為 777
cd /opt/space/soft/docker/
sudo mkdir -p elasticsearch/{config,data,logs,plugins}
sudo mkdir -p kibana/{config,data,logs}
sudo mkdir -p logstash/{config,pipeline,logs}
sudo docker cp elasticsearch:/usr/share/elasticsearch/config ./elasticsearch/
sudo docker cp kibana:/usr/share/kibana/config ./kibana/
sudo docker cp logstash:/usr/share/logstash/config ./logstash/
sudo docker cp logstash:/usr/share/logstash/pipeline ./logstash/
sudo chmod 777 -R /opt/space/soft/docker/elasticsearch/*
sudo chmod 777 -R /opt/space/soft/docker/kibana/*
sudo chmod 777 -R /opt/space/soft/docker/logstash/*
- 修改jvm參數(shù)（作者電腦配置不夠高，所以需要進行修改）
3.1 修改elasticsearch的jvm參數(shù)
打開/opt/space/soft/docker/elasticsearch/config/jvm.options文件，按圖中所示進行配置。
3.2 修改logstash的jvm參數(shù)
打開/opt/space/soft/docker/logstash/config/jvm.options文件，按圖中所示進行配置。
- 放開docker-compose-elk.yml文件內(nèi)掛載數(shù)據(jù)卷的注釋，并執(zhí)行下列命令。
docker-compose -f docker-compose-elk.yml stop
docker-compose -f docker-compose-elk.yml rm
docker-compose -f docker-compose-elk.yml up -d
配置elasticsearch和kibana并開啟ssl
配置基礎(chǔ)數(shù)據(jù)認證
參考文檔
- 進入elasticsearch的docker容器內(nèi)
docker exec -it elasticsearch bash
- 生成elastic-stack-ca.p12
./bin/elasticsearch-certutil ca
- 生成elastic-certificates.p12
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
4. 拷貝elastic-certificates.p12到目錄/usr/share/elasticsearch/config/certs，刪除elastic-stack-ca.p12
mv elastic-certificates.p12 config/certs/
mv elastic-stack-ca.p12 ./config/
elastic-stack-ca.p12文件保留下來，后面有用
5.設(shè)置elasticsearch.yml配置文件
安裝如圖所示進行修改
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/elastic-certificates.p12
truststore.path: certs/elastic-certificates.p12
6.如果certificate設(shè)置了密碼，需要執(zhí)行一下兩個命令
./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
此處可能出現(xiàn)以下錯誤，需要退出容器，給提示的文件設(shè)置所屬用戶
sudo chown linux當前登錄用戶名 elasticsearch/config/elasticsearch.keystore
- 退出elasticsearch容器，給certificates文件賦予777權(quán)限
sudo chmod 777 elasticsearch/config/certs/elastic-certificates.p12
- 重啟elasticsearch容器
docker restart elasticsearch
配置elasticsearch和kibana開啟https訪問
參考文檔
- 再次進入elasticsearch的docker容器內(nèi)
docker exec -it elasticsearch bash
- 生成elasticsearch-ssl-http.zip
命令所需操作過長，此處不在截圖
elasticsearch@elasticsearch:~$ ./bin/elasticsearch-certutil http
## Elasticsearch HTTP Certificate Utility
The 'http' command guides you through the process of generating certificates
for use on the HTTP (Rest) interface for Elasticsearch.
This tool will ask you a number of questions in order to generate the right
set of files for your needs.
## Do you wish to generate a Certificate Signing Request (CSR)?
A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you don't have
access to the keys for that CA).
If you are in a corporate environment with a central security team, then you
may have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organisation may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.
If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control. This is a
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.
## 生成CSR 輸入n
Generate a CSR? [y/N]n
## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?
If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.
If you do not have an existing CA, one will be generated for you.
## 是否使用存在的ca 輸入y（在基礎(chǔ)配置時生成了）
Use an existing CA? [y/N]y
## What is the path to your CA?
Please enter the full pathname to the Certificate Authority that you wish to
use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
(.jks) or PEM (.crt, .key, .pem) format.
## 輸入ca文件的地址
CA Path: /usr/share/elasticsearch/config/elastic-stack-ca.p12
Reading a PKCS12 keystore requires a password.
It is possible for the keystore's password to be blank,
in which case you can simply press <ENTER> at the prompt
## 輸入文件設(shè)置的密碼
Password for elastic-stack-ca.p12:
## How long should your certificates be valid?
Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.
Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.
You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)
## 設(shè)置過期時間
For how long should your certificate be valid? [5y] 5y
## Do you wish to generate one certificate per node?
If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.
Alternatively, you may wish to generate a single certificate that is valid
across all the hostnames or addresses in your cluster.
If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.
However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.
## 是否為每一個節(jié)點生成證書 輸入n
Generate a certificate per node? [y/N]n
## Which hostnames will be used to connect to your nodes?
These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.
You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.
If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.
## 節(jié)點的hostname,設(shè)置為elasticsearch，敲兩次回車
Enter all the hostnames that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.
elasticsearch
You entered the following hostnames.
- elasticsearch
## 配置是否正確
Is this correct [Y/n]y
## Which IP addresses will be used to connect to your nodes?
If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.
If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press <ENTER> to skip this step.
## 節(jié)點的ip(可以在宿主機通過命令docker inspect elasticsearch查看),設(shè)置為172.27.0.2，敲兩次回車
Enter all the IP addresses that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.
172.27.0.2
You entered the following IP addresses.
- 172.27.0.2
## 配置是否正確
Is this correct [Y/n]y
## Other certificate options
The generated certificate will have the following additional configuration
values. These values have been selected based on a combination of the
information you have provided above and secure defaults. You should not need to
change these values unless you have specific requirements.
Key Name: elasticsearch
Subject DN: CN=elasticsearch
Key Size: 2048
## 是否更改任意項
Do you wish to change any of these options? [y/N]n
## What password do you want for your private key(s)?
Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.
If you wish to use a blank password, simply press <enter> at the prompt below.
## 輸入生成文件的密碼（可不設(shè)置，設(shè)置需要在后面進行配置）
Provide a password for the "http.p12" file: [<ENTER> for none]
## 再次輸入生成文件的密碼
Repeat password to confirm:
## Where should we save the generated files?
A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.
These files will be included in a single zip archive.
## 生成壓縮文件的地址和名稱，直接敲回車即可
What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]
Zip file written to /usr/share/elasticsearch/elasticsearch-ssl-http.zip
- 移動elasticsearch-ssl-http.zip壓縮包
mv elasticsearch-ssl-http.zip ./config/
- 解壓，并移動elasticsearc的http.p12文件，刪除其余文件
unzip config/elasticsearch-ssl-http.zip
mv elasticsearch/http.p12 ./config/certs/
rm -rf elasticsearch/ kibana/
- 設(shè)置密碼是需要執(zhí)行該命令，沒有設(shè)置則跳過
./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
7. 退出elasticsearch容器，給http.p12文件賦予777權(quán)限
sudo chmod 777 elasticsearch/config/certs/http.p12
- 重啟elasticsearch容器
docker restart elasticsearch
10.進入容器給elastic用戶設(shè)置密碼，驗證https訪問
修改密碼參考資料
bin/elasticsearch-reset-password -u elastic -i
瀏覽器訪問
https://宿主機IP:映射端口
作者這里宿主機IP為192.168.0.100 映射端口為21033
故訪問https://192.168.0.100:21033/
訪問需要帳號密碼進行驗證，帳號為elastic 密碼為剛剛設(shè)置的密碼
登錄成功后可看到elasticsearch配置
11. 復(fù)制壓縮包（第二步生成的）內(nèi)kibana文件夾下elasticsearch-ca.pem到kibana的配置文件夾內(nèi)
12. 配置kibana_system用戶的密碼
進入elasticsearch容器內(nèi)，執(zhí)行下列命令
bin/elasticsearch-reset-password -u kibana_system -i
- 生成kibana用https訪問的公鑰和私鑰
./bin/elasticsearch-certutil csr -name kibana-server
會生成一個壓縮文件，解壓后有一下文件
/kibana-server
|_ kibana-server.csr
|_ kibana-server.key
- 將kibana-server.csr和kibana-server.key兩個文件拷貝到kibana配置文件夾內(nèi)
- 執(zhí)行下列命令，生成
openssl x509 -req -days 3650 -in kibana-server.csr -signkey kibana-server.key -out kibana-server.crt
- 打開kibana配置文件kibana.yml，添加和修改以下配置（IP自查）
server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "https://172.27.0.2:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/elasticsearch-ca.pem"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "12步設(shè)置的密碼"
server.ssl.certificate: "/usr/share/kibana/config/kibana-server.crt"
server.ssl.key: "/usr/share/kibana/config/kibana-server.key"
server.ssl.enabled: true
# 設(shè)置中文訪問
i18n.locale: "zh-CN"
- 給生成的四個文件設(shè)置777權(quán)限
sudo chmod 777 elasticsearch-ca.pem kibana-server.csr kibana-server.key kibana-server.crt
- 重啟kibana
- 登錄kibana控制臺驗證
瀏覽器訪問
https://宿主機IP:映射端口
作者這里宿主機IP為192.168.0.100 映射端口為21041
故訪問https://192.168.0.100:21041/
訪問需要帳號密碼進行驗證，帳號為elastic 密碼為剛剛設(shè)置的密碼，登錄成功后選擇自己瀏覽
配置logstash
- 設(shè)置elasticsearch內(nèi)置用戶logstash_system密碼（上面已經(jīng)設(shè)置過兩個用戶的密碼了，此處就當留一個考點吧）
- 生成logstash和elasticsearch之間的安全認證文件（文件elastic-certificates.p12大家還有印象嗎？）
#如果設(shè)置有密碼此命令需要輸入密碼
openssl pkcs12 -in elasticsearch/config/certs/elastic-certificates.p12 -cacerts -nokeys -chain -out logstash.pem
- 移動logstash.pem文件到logstash配置文件目錄下，并設(shè)置777權(quán)限
sudo chmod 777 logstash.pem
- 配置logstash.yml文件
http.host: "0.0.0.0"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 第一步設(shè)置的密碼
#這里必須用 https 這里必須用 https 這里必須用 https
xpack.monitoring.elasticsearch.hosts: [ "https://172.27.0.2:9200" ]
#你的ca.pem 的所在路徑
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/config/logstash.pem"
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
# 探嗅 es節(jié)點，設(shè)置為 false
xpack.monitoring.elasticsearch.sniffing: false
- 使用kabina配置一個新的elasticsearch用戶給logstash使用
- 在pipeline目錄下，增加logstash.conf文件
配置如下
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
tcp {
port => 21068
codec => json_lines
}
}
output {
elasticsearch {
hosts => ["https://172.27.0.2:9200"]
index => "自定義索引名稱-%{+YYYY.MM.dd}"
user => "自定義用戶"
password => "自定義密碼"
ssl_enabled => true
ssl_certificate_authorities => ["/usr/share/logstash/config/logstash.pem"]
}
}
- 重啟logstash
創(chuàng)建springboot項目進行測試
- 在pom文件內(nèi)添加依賴
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>7.4</version>
</dependency>
- 配置logback.xml文件，添加以下appender
destination配置問logstash的地址，需要哪些數(shù)據(jù)請自定義。
<springProperty scope="context" name="appName" source="spring.application.name"/>
<springProperty scope="context" name="port" source="server.port"/>
<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>ip:port</destination>
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<timestamp>
<timeZone>Asia/Shanghai</timeZone>
</timestamp>
<providers>
<pattern>
<pattern>
{
"index":"index-%d{yyyy-MM-dd}",
"appName":"${appName}",
"pid": "${PID:-}",
"port":"${port}",
"timestamp":"%d{yyyy-MM-dd HH:mm:ss.SSS}",
"thread":"%thread",
"level":"%level",
"class": "%logger",
"message": "%message",
"stack_trace":"%exception"
}
</pattern>
</pattern>
</providers>
</encoder>
<keepAliveDuration>5 minutes</keepAliveDuration>
</appender>
- 將新增加appender的名字添加到root內(nèi)
<root level="INFO">
<appender-ref ref="ERROR"/>
<appender-ref ref="WARN"/>
<appender-ref ref="INFO"/>
<appender-ref ref="DEBUG"/>
<appender-ref ref="LOGSTASH"/>
<appender-ref ref="STDOUT"/>
</root>
4.啟動驗證文章來源：http://www.zghlxwxcb.cn/news/detail-715773.html
kibana創(chuàng)建視圖，查詢?nèi)罩?/h3>
文章來源地址http://www.zghlxwxcb.cn/news/detail-715773.html
到了這里，關(guān)于docker-compose部署elk（8.9.0）并開啟ssl認證的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
