Elasticsearch+Kibana集群部署（3節(jié)點(diǎn)）

$$lindor??良民筆記$$

前言

?? 僅作為筆記并記錄elk搭建過程和搭建中遇到的問題，轉(zhuǎn)載請注明出處，目前該章節(jié)只講述了 elasticsearch+Kibana 的安裝過程，以及安裝中的一些簡單報(bào)錯；適合有適當(dāng)基礎(chǔ)的同學(xué)，理論表的比較少。

?

項(xiàng)目地址：

• elastic官網(wǎng)地址：https://www.elastic.co/cn/

  elastic產(chǎn)品地址：https://www.elastic.co/cn/elastic-stack

• 清華大學(xué)yum源地址：https://mirrors.tuna.tsinghua.edu.cn/elasticstack/

?

準(zhǔn)備工具：

• ELK版本：elasticsearch-7.7.1-x86_64.rpm

• Filebeat版本：filebeat-7.7.1-x86_64.rpm

• Kibana版本：kibana-7.7.1-x86_64.rpm

• JDK版本：java-1.8.0-openjdk

• Logstash版本：logstash-8.5.0-linux-x86_64.rpm

?

節(jié)點(diǎn)分布：

?

一、Elasticsearch部署

• 只展示單臺ES節(jié)點(diǎn) ，其他ES節(jié)點(diǎn)步驟一致

• 安裝JDK

  1.安裝
  [root@localhost ~]# yum -y install java-1.8.0-openjdk
  
  2.驗(yàn)證
  root@localhost ~]# java -version
  openjdk version "1.8.0_352"
  OpenJDK Runtime Environment (build 1.8.0_352-b08)
  OpenJDK 64-Bit Server VM (build 25.352-b08, mixed mode)
  
  > yum安裝的jdk 不用配置環(huán)境變量。手動下載linux的包需要配置環(huán)境變量
  

• 安裝Elasticsearch

  我這里提前上傳到了/root/ 目錄下，直接本地安裝即可。下載可參考項(xiàng)目地址[^1]

  1.安裝
  創(chuàng)建elk用戶并授權(quán)elk文件夾權(quán)限
  [root@localhost ~]# ls
  elasticsearch-7.7.1-x86_64.rpm  
  
  [root@localhost ~]# adduser elasticsearch    #創(chuàng)建elasticsearch用戶
  [root@localhost ~]# yum -y localinstall elasticsearch-7.7.1-x86_64.rpm    #安裝elasticsearch
  [root@localhost ~]# chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/   #授權(quán)所有權(quán)給elasticsearch用戶
  [root@localhost ~]# mkdir -p /home/elasticsearch/{data,logs}     #創(chuàng)建data/log文件夾
  [root@localhost ~]# chown -R elasticsearch:elasticsearch /home/elasticsearch    #授權(quán)文件夾所屬用戶為elasticsearch
  [root@localhost ~]# su elasticsearch         					#切換elasticsearch用戶
  [root@localhost ~]# /usr/share/elasticsearch/bin/elasticsearch 			#前臺啟動查看是否報(bào)錯
  
  
  2.elasticsearch安裝路徑
  [root@localhost ~]# whereis elasticsearch
  elasticsearch: /etc/elasticsearch /usr/share/elasticsearch
  配置文件路徑：/etc/elasticsearch 
  安裝程序路徑：/usr/share/elasticsearch
  

  ?

• Elasticsearch 系統(tǒng)優(yōu)化

  優(yōu)化默認(rèn)軟限制或硬限制 參考：https://access.redhat.com/solutions/406663

  在/etc/security/limits.d/20-nproc.conf 下添加如下配置
  優(yōu)化配置：
  [root@localhost limits.d]# vim /etc/security/limits.d/20-nproc.conf
  
  elasticsearch    soft    nofile    65535   #elasticsearch  代表你創(chuàng)建的es用戶我這里是elasticsearch
  elasticsearch    hard    nofile    65535
  
  elasticsearch    soft    nproc    4096
  elasticsearch    hard    nproc    4096
  
  elasticsearch    soft    memlock    unlimited
  elasticsearch    hard    memlock    unlimited
  ?
  
  

• Elasticsearch節(jié)點(diǎn)配置

  一共3個節(jié)點(diǎn) 分別是 master、node1、node2，我這里用IP命名

  主要配置文件在：/etc/elasticsearch/elasticsearch.yml

  master

  [root@localhost ~]# grep -Ev "^$|^[#;]"  /etc/elasticsearch/elasticsearch.yml
  path.data: /var/log/elasticsearch/data
  path.logs: /var/log/elasticsearch/logs
  cluster.name: els
  node.name: 10.0.0.1
  network.host: 10.0.0.1
  node.master: true   #搶占master
  http.port: 9200
  http.cors.enabled: true
  http.cors.allow-origin: '*'
  discovery.seed_hosts: ['10.0.0.1', '10.0.0.2','10.0.0.3']
  cluster.initial_master_nodes: ['10.0.0.1', '10.0.0.2','10.0.0.3']
  discovery.zen.minimum_master_nodes: 3
  indices.memory.index_buffer_size: 20%
  indices.query.bool.max_clause_count: 100000000
  xpack.security.enabled: true
  xpack.security.transport.ssl.enabled: true
  xpack.security.transport.ssl.verification_mode: certificate
  xpack.security.transport.ssl.client_authentication: required
  xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12	  #證書配置
  xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12  #證書配置
  

  node1

  [root@localhost ~]# grep -Ev "^$|^[#;]"  /etc/elasticsearch/elasticsearch.yml
  path.data: /var/log/elasticsearch/data
  path.logs: /var/log/elasticsearch/logs
  cluster.name: els
  node.name: 192.168.169.40
  network.host: 192.168.169.40
  node.master: true   #搶占master
  http.port: 9200
  http.cors.enabled: true
  http.cors.allow-origin: '*'
  discovery.seed_hosts: ['10.0.0.1', '10.0.0.2','10.0.0.3']
  cluster.initial_master_nodes: ['10.0.0.1', '10.0.0.2','10.0.0.3']
  discovery.zen.minimum_master_nodes: 3
  indices.memory.index_buffer_size: 20%
  indices.query.bool.max_clause_count: 100000000
  xpack.security.enabled: true
  xpack.security.transport.ssl.enabled: true
  xpack.security.transport.ssl.verification_mode: certificate
  xpack.security.transport.ssl.client_authentication: required
  xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12	  #證書配置
  xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12  #證書配置
  

  node2

  [root@localhost ~]# grep -Ev "^$|^[#;]"  /etc/elasticsearch/elasticsearch.yml
  path.data: /var/log/elasticsearch/data
  path.logs: /var/log/elasticsearch/logs
  cluster.name: els
  node.name: 10.0.0.3
  network.host: 10.0.0.3
  node.master: true   #搶占master
  http.port: 9200
  http.cors.enabled: true
  http.cors.allow-origin: '*'
  discovery.seed_hosts: ['10.0.0.1', '10.0.0.2','10.0.0.3']
  cluster.initial_master_nodes: ['10.0.0.1', '10.0.0.2','10.0.0.3']
  discovery.zen.minimum_master_nodes: 3
  indices.memory.index_buffer_size: 20%
  indices.query.bool.max_clause_count: 100000000
  xpack.security.enabled: true
  xpack.security.transport.ssl.enabled: true
  xpack.security.transport.ssl.verification_mode: certificate
  xpack.security.transport.ssl.client_authentication: required
  xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12   #證書配置
  xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12   #證書配置
  

  注：在沒做好基礎(chǔ)配置前，安裝好后先切換到ES用戶前臺啟動一次，看看是否報(bào)錯。然后在做好基礎(chǔ)配置后，一定要切換到ES用戶，前臺啟動看看是否報(bào)錯等，在接著往下

  Elasticsearch 設(shè)置證書和密鑰

  在/etc/elasticsearch/elasticsearch.yml下 添加如下配置

  xpack.security.enabled: true
  xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12   #證書配置
  xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12   #證書配置
  

  生成如下證書，并賦予證書權(quán)限，有疑問請?zhí)D(zhuǎn)到報(bào)錯篇

  #生成證書和證書密鑰，證書生成后默認(rèn)路徑在/usr/share/elasticsearch/下
  [root@localhost ~]# sh /usr/share/elasticsearch/bin/elasticsearch-certutil ca  #生成證書，直接全部回車到最后
  [root@localhost ~]# sh /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12   #生成證書對應(yīng)的密鑰，在將密鑰分發(fā)到/etc/elasticsearch/
  [root@localhost ~]# ls /usr/share/elasticsearch/
  bin  elastic-certificates.p12  elastic-stack-ca.p12  jdk  lib  LICENSE.txt  modules  NOTICE.txt  plugins  README.asciidoc
  
  #拷貝證書到/etc/elasticsearch/下，和配置文件對應(yīng)，并授權(quán)
  [root@localhost ~]# cp /usr/share/elasticsearch/elastic-certificates.p12  /etc/elasticsearch/
  [root@localhost ~]# chomd 777 /etc/elasticsearch/elastic-certificates.p12
  [root@localhost ~]# chown -R elasticsearch:elasticsearch /etc/elasticsearch/elastic-certificates.p12
  [root@localhost ~]# ls /etc/elasticsearch/
  elastic-certificates.p12  elasticsearch.yml  jvm.options.d      role_mapping.yml  users
  elasticsearch.keystore    jvm.options        log4j2.properties  roles.yml         users_roles
  
  

  注：設(shè)置證書在集群沒起來的時(shí)候就可以配置，配置完證書后**把密鑰證書 elastic-certificates.p12 **分發(fā)證書到各個節(jié)點(diǎn)的/etc/elasticsearch/下，路徑可自定義。

  ?

  Elasticsearch 生成密碼

  常見的生成密碼有兩種，我只展示第一種；

  Plan A：隨機(jī)生成所有密碼

  Plan B：自定義生成密碼

  隨機(jī)生成密碼如下：

  命令：
  sh /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto
  
  演示：
  [root@localhost bin]# sh /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto
  Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
  The passwords will be randomly generated and printed to the console.
  Please confirm that you would like to continue [y/N] y
  
  
  Changed password for user apm_system
  PASSWORD apm_system = sMZg4sW5bBbfL1fRjDPP
  
  Changed password for user kibana
  PASSWORD kibana = qZjB60sGzxfBcPrTxdQT
  
  Changed password for user logstash_system
  PASSWORD logstash_system = raxBaIRutgxxwRqe63c1
  
  Changed password for user beats_system
  PASSWORD beats_system = 86NyKgnMkaDrb9gBSyr4
  
  Changed password for user remote_monitoring_user
  PASSWORD remote_monitoring_user = sSQycnFqnTeEuxBZN7HS
  
  Changed password for user elastic
  PASSWORD elastic = Y3NpRblUxipGz9YCN6gg
  
  [root@localhost bin]#
  

  注：生成密鑰后需要做好保存，在集群沒起來前，生成密鑰時(shí)會報(bào)錯。集群起來后在master節(jié)點(diǎn)生成密碼即可；

• Elasticsearch驗(yàn)證

  通過curl的方式，查看每個節(jié)點(diǎn)的狀態(tài)是否正常和集群是否正常
  http://10.0.0.1:9200

  {
    "name" : "10.0.0.1",
    "cluster_name" : "els",
    "cluster_uuid" : "rCoR2r6oTzmcozBuUImupA",
    "version" : {
      "number" : "7.7.1",
      "build_flavor" : "default",
      "build_type" : "rpm",
      "build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",
      "build_date" : "2020-05-28T16:30:01.040088Z",
      "build_snapshot" : false,
      "lucene_version" : "8.5.1",
      "minimum_wire_compatibility_version" : "6.8.0",
      "minimum_index_compatibility_version" : "6.0.0-beta1"
    },
    "tagline" : "You Know, for Search"
  }
  

  http://10.0.0.2:9200

  {
    "name" : "10.0.0.2",
    "cluster_name" : "els",
    "cluster_uuid" : "rCoR2r6oTzmcozBuUImupA",
    "version" : {
      "number" : "7.7.1",
      "build_flavor" : "default",
      "build_type" : "rpm",
      "build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",
      "build_date" : "2020-05-28T16:30:01.040088Z",
      "build_snapshot" : false,
      "lucene_version" : "8.5.1",
      "minimum_wire_compatibility_version" : "6.8.0",
      "minimum_index_compatibility_version" : "6.0.0-beta1"
    },
    "tagline" : "You Know, for Search"
  }
  

  http://10.0.0.3:9200

   {
     "name" : "10.0.0.3",
     "cluster_name" : "els",
     "cluster_uuid" : "rCoR2r6oTzmcozBuUImupA",
     "version" : {
       "number" : "7.7.1",
       "build_flavor" : "default",
       "build_type" : "rpm",
       "build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",
       "build_date" : "2020-05-28T16:30:01.040088Z",
       "build_snapshot" : false,
       "lucene_version" : "8.5.1",
       "minimum_wire_compatibility_version" : "6.8.0",
       "minimum_index_compatibility_version" : "6.0.0-beta1"
     },
     "tagline" : "You Know, for Search"
   }
  

  基本正常，接下來查看集群狀態(tài)
  http://10.0.0.1:9200/_cluster/health?pretty

  {
    "cluster_name" : "els",  	#集群名稱
    "status" : "green",     	#集群狀態(tài)，green表示所有主分片和副本分片%100可用（屬于正常）
    "timed_out" : false,		#超時(shí)
    "number_of_nodes" : 3,	#集群節(jié)點(diǎn)3個
    "number_of_data_nodes" : 3,
    "active_primary_shards" : 70,
    "active_shards" : 140,
    "relocating_shards" : 2,
    "initializing_shards" : 0,
    "unassigned_shards" : 0,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 100.0
  }
  

  檢查端口是否正常

  [root@localhost ~]# netstat -anpt
  [root@localhost ~]# lsof -i:9200
  

  測試各個節(jié)點(diǎn)正常，集群正常，端口正常，自此es集群部署完畢

?

• Elasticsearch報(bào)錯

• 配置好證書后，切換elasticsearch用戶啟動ES時(shí)，收到如下報(bào)錯

  [ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [192.168.169.12] uncaught exception in thread [main]
  org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - not permitted to read truststore file [/etc/elasticsearch/elastic-certificates.p12]]; nested: AccessDeniedException[/etc/elasticsearch/elastic-certificates.p12];
  
  #報(bào)錯原因：無法加載/etc/elasticsearch/elastic-certificates.p12證書
  
  #因?yàn)槭侵苯由傻淖C書，未改動權(quán)限，直接copy到/etc/elasticsearch目錄下，查看了下權(quán)限，無法執(zhí)行，不屬于elasticsearch用戶。
  #解決辦法：授權(quán)給elasticsearch用戶，加權(quán)到777在更改。
  chomd 777 /etc/elasticsearch/elastic-certificates.p12
  chown -R elasticsearch:elasticsearch /etc/elasticsearch/elastic-certificates.p12
  #再啟動時(shí)問題解決
  

?

二、kibana部署

• kibana安裝

  依舊是提前下載好了并上傳到了10.0.0.1的/root/下

  [root@localhost ~]# ls
  kibana-7.7.1-x86_64.rpm
  
  #直接本地安裝
  [root@localhost ~]# yum -y localinstall kibana-7.7.1-x86_64.rpm
  
  #文件路徑
  [root@localhost ~]# whereis kibana
  kibana: /etc/kibana /usr/share/kibana
  
  #配置文件路徑：/etc/kibana
  #安裝程序路徑：/usr/share/kibana
  

• kibana配置

  配置文件是 /etc/kibana/kibana.yml 直接編輯找到相關(guān)配置更改即可，參考如下

  [root@localhost kibana]# grep -Ev "^$|^[#;]" /etc/kibana/kibana.yml
  server.port: 5601
  server.host: "0.0.0.0"
  server.maxPayloadBytes: 10485760
  elasticsearch.hosts: ["http://192.168.169.41:9200","http://192.168.169.40:9200","http://192.168.169.39:9200"]
  #kibana.index: ".kibana"      //參考kibana報(bào)錯，可解決
  elasticsearch.username: "elastic"
  elasticsearch.password: "Y3NpRblUxipGz9YCN6gg"
  i18n.locale: "zh-CN"       #編碼改為中國
  

• 啟動kibana，進(jìn)入web界面

  啟動命令如下;比較粗暴直接在root下啟動
  [root@localhost ~]# sh /usr/share/kibana/bin/kibana --allow-root
  
  沒有報(bào)錯，直接系統(tǒng)啟動
  [root@localhost ~]# systemctl start kibana
  
  

  

  自此kibana安裝完畢，安裝完畢還沒有數(shù)據(jù)，需要配合filebeat、logstash或者auditbeat,推送日志數(shù)據(jù)到es中，然后建立索引，并配合面板進(jìn)行展示即可

• kibana報(bào)錯

  基礎(chǔ)配置做好時(shí)啟動報(bào)如下錯誤文章來源地址http://www.zghlxwxcb.cn/news/detail-492320.html

  [root@localhost ~]# sh /usr/share/kibana/bin/kibana --allow-root
    log   [07:47:48.360] [warning][plugins-discovery] Expect plugin "id" in camelCase, but found: apm_oss
    log   [07:47:48.368] [warning][plugins-discovery] Expect plugin "id" in camelCase, but found: file_upload
    log   [07:47:48.369] [warning][plugins-discovery] Expect plugin "id" in camelCase, but found: triggers_actions_ui
    log   [07:47:53.145] [warning][config][deprecation] Setting [elasticsearch.username] to "elastic" is deprecated. You should use the "kibana" user instead.
    log   [07:47:53.145] [warning][config][deprecation] Setting [monitoring.username] to "elastic" is deprecated. You should use the "kibana" user instead.
    log   [07:47:53.148] [fatal][root] { Error: Unknown configuration key(s): "index". Check for spelling errors and ensure that expected plugins are installed.
      at ensureValidConfiguration (/usr/share/kibana/src/core/server/legacy/config/ensure_valid_configuration.js:46:11) code: 'InvalidConfig', processExitCode: 64, cause: undefined }
  
   FATAL  Error: Unknown configuration key(s): "index". Check for spelling errors and ensure that expected plugins are installed.
  
  報(bào)錯原因：Unknown configuration key(s): "index"，是因?yàn)榇隧?xiàng)配置錯誤導(dǎo)致，官網(wǎng)8.0版本的kibana中 已經(jīng)沒有使用該配置，所以刪除該配置即可。
  參考：https://discuss.elastic.co/t/kibana-8-0-0-unknown-configuration-key-s-kibana-index/299228/1
  

到了這里，關(guān)于Elasticsearch+Kibana集群部署(3節(jié)點(diǎn))的文章就介紹完了。如果您還想了解更多內(nèi)容，請?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！