Elastic Stack 環(huán)境配置與框架簡介

這篇具有很好參考價值的文章主要介紹了Elastic Stack 環(huán)境配置與框架簡介。希望對大家有所幫助。如果存在錯誤或未考慮完全的地方，請大家不吝賜教，您也可以點擊"舉報違法"按鈕提交疑問。

簡介

什么是Elastic Stack

Elasticasearch

Logstash

Kibana

Beats

框架圖

下載

配置

一、安裝java環(huán)境

啟動

Elasticsearch

Kibana

FileBeat

Logstash

測驗

簡介

什么是Elastic Stack

Elastic Stack縮寫為elk，它由三個軟件組成：Elasticsearch、Logstash、Kibana，因此縮寫為elk，隨著版本的變化，添加了一個新軟件Beats，因此其全名現(xiàn)在為ELKB

Elasticasearch

Elasticasearch是一個基于java的開源分布式搜索引擎。其特點如下：

分布式
無需配置
自動發(fā)現(xiàn)
索引復制機制
Restful風格界面
多個數(shù)據(jù)源
自動搜索加載

Elasticasearch為所有類型的數(shù)據(jù)提供近乎實時的搜索和分析。無論您擁有結構化或非結構化文本、數(shù)字數(shù)據(jù)或地理空間數(shù)據(jù)，Elasticsearch都可以以支持快速搜索的方式高效地存儲和索引。

Logstash

Logstash是一個輕量級的日志收集系統(tǒng)。它以管道的形式工作。它可以過濾從管道輸入的數(shù)據(jù)，并在輸出時將結果傳輸?shù)饺魏畏掌?/p>

Kibana

Kibana是Elasticsearch的可視化服務。它可以直觀地顯示Elasticsearch生成的數(shù)據(jù)，并支持多樣化的分析和演示

Beats

Beats是Elastic開發(fā)的客戶端程序，用于存儲在監(jiān)控服務器中。它主要用于數(shù)據(jù)收集，可以將收集的數(shù)據(jù)發(fā)送到Elasticsearch或Logstash

框架圖

elastic stack 搭建,# Elastic Stack,大數(shù)據(jù),elasticsearch

?Beats（FileBeats）

Beats->Logstash->Elasticsearch->Kibana

elastic stack 搭建,# Elastic Stack,大數(shù)據(jù),elasticsearch

下載

本文中使用的beats功能：filebeat

	Name	Link
1	Elasticasearch	download
2	Logstash	download
3	Kibana	download
4	beats	download

配置

一、安裝java環(huán)境

安裝JDK

sudo apt-get install openjdk-8-jdk

檢查Java

$ java -version
openjdk version "1.8.0_352"
OpenJDK Runtime Environment (build 1.8.0_352-8u352-ga-1~22.04-b08)
OpenJDK 64-Bit Server VM (build 25.352-b08, mixed mode)

關閉防火墻

sudo ufw disable

啟動

Elasticsearch

打開配置文件

vim config/elasticsearch.yml

打開以下兩個字段并設置網(wǎng)絡。主機設置為0.0.0.0，以便Internet可以訪問此計算機

# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200

關閉GeoIp采集

ingest.geoip.downloader.enabled: false

關閉SSH身份驗證。你可以檢查自己的情況。如果您在沒有證書的情況下通過HTTPS訪問瀏覽器，將被阻止

xpack.security.enabled: false
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

添加跨域訪問

http.cors.enabled: true
http.cors.allow-origin: "*"

修改jvm虛擬機的映射內(nèi)存以提高文件處理速度

vim config/jvm.options

修改如下：

如果使用Java 19，可以添加--enable-preview參數(shù)來改進MMapDirectory

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## which should be named with .options suffix, and the min and
## max should be set to the same value. For example, to set the
## heap to 4 GB, create a new file in the jvm.options.d
## directory containing these lines:
##
-Xms4g
-Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/8.5/heap-size.html
## for more information
##
################################################################

修改虛擬MAP大小

sudo vim /etc/sysctl.conf

將此內(nèi)容添加到文件并保存

vm.max_map_count=262144

重置sysctl

sysctl -p

啟動

./bin/elasticsearch

在瀏覽器中打開以下地址，并將IP更改為服務器的IP（非環(huán)回地址）

http://x.x.x.x:9200/

如果成功，瀏覽器將返回以下輸出：

{
  "name" : "xxxx",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "mkUeEpucQEWlPVQpJXwOjw",
  "version" : {
    "number" : "8.5.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "a846182fa16b4ebfcc89aa3c11a11fd5adf3de04",
    "build_date" : "2022-11-17T18:56:17.538630285Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Kibana

配置端口和地址。使用0.0.0.0作為通用地址

vim config/kibana.yaml

添加以下內(nèi)容：

# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

啟動

./bin/kinaba

通過瀏覽器訪問以下地址：

http://x.x.x.x:5601/

如果啟動正常，你可以看到以下頁面：

elastic stack 搭建,# Elastic Stack,大數(shù)據(jù),elasticsearch

FileBeat

首先，創(chuàng)建測試日志文件并輸入測試內(nèi)容：

touch test.log
echo test message > test.log

打開filebeat。yml并修改以下內(nèi)容
測驗log是剛剛創(chuàng)建的文件，你可以根據(jù)你的要求更改：

# filestream is an input for collecting log messages from files.
- type: filestream
 
  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id
 
  # Change to true to enable this input configuration.
  enabled: true
 
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /home/logserver_bzm/test.log

屏蔽elastic search輸出

#output.elasticsearch:
  # Array of hosts to connect to.
  #  hosts: ["localhost:9200"]

打開logstash

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

啟動

./filebeat -e -c filebeat.yml -d "publish"

Logstash

首先創(chuàng)建一個配置文件來配置logstash

vim ./config/logstash.conf

寫下以下內(nèi)容
"logstash的簡單配置。輸入部分用于配置filebeat，過濾部分用于設置過濾屏幕以過濾filebeat的輸入，輸出部分將消息發(fā)送到彈性搜索"

input {
        beats {
                port => 5044   
                codec => "json"
        }
}
 
filter {
grok {
match => { "message" => "(?<text>(.*))" }
}
}
 
output {
        stdout { codec => rubydebug }
        elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "test1-nginx-access-%{+YYYY.MM.dd}"
                }
}

啟動

./bin/logstash -f ./config/logstash.conf

測驗

完成所有配置并啟動四項服務后，在瀏覽器中輸入Kibana的地址
在打開的頁面中選擇菜單圖標

elastic stack 搭建,# Elastic Stack,大數(shù)據(jù),elasticsearch

下拉并找到“Dev Tools”

elastic stack 搭建,# Elastic Stack,大數(shù)據(jù),elasticsearch

運行控制臺代碼。如果一切正常，你可以看到以下輸出：

elastic stack 搭建,# Elastic Stack,大數(shù)據(jù),elasticsearch

可以從輸出中看到你在filebeat里配置的日志文件里的內(nèi)容

{
  "took": 58,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "test1-nginx-access-2022.11.25",
        "_id": "P93orIQBWUNz5yg551ec",
        "_score": 1,
        "_source": {
          "@version": "1",
          "input": {
            "type": "filestream"
          },
          "@timestamp": "2022-11-25T03:49:18.038Z",
          "log": {
            "file": {
              "path": "/home/logserver_bzm/test.log"
            },
            "offset": 0
          },
          "text": "test message",
          "tags": [
            "_jsonparsefailure",
            "beats_input_codec_json_applied"
          ],
          "message": "test message",
          "ecs": {
            "version": "8.0.0"
          }
        }
      }
    ]
  }
}

很明顯，輸出中的消息部分與日志中的內(nèi)容一致

你還可以看到logstash的輸出

? elastic stack 搭建,# Elastic Stack,大數(shù)據(jù),elasticsearch 文章來源地址http://www.zghlxwxcb.cn/news/detail-530034.html