- 編寫(xiě)logstash數(shù)據(jù)轉(zhuǎn)換的配置文件
export-csv.yml
需要根據(jù)es中數(shù)據(jù)和導(dǎo)出的原始數(shù)據(jù)格式對(duì)應(yīng)的clickhouse中字段類型對(duì)數(shù)據(jù)的要求在filter中對(duì)數(shù)據(jù)的處理input { elasticsearch { hosts => "localhost:9200" index => "test" } } # 通過(guò)filter對(duì)數(shù)據(jù)做ETL以符合clickhouse要求 filter { # 將long類型毫秒值轉(zhuǎn)為指定時(shí)間格式 ruby { code => "event.set('timestamp',Time.at((event.get('timestamp').to_i)/1000).strftime('%Y-%m-%d %H:%M:%S'))" } # 將指定字段類型轉(zhuǎn)為符合clickhouse要求的，需要根據(jù)es中數(shù)據(jù)和導(dǎo)出的原始數(shù)據(jù)格式對(duì)應(yīng)的clickhouse中字段類型決定 mutate { convert => { "dstType" => "integer" "downStreamOct" => "integer" "totalOct" => "integer" "upStreamOct" => "integer" "srcType" => "integer" } } } output { csv { fields => ["srcRegion","dstType","downStreamOct","totalOct","upStreamOct","srcType","timestamp"] path => "/tmp/test.csv" } } - 啟動(dòng)logstash，
-f指定配置文件不使用logstash默認(rèn)配置文件路徑./logstash-6.8.6/bin/logstash -f test.yml - clickhouse中創(chuàng)建表，字段類型需要注意根據(jù)es中字段類型和es導(dǎo)出的數(shù)據(jù)格式?jīng)Q定，字段類型決定了可以寫(xiě)入clickhouse的該字段數(shù)據(jù)的格式
CREATE TABLE traffic.traffic_monitor ( -- 指定字段類型 `srcRegion` String, `dstType` UInt8, `downStreamOct` UInt64, `totalOct` UInt64, `upStreamOct` UInt64, `srcType` UInt8, `timestamp` DateTime, ) -- 指定存儲(chǔ)引擎、分區(qū)字段和排序字段 ENGINE = MergeTree() PARTITION BY toYYYYMMDD(timestamp) ORDER BY (timestamp) - 通過(guò)csv文件將數(shù)據(jù)導(dǎo)入clickhouse
csv首行有字段名稱：clickhouse-client --query "INSERT INTO traffic.traffic_monitor FORMAT CSVWithNames" < test.csv
csv首行無(wú)字段名稱：clickhouse-client --query "INSERT INTO traffic.traffic_monitor FORMAT CSV" < test.csv
文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-538883.html
文章來(lái)源：http://www.zghlxwxcb.cn/news/detail-538883.html
到了這里，關(guān)于通過(guò)logstash(6.8.6)將es(6.8.6)數(shù)據(jù)導(dǎo)入clickhouse的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
