目錄
一、例子說明
1.1、概述
1.1、所需環(huán)境
1.2、執(zhí)行流程?
二、部署環(huán)境
2.1、中間件部署
2.1.1部署kakfa
2.1.1.1 上傳解壓kafka安裝包
2.1.1.2?修改zookeeper.properties
2.1.1.3?修改server.properties
2.1.1.3 啟動(dòng)kafka
2.1.2、部署flink
2.1.2.1 上傳解壓flink安裝包
?2.1.2.1 修改flink配置
2.1.2.3?flink單節(jié)點(diǎn)啟動(dòng)與停止命令
2.1.3、部署doris
2.1.3.1 下載安裝包并上傳服務(wù)器
?2.1.3.2?配置doris fe（前端）
?2.1.3.3?啟動(dòng)doris fe（前端）?
?2.1.3.4 配置doris be（后端）??
??2.1.3.5?doris啟動(dòng) be（后端）?
??2.1.3.5?doris啟動(dòng)成功驗(yàn)證
2.1.3.6?doris的be在fe上注冊?
2.1.3.6?通過doris的fe的Web UI頁面創(chuàng)建數(shù)據(jù)庫表
2.1.4、部署spring-boot的syslog-kafka-es-avro
?2.1.4.1、syslog-kafka-es-avro基于netty已UDP方式監(jiān)聽syslog
2.1.4.2、syslog-kafka-es-avro已a(bǔ)vro格式保存數(shù)據(jù)到kafka
2.1.5、部署spring-boot的flink-do-doris
?2.1.5.1、flink-do-doris主類
三、效果驗(yàn)證
3.1、發(fā)送syslog日志，syslog-kafka-es-avro監(jiān)聽處理，存儲到kafka
3.2?、查看flink消費(fèi)kafka
3.3、 在doris上查看入庫詳情
一、例子說明
1.1、概述
這是個(gè)例子，將輸入寫入kafka，flink消費(fèi)kafka，并實(shí)時(shí)寫入doris。
1.1、所需環(huán)境
| 軟件 | 版本 | 備注 | |
| kafka_2.12-3.5.0 | kafka_2.12-3.5.0 | 使用自帶的zookeeper | |
| flink-1.17.1 | flink-1.17.1 | ||
| jdk | 1.8.0_202 | ||
| doris | 1.2.6 ( Stable ) | ||
spring-boot |
2.1.17.RELEASE | syslog-kafka-es-avro |
|
| spring-boot | 2.4.5 | flink-do-doris |
|
flink-doris-connector-1.17 |
1.4.0 |
||
| elasticsearch | 7.6.2 | ||
| 基礎(chǔ)目錄 | /home | ||
| 服務(wù)器 | 10.10.10.99 | centos 7.x |
1.2、執(zhí)行流程
?①、工具發(fā)送數(shù)據(jù)
?②、spring-boot基于netty開啟某端口監(jiān)聽，接收發(fā)送的消息內(nèi)容，進(jìn)行數(shù)據(jù)清洗、標(biāo)準(zhǔn)化
?③、kafka product組件接收上一步產(chǎn)生的數(shù)據(jù)，已a(bǔ)vro格式保存到kafka某topic上。
?④、flink實(shí)時(shí)消費(fèi)kafka某topic，以流的方式進(jìn)行處理，輸出源設(shè)置為doris
?⑤、終端數(shù)據(jù)可在doris的fe頁面上實(shí)時(shí)查詢。
二、部署環(huán)境
2.1、中間件部署
2.1.1部署kakfa
2.1.1.1 上傳解壓kafka安裝包
將安裝包kafka_2.12-3.5.0.tar.gz上傳到/home目錄
tar -zxvf?kafka_2.12-3.5.0.tar.gz
mv?kafka_2.12-3.5.0 kafka
2.1.1.2?修改zookeeper.properties
路徑：/home/kafka/config/zookeeper.properties
dataDir=/home/kafka/zookeeper
clientPort=2181
maxClientCnxns=0
admin.enableServer=false
# admin.serverPort=8080
2.1.1.3?修改server.properties
路徑：/home/kafka/config/server.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. ?See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. ?You may obtain a copy of the License at
#
# ? ?http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.#
# This configuration file is intended for use in ZK-based mode, where Apache ZooKeeper is required.
# See kafka.server.KafkaConfig for additional details and defaults
############################## Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0############################# Socket Server Settings #############################文章來源：http://www.zghlxwxcb.cn/news/detail-625271.html
# The address the socket server listens on. If not configured, the host name will be equal to the value of
# java.net.InetAddress.getCanonicalHostName(), with 文章來源地址http://www.zghlxwxcb.cn/news/detail-625271.html
到了這里，關(guān)于flink+kafka+doris+springboot集成例子的文章就介紹完了。如果您還想了解更多內(nèi)容，請?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
