CentOs7.x安裝部署SeaTunnelWeb遇到的坑
1. 環(huán)境
??采用的是虛擬機(jī)上安裝CentOs7.x，安裝java15環(huán)境，安裝mysql8.0.28，這兩步驟省略，之前的文章里面有講，屬于基礎(chǔ)環(huán)境配置，也比較簡(jiǎn)單，說(shuō)明我用的是一臺(tái)CentOs7.x的虛擬機(jī)實(shí)例，都安裝在一臺(tái)上面，需要開(kāi)放8081、3306和5801的防火墻，不然會(huì)網(wǎng)絡(luò)訪問(wèn)不到。
2. SeaTunnel安裝部署
2.1下載安裝包
export version="2.3.3"
wget "https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz"
tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"
2.2 設(shè)置環(huán)境變量
vi /etc/profile.d/seatunnel.sh
# 新增如下變量
export SEATUNNEL_HOME=/root/apache-seatunnel-2.3.3 #這里設(shè)置的是seatunnel的解壓目錄即可
export PATH=$PATH:$SEATUNNEL_HOME/bin
然后執(zhí)行source /etc/profile
2.3 安裝連接器插件
??進(jìn)入/root/apache-seatunnel-2.3.3路徑下執(zhí)行安裝連接器插件
sh bin/install-plugin.sh 2.3.3
??可以修改/root/apache-seatunnel-2.3.3/connectors下的plugin-mapping.properties文件，根據(jù)需要加入連接器插件，把不需要的連接器插件名稱(chēng)刪除留下需要的即可，然后執(zhí)行上面的安裝命令，由于我沒(méi)有修改這個(gè)文件，默認(rèn)安裝的是全部的連接器插件，所以需要一點(diǎn)時(shí)間，下載速度取決于網(wǎng)速。
2.4 拷貝jar包到lib下
2.5 啟動(dòng)命令
??進(jìn)入/root/apache-seatunnel-2.3.3路徑下執(zhí)行啟動(dòng)命令：
sh bin/seatunnel-cluster.sh -d -DJvmOption="-Xms1G -Xmx1G"
或者
nohup sh bin/seatunnel-cluster.sh 2>&1 &
??jps查看seatunnel的進(jìn)程如下：
??查看logs下的日志均正常輸出沒(méi)有報(bào)錯(cuò)，說(shuō)明啟動(dòng)成功，然后執(zhí)行下面的官方demo沒(méi)有報(bào)錯(cuò)，說(shuō)明seatunnel啟動(dòng)是正常的。
2.6 執(zhí)行官方client提交任務(wù)demo
??進(jìn)入/root/apache-seatunnel-2.3.3路徑下執(zhí)行啟動(dòng)命令：
$SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template
??該命令來(lái)自于官網(wǎng)，執(zhí)行結(jié)果如下：
[root@es1 apache-seatunnel-2.3.3]# $SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/root/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /root/apache-seatunnel-2.3.3/config/seatunnel.yaml
12月 19, 2023 12:11:20 上午 org.apache.seatunnel.engine.common.config.SeaTunnelConfig
信息: seatunnel.home is /root/apache-seatunnel-2.3.3
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/root/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /root/apache-seatunnel-2.3.3/config/hazelcast.yaml
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/root/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /root/apache-seatunnel-2.3.3/config/hazelcast-client.yaml
2023-12-19 00:11:21,149 INFO com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
2023-12-19 00:11:21,233 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-12-19 00:11:21,234 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/root/apache-seatunnel-2.3.3/starter/seatunnel-starter.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2023-12-19 00:11:21,294 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
2023-12-19 00:11:21,298 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to [localhost]:5801
2023-12-19 00:11:21,352 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-12-19 00:11:21,352 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [localhost]:5801:772efc0a-4c18-4a4b-baa7-b82b9ae4a395, server version: 5.1, local address: /127.0.0.1:36095
2023-12-19 00:11:21,356 INFO com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-12-19 00:11:21,384 INFO com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel] [5.1]
Members [1] {
Member [localhost]:5801 - 772efc0a-4c18-4a4b-baa7-b82b9ae4a395
}
2023-12-19 00:11:21,421 INFO com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
2023-12-19 00:11:21,706 INFO org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[]
2023-12-19 00:11:21,733 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: /root/apache-seatunnel-2.3.3/config/v2.batch.config.template
2023-12-19 00:11:21,799 INFO org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
2023-12-19 00:11:21,848 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
"env" : {
"execution.parallelism" : 2,
"job.mode" : "BATCH",
"checkpoint.interval" : 10000
},
"source" : [
{
"schema" : {
"fields" : {
"name" : "string",
"age" : "int"
}
},
"row.num" : 16,
"parallelism" : 2,
"result_table_name" : "fake",
"plugin_name" : "FakeSource"
}
],
"sink" : [
{
"plugin_name" : "Console"
}
]
}
2023-12-19 00:11:21,885 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:21,886 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:21,895 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /root/apache-seatunnel-2.3.3/connectors/seatunnel
2023-12-19 00:11:21,911 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar
2023-12-19 00:11:21,912 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Console at: file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar
2023-12-19 00:11:21,915 INFO org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.
2023-12-19 00:11:21,915 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:21,915 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:21,916 WARN org.apache.seatunnel.engine.core.parse.ConfigParserUtil - This configuration is not recommended. A source/transform(FakeSource) is configured with 'result_table_name' option value of 'fake', but subsequent transform/sink(Console) is not configured with 'source_table_name' option.
2023-12-19 00:11:21,919 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources.
2023-12-19 00:11:21,919 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:21,953 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /root/apache-seatunnel-2.3.3/connectors/seatunnel
2023-12-19 00:11:21,970 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar
2023-12-19 00:11:21,974 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='FakeSource'} from classpath
2023-12-19 00:11:22,003 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all transforms.
2023-12-19 00:11:22,003 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sinks.
2023-12-19 00:11:22,004 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:22,011 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:22,090 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Start submit job, job id: 789162834679300097, with plugin jar [file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar, file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar]
2023-12-19 00:11:22,893 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Submit job finished, job id: 789162834679300097, job name: SeaTunnel
2023-12-19 00:11:22,956 WARN org.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run
2023-12-19 00:11:24,370 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Job (789162834679300097) end with state FINISHED
2023-12-19 00:11:24,416 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand -
***********************************************
Job Statistic Information
***********************************************
Start Time : 2023-12-19 00:11:21
End Time : 2023-12-19 00:11:24
Total Time(s) : 2
Total Read Count : 32
Total Write Count : 32
Total Failed Count : 0
***********************************************
2023-12-19 00:11:24,416 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-12-19 00:11:24,422 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [localhost]:5801:772efc0a-4c18-4a4b-baa7-b82b9ae4a395, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:36095->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-12-19 00:11:24.411, lastWriteTime=2023-12-19 00:11:24.371, closedTime=2023-12-19 00:11:24.420, connected server version=5.1}
2023-12-19 00:11:24,422 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-12-19 00:11:24,431 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-12-19 00:11:24,433 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-12-19 00:11:24,433 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ......
2023-12-19 00:11:24,438 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal
3. SeaTunnel-Web安裝部署
3.1 下載安裝包
??安裝包下在地址如下：
https://seatunnel.apache.org/download
??解壓：
tar -zxvf apache-seatunnel-web-bin-${project.version}.tar.gz
??解壓目錄如下：
3.2 初始化數(shù)據(jù)庫(kù)腳本或修改配置application.yml中的數(shù)據(jù)庫(kù)連接信息
3.2.1 不使用它的初始化腳本自動(dòng)創(chuàng)建
??事先手動(dòng)執(zhí)行好腳本，然后在修改application.yml文件中的數(shù)據(jù)庫(kù)連接信息
3.2.2 使用腳本初始化數(shù)據(jù)庫(kù)
??首先，設(shè)置變量如下：
export HOSTNAME="localhost"
export PORT="3306"
export USERNAME="root"
export PASSWORD="123456"
??然后執(zhí)行：
sh apache-seatunnel-web-bin-2.3.3/script/init_sql.sh
??如果環(huán)境變量有沖突需要改下名字和init_sql.sh中的環(huán)境變量的名字，可以加一個(gè)前綴可以加上前綴`STWEB_ ，然后把init_sql.sh中上面的那幾個(gè)環(huán)境變量也加上這個(gè)前綴即可，然后執(zhí)行初始化命令
3.3 修改端口與數(shù)據(jù)源
vim conf/application.yml修改端口號(hào)和數(shù)據(jù)源信息
3.4 兩個(gè)文件拷貝配置
??需要將apache-seatunnel-2.3.3/config/hazelcast-client和apache-seatunnel-2.3.3/connectors/plugin-mapping.properties文件拷貝到apache-seatunnel-web-bin-2.3.3的conf下。
3.5 拷貝jar到lib下
3.6 啟動(dòng)
sh bin/seatunnel-backend-daemon.sh start
jps查看java進(jìn)程如下：
??這里有個(gè)很坑的地方就是，執(zhí)行命令如果是進(jìn)入bin路徑執(zhí)行，訪問(wèn)首頁(yè)的時(shí)候就會(huì)有問(wèn)題，訪問(wèn)首頁(yè)就會(huì)報(bào)404的錯(cuò)
sh seatunnel-backend-daemon.sh start
??訪問(wèn)首頁(yè)報(bào)錯(cuò)如下：
3.7 訪問(wèn)首頁(yè)
??訪問(wèn)ip:8081/ui，此端口為conf/application.yml中配置的端口
http://192.168.1.4:8081/
??登錄不上有可能是沒(méi)有啟動(dòng)mysql數(shù)據(jù)庫(kù)
service mysqld start #使用service啟動(dòng)mysql服務(wù)
service mysqld status #查看MYSQL運(yùn)行狀態(tài)
service mysqld stop #使用service停止mysql服務(wù)
service mysqld restart #使用service重啟mysql服務(wù)
systemctl enable mysqld.service #設(shè)置MySQL服務(wù)開(kāi)機(jī)自動(dòng)啟動(dòng)
systemctl is-enabled mysqld.service #確認(rèn)MySQL服務(wù)已經(jīng)被設(shè)置為自動(dòng)啟動(dòng)
3.8 執(zhí)行myslq-jdbc–>mysl-jdbc的單表數(shù)據(jù)同步
??執(zhí)行是成功的，但是在我的CentOs7.x的虛擬機(jī)上我是沒(méi)有安裝hadoop3.1.3的環(huán)境，執(zhí)行看日志是沒(méi)有報(bào)錯(cuò)的，官方說(shuō)是這個(gè)hadoop的環(huán)境不是必須的也是有道理的，但是之前的文章在本地調(diào)試的時(shí)候沒(méi)有hadoop環(huán)境會(huì)報(bào)錯(cuò)，所以建議需要本地編譯構(gòu)建的小伙伴還是安裝一個(gè)hadoop，以免執(zhí)行任務(wù)報(bào)錯(cuò)。文章來(lái)源：http://www.zghlxwxcb.cn/news/detail-786044.html
4. 總結(jié)
??本次分享就到此結(jié)束了，踩了一些坑，還是請(qǐng)教了官方的大佬才知道的是哪里有問(wèn)題，希望我的分享對(duì)你有所幫助，請(qǐng)一鍵三連，么么么噠！文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-786044.html
到了這里，關(guān)于CentOs7.x安裝部署SeaTunnelWeb遇到的坑的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
