1、下載解壓安裝包到對(duì)應(yīng)目錄
> [zhangflink@9wmwtivvjuibcd2e package]$ tar -zxvf flink-1.16.0-bin-scala_2.12.tgz -C ../software/
2、修改配置文件
[zhangflink@9wmwtivvjuibcd2e software]$ vim flink/conf/flink-conf.yaml
#設(shè)置jobmanager的機(jī)器地址
jobmanager.rpc.address: flinkv2
#設(shè)置允許訪問(wèn)jobmanager的機(jī)器地址,相當(dāng)于白名單,0.0.0.0表示允許所有機(jī)器訪問(wèn)
jobmanager.bind-host: 0.0.0.0
#設(shè)置允許訪問(wèn)taskmanager的機(jī)器地址,相當(dāng)于白名單,0.0.0.0表示允許所有機(jī)器訪問(wèn)
taskmanager.bind-host: 0.0.0.0
#設(shè)置taskmanager的機(jī)器地址,集群設(shè)置為當(dāng)前機(jī)器所在地址,jobmanager那臺(tái)機(jī)器不修改保持localhost地址。
taskmanager.host: flinkv1
#設(shè)置webui頁(yè)面訪問(wèn)地址,如果沒(méi)有設(shè)置或者IP地址錯(cuò)誤會(huì)照成,集群即使啟動(dòng)成功,頁(yè)面無(wú)法訪問(wèn)。
rest.bind-address: 0.0.0.0
[zhangflink@9wmwtivvjuibcd2e software]$ vim flink/conf/workers
#設(shè)置taskmanager地址
flinkv1
flinkv3
3、分發(fā)所有節(jié)點(diǎn)配置文件
[zhangflink@9wmwtivvjuibcd2e software]$ xsync flink/conf/
4、啟動(dòng)集群
啟動(dòng)集群在jobmanager那臺(tái)機(jī)器啟動(dòng)
[zhangflink@9wmwtivvjuibcd2e-0001 flink]$ bin/start-cluster.sh
啟動(dòng)成功jobmanager會(huì)出現(xiàn)如下進(jìn)程
啟動(dòng)成功taskmanager會(huì)出現(xiàn)如下進(jìn)程
訪問(wèn)頁(yè)面http://localhost:8081/#/overviewi頁(yè)面slots卡槽數(shù)正確展示說(shuō)明啟動(dòng)成功
5、配置錯(cuò)誤遇到的問(wèn)題
(1)如果機(jī)器的taskmanager節(jié)點(diǎn)沒(méi)有配置當(dāng)前節(jié)點(diǎn)機(jī)器IP地址那么會(huì)出現(xiàn)以下報(bào)錯(cuò)
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connecting to remote task manager 'localhost/127.0.0.1:42231' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.connect(PartitionRequestClientFactory.java:169) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.connectWithRetries(PartitionRequestClientFactory.java:135) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:96) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:95) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:186) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.internalRequestPartitions(SingleInputGate.java:342) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:312) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.taskmanager.InputGateWithMetrics.requestPartitions(InputGateWithMetrics.java:115) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935) [flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914) [flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728) [flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) [flink-dist-1.16.0.jar:1.16.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒絕連接: localhost/127.0.0.1:42231
Caused by: java.net.ConnectException: finishConnect(..) failed: 拒絕連接
at org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.shaded.netty4.io.netty.channel.unix.Socket.finishConnect(Socket.java:320) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[flink-dist-1.16.0.jar:1.16.0]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[flink-dist-1.16.0.jar:1.16.0]
... 1 more
2023-07-16 22:28:54,030 WARN org.apache.flink.runtime.taskmanager.Task [] - Sink: Print to Std. Out (2/2)#0 (40e492979c92fe282bc6ccbae6837fc3_0a448493b4782967b150582570326227_1_0) switched from RUNNING to FAILED with failure cause: org.apache.flink.runtime.io.network.partition.consumer.PartitionConnectionException: Connection for partition 76fdc87b7372a6e8895421b938589124#0@40e492979c92fe282bc6ccbae6837fc3_bc764cd8ddf7a0cff126f51c16239658_0_0 not reachable.
at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:190)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.internalRequestPartitions(SingleInputGate.java:342)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:312)
at org.apache.flink.runtime.taskmanager.InputGateWithMetrics.requestPartitions(InputGateWithMetrics.java:115)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connecting to remote task manager 'localhost/127.0.0.1:42231' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.connect(PartitionRequestClientFactory.java:169)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.connectWithRetries(PartitionRequestClientFactory.java:135)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:96)
at org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:95)
at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:186)
... 16 more
Caused by: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒絕連接: localhost/127.0.0.1:42231
Caused by: java.net.ConnectException: finishConnect(..) failed: 拒絕連接
at org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
at org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
at org.apache.flink.shaded.netty4.io.netty.channel.unix.Socket.finishConnect(Socket.java:320)
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470)
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
解決方法,檢查以下配置是否正確。文章來(lái)源:http://www.zghlxwxcb.cn/news/detail-786418.html
#設(shè)置taskmanager的機(jī)器地址,集群設(shè)置為當(dāng)前機(jī)器所在地址,jobmanager那臺(tái)機(jī)器不修改保持localhost地址。
taskmanager.host: flinkv1
(2)如果出現(xiàn):無(wú)法連接到taskmanage或者resourcemanager的類(lèi)似拒絕連接的報(bào)錯(cuò),說(shuō)明連接白名單沒(méi)有放開(kāi)。這種情況也會(huì)照成,集群?jiǎn)?dòng)成功但是WEBUI上面Available Task Slots數(shù)不顯示,taskmanager等信息不顯示。
Could not connect to rpc endpoint under address akka.tcp://flink@flinkv2:6123/user/rpc/resourcemanager_*.
解決方法:設(shè)置taskmanager和jobmanager允許所有機(jī)器連接。文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-786418.html
#設(shè)置允許訪問(wèn)jobmanager的機(jī)器地址,相當(dāng)于白名單,0.0.0.0表示允許所有機(jī)器訪問(wèn)
jobmanager.bind-host: 0.0.0.0
#設(shè)置允許訪問(wèn)taskmanager的機(jī)器地址,相當(dāng)于白名單,0.0.0.0表示允許所有機(jī)器訪問(wèn)
taskmanager.bind-host: 0.0.0.0
到了這里,關(guān)于Flink本地集群部署啟動(dòng)&常見(jiàn)問(wèn)題的解決方法的文章就介紹完了。如果您還想了解更多內(nèi)容,請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章,希望大家以后多多支持TOY模板網(wǎng)!