尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記02【Flink部署】

這篇具有很好參考價值的文章主要介紹了尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記02【Flink部署】。希望對大家有所幫助。如果存在錯誤或未考慮完全的地方，請大家不吝賜教，您也可以點擊"舉報違法"按鈕提交疑問。

尚硅谷大數(shù)據(jù)技術(shù)-教程-學(xué)習(xí)路線-筆記匯總表【課程資料下載】

視頻地址：尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程從入門到精通_嗶哩嗶哩_bilibili

尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記01【Flink概述、Flink快速上手】

尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記02【Flink部署】

尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記03【】

尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記04【】

尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記05【】

尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記06【】

尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記07【】

尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記08【】

基礎(chǔ)篇

第03章 Flink部署

P011【011_Flink部署_集群角色】03:07

P012【012_Flink部署_集群搭建_集群啟動】14:22

P013【013_Flink部署_集群搭建_WebUI提交作業(yè)】13:58

P014【014_Flink部署_集群搭建_命令行提交作業(yè)】03:46

P015【015_Flink部署_部署模式介紹】10:17

P016【016_Flink部署_Standalone運行模式】08:16

P017【017_Flink部署_YARN運行模式_環(huán)境準(zhǔn)備】07:41

P018【018_Flink部署_YARN運行模式_會話模式】18:11

P019【019_Flink部署_YARN運行模式_會話模式的停止】04:10

P020【020_Flink部署_YARN運行模式_單作業(yè)模式】09:49

P021【021_Flink部署_YARN運行模式_應(yīng)用模式】12:51

P022【022_Flink部署_歷史服務(wù)器】08:11

基礎(chǔ)篇

第03章 Flink部署

P011【011_Flink部署_集群角色】03:07

第 3 章 Flink 部署

3.1 集群角色

P012【012_Flink部署_集群搭建_集群啟動】14:22

表3-1 集群角色分配

節(jié)點服務(wù)器

hadoop102

hadoop103

hadoop104

角色

JobManager

TaskManager

TaskManager

TaskManager

表3-1 集群角色分配
節(jié)點服務(wù)器	hadoop102	hadoop103	hadoop104
角色	JobManager TaskManager	TaskManager	TaskManager

[atguigu@node001 module]$ cd flink
[atguigu@node001 flink]$ cd flink-1.17.0/
[atguigu@node001 flink-1.17.0]$ bin/start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host node001.
Starting taskexecutor daemon on host node001.
Starting taskexecutor daemon on host node002.
Starting taskexecutor daemon on host node003.
[atguigu@node001 flink-1.17.0]$ jpsall 
================ node001 ================
3408 Jps
2938 StandaloneSessionClusterEntrypoint
3276 TaskManagerRunner
================ node002 ================
2852 TaskManagerRunner
2932 Jps
================ node003 ================
2864 TaskManagerRunner
2944 Jps
[atguigu@node001 flink-1.17.0]$

P013【013_Flink部署_集群搭建_WebUI提交作業(yè)】13:58

3.2.2 向集群提交作業(yè)

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.4</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <artifactSet>
                            <excludes>
                                <exclude>com.google.code.findbugs:jsr305</exclude>
                                <exclude>org.slf4j:*</exclude>
                                <exclude>log4j:*</exclude>
                            </excludes>
                        </artifactSet>
                        <filters>
                            <filter>
                                <!-- Do not copy the signatures in the META-INF folder.
                                Otherwise, this might cause SecurityExceptions when using the JAR. -->
                                <artifact>*:*</artifact>
                                <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <transformers combine.children="append">
                            <transformer
                                    implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer">
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

com.atguigu.wc.WordCountStreamUnboundedDemo

P014【014_Flink部署_集群搭建_命令行提交作業(yè)】03:46

3.2.2 向集群提交作業(yè)

4）命令行提交作業(yè)

連接成功
Last login: Fri Jun 16 14:44:01 2023 from 192.168.10.1
[atguigu@node001 ~]$ cd /opt/module/flink/flink-1.17.0/
[atguigu@node001 flink-1.17.0]$ cd bin
[atguigu@node001 bin]$ ./start-cluster.sh 
Starting cluster.
Starting standalonesession daemon on host node001.
Starting taskexecutor daemon on host node001.
Starting taskexecutor daemon on host node002.
Starting taskexecutor daemon on host node003.
[atguigu@node001 bin]$ jpsall 
================ node001 ================
2723 TaskManagerRunner
2855 Jps
2380 StandaloneSessionClusterEntrypoint
================ node002 ================
2294 TaskManagerRunner
2367 Jps
================ node003 ================
2292 TaskManagerRunner
2330 Jps
[atguigu@node001 bin]$ cd ..
[atguigu@node001 flink-1.17.0]$ bin/flink run -m node001:8081 -c com.atguigu.wc.WordCountStreamUnboundedDemo ./
bin/        conf/       examples/   lib/        LICENSE     licenses/   log/        NOTICE      opt/        plugins/    README.txt  
[atguigu@node001 flink-1.17.0]$ bin/flink run -m node001:8081 -c com.atguigu.wc.WordCountStreamUnboundedDemo ../
flink-1.17.0/ jar/          
[atguigu@node001 flink-1.17.0]$ bin/flink run -m node001:8081 -c com.atguigu.wc.WordCountStreamUnboundedDemo ../jar/FlinkTutorial-1.17-1.0-SNAPSHOT.jar 
Job has been submitted with JobID 59ae9d6532523b0c48cdb8b6c9105356

P015【015_Flink部署_部署模式介紹】10:17

3.3 部署模式

在一些應(yīng)用場景中，對于集群資源分配和占用的方式，可能會有特定的需求。Flink為各種場景提供了不同的部署模式，主要有以下三種：會話模式（Session Mode）、單作業(yè)模式（Per-Job Mode）、應(yīng)用模式（Application Mode）。

它們的區(qū)別主要在于：集群的生命周期以及資源的分配方式；以及應(yīng)用的main方法到底在哪里執(zhí)行——客戶端（Client）還是JobManager。

P016【016_Flink部署_Standalone運行模式】08:16

3.4 Standalone運行模式（了解）

獨立模式是獨立運行的，不依賴任何外部的資源管理平臺；當(dāng)然獨立也是有代價的：如果資源不足，或者出現(xiàn)故障，沒有自動擴展或重分配資源的保證，必須手動處理。所以獨立模式一般只用在開發(fā)測試或作業(yè)非常少的場景下。

[atguigu@node001 ~]$ cd /opt/module/flink/flink-1.17.0/bin
[atguigu@node001 bin]$ ./stop-cluster.sh 
Stopping taskexecutor daemon (pid: 2723) on host node001.
Stopping taskexecutor daemon (pid: 2294) on host node002.
Stopping taskexecutor daemon (pid: 2292) on host node003.
Stopping standalonesession daemon (pid: 2380) on host node001.
[atguigu@node001 bin]$ jpsall 
================ node001 ================
5120 Jps
================ node002 ================
3212 Jps
================ node003 ================
3159 Jps
[atguigu@node001 bin]$ ls
bash-java-utils.jar  flink             historyserver.sh          kubernetes-session.sh      sql-client.sh      start-cluster.sh           stop-zookeeper-quorum.sh  zookeeper.sh
config.sh            flink-console.sh  jobmanager.sh             kubernetes-taskmanager.sh  sql-gateway.sh     start-zookeeper-quorum.sh  taskmanager.sh
find-flink-home.sh   flink-daemon.sh   kubernetes-jobmanager.sh  pyflink-shell.sh           standalone-job.sh  stop-cluster.sh            yarn-session.sh
[atguigu@node001 bin]$ cd ../lib/
[atguigu@node001 lib]$ ls
flink-cep-1.17.0.jar              flink-dist-1.17.0.jar        flink-table-api-java-uber-1.17.0.jar   FlinkTutorial-1.17-1.0-SNAPSHOT.jar  log4j-core-2.17.1.jar
flink-connector-files-1.17.0.jar  flink-json-1.17.0.jar        flink-table-planner-loader-1.17.0.jar  log4j-1.2-api-2.17.1.jar             log4j-slf4j-impl-2.17.1.jar
flink-csv-1.17.0.jar              flink-scala_2.12-1.17.0.jar  flink-table-runtime-1.17.0.jar         log4j-api-2.17.1.jar
[atguigu@node001 lib]$ cd ../
[atguigu@node001 flink-1.17.0]$ bin/standalone-job.sh start --job-classname com.atguigu.wc.WordCountStreamUnboundedDemo
Starting standalonejob daemon on host node001.
[atguigu@node001 flink-1.17.0]$ jpsall 
================ node001 ================
5491 StandaloneApplicationClusterEntryPoint
5583 Jps
================ node002 ================
3326 Jps
================ node003 ================
3307 Jps
[atguigu@node001 flink-1.17.0]$ bin/taskmanager.sh 
Usage: taskmanager.sh (start|start-foreground|stop|stop-all)
[atguigu@node001 flink-1.17.0]$ bin/taskmanager.sh start
Starting taskexecutor daemon on host node001.
[atguigu@node001 flink-1.17.0]$ jpsall 
================ node001 ================
5491 StandaloneApplicationClusterEntryPoint
5995 Jps
5903 TaskManagerRunner
================ node002 ================
3363 Jps
================ node003 ================
3350 Jps
[atguigu@node001 flink-1.17.0]$ bin/taskmanager.sh stop
Stopping taskexecutor daemon (pid: 5903) on host node001.
[atguigu@node001 flink-1.17.0]$ bin/standalone-job.sh stop
No standalonejob daemon (pid: 5491) is running anymore on node001.
[atguigu@node001 flink-1.17.0]$ xcall jps
=============== node001 ===============
6682 Jps
=============== node002 ===============
3429 Jps
=============== node003 ===============
3419 Jps
[atguigu@node001 flink-1.17.0]$

P017【017_Flink部署_YARN運行模式_環(huán)境準(zhǔn)備】07:41

3.5 YARN運行模式（重點）

YARN上部署的過程是：客戶端把Flink應(yīng)用提交給Yarn的ResourceManager，Yarn的ResourceManager會向Yarn的NodeManager申請容器。在這些容器上，F(xiàn)link會部署JobManager和TaskManager的實例，從而啟動集群。Flink會根據(jù)運行在JobManger上的作業(yè)所需要的Slot數(shù)量動態(tài)分配TaskManager資源。

[atguigu@node001 flink-1.17.0]$ source /etc/profile.d/my_env.sh 
[atguigu@node001 flink-1.17.0]$ myhadoop.sh s
Input Args Error...
[atguigu@node001 flink-1.17.0]$ myhadoop.sh start
 ================ 啟動 hadoop集群 ================
 ---------------- 啟動 hdfs ----------------
Starting namenodes on [node001]
Starting datanodes
Starting secondary namenodes [node003]
 --------------- 啟動 yarn ---------------
Starting resourcemanager
Starting nodemanagers
 --------------- 啟動 historyserver ---------------
[atguigu@node001 flink-1.17.0]$ jpsall 
================ node001 ================
9200 JobHistoryServer
8416 NameNode
8580 DataNode
9284 Jps
8983 NodeManager
================ node002 ================
3892 ResourceManager
3690 DataNode
4365 Jps
4015 NodeManager
================ node003 ================
3680 DataNode
3778 SecondaryNameNode
3911 NodeManager
4044 Jps
[atguigu@node001 flink-1.17.0]$

P018【018_Flink部署_YARN運行模式_會話模式】18:11

[atguigu@node001 bin]$ ./yarn-session.sh --help
[atguigu@node001 bin]$ ./yarn-session.sh
[atguigu@node001 bin]$ ./yarn-session.sh -d -nm test

P019【019_Flink部署_YARN運行模式_會話模式的停止】04:10

3.5.3 單作業(yè)模式部署

在YARN環(huán)境中，由于有了外部平臺做資源調(diào)度，所以我們也可以直接向YARN提交一個單獨的作業(yè)，從而啟動一個Flink集群。

P020【020_Flink部署_YARN運行模式_單作業(yè)模式】09:49

3.5.3 單作業(yè)模式部署

（1）執(zhí)行命令提交作業(yè)

P021【021_Flink部署_YARN運行模式_應(yīng)用模式】12:51

3.5.4 應(yīng)用模式部署

應(yīng)用模式同樣非常簡單，與單作業(yè)模式類似，直接執(zhí)行flink run-application命令即可。

[atguigu@node001 flink-1.17.0]$ bin/flink run-application -t yarn-application -c com.atguigu.wc.WordCountStreamUnboundedDemo ./FlinkTutorial-1.17-1.0-SNAPSHOT.jar 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/flink/flink-1.17.0/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2023-06-19 14:31:05,693 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-atguigu.
2023-06-19 14:31:05,693 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-atguigu.
2023-06-19 14:31:06,142 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/opt/module/flink/flink-1.17.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2023-06-19 14:31:06,632 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at node002/192.168.10.102:8032
2023-06-19 14:31:07,195 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar

[atguigu@node001 flink-1.17.0]$ bin/flink run-application -t yarn-application -c com.atguigu.wc.WordCountStreamUnboundedDemo ./FlinkTutorial-1.17-1.0-SNAPSHOT.jar
SLF4J: Class path contains multiple SLF4J bindings.

[atguigu@node001 flink-1.17.0]$ bin/flink run-application -t yarn-application -Dyarn.provided.lib.dirs="hdfs://node001:8020/flink-dist" -c com.atguigu.wc.WordCountStreamUnboundedDemo hdfs://node001:8020/flink-jars/FlinkTutorial-1.17-1.0-SNAPSHOT.jar

P022【022_Flink部署_歷史服務(wù)器】08:11

3.6 K8S 運行模式（了解）

容器化部署是如今業(yè)界流行的一項技術(shù)，基于Docker鏡像運行能夠讓用戶更加方便地對應(yīng)用進(jìn)行管理和運維。容器管理工具中最為流行的就是Kubernetes（k8s），而Flink也在最近的版本中支持了k8s部署模式。基本原理與YARN是類似的，具體配置可以參見官網(wǎng)說明，這里我們就不做過多講解了。

3.7 歷史服務(wù)器

運行 Flink job 的集群一旦停止，只能去 yarn 或本地磁盤上查看日志，不再可以查看作業(yè)掛掉之前的運行的 Web UI，很難清楚知道作業(yè)在掛的那一刻到底發(fā)生了什么。如果我們還沒有 Metrics 監(jiān)控的話，那么完全就只能通過日志去分析和定位問題了，所以如果能還原之前的 Web UI，我們可以通過 UI 發(fā)現(xiàn)和定位一些問題。

Flink提供了歷史服務(wù)器，用來在相應(yīng)的 Flink 集群關(guān)閉后查詢已完成作業(yè)的統(tǒng)計信息。我們都知道只有當(dāng)作業(yè)處于運行中的狀態(tài)，才能夠查看到相關(guān)的WebUI統(tǒng)計信息。通過 History Server 我們才能查詢這些已完成作業(yè)的統(tǒng)計信息，無論是正常退出還是異常退出。

此外，它對外提供了 REST API，它接受 HTTP 請求并使用 JSON 數(shù)據(jù)進(jìn)行響應(yīng)。Flink 任務(wù)停止后，JobManager 會將已經(jīng)完成任務(wù)的統(tǒng)計信息進(jìn)行存檔，History Server 進(jìn)程則在任務(wù)停止后可以對任務(wù)統(tǒng)計信息進(jìn)行查詢。比如：最后一次的 Checkpoint、任務(wù)運行時的相關(guān)配置。

[atguigu@node001 flink-1.17.0]$ bin/historyserver.sh start
Starting historyserver daemon on host node001.
[atguigu@node001 flink-1.17.0]$ bin/flink run -t yarn-per-job -d -c com.atguigu.wc.WordCountStreamUnboundedDemo ../jar/FlinkTutorial-1.17-1.0-SNAPSHOT.jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/flink/flink-1.17.0/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

文章來源地址http://www.zghlxwxcb.cn/news/detail-501211.html

到了這里，關(guān)于尚硅谷大數(shù)據(jù)Flink1.17實戰(zhàn)教程-筆記02【Flink部署】的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！