Sparkthrift Server 啟動(dòng)命令調(diào)優(yōu)及問題報(bào)錯(cuò)解決

這篇具有很好參考價(jià)值的文章主要介紹了Sparkthrift Server 啟動(dòng)命令調(diào)優(yōu)及問題報(bào)錯(cuò)解決。希望對大家有所幫助。如果存在錯(cuò)誤或未考慮完全的地方，請大家不吝賜教，您也可以點(diǎn)擊"舉報(bào)違法"按鈕提交疑問。

1、sparkthrift Server 啟動(dòng)命令

因?yàn)槭菧y試命令，所以你需要和正式服務(wù)進(jìn)行區(qū)別，不改變節(jié)點(diǎn)的情況下需要改變服務(wù)名稱和服務(wù)端口。

sbin/start-thriftserver.sh \
--name spark_sql_thriftserver2 \
--master yarn --deploy-mode client \
--driver-cores 4 --driver-memory 8g \
--executor-cores 4 --executor-memory 6g \
--conf spark.scheduler.mode=FAIR \
--conf spark.hadoop.fs.hdfs.impl.disable.cache=true \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
--hiveconf hive.server2.thrift.bind.host=`hostname -i` \
--hiveconf hive.server2.thrift.port=10002

命令解析：

–name spark_sql_thriftserver2
指定服務(wù)名稱為 spark_sql_thriftserver2
效果如圖：
–master yarn --deploy-mode client
指定 spark Job 提交的運(yùn)行模式為 yarn-client。
提交至 yarn 的運(yùn)行模式有兩種：yarn-client和yarn-cluster；yarn-client 的模式 driver 運(yùn)行在本機(jī)，yarn-cluster 的模式 driver 提交給 yarn 進(jìn)行指定，運(yùn)行在 applicationMaster 所在節(jié)點(diǎn)。
因?yàn)?sparkthrift Server 是交互式的任務(wù)，需要固定節(jié)點(diǎn)和端口，所以只能使用 yarn-client 模式。
–driver-cores 4 --driver-memory 8g
指定主程序運(yùn)行的環(huán)境配置，這里需要考慮本機(jī)的資源進(jìn)行合理配置，因?yàn)?sparkthrift Server 是常駐任務(wù)，并且提供連接查詢數(shù)據(jù)，所以考慮給大一點(diǎn)，cpu給4個(gè)，內(nèi)存給8g。
–executor-cores 4 --executor-memory 6g
指定啟動(dòng)每個(gè) executor 需要分配的資源，同樣需要考慮yarn每個(gè)節(jié)點(diǎn)的剩余資源進(jìn)行合理分配；cpu給4個(gè)，內(nèi)存給6g。
–conf spark.scheduler.mode=FAIR
配置 spark 中 job 池的調(diào)度模式，該模式有兩種：FIFO和FAIR。
FIFO：每個(gè)job沒有優(yōu)先級，不做任務(wù)調(diào)度，順序執(zhí)行。
FAIR：判斷每個(gè)job的優(yōu)先權(quán)重后，優(yōu)先級高的，權(quán)重大的先調(diào)度執(zhí)行。
當(dāng)然每個(gè)池子都可以配置不同的模式（，權(quán)重，最小的資源使用量），參考如圖：
–conf spark.hadoop.fs.hdfs.impl.disable.cache=true
禁止該任務(wù)使用cache緩存。
–conf spark.serializer=org.apache.spark.serializer.KryoSerializer
指定序列化類為Kryo，這個(gè)參數(shù)非必要，我測試的時(shí)候發(fā)現(xiàn)默認(rèn)的序列化類就是這個(gè)。
–hiveconf hive.server2.thrift.bind.host=hostname -i
指定提供查詢的server2服務(wù)的地址：即本機(jī)ip即可。（hostname=>本機(jī)別稱，hostname -i=>本機(jī)ip）
–hiveconf hive.server2.thrift.port=10002
指定提供查詢的server2服務(wù)的端口為：10002。（注意查看一下該端口是否被其他服務(wù)占用，lsof -i=>查詢所有端口使用情況，lsof -i:端口=>查詢某個(gè)端口的使用情況）

2、實(shí)際生產(chǎn)過程中的報(bào)錯(cuò)解決

2.1、Kryo serialization failed: Buffer overflow. Available: 0, required: 2428400. To avoid this, increase spark.kryoserializer.buffer.max value

查詢一張300多萬條數(shù)據(jù)的表時(shí)（查表直接用的是 select * from tablename），緩沖一段時(shí)間后報(bào)錯(cuò)：

Caused by: org.apache.spark.SparkException: 
Kryo serialization failed: Buffer overflow. Available: 0, required: 2428400. To avoid this, increase spark.kryoserializer.buffer.max value. at 
	org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:350) at 
	org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:393) 
	...

報(bào)錯(cuò)信息提示：序列化的緩沖內(nèi)存不足，需要擴(kuò)大緩存內(nèi)存的容量。
這里應(yīng)該時(shí) driver 解析 task 發(fā)送過來的數(shù)據(jù)，task 任務(wù)會(huì)將數(shù)據(jù)序列化后進(jìn)行網(wǎng)絡(luò)傳輸，driver 接收到數(shù)據(jù)流后對其反序列化解析數(shù)據(jù)，才能將數(shù)據(jù)實(shí)際呈現(xiàn)，至于是寫buffer 還是讀buffer 的內(nèi)存不足這里不做深入。
直接根據(jù)其提示，配置相應(yīng)的參數(shù)：

--conf spark.kryoserializer.buffer=64m //這個(gè)指定序列化的默認(rèn)緩沖容量（這個(gè)配置可有可無，不影響）
--conf spark.kryoserializer.buffer.max=1024m //指定序列化的最大緩沖容量

2.2、java.lang.OutOfMemoryError: GC overhead limit exceeded

銜接2.1，解決了序列化緩沖的問題，查詢同一張表，緩沖了很長一段時(shí)間后，任務(wù)失敗，連同 sparkthrift 任務(wù)一同掛掉了。
查詢 sparkthrift 任務(wù)的執(zhí)行日志可見主要報(bào)錯(cuò)信息：

23/03/14 11:18:29 WARN TransportChannelHandler: Exception in connection from /192.168.1.120:60792
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOfRange(Arrays.java:3664)
        at java.lang.String.<init>(String.java:207)
        at java.lang.StringBuilder.toString(StringBuilder.java:407)
        at java.lang.Class.getDeclaredMethod(Class.java:2130)
        at java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1611)
        at java.io.ObjectStreamClass.access$2400(ObjectStreamClass.java:79)
        at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:531)
        at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:494)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391)
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2001)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1848)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2158)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1665)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2403)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2327)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2185)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1665)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:501)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:459)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:109)
        at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$deserialize$2(NettyRpcEnv.scala:299)
        at org.apache.spark.rpc.netty.NettyRpcEnv$$Lambda$835/115143278.apply(Unknown Source)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
        at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:352)
        at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$deserialize$1(NettyRpcEnv.scala:298)
        at org.apache.spark.rpc.netty.NettyRpcEnv$$Lambda$834/1059022105.apply(Unknown Source)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
        at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:298)

有報(bào)錯(cuò)信息可以得知是 GC 超限了。
復(fù)現(xiàn)問題的時(shí)候，任務(wù)執(zhí)行期間，用 top 命令查詢任務(wù)資源消耗情況。發(fā)現(xiàn) sparkthrift 任務(wù)內(nèi)存使用情況比較穩(wěn)定，穩(wěn)定維持在20%以下，但是cpu的使用率卻高達(dá)1200%以上。
同時(shí)可見報(bào)錯(cuò)信息主要來自于 java.io.ObjectInputStream 這個(gè)類，該類應(yīng)該是 driver 端對數(shù)據(jù)流進(jìn)行反序列化解析讀取數(shù)據(jù)時(shí)調(diào)用的。
個(gè)人推測應(yīng)該是 driver 端解析數(shù)據(jù)時(shí)，內(nèi)存使用率爆滿，但是并沒有多余閑置的內(nèi)存可以釋放，所以 cpu 反復(fù) GC 無果，最后報(bào)錯(cuò) GC 超限了。
所以根據(jù)推斷，我調(diào)整了擴(kuò)大了 driver 端內(nèi)存的容量：

--driver-cores 4 --driver-memory 12g //這個(gè)擴(kuò)大了driver的內(nèi)存大小，同時(shí)可以考慮調(diào)大cpu的數(shù)量，畢竟測試的時(shí)候cpu使用率都達(dá)到1200%以上了

2.3、Job aborted due to stage failure: Total size of serialized results of 7 tasks (1084.0 MiB) is bigger than spark.driver.maxResultSize (1024.0 MiB)

銜接2.2，GC超限問題解決后，查詢同一張表，秒出報(bào)錯(cuò)信息：

SQL 錯(cuò)誤: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 7 tasks (1084.0 MiB) is bigger than spark.driver.maxResultSize (1024.0 MiB)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:361)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:263)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:78)
	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:62)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:43)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:263)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:258)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:272)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 7 tasks (1084.0 MiB) is bigger than spark.driver.maxResultSize (1024.0 MiB)
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2258)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2207)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2206)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2206)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1079)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1079)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1079)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2445)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2387)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2376)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2196)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2217)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2236)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2261)
	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
	at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:390)
	at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3696)
	at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2965)
	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685)
	at org.apache.spark.sql.Dataset.collect(Dataset.scala:2965)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:334)
	... 16 more

由報(bào)錯(cuò)信息可知：某個(gè) task 返回給 driver 的結(jié)果集已經(jīng)超出了默認(rèn)的大小，那么直接調(diào)大結(jié)果集的最大容量即可。
一開始我擴(kuò)大了一倍到2g，但是后來2g又不夠了，再給3g，繼續(xù)不夠，得最終直接給到6g算了，反正資源充足。

--conf spark.driver.maxResultSize=6g //如果需要不限制結(jié)果集大小的話，直接該參數(shù) =0 即可。

但是該參數(shù)應(yīng)該并不適合無限制放大，其他博主的方案是要減小分區(qū)的數(shù)量（寫spark程序時(shí)需要進(jìn)行優(yōu)化考慮），以減小最后 driver 端的內(nèi)存壓力。

2.4、java.lang.OutOfMemoryError: Java heap space

當(dāng)我查詢更大的表時(shí)（數(shù)據(jù)量7000W行），最終還是支持不住，報(bào)錯(cuò)了：

SQL 錯(cuò)誤: org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.OutOfMemoryError: Java heap space
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:361)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:263)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:78)
	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:62)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:43)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:263)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:258)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:272)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.spark.sql.execution.SparkPlan$$anon$1.next(SparkPlan.scala:373)
	at org.apache.spark.sql.execution.SparkPlan$$anon$1.next(SparkPlan.scala:369)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at org.apache.spark.sql.execution.SparkPlan$$anon$1.foreach(SparkPlan.scala:369)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeCollect$1(SparkPlan.scala:391)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeCollect$1$adapted(SparkPlan.scala:390)
	at org.apache.spark.sql.execution.SparkPlan$$Lambda$3241/463491470.apply(Unknown Source)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:390)
	at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3696)
	at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2965)
	at org.apache.spark.sql.Dataset$$Lambda$2425/999790851.apply(Unknown Source)
	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
	at org.apache.spark.sql.Dataset$$Lambda$1881/1956032123.apply(Unknown Source)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
	at org.apache.spark.sql.execution.SQLExecution$$$Lambda$1889/493204045.apply(Unknown Source)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
	at org.apache.spark.sql.execution.SQLExecution$$$Lambda$1882/1152849040.apply(Unknown Source)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685)
	at org.apache.spark.sql.Dataset.collect(Dataset.scala:2965)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:334)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:263)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3$$Lambda$2021/540239691.apply$mcV$sp(Unknown Source)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:78)
	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:62)

這里總結(jié)一下，Spark 常見的兩類 OOM 問題：Driver OOM 和 Executor OOM。
如果是 driver 端的 OOM，可以考慮減少分區(qū)的數(shù)量和擴(kuò)大 driver 端的內(nèi)存容量。
如果發(fā)生在 executor，可以通過增加分區(qū)數(shù)量，減少每個(gè) executor 負(fù)載。但是此時(shí)，會(huì)增加 driver 的負(fù)載。所以，可能同時(shí)需要增加 driver 內(nèi)存。定位問題時(shí)，一定要先判斷是哪里出現(xiàn)了 OOM ，對癥下藥，才能事半功倍。

3、問題留言

最后，我懷疑，是不是 spark 不適合一次性返回這么大的數(shù)據(jù)量（select * from tablename 這種方式），畢竟這么大數(shù)據(jù)量都是要進(jìn)行網(wǎng)絡(luò)傳輸?shù)模瑹o論如何 driver 端的壓力都會(huì)是巨大的。如果有大神知道，望留言解答，蟹蟹各位！文章來源地址http://www.zghlxwxcb.cn/news/detail-729326.html

到了這里，關(guān)于Sparkthrift Server 啟動(dòng)命令調(diào)優(yōu)及問題報(bào)錯(cuò)解決的文章就介紹完了。如果您還想了解更多內(nèi)容，請?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！