陽(yáng)陽(yáng)的一周，算是挺過來了，現(xiàn)在只剩感冒了，迷迷糊糊的干了一周，混口飯吃不容易呀！簡(jiǎn)單記錄一下遇到的問題吧！
連接hive(星環(huán))數(shù)據(jù)庫(kù)失敗
-
方案一 ： 海豚調(diào)度2.0.5使用的hive包是2.0版本,星環(huán)庫(kù)包裝的是hive 1.0版本，因此連接不上,將hive包降為1.0（
<hive.jdbc.version>1.1.0</hive.jdbc.version>），同時(shí)修改VALIDATION_QUERY為"select 1 from system.dual"即可
org.apache.dolphinscheduler.spi.utils.Constants
-
方案二(推薦) ： 刪除
hive-jdbcjar包，引入星環(huán)驅(qū)動(dòng)包inceptor-driver-4.8.3.jar（下了半天沒成功，如果連接星環(huán)庫(kù)肯定項(xiàng)目里面有這個(gè)包，直接拿過來吧)，連接過程中若是包某類或方法不存在，則為jar包沖突，需要繼續(xù)刪除hive相關(guān)包（目前遇到的只有hive1.0的service包沖突，2.0只刪除了jdbc包，其它沒報(bào)沖突），同樣也需要修改VALIDATION_QUERY為"select 1 from system.dual"，同上
或<dependency> <groupId>inceptor.hive</groupId> <artifactId>inceptor.driver</artifactId> <version>4.8.3</version> </dependency>
推薦理由：hive1.0 不支持存儲(chǔ)過程調(diào)用方法，會(huì)報(bào)錯(cuò)，詳情如下<dependency> <groupId>com.transwarp</groupId> <artifactId>inceptor-driver</artifactId> <version>4.8.3</version> </dependency>[ERROR] 2022-12-12 17:54:44.313 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.procedure.ProcedureTask:[123] - procedure task error java.sql.SQLException: Method not supported at org.apache.hive.jdbc.HiveConnection.prepareCall(HiveConnection.java:922) at com.zaxxer.hikari.pool.ProxyConnection.prepareCall(ProxyConnection.java:316) at com.zaxxer.hikari.pool.HikariProxyConnection.prepareCall(HikariProxyConnection.java) at org.apache.dolphinscheduler.plugin.task.procedure.ProcedureTask.handle(ProcedureTask.java:107) at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:191) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
執(zhí)行星環(huán)(hive) sql節(jié)點(diǎn)，工作流實(shí)例一直執(zhí)行中，任務(wù)實(shí)例則為提交狀態(tài)，一直不執(zhí)行
原因是common.properties 配置文件中的resource.storage.type=HDFS，配置了HDFS存儲(chǔ)，但是并未按照hadoop集群，master一直嘗試連接hdfs服務(wù)器，因此出現(xiàn)上述情況。將resource.storage.type=HDFS改為resource.storage.type=NONE，重啟服務(wù)即可
執(zhí)行星環(huán)(hive) 存儲(chǔ)過程節(jié)點(diǎn)，超過30秒變報(bào)超時(shí)錯(cuò)
-
錯(cuò)誤詳情
org.apache.dolphinscheduler.plugin.task.procedure.ProcedureTask [ERROR] 2022-12-22 20:49:10.275 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.procedure.ProcedureTask:[123] - procedure task error java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.hive.jdbc.HivePreparedStatement2.executeInternal(HivePreparedStatement2.java:158) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:419) at org.apache.hive.jdbc.HivePreparedStatement2.execute(HivePreparedStatement2.java:165) at org.apache.hive.jdbc.HiveCallableStatement.execute(HiveCallableStatement.java:41) at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) at com.zaxxer.hikari.pool.HikariProxyCallableStatement.execute(HikariProxyCallableStatement.java) at org.apache.dolphinscheduler.plugin.task.procedure.ProcedureTask.handle(ProcedureTask.java:116) at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:191) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376) at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453) at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435) at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hive.service.cli.thrift.TCLIService$Client.recv_ExecutePreCompiledStatement(TCLIService.java:763) at org.apache.hive.service.cli.thrift.TCLIService$Client.ExecutePreCompiledStatement(TCLIService.java:750) at org.apache.hive.jdbc.HivePreparedStatement2.executeInternal(HivePreparedStatement2.java:141) ... 14 common frames omitted -
解決辦法：修改
HiveDataSourceClient類中的hive連接超時(shí)時(shí)間
文章來源：http://www.zghlxwxcb.cn/news/detail-493087.html -
HiveDataSourceClient文章來源地址http://www.zghlxwxcb.cn/news/detail-493087.html
/* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package org.apache.dolphinscheduler.plugin.datasource.hive; import static org.apache.dolphinscheduler.spi.task.TaskConstants.HADOOP_SECURITY_AUTHENTICATION_STARTUP_STATE; import static org.apache.dolphinscheduler.spi.task.TaskConstants.JAVA_SECURITY_KRB5_CONF; import static org.apache.dolphinscheduler.spi.task.TaskConstants.JAVA_SECURITY_KRB5_CONF_PATH; import java.io.IOException; import java.lang.reflect.Field; import java.sql.Connection; import java.sql.SQLException; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; import org.apache.dolphinscheduler.plugin.datasource.api.client.CommonDataSourceClient; import org.apache.dolphinscheduler.plugin.datasource.api.provider.JdbcDataSourceProvider; import org.apache.dolphinscheduler.plugin.datasource.utils.CommonUtil; import org.apache.dolphinscheduler.spi.datasource.BaseConnectionParam; import org.apache.dolphinscheduler.spi.enums.DbType; import org.apache.dolphinscheduler.spi.utils.Constants; import org.apache.dolphinscheduler.spi.utils.PropertyUtils; import org.apache.dolphinscheduler.spi.utils.StringUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.security.UserGroupInformation; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import com.zaxxer.hikari.HikariDataSource; import sun.security.krb5.Config; public class HiveDataSourceClient extends CommonDataSourceClient { private static final Logger logger = LoggerFactory.getLogger(HiveDataSourceClient.class); private ScheduledExecutorService kerberosRenewalService; private Configuration hadoopConf; protected HikariDataSource oneSessionDataSource; private UserGroupInformation ugi; public HiveDataSourceClient(BaseConnectionParam baseConnectionParam, DbType dbType) { super(baseConnectionParam, dbType); } @Override protected void preInit() { logger.info("PreInit in {}", getClass().getName()); this.kerberosRenewalService = Executors.newSingleThreadScheduledExecutor(); } @Override protected void initClient(BaseConnectionParam baseConnectionParam, DbType dbType) { logger.info("Create Configuration for hive configuration."); this.hadoopConf = createHadoopConf(); logger.info("Create Configuration success."); logger.info("Create UserGroupInformation."); this.ugi = createUserGroupInformation(baseConnectionParam.getUser()); logger.info("Create ugi success."); super.initClient(baseConnectionParam, dbType); this.oneSessionDataSource = JdbcDataSourceProvider.createOneSessionJdbcDataSource(baseConnectionParam, dbType); logger.info("Init {} success.", getClass().getName()); } @Override protected void checkEnv(BaseConnectionParam baseConnectionParam) { super.checkEnv(baseConnectionParam); checkKerberosEnv(); } private void checkKerberosEnv() { String krb5File = PropertyUtils.getString(JAVA_SECURITY_KRB5_CONF_PATH); Boolean kerberosStartupState = PropertyUtils.getBoolean(HADOOP_SECURITY_AUTHENTICATION_STARTUP_STATE, false); if (kerberosStartupState && StringUtils.isNotBlank(krb5File)) { System.setProperty(JAVA_SECURITY_KRB5_CONF, krb5File); try { Config.refresh(); Class<?> kerberosName = Class.forName("org.apache.hadoop.security.authentication.util.KerberosName"); Field field = kerberosName.getDeclaredField("defaultRealm"); field.setAccessible(true); field.set(null, Config.getInstance().getDefaultRealm()); } catch (Exception e) { throw new RuntimeException("Update Kerberos environment failed.", e); } } } private UserGroupInformation createUserGroupInformation(String username) { String krb5File = PropertyUtils.getString(Constants.JAVA_SECURITY_KRB5_CONF_PATH); String keytab = PropertyUtils.getString(Constants.LOGIN_USER_KEY_TAB_PATH); String principal = PropertyUtils.getString(Constants.LOGIN_USER_KEY_TAB_USERNAME); try { UserGroupInformation ugi = CommonUtil.createUGI(getHadoopConf(), principal, keytab, krb5File, username); try { Field isKeytabField = ugi.getClass().getDeclaredField("isKeytab"); isKeytabField.setAccessible(true); isKeytabField.set(ugi, true); } catch (NoSuchFieldException | IllegalAccessException e) { logger.warn(e.getMessage()); } kerberosRenewalService.scheduleWithFixedDelay(() -> { try { ugi.checkTGTAndReloginFromKeytab(); } catch (IOException e) { logger.error("Check TGT and Renewal from Keytab error", e); } }, 5, 5, TimeUnit.MINUTES); return ugi; } catch (IOException e) { throw new RuntimeException("createUserGroupInformation fail. ", e); } } protected Configuration createHadoopConf() { Configuration hadoopConf = new Configuration(); hadoopConf.setBoolean("ipc.client.fallback-to-simple-auth-allowed", true); return hadoopConf; } protected Configuration getHadoopConf() { return this.hadoopConf; } @Override public Connection getConnection() { try { oneSessionDataSource.setConnectionTimeout(172800L);//設(shè)置連接超時(shí)時(shí)間 2天 //oneSessionDataSource.setIdleTimeout(60000L);// 非必須（空閑超時(shí)時(shí)間）,保持默認(rèn)值就行 //oneSessionDataSource.setMaxLifetime(600000L);// 非必須（最大生命周期），保持默認(rèn)值就行 return oneSessionDataSource.getConnection(); } catch (SQLException e) { logger.error("get oneSessionDataSource Connection fail SQLException: {}", e.getMessage(), e); return null; } } @Override public void close() { super.close(); logger.info("close HiveDataSourceClient."); kerberosRenewalService.shutdown(); this.ugi = null; this.oneSessionDataSource.close(); this.oneSessionDataSource = null; } }
http執(zhí)行報(bào)錯(cuò)：java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory
-
錯(cuò)誤詳情
[ERROR] 2022-12-26 11:37:12.532 org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread:[209] - task scheduler failure java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:912) at org.apache.dolphinscheduler.plugin.task.http.HttpTask.createHttpClient(HttpTask.java:333) at org.apache.dolphinscheduler.plugin.task.http.HttpTask.handle(HttpTask.java:98) at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:191) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) -
解決版本：刪除星環(huán)驅(qū)動(dòng)包中的http目錄
到了這里，關(guān)于20221225 海豚調(diào)度2.0.5 星環(huán)驅(qū)動(dòng)包踩坑（一）的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
