flink sql client读取hive时卡住

问题复现如下：

查看$FLINK_HOME/log/flink-appleyuchi-sql-client-Desktop.log

2020-12-23 11:48:56,811 INFO org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn [] - Opening socket connection to server Laptop/192.168.0.103:2182. Will not attempt to authenticate using SASL (unknown error)
2020-12-23 11:48:56,814 INFO org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn [] - Socket error occurred: Laptop/192.168.0.103:2182: Connection refused
2020-12-23 11:48:56,915 INFO org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn [] - Opening socket connection to server Desktop/192.168.0.102:2181. Will not attempt to authenticate using SASL (unknown error)
2020-12-23 11:48:56,915 INFO org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn [] - Socket error occurred: Desktop/192.168.0.102:2181: Connection refused
2020-12-23 11:48:57,016 INFO org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn [] - Opening socket connection to server Laptop/192.168.0.103:2183. Will not attempt to authenticate using SASL (unknown error)
2020-12-23 11:48:57,019 INFO org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn [] - Socket error occurred: Laptop/192.168.0.103:2183: Connection refused

解决方案：

启动zookeeper集群

########################################################################################################################################################################

然后重启Flink SQL Client发现读取hive时再次卡住,

再次查看$FLINK_HOME/log/flink-appleyuchi-sql-client-Desktop.log

2020-12-23 11:01:07,786 WARN  org.apache.flink.table.client.cli.CliClient                  [] - Could not execute SQL statement.
org.apache.flink.table.client.gateway.SqlExecutionException: Error while submitting job.at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeQueryInternal$7(LocalExecutor.java:554) ~[flink-sql-client_2.12-1.12.0.jar:1.12.0]at org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:257) ~[flink-sql-client_2.12-1.12.0.jar:1.12.0]at org.apache.flink.table.client.gateway.local.LocalExecutor.executeQueryInternal(LocalExecutor.java:549) ~[flink-sql-client_2.12-1.12.0.jar:1.12.0]at org.apache.flink.table.client.gateway.local.LocalExecutor.executeQuery(LocalExecutor.java:365) ~[flink-sql-client_2.12-1.12.0.jar:1.12.0]at org.apache.flink.table.client.cli.CliClient.callSelect(CliClient.java:634) ~[flink-sql-client_2.12-1.12.0.jar:1.12.0]at org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:324) ~[flink-sql-client_2.12-1.12.0.jar:1.12.0]at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_131]at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:216) [flink-sql-client_2.12-1.12.0.jar:1.12.0]at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:141) [flink-sql-client_2.12-1.12.0.jar:1.12.0]at org.apache.flink.table.client.SqlClient.start(SqlClient.java:114) [flink-sql-client_2.12-1.12.0.jar:1.12.0]at org.apache.flink.table.client.SqlClient.main(SqlClient.java:196) [flink-sql-client_2.12-1.12.0.jar:1.12.0]
Caused by: java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_131]at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_131]at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeQueryInternal$7(LocalExecutor.java:552) ~[flink-sql-client_2.12-1.12.0.jar:1.12.0]... 10 more
Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:366) ~[flink-dist_2.12-1.12.0.jar:1.12.0]at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) ~[?:1.8.0_131]at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) ~[?:1.8.0_131]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_131]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_131]at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:361) ~[flink-dist_2.12-1.12.0.jar:1.12.0]at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_131]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_131]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_131]at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1168) ~[flink-dist_2.12-1.12.0.jar:1.12.0]at org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211) ~[flink-dist_2.12-1.12.0.jar:1.12.0]at org.apache.flink.runtime.concurrent.FutureUtils.lambda$orTimeout$15(FutureUtils.java:549) ~[flink-dist_2.12-1.12.0.jar:1.12.0]at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_131]at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_131]at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_131]at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_131]at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]
Caused by: java.util.concurrent.TimeoutExceptionat org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1168) ~[flink-dist_2.12-1.12.0.jar:1.12.0]at org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211) ~[flink-dist_2.12-1.12.0.jar:1.12.0]at org.apache.flink.runtime.concurrent.FutureUtils.lambda$orTimeout$15(FutureUtils.java:549) ~[flink-dist_2.12-1.12.0.jar:1.12.0]at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_131]at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_131]at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_131]at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_131]at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]

解决方案:

看得我一脸蒙比，机缘巧合下手贱

启动了Flink集群(HA)

问题解决了．

##########################################################################################################################################################

综上，Flink操作hive卡住问题有两种情况

①Flink这边的问题，

启动Zookeeper集群

启动Flink集群(HA)

②Hive本身操作表格和schema会卡住

Reference:

[1]Hive drop table时候卡死-各种情况的解决方案(持续更新中)

flink sql client读取hive时卡住相关推荐

【阿里云EMR实战篇】以EMR测试集群版本为例，详解 Flink SQL Client 集成 Hive 使用步骤
简介: 以测试集群版本为例(EMR-4.4.1)-- Flink SQL Client 集成 Hive 使用文档作者:林志成,阿里云EMR产品团队技术支持,拥有多年开源大数据经验 1.以测试集群版本 ...
Flink SQL Client方言切换与datagen-＞Hive(DDL形式+streaming形式)
概述本文是对[1]的完整复现,补充了[1]中缺失的大量细节. 切换方言切换目标 Flink SQL Client命令切换为hive SET table.sql-dialect=hive; 切换为 ...
flink sql client讀取kafka數據的timestamp(DDL方式)
实验目的 Kafka的数据能让Flink SQL Client读取到本文是对[1]的详细记载具体操作步骤 ①啓動hadoop集羣,離開安全模式 ②各个节点都关闭防火墙: service firew ...
Flink实战（八十）：flink-sql使用（七）Flink SQL Clien读取Kafka数据流式写入Hive（用hive 管理kafka元数据）
声明:本系列博客是根据SGG的视频整理而成,非常适合大家入门学习. <2021年最新版大数据面试题全面开启更新> 版本说明: Flink 1.11.2 Kafka 2.4.0 Hive 3 ...
Flink SQL Client读Kafka+流计算(DDL方式+代碼嵌入DDL/SQL方式)
#################################################################################################### ...
Flink SQL Client CLI 使用入门
参照:https://blog.csdn.net/boling_cavalry/article/details/105964425 一.SQL Client Flink 的 Table & S ...
[Spark][Hive][Python][SQL]Spark 读取Hive表的小例子
[Spark][Hive][Python][SQL]Spark 读取Hive表的小例子 $ cat customers.txt 1 Ali us 2 Bsb ca 3 Carls mx $ hive ...
Flink SQL Client注册SCALA UDF完整流程
UDF的完整maven工程與SQL https://github.com/appleyuchi/Flink_SQL_Client_UDF 完整操作步骤 ①mvn scala:compile packa ...
Flink SQL Client注册JAVA UDF完整流程
概述听大佬说[1]里面有flink sql client注册udf的方法去看了一眼,全是文字,闹心,索性琢磨了一下,记录下来． UDF的完整maven工程 https://github.com/a ...

flink sql client读取hive时卡住

flink sql client读取hive时卡住相关推荐

最新文章

热门文章