Spark _26_Spark On Hive的配置

网上的配置大多如下：

Spark On Hive的配置

在Spark客户端配置Hive On Spark

在Spark客户端安装包下spark-1.6.0/conf中创建文件hive-site.xml：

配置hive的metastore路径

<configuration>

<property>

<name>hive.metastore.uris</name>

<value>thrift://node1:9083</value>

</property>

</configuration>

启动Hive的metastore服务

hive --service metastore

启动zookeeper集群，启动HDFS集群。
启动SparkShell 读取Hive中的表总数，对比hive中查询同一表查询总数测试时间。

./spark-shell

--master spark://node1:7077,node2:7077

--executor-cores 1

--executor-memory 1g

--total-executor-cores 1

import org.apache.spark.sql.hive.HiveContext

val hc = new HiveContext(sc)

hc.sql("show databases").show

hc.sql("user default").show

hc.sql("select count(*) from jizhan").show

然后，我是没搞定：

不过另有他法：

配置之后，直接连接spark-sql

[root@henu1 bin]# ./spark-sql
2019-10-28 21:15:12 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library f
or your platform... using builtin-java classes where applicable2019-10-28 21:15:14 INFO  metastore:376 - Trying to connect to metastore with URI thri
ft://henu2:90832019-10-28 21:15:14 INFO  metastore:472 - Connected to metastore.
2019-10-28 21:15:20 INFO  SessionState:641 - Created local directory: /tmp/a85f1a6b-85
d3-4bf1-b366-14e375c76622_resources2019-10-28 21:15:20 INFO  SessionState:641 - Created HDFS directory: /tmp/hive/root/a8
5f1a6b-85d3-4bf1-b366-14e375c766222019-10-28 21:15:20 INFO  SessionState:641 - Created local directory: /tmp/root/a85f1a
6b-85d3-4bf1-b366-14e375c766222019-10-28 21:15:20 INFO  SessionState:641 - Created HDFS directory: /tmp/hive/root/a8
5f1a6b-85d3-4bf1-b366-14e375c76622/_tmp_space.db2019-10-28 21:15:20 INFO  SparkContext:54 - Running Spark version 2.3.1
2019-10-28 21:15:20 INFO  SparkContext:54 - Submitted application: SparkSQL::192.168.2
48.2412019-10-28 21:15:20 INFO  SecurityManager:54 - Changing view acls to: root
2019-10-28 21:15:20 INFO  SecurityManager:54 - Changing modify acls to: root
2019-10-28 21:15:20 INFO  SecurityManager:54 - Changing view acls groups to:
2019-10-28 21:15:20 INFO  SecurityManager:54 - Changing modify acls groups to:
2019-10-28 21:15:20 INFO  SecurityManager:54 - SecurityManager: authentication disable
d; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()2019-10-28 21:15:21 INFO  Utils:54 - Successfully started service 'sparkDriver' on por
t 33507.2019-10-28 21:15:21 INFO  SparkEnv:54 - Registering MapOutputTracker
2019-10-28 21:15:21 INFO  SparkEnv:54 - Registering BlockManagerMaster
2019-10-28 21:15:21 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.stora
ge.DefaultTopologyMapper for getting topology information2019-10-28 21:15:21 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint u
p2019-10-28 21:15:21 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockm
gr-d55cf82a-aff1-458b-a83a-e5d6a4210af92019-10-28 21:15:21 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2019-10-28 21:15:21 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2019-10-28 21:15:21 INFO  log:192 - Logging initialized @11266ms
2019-10-28 21:15:22 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
2019-10-28 21:15:22 INFO  Server:414 - Started @11505ms
2019-10-28 21:15:22 INFO  AbstractConnector:278 - Started ServerConnector@6a9b50cc{HTT
P/1.1,[http/1.1]}{0.0.0.0:4040}2019-10-28 21:15:22 INFO  Utils:54 - Successfully started service 'SparkUI' on port 40
40.2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7
be3a9ce{/jobs,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1
abfe081{/jobs/json,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2
a685eba{/jobs/job,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1
07f4980{/jobs/job/json,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7
5a118e6{/stages,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1
d540566{/stages/json,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6
014a9ba{/stages/stage,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4
a14c44f{/stages/stage/json,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@f
08fdce{/stages/pool,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6
bda1d19{/stages/pool/json,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2
8c86134{/storage,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4
492eede{/storage/json,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c
bc8d0f{/storage/rdd,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3
7b57b54{/storage/rdd/json,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5
c1f6d57{/environment,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@f
288c14{/environment/json,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6
794ac0b{/executors,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7
be71476{/executors/json,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5
cb5bb88{/executors/threadDump,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1
7b6d426{/executors/threadDump/json,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5
580d62f{/static,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c
f67838{/,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6
137cf6e{/api,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3
3a3c44a{/jobs/job/kill,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5
fcfca62{/stages/stage/kill,null,AVAILABLE,@Spark}2019-10-28 21:15:22 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http:/
/henu1:40402019-10-28 21:15:22 INFO  Executor:54 - Starting executor ID driver on host localhost
2019-10-28 21:15:22 INFO  Utils:54 - Successfully started service 'org.apache.spark.ne
twork.netty.NettyBlockTransferService' on port 35317.2019-10-28 21:15:22 INFO  NettyBlockTransferService:54 - Server created on henu1:35317
2019-10-28 21:15:22 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlock
ReplicationPolicy for block replication policy2019-10-28 21:15:22 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManage
rId(driver, henu1, 35317, None)2019-10-28 21:15:22 INFO  BlockManagerMasterEndpoint:54 - Registering block manager he
nu1:35317 with 413.9 MB RAM, BlockManagerId(driver, henu1, 35317, None)2019-10-28 21:15:22 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManager
Id(driver, henu1, 35317, None)2019-10-28 21:15:22 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(d
river, henu1, 35317, None)2019-10-28 21:15:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5
8f39564{/metrics/json,null,AVAILABLE,@Spark}2019-10-28 21:15:23 INFO  SharedState:54 - loading hive config file: file:/opt/spark/c
onf/hive-site.xml2019-10-28 21:15:23 INFO  SharedState:54 - Setting hive.metastore.warehouse.dir ('null
') to the value of spark.sql.warehouse.dir ('file:/opt/spark/bin/spark-warehouse').2019-10-28 21:15:23 INFO  SharedState:54 - Warehouse path is 'file:/opt/spark/bin/spar
k-warehouse'.2019-10-28 21:15:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7
d3fb0ef{/SQL,null,AVAILABLE,@Spark}2019-10-28 21:15:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7
dbe2ebf{/SQL/json,null,AVAILABLE,@Spark}2019-10-28 21:15:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4
fa9ab6{/SQL/execution,null,AVAILABLE,@Spark}2019-10-28 21:15:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2
d3ef181{/SQL/execution/json,null,AVAILABLE,@Spark}2019-10-28 21:15:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7
4e6094b{/static/sql,null,AVAILABLE,@Spark}2019-10-28 21:15:23 INFO  HiveUtils:54 - Initializing HiveMetastoreConnection version
1.2.1 using Spark classes.2019-10-28 21:15:23 INFO  HiveClientImpl:54 - Warehouse location for Hive client (vers
ion 1.2.2) is file:/opt/spark/bin/spark-warehouse2019-10-28 21:15:23 INFO  metastore:291 - Mestastore configuration hive.metastore.ware
house.dir changed from /user/hive/warehouse to file:/opt/spark/bin/spark-warehouse2019-10-28 21:15:23 INFO  metastore:376 - Trying to connect to metastore with URI thri
ft://henu2:90832019-10-28 21:15:23 INFO  metastore:472 - Connected to metastore.
2019-10-28 21:15:24 INFO  StateStoreCoordinatorRef:54 - Registered StateStoreCoordinat
or endpointspark-sql>

后续搞成了，再来补充！！！

Spark _26_Spark On Hive的配置相关推荐

spark基础之Spark SQL和Hive的集成以及ThriftServer配置
如果希望Maven编译Spark时支持Hive,需要给定-Phive -Phive-thriftserver.比如比如:mvn -Pyarn -Phadoop-2.6 -Dhadoop.version ...
Spark SQL整合Hive
Spark SQL官方释义 Spark SQL is Apache Spark's module for working with structured data. 一.使用Spark SQL访问Hi ...
Hive on Spark和Spark sql on Hive，你能分的清楚么
摘要:结构上Hive On Spark和SparkSQL都是一个翻译层,把一个SQL翻译成分布式可执行的Spark程序. 本文分享自华为云社区<Hive on Spark和Spark sql o ...
Spark SQL操作Hive表
Spark SQL支持从Hive存储中读写数据.然而,Hive存在很多的依赖,而这些依赖又不包含在默认的各类Spark发型版本中.如果将Hive的依赖放入classpath中,Spark将自动加载它们 ...
使用Spark SQL读取Hive上的数据
Spark SQL主要目的是使得用户可以在Spark上使用SQL,其数据源既可以是RDD,也可以是外部的数据源(比如Parquet.Hive.Json等).Spark SQL的其中一个分支就是Spar ...
Spark SQL 与 Hive 的第一场会师
"你好,一杯热美式,加 2 份shot, 1 份焦糖,谢谢" L 跨进汇智国际中心大厦的 Starbucks, 拿着 iPhone 对着点餐机轻轻一扫,对黑带服务员小妹抛出一个笑脸 ...
spark sql读写hive的过程
Spark sql读写hive需要hive相关的配置,所以一般将hive-site.xml文件放到spark的conf目录下.代码调用都是简单的,关键是源码分析过程,spark是如何与hive交互的. ...
大数据学习系列之七 ----- Hadoop+Spark+Zookeeper+HBase+Hive集群搭建图文详解
引言在之前的大数据学习系列中,搭建了Hadoop+Spark+HBase+Hive 环境以及一些测试.其实要说的话,我开始学习大数据的时候,搭建的就是集群,并不是单机模式和伪分布式.至于为什么先写单 ...
spark为什么比hive速度快？
spark是什么? spark是针对于大规模数据处理的统一分析引擎,通俗点说就是基于内存计算的框架 spark和hive的区别? spark的job输出结果可保存在内存中,而MapReduce的job ...

Spark _26_Spark On Hive的配置

Spark On Hive的配置

Spark _26_Spark On Hive的配置相关推荐

最新文章

热门文章