CDH5.15.0+spark1.6.0+hive1.1集群与zeppelin0.8.1+spark-notebook打通踩坑总结
2019独角兽企业重金招聘Python工程师标准>>>
二进制all包多为spark2 scala2.11的所以源码编译本地相关版本兼容的包的及其它hadoop hive yarn 版本,源码git下载编译排错见前边文章,下为编译合适版本后的安装过程:
1.zeppelin081/conf/zeppelin-env.sh:
export MASTER=local[2]
#yarn-client
#export SCALA_HOME=/usr/share/scala
export SCALA_HOME=/opt/soft/scala-2.10.5
export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
#export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
if [ -n "$HADOOP_HOME" ]; thenexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
fi#export SPARK_CONF_DIR=/etc/spark2/conf
export SPARK_CONF_DIR=/etc/spark/conf
export HIVE_CONF_DIR=/etc/hive/conf
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-$SPARK_CONF_DIR/yarn-conf}
HIVE_CONF_DIR=${HIVE_CONF_DIR:-/etc/hive/conf}
if [ -d "$HIVE_CONF_DIR" ]; thenHADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HIVE_CONF_DIR"
fi
export HADOOP_CONF_DIRexport ZEPPELIN_INTP_CLASSPATH_OVERRIDES=/etc/hive/conf#export ZEPPELIN_INTP_CLASSPATH_OVERRIDES=:/etc/hive/conf:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/parcels/CDH/lib/spark/lib/spark-assembly-1.6.0-cdh5.15.0-hadoop2.6.0-cdh5.15.0.jar:/opt/cloudera/parcels/CDH/jars/*:/opt/cloudera/parcels/CDH/lib/hive/lib/*:/opt/soft/zeppelin081/interpreter/spark/spark-interpreter-0.8.1.jar
2.ln -s /etc/hive/conf/hive-site.xml conf/
3.修改conf/zeppelin-site.xml 的启动端口号
4.bin/zeppelin-daemon.sh restart 启动 ,自动生成相关log run 和webapp目录
5.看日志报错:
vi logs/zeppelin-root-master.log:
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common
/collect/Queues
Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Queues
解决:替换相关guava包 对应CDH lib目录相关版本
cp /opt/cloudera/parcels/CDH/lib/hive/lib/guava-14.0.1.jar lib/
还报错要guava-21
vi logs/zeppelin-root-master.out:
MultiException[java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/Ve
rsioned, java.lang.NoClassDefFoundError: org/glassfish/jersey/jackson/intern
al/jackson/jaxrs/json/JacksonJaxbJsonProvider]
解决:替换相关jackson 包对应CDH lib目录相关版本
ls lib/|grep jackson
google-http-client-jackson-1.23.0.jar
google-http-client-jackson2-1.23.0.jar
jackson-annotations-2.8.0.jar.bak
jackson-core-2.8.10.jar.bak
jackson-core-asl-1.9.13.jar
jackson-databind-2.8.11.1.jar.bak
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations-2.8.10.jar.bak
jackson-xc-1.8.8.jar
jersey-media-json-jackson-2.27.jar
[root@master zeppelin081]# cp /opt/cloudera/parcels/CDH/jars/jackson-annotations-2.1.0.jar lib/
[root@master zeppelin081]# cp /opt/cloudera/parcels/CDH/jars/jackson-core-2.1.0.jar lib/
[root@master zeppelin081]# cp /opt/cloudera/parcels/CDH/jars/jackson-databind-2.1.0.jar lib/
[root@master zeppelin081]# cp /opt/cloudera/parcels/CDH/jars/jackson-module-jaxb-annotations-2.1.0.jar lib/
试多版本都不行后查要scala版:
cp /opt/cloudera/parcels/CDH/jars/jackson*2.2.3*.jar lib/
[root@master zeppelin081]# ls lib/jackson-
jackson-annotations-2.1.0.jar.bak
jackson-annotations-2.2.2.jar.bak
jackson-annotations-2.2.3.jar
jackson-annotations-2.3.1.jar.bak
jackson-annotations-2.8.0.jar.bak
jackson-core-2.1.0.jar.bak
jackson-core-2.2.2.jar.bak
jackson-core-2.2.3.jar
jackson-core-2.8.10.jar.bak
jackson-core-asl-1.9.13.jar
jackson-databind-2.1.0.jar.bak
jackson-databind-2.2.2.jar.bak
jackson-databind-2.2.3.jar
jackson-databind-2.8.11.1.jar.bak
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations-2.1.0.jar.bak
jackson-module-jaxb-annotations-2.8.10.jar.bak
jackson-module-scala_2.10-2.2.3.jar
jackson-xc-1.8.8.jar
终于搞定!!!
======然后开始打通测试各插件=======
spark interpreter:
master yarn-client
Dependencies
artifact | exclude |
---|---|
/usr/share/java/mysql-connector-java.jar | |
/opt/cloudera/parcels/CDH/lib/spark/lib/spark-assembly-1.6.0-cdh5.15.0-hadoop2.6.0-cdh5.15.0.jar |
sc.getConf.toDebugString.split("\n").foreach(println)
sqlContext.sql("show tables").show
%sql
select area,count(cid) from default.dimcity
group by area
presto interpreter:(new jdbc)
default.driver com.facebook.presto.jdbc.PrestoDriver
default.url jdbc:presto://master:19000/hive/
default.user root
Dependencies
artifact | exclude |
---|---|
com.facebook.presto:presto-jdbc:0.100 |
%presto
-- SHOW SCHEMAS
select area,count(cid) from default.dimcity
group by area
phoenix interpreter:(new jdbc)
default.driver org.apache.phoenix.jdbc.PhoenixDriver
default.url jdbc:phoenix:master:2181:/hbase
default.user hdfs
Dependencies
artifact | exclude |
---|---|
org.apache.phoenix:phoenix-core:4.7.0-HBase-1.1 | |
org.apache.phoenix:phoenix-server-client:4.7.0-HBase-1.1 |
%phoenix
-- !tables -- 不支持
-- SHOW SCHEMAS -- 不支持
select * from SYSTEM.CATALOG --dim_channels --tc_district //HBASE 表名 必须大写才支持
hbase interpreter:
hbase.home | /opt/cloudera/parcels/CDH/lib/hbase |
hbase.ruby.sources | lib/ruby |
zeppelin.hbase.test.mode | false |
Dependencies (因zepplin自动编译为hbase1.0,要不指定版本重编,要不加载下边包覆盖)
artifact | exclude |
---|---|
/opt/cloudera/parcels/CDH/lib/hbase/lib/hbase-client-1.2.0-cdh5.15.0.jar | |
/opt/cloudera/parcels/CDH/lib/hbase/lib/hbase-common-1.2.0-cdh5.15.0.jar | |
/opt/cloudera/parcels/CDH/lib/hbase/lib/hbase-protocol-1.2.0-cdh5.15.0.jar |
%hbase
desc 'car_brand'
list
elasticsearch interpreter: 参http://cwiki.apachecn.org/pages/viewpage.action?pageId=10030782 默认transport 9300
elasticsearch.client.type http
elasticsearch.cluster.name tuanchees
elasticsearch.host 172.16.60.182
elasticsearch.port 9200
%elasticsearch
search /
file interpreter:
hdfs.maxlength 1000
hdfs.url http://master:50070/webhdfs/v1/
hdfs.user
Dependencies
artifact exclude
/opt/cloudera/parcels/CDH/jars/jersey-client-1.9.jar
/opt/cloudera/parcels/CDH/jars/jersey-core-1.9.jar
/opt/cloudera/parcels/CDH/jars/jersey-guice-1.9.jar
/opt/cloudera/parcels/CDH/jars/jersey-server-1.9.jar
/opt/cloudera/parcels/CDH/jars/jersey-json-1.9.jar
%file
ls /
flink interpreter:
host localhost
port 6123
%flink
val text = benv.fromElements("In the time of chimpanzees, I was a monkey", // some lines of text to analyze
"Butane in my veins and I'm out to cut the junkie",
"With the plastic eyeballs, spray paint the vegetables",
"Dog food stalls with the beefcake pantyhose",
"Kill the headlights and put it in neutral",
"Stock car flamin' with a loser in the cruise control",
"Baby's in Reno with the Vitamin D",
"Got a couple of couches, sleep on the love seat",
"Someone came in sayin' I'm insane to complain",
"About a shotgun wedding and a stain on my shirt",
"Don't believe everything that you breathe",
"You get a parking violation and a maggot on your sleeve",
"So shave your face with some mace in the dark",
"Savin' all your food stamps and burnin' down the trailer park",
"Yo, cut it")
val counts = text.flatMap{ _.toLowerCase.split("\\W+") }.map { (_,1) }.groupBy(0).sum(1)counts.collect().foreach(println(_))// // Streaming Example
// case class WordWithCount(word: String, count: Long)// val text = env.socketTextStream(host, port, '\n')// val windowCounts = text.flatMap { w => w.split("\\s") }
// .map { w => WordWithCount(w, 1) }
// .keyBy("word")
// .timeWindow(Time.seconds(5))
// .sum("count")// windowCounts.print()
// // Batch Example
// case class WordWithCount(word: String, count: Long)// val text = env.readTextFile(path)// val counts = text.flatMap { w => w.split("\\s") }
// .map { w => WordWithCount(w, 1) }
// .groupBy("word")
// .sum("count")// counts.writeAsCsv(outputPath)
=========spark-notebook============
spark-notebook相对简单下载解压Scala [2.10.5] Spark [1.6.0] Hadoop [2.6.0] {Hive ✓} {Parquet ✓}
一样连接 hive-site.xml: ln -s /etc/hive/conf/hive-site.xml conf/
改端口:vi conf/application.ini
然后可以完全不动直接启后在改配置,但方便重启,写了个脚本
bin/start.sh
#!/bin/bash
export MASTER=local[2]
#yarn-client
#export SCALA_HOME=/usr/share/scala
export SCALA_HOME=/opt/soft/scala-2.10.5
export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
if [ -n "$HADOOP_HOME" ]; thenexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
fiexport SPARK_CONF_DIR=/etc/spark/conf
export HIVE_CONF_DIR=/etc/hive/conf
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-$SPARK_CONF_DIR/yarn-conf}
HIVE_CONF_DIR=${HIVE_CONF_DIR:-/etc/hive/conf}
if [ -d "$HIVE_CONF_DIR" ]; thenHADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HIVE_CONF_DIR"
fi
export HADOOP_CONF_DIRworkdir=/opt/soft/spark-notebook
kill -9 `cat ${workdir}/RUNNING_PID`
rm -rf ${workdir}/derby.log ${workdir}/metastore_db ${workdir}/RUNNING_PID
${workdir}/bin/spark-notebook > snb.log 2>&1 &
开始一直连不上HIVE ,后来配置notebook metadata如下:(notebook metadata的配置参考http://master151:9002/assets/docs/clusters_clouds.html)
{"name": "test","user_save_timestamp": "1970-01-01T08:00:00.000Z","auto_save_timestamp": "1970-01-01T08:00:00.000Z","language_info": {"name": "scala","file_extension": "scala","codemirror_mode": "text/x-scala"},"trusted": true,"customLocalRepo": null,"customRepos": null,"customDeps": null,"customImports": ["import scala.util._","import org.apache.spark.SparkContext._"],"customArgs": null,"customSparkConf": {"spark.master": "local[2]","hive.metastore.warehouse.dir": "/user/hive/warehouse","hive.metastore.uris": "thrift://master:9083","spark.sql.hive.metastore.version": "1.1.0","spark.sql.hive.metastore.jars": "/opt/cloudera/parcels/CDH/lib/hadoop/../hive/lib/*","hive.metastore.schema.verification": "false","spark.jars": "/usr/share/java/mysql-connector-java.jar,/opt/cloudera/parcels/CDH/lib/spark/lib/spark-assembly-1.6.0-cdh5.15.0-hadoop2.6.0-cdh5.15.0.jar","spark.driver.extraClassPath": "/etc/spark/conf:/etc/spark/conf/yarn-conf:/etc/hadoop/conf:/etc/hive/conf:/opt/cloudera/parcels/CDH/lib/hadoop/*:/opt/cloudera/parcels/CDH/lib/hadoop/../hadoop-hdfs/*:/opt/cloudera/parcels/CDH/lib/hadoop/../hadoop-mapreduce/*:/opt/cloudera/parcels/CDH/lib/hadoop/../hadoop-yarn/*:/opt/cloudera/parcels/CDH/lib/hadoop/../hive/lib/*:/opt/cloudera/parcels/CDH/jars/*:/opt/soft/spark-notebook/lib/*","spark.executor.extraClassPath": "/etc/spark/conf:/etc/spark/conf/yarn-conf:/etc/hadoop/conf:/etc/hive/conf:/opt/cloudera/parcels/CDH/lib/hadoop/*:/opt/cloudera/parcels/CDH/lib/hadoop/../hadoop-hdfs/*:/opt/cloudera/parcels/CDH/lib/hadoop/../hadoop-mapreduce/*:/opt/cloudera/parcels/CDH/lib/hadoop/../hadoop-yarn/*:/opt/cloudera/parcels/CDH/lib/hadoop/../hive/lib/*:/opt/cloudera/parcels/CDH/jars/*:/opt/soft/spark-notebook/lib/*"},"kernelspec": {"name": "spark","display_name": "Scala [2.10.5] Spark [1.6.0] Hadoop [2.6.0] {Hive ✓} {Parquet ✓}"}
}
报错:
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/AlreadyExistsException
经查原来spark-notebook编绎时自动使用HIVE1.2 metastore,cdh使用的1.1,版本不兼容
转载于:https://my.oschina.net/hblt147/blog/3015713
CDH5.15.0+spark1.6.0+hive1.1集群与zeppelin0.8.1+spark-notebook打通踩坑总结相关推荐
- Hadoop2.2.0 HA高可用分布式集群搭建(hbase,hive,sqoop,spark)
1 需要软件 Hadoop-2.2.0 Hbase-0.96.2(这里就用这个版本,跟Hadoop-2.2.0是配套的,不用覆盖jar包什么的) Hive-0.13.1 Zookeepr-3.4.6( ...
- 生产环境elasticsearch5.0.1和6.3.2集群的部署配置详解
线上环境elasticsearch5.0.1集群的配置部署 es集群的规划: 硬件: 7台8核.64G内存.2T ssd硬盘加1台8核16G的阿里云服务器 其中一台作为kibana+kafka连接查询 ...
- redis 4.0.8 源码包安装集群
系统:centos 6.9 软件版本:redis-4.0.8,rubygems-2.7.7,gcc version 4.4.7 20120313,openssl-1.1.0h,zlib-1.2.11 ...
- mysql mgr简介_mysql8.0初探:(二)MySQL Group Replication-MGR集群简介
mysql8.0初探:(二)MySQL Group Replication-MGR集群简介 发布时间:2020-06-12 23:59:17 来源:51CTO 阅读:49832 作者:arthur37 ...
- CentOS 7上搭建Spark3.0.1+ Hadoop3.2.1分布式集群
CentOS 7上搭建Spark3.0.1+ Hadoop3.2.1分布式集群 VMWare 安装CentOS 7 使用Xshell连接虚拟机 集群设置 安装JDK 1.8 SSH 免密登陆 安装ha ...
- 【VMware vSAN 7.0】5.5 配置 vSAN 集群的许可证设置
[VMware vSAN 7.0]5.5 配置 vSAN 集群的许可证设置-我们有软硬件解决方案 IT干货 2021-03-31 16:36:53 213 收藏 1 分类专栏: 1.服务器虚拟化集群方 ...
- CentOS7.0下Hadoop2.7.3的集群搭建
集群服务器规划 使用3台CentOS-6.8虚拟机进行集群搭建 服务 ip 主机名称 用户 HDFS YARN hadoop1 192.168.1.40 hadoop1 root NameNode, ...
- Ubuntu环境下用docker从0到1部署Elasticsearch 7集群
目录 一.镜像准备 二.配置准备 1.ubuntu安装ssh 2. 开放22号端口 3. 配置密码可访问 4. 登录到ubuntu服务器 三.搭建集群准备 1. FileZilla工具网盘下载 2. ...
- [转]redis 5.0.5 5分钟搭建redis集群
环境:centos 7 1:下载并安装redis $ wget http://download.redis.io/releases/redis-5.0.5.tar.gz$ tar xzf ...
最新文章
- IT工作者,你们的爱情是这样的吗?
- 爱情这东西,真的有报应吗…
- 笔记-信息系统安全管理-信息系统安全技术体系
- 在SAP云平台测试帐号中启用Kyma运行环境
- 俱乐部通知[即日起启用微软Live Meeting]
- 数组 最大值,课堂测试
- es search delete
- 小米系统服务器,小米服务框架
- IOS日历控件JTCalendar
- JAVA退房计算价格怎么写,JAVA宾馆客房管理系统报告.doc
- EtherCAT总线运动控制器中简单易用的直线插补
- 弹出登录框 您未被授权查看该页 的解决办法
- 总结IDA调试app so库的三种方法
- 使用数字芯片将5V转3.3V方案电路图
- Python爬取某站上海租房图片!
- android scrcpy 源代码分析,Scrcpy投屏原理浅析-设备控制篇
- Stata 新命令 readWind:快速读入并转换万德数据
- C语言:fscanf函数与fprintf函数——格式化读写函数
- 【MySQL】SQL执行顺序
- 牛奶可乐经济学之Q3:为什么在高档餐厅,侍者的工资比助理厨师要高?
热门文章
- newman执行测试_Postman+Newman执行自动化测试
- SSM整合-异常处理器及项目异常处理方案
- 20220528【聊聊假芯片】贪便宜往往吃大亏,盘点下那些假的内存卡和固态硬盘
- 想哪写哪_随笔20191203
- 黄世仁之监听程序前后台切换
- #单调队列#洛谷 2698 [USACO12MAR]花盆Flowerpot
- Tomcat异常,tomcat.util.http.fileupload.FileUploadBase$SizeLimitExceededException
- IOS系统通话录音功能的实现方案
- golang框架gin的日志处理和zap lumberjack日志使用
- 清华李建忠设计模式课程感想