http://blog.csdn.net/pipisorry/article/details/52916307

路径错误

spark FileNotFoundError: [Errno 2] No such file or directory: '~/ENV/spark/./bin/spark-submit'

居然是因为这个引发的错误：SPARK_HOME = ~/ENV/spark应该改成SPARK_HOME = /home/pipi/ENV/spark

之前就不会有这个错误，不知道为啥？

spark.textFile读取用户主目录下的文件出错

org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/home/pipi/files/...SparkLDA/～/files

文件名改成绝对路径就不会出错了，而不能使用主目录标志~。

importerror no module named 'pyspark'

可能出错原因：

1 设置的路径应该在import语句之前

2 pyspark_python路径出错，且如果在pycharm中修改后要重启pycharm才能生效，否则还是原来的路径（可能是因为之前的路径也保存在pycharm解释器中了？）

PYSPARK_PYTHON = /home/pipi/opt/anaconda3/bin/python

Note: 并不用设置py4j的路径。

皮皮blog

系统环境错误

1 在~/.bashrc中设置的系统环境只能在terminal shell下运行spark程序才有效，因为.bashrc is only read for interactive shells.

如果要在当前用户整个系统中都有效（包括pycharm等等IDE），就应该将系统环境变量设置在~/.profile文件中。如果是设置所有用户整个系统，修改/etc/profile或者/etc/environment吧。

2 当然也可以在代码中设置，这样不管环境有没有问题了

# spark environment settings
import sys, os
os.environ['SPARK_HOME'] = conf.get(SECTION, 'SPARK_HOME')
sys.path.append(os.path.join(conf.get(SECTION, 'SPARK_HOME'), 'python'))
os.environ["PYSPARK_PYTHON"] = conf.get(SECTION, 'PYSPARK_PYTHON')
os.environ['SPARK_LOCAL_IP'] = conf.get(SECTION, 'SPARK_LOCAL_IP')
os.environ['JAVA_HOME'] = conf.get(SECTION, 'JAVA_HOME')
os.environ['PYTHONPATH'] = '$SPARK_HOME/python/lib/py4j-0.10.3-src.zip:$PYTHONPATH'

3 在$SPARK_HOME/conf/spark-env.sh中设置这些变量好像也只是在terminal中的shell环境中才有效

JAVA_HOME is not set

Exception: Java gateway process exited before sending the driver its port number

但是在命令行中是有的

pipi@pipicmp:~$ echo $JAVA_HOME
/home/pipi/ENV/jdk

解决方法1：在py代码中加入JAVA_HOME到os中

JAVA_HOME = /home/pipi/ENV/jdk

os.environ['JAVA_HOME'] = conf.get(SECTION, 'JAVA_HOME')

解决方法2：或者在hadoop中配置好JAVA_HOME

hadoop中配置JAVA_HOME

pika:~$sudo vim /usr/local/hadoop-2.6.4/libexec/hadoop-config.sh

在大概161行位置加上export JAVA_HOME=/opt/jdk1.8.0_73

KeyError: 'SPARK_HOME'

from pyspark.shell import sqlContext 语句出错

raise KeyError(key) from None KeyError: 'SPARK_HOME'

解决方法1：

代码中将SPARK_HOME加入到os.env中

os.environ['SPARK_HOME'] = conf.get(SECTION, 'SPARK_HOME')

解决方法2：

import shell前要将SPARK_HOME设置放在其前面

os.environ['SPARK_HOME'] = conf.get(SECTION, 'SPARK_HOME')

解决方法3：

只要将SPARK_HOME加入到~/.profile就可以了，这样os.env就可以自动读取到。

export envfilename=~/.bashrc
export envfilename2=~/.profile

(echo "export SPARK_HOME=$ENV/spark
#export PYSPARK_PYTHON=python3
#export PATH=\${SPARK_HOME}/bin:\$PATH") | sudo tee -a $envfilename $envfilename2
. $envfilename
. $envfilename2

pyspark different python version

Exception: Python in worker has different version 2.7 than that in driver 3.5, PySpark cannot run with different minor versions

解决：设置PYSPARK_PYTHON

export ENV=~/ENV
export PYSPARK_PYTHON=$ENV/ubuntu_env/bin/python
export envfilename=~/.profile
(echo "

export SPARK_HOME=$ENV/spark
export PYSPARK_PYTHON=$PYSPARK_PYTHON
#export PATH=\${SPARK_HOME}/bin:\$PATH
") | sudo tee -a $envfilename
. $envfilename

loopback address: 127.0.1.1

WARN Utils: Your hostname, pipicmp resolves to a loopback address: 127.0.1.1; using 192.168.199.210 instead (on interface enp2s0)

解决：这个只是个warn，可以不解决，如果不想有的话，可以这样自动设置

(echo "export SPARK_LOCAL_IP=$(ifconfig | grep 'inet addr:'| grep -v '127.0.0.1' | cut -d : -f 2 | awk '{ print $1 }')") >> $SPARK_HOME/conf/spark-env.sh

皮皮blog

运行错误

ValueError: Cannot run multiple SparkContexts at once

Welcome to
      ____              __
     / __/__ ___ _____/ /__
    _\ \/ _ \/ _ `/ __/ '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.0.1
      /_/

Using Python version 3.5.2 (default, Sep 10 2016 08:21:44)
SparkSession available as 'spark'.

ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=pyspark-shell, master=local[*]) created by <module> at <frozen importlib._bootstrap>:222

原因是：from pyspark.shell import sqlContext

引入的包中也定义了一个sc = spark.sparkContext导致和本代码中定义重复了。

py4j.java_gateway:Answer received: !yv

DEBUG:py4j.java_gateway:Command to send: m

未解决！！不过只是在logging中存在，debug信息，好像也没影响。

py4j.protocol.Py4JJavaError

py4j.protocol.Py4JJavaError: An error occurred while calling o49.trainLDAModel.

原因：corpus = data.zipWithIndex().map(lambda x: [x[-1], x[0:-1]]).cache()

lambda函数中的x下标在pyspark不能这样写？

解决：改成corpus = data.zipWithIndex().map(lambda x: [x[1], x[0]]).cache()就不报这个错误了，也是醉了。

py4j.protocol.Py4JError

LDA引入自from pyspark.mllib.clustering import LDA

py4j.protocol.Py4JError: An error occurred while calling o39.trainLDAModel. Trace:
py4j.Py4JException: Method trainLDAModel([class org.apache.spark.api.java.JavaRDD, class java.lang.Integer, class java.lang.Integer, class java.util.ArrayList, class java.lang.Double, class java.lang.Integer, class java.lang.Integer, class java.lang.String]) does not exist

原因：lda_model = LDA.train(corpus, maxIterations=max_iter, seed=seed, checkpointInterval=checkin_point_interval, k=K, optimizer=optimizer, docConcentration=[alpha], topicConcentration=beta)

解决：lda_model = LDA.train(corpus, maxIterations=max_iter, seed=seed, checkpointInterval=checkin_point_interval, k=K, optimizer=optimizer, docConcentration=alpha, topicConcentration=beta)

TypeError: Invalid param value given for param "docConcentration".

这个错误与上一个错误相关啊，呵呵了。。。

LDA引入自from pyspark.ml.clustering import LDA

TypeError: Invalid param value given for param "docConcentration".Could not convert 5.0 to list of floats.

原因：lda_model = LDA.train(corpus, maxIterations=max_iter, seed=seed, checkpointInterval=checkin_point_interval, k=K, optimizer=optimizer, docConcentration=alpha, topicConcentration=beta)

解决：lz设置docConcentration和TopicConcentration时，只能使用docConcentration=[value]和TopicConcentration=value两种不同的初始化方式，不知道为啥。。。否则出错。

lda_model = LDA.train(corpus, maxIterations=max_iter, seed=seed, checkpointInterval=checkin_point_interval, k=K, optimizer=optimizer, docConcentration=[alpha], topicConcentration=beta)

spark输出太多warning messages

WARN Executor: 2 block locks were not released by TID =

Lock release errors occur frequently in executor logs

原因：If there are any releasedLocks (after calling BlockManager.releaseAllLocksForTask earlier) and spark.storage.exceptionOnPinLeak is enabled (it is not by default) with no exception having been thrown while the task was running, a SparkException is thrown:
[releasedLocks] block locks were not released by TID = [taskId]:
[releasedLocks separated by comma]
Otherwise, if spark.storage.exceptionOnPinLeak is disabled or an exception was thrown by the task, the following WARN message is displayed in the logs instead:
WARN Executor: [releasedLocks] block locks were not released by TID = [taskId]:
[releasedLocks separated by comma]
Note If there are any releaseLocks, they lead to a SparkException or WARN message in the logs.
[jaceklaskowski/mastering-apache-spark-book/spark-executor-taskrunner.adoc]

mapWithState causes block lock warning?

The warning was added by: SPARK-12757 Add block-level read/write locks to BlockManager?

[connectedComponents() raises lots of warnings that say "block locks were not released by TID = ..."]

[ Lock release errors occur frequently in executor logs]

解决：终于在调试log时候发现问题解决了

在简略Spark输出设置时[Spark安装和配置 ]修改过$SPARK_HOME/conf/log4j.properties.template文件只输出WARN信息，就算改成了ERROR，信息也还是会自动修改成WARN输出出来，不过多了一条提示：

Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel).

就在这时发现了一个解决方案：

根据提示在代码中加入sc.setLogLevel('ERROR')就可以解决了！

from: http://blog.csdn.net/pipisorry/article/details/52916307

ref:

Spark：相关错误总结相关推荐

Hadoop/Spark相关面试问题总结
Hadoop/Spark相关面试问题总结面试回来之后把当中比較重要的问题记了下来写了个总结: (答案在后面) 1.简答说一下hadoop的map-reduce编程模型 2.hadoop的TextIn ...
Scrapy运行中常见网络相关错误
错误类型错误名称 defer.TimeoutError 当延迟超时时,默认情况下会引发此错误(链接) TimeoutError 用户超时导致连接失败 DNSLookupError DNS查找失败 C ...
常见的 ChatGPT 相关错误提示及解决办法
我汇总了一些大家在使用 ChatGPT 过程中,可能会遇到的一些问题以及相关错误提示,大家可以根据遇到的情况查看解决办法. 注册登陆问题: 错误提示:Too many signups from the ...
朱雀雀openfire+spark相关的中文资料
难得找几篇openfire+spark相关的中文资料从朱雀雀的博客里找到几篇文章,很有启发,也在这里转载一下. 开发你自己的XMPP IM - [J2EE] 最近没在Blog 上露脸,为撒类?应师兄 ...
Power BI中配置Python运行环境，及相关错误问题解决
Power BI Desktop中支持python,在Power BI的数据获取.数据转换.数据可视化.数据输出的全流程中,都可以支持使用Python.关于如何全流程使用Python,可以参见文章ht ...
spark环境安装及配置详细（附带相关错误及解决办法）
安装spark之前需要成功安装scala 解压安装包更改名字修改环境变量使环境变量生效修改配置文件进入spark目录进入conf 查看文件更改文件名称将spark-env.sh.tem ...
【大数据面试题】（八）Spark 相关面试题
1.spark中的RDD是什么,有哪些特性? 答:RDD(Resilient Distributed Dataset)叫做分布式数据集,是spark中最基本的数据抽象,它代表一个不可变,可分区,里面的 ...
spark常见错误解决方案
1.用./bin/spark-shell启动spark时遇到异常:java.net.BindException: Can't assign requested address: Service 'sp ...
spark相关面试题总结
(根据博客总结并不断增加自己的内容) 1.spark中的RDD是什么,有哪些特性? 答:RDD(Resilient Distributed Dataset)叫做分布式数据集,是spark中最基本的数据 ...
spark 序列化错误集群提交时_spark 面试题（2）
33.选择题二.选择题 1. Spark 的四大组件下面哪个不是 (D ) A.Spark Streaming B. Mlib C Graphx D.Spark R 2.下面哪个端口不是 ...

Spark：相关错误总结