spark运行出现py4j.protocol.Py4JError异常

spark配置启动过程中出现：py4j.protocol.Py4JError: An error occurred while calling None.None. Trace:
Authentication error: unexpected command.
异常的代码：

ssh://root@192.168.141.130:22/root/miniconda2/envs/ai/bin/python3.6 -u /root/Desktop/kafka_test.py
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/export/servers/spark-2.3.4-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/export/servers/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2019-10-17 14:56:39 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):File "/root/Desktop/kafka_test.py", line 15, in <module>spark = SparkSession.builder.appName("wordcount").getOrCreate()File "/root/miniconda2/envs/ai/lib/python3.6/site-packages/pyspark/sql/session.py", line 173, in getOrCreatesc = SparkContext.getOrCreate(sparkConf)File "/root/miniconda2/envs/ai/lib/python3.6/site-packages/pyspark/context.py", line 363, in getOrCreateSparkContext(conf=conf or SparkConf())File "/root/miniconda2/envs/ai/lib/python3.6/site-packages/pyspark/context.py", line 129, in __init__SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)File "/root/miniconda2/envs/ai/lib/python3.6/site-packages/pyspark/context.py", line 312, in _ensure_initializedSparkContext._gateway = gateway or launch_gateway(conf)File "/root/miniconda2/envs/ai/lib/python3.6/site-packages/pyspark/java_gateway.py", line 46, in launch_gatewayreturn _launch_gateway(conf)File "/root/miniconda2/envs/ai/lib/python3.6/site-packages/pyspark/java_gateway.py", line 139, in _launch_gatewayjava_import(gateway.jvm, "org.apache.spark.SparkConf")File "/root/miniconda2/envs/ai/lib/python3.6/site-packages/py4j/java_gateway.py", line 176, in java_importreturn_value = get_return_value(answer, gateway_client, None, None)File "/root/miniconda2/envs/ai/lib/python3.6/site-packages/py4j/protocol.py", line 324, in get_return_valueformat(target_id, ".", name, value))
py4j.protocol.Py4JError: An error occurred while calling None.None. Trace:
Authentication error: unexpected command.

出现异常的原因：虚拟环境中安装的py4j的版本和spark中Python中lib包中的版本不一致引起的

虚拟环境中的py4j的版本为0.10.6版

spark安装包中python目录下lib中的py4j版本为0.10.7版：
查看路径：/export/servers/spark-2.3.4-bin-hadoop2.7/python/lib

解决方案：
卸载虚拟环境中的py4j，安装和spark中一致的包即可。

# 卸载虚拟环境中的安装包
conda uninstall py4j
# 安装新的版本
conda install py4j=0.10.7

spark运行出现py4j.protocol.Py4JError异常相关推荐

成功解决raise Py4JError py4j.protocol.Py4JError: py4j.protocol.Py4JError: Could not find py4j jar at
成功解决raise Py4JError("Could not find py4j jar at {0}".format(jarpath)) py4j.protocol.Py4JEr ...
from py4j.protocol import Py4JError ModuleNotFoundError 错误
from py4j.protocol import Py4JError ModuleNotFoundError: No module named 'py4j'报错了!!! 学习林子雨老师的pySpar ...
Spark运行程序异常信息： org.apache.spark.SparkException: Task not serializable 解决办法
Spark运行程序异常信息: org.apache.spark.SparkException: Task not serializable 解决办法参考文章: (1)Spark运行程序异常信息: o ...
Spark: py4j.protocol.Py4JJavaError: An error occurred while calling o91.showString.
@创建于:2022.06.15 @修改于:2022.06.15 利用Spark的yarn模式(把.py文件上传到hadoop平台),执行过程中发现了下面的问题.py4j.protocol.Py4JJa ...
Unsupported protocol: t3 异常的处理
最近领导要求用JMX写一个监控WEBLOGIC运行的东西, 20162514578912==10.4.32.5==7001==weblogic==weblogic123==serverruntime ...
py4j.protocol.Py4JJavaError错误
py4j.protocol.Py4JJavaError: An error occurred while calling o30.toDebugString. : org.apache.hadoop. ...
Spark运行模式Local+Standalone+Yarn+mesos
Spark运行模式Local+Standalone+Yarn+mesos bin/spark-submit --help 注意: --master MASTER_URL spark://host:po ...
JSP Tomcat8.0运行连接池时发生异常【AbstractMethodError oracle.jdbc.driver.T4CConnection.isValid(I)Z】...
原创 2015年12月28日 11:38:01 2004 一.Tomcat8.0运行连接池时发生异常: AbstractMethodError oracle.jdbc.driver.T4CConnec ...
Spark详解（十一）：Spark运行架构原理分析
1. Spark 运行架构总体分析 1.1 总体介绍 Spark应用程序的运行架构基本上由三部分组成,包括SparkContext(驱动程序).ClusterManger(集群资源管理器)和Execu ...

spark运行出现py4j.protocol.Py4JError异常

spark运行出现py4j.protocol.Py4JError异常相关推荐

最新文章

热门文章