py4j.protocol.Py4JJavaError: An error occurred while calling o90.save

环境:

Ubuntu19.10

anaconda3-python3.6.10

scala 2.11.8

apache-hive-3.0.0-bin

hadoop-2.7.7

spark-2.3.1-bin-hadoop2.7

java version "1.8.0_131"

Mysql Server version: 8.0.19-0ubuntu0.19.10.3 (Ubuntu)

driver:mysql-connector-java-8.0.20.jar

[Driver link|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.20]

使用的代码是:

import pandas as pd
from pyspark.sql import SparkSession
from pyspark import SparkContext
from pyspark.sql import SQLContextdef map_extract(element):file_path, content = elementyear = file_path[-8:-4]return [(year, i) for i in content.split("\n") if i]spark = SparkSession\.builder\.appName("PythonTest")\.getOrCreate()res = spark.sparkContext.wholeTextFiles('hdfs://Desktop:9000/user/mercury/names',minPartitions=40)  \.map(map_extract) \.flatMap(lambda x: x) \.map(lambda x: (x[0], int(x[1].split(',')[2]))) \.reduceByKey(lambda x,y:x+y)df = res.toDF(["key","num"])  #把已有数据列改成和目标mysql表的列的名字相同
# print(dir(df))
df.printSchema()
print(df.show())
df.printSchema()df.write.format("jdbc").options(url="jdbc:mysql://127.0.0.1:3306/leaf",driver="com.mysql.cj.jdbc.Driver",dbtable="spark",user="appleyuchi",password="appleyuchi").mode('append').save()

提交方式是(下面两种方式都能复现bug):

①pyspark --master yarn（然后在交互是模式中输入交互式代码）

②spark-submit --master yarn --deploy-mode cluster 源码.py

③pyspark --master yarn --conf spark.executor.extraClassPath=/home/appleyuchi/bigdata/apache-hive-3.0.0-bin/lib/mysql-connector-java-8.0.20.jar

同样会报告类似的错误

解决方案:

https://gitee.com/appleyuchi/cluster_configuration/blob/master/物理环境配置流程-必须先看.txt

Reference:

[1]https://zhuanlan.zhihu.com/p/136777424

py4j.protocol.Py4JJavaError: An error occurred while calling o90.save相关推荐

Spark: py4j.protocol.Py4JJavaError: An error occurred while calling o91.showString.
@创建于:2022.06.15 @修改于:2022.06.15 利用Spark的yarn模式(把.py文件上传到hadoop平台),执行过程中发现了下面的问题.py4j.protocol.Py4JJa ...
py4j.protocol.Py4JJavaError错误
py4j.protocol.Py4JJavaError: An error occurred while calling o30.toDebugString. : org.apache.hadoop. ...
An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
最近公司上云,所有大数据环境都迁移到云上.在进行项目迁移时候发现运行报错 py4j.protocol.Py4JJavaError: An error occurred while calling No ...
spark运行出现py4j.protocol.Py4JError异常
spark配置启动过程中出现:py4j.protocol.Py4JError: An error occurred while calling None.None. Trace: Authentica ...
An internal error occurred during: Launching xxx on WebLogic10.x.
An internal error occurred during: "Launching xxx on WebLogic10.x". java.lang.NullPointerE ...
An internal error occurred during: quot;J2EE Component Mapping Updatequot;.
1.错误描写叙述 An internal error occurred during: "J2EE Component Mapping Update". java.lang.Nul ...
java初始化虚拟机错误_异常 - 虚拟机初始化错误 - Error occurred during initialization of VM...
1 环境配置信息 1.1 服务器配置信息服务器是物理机, 配置信息如下: CPU型号 CPU个数 CPU核数 CPU线程数内存 Intel(R) Xeon(R) CPU E5-2630 v4 @ ...
A generic error occurred in GDI+的解决方案
转自智慧光原文A generic error occurred in GDI+. 解决方法使用image1.RotateFlip(RotateFlipType.Rotate90FlipNone)方法 ...
成功解决raise Py4JError py4j.protocol.Py4JError: py4j.protocol.Py4JError: Could not find py4j jar at
成功解决raise Py4JError("Could not find py4j jar at {0}".format(jarpath)) py4j.protocol.Py4JEr ...

py4j.protocol.Py4JJavaError: An error occurred while calling o90.save

py4j.protocol.Py4JJavaError: An error occurred while calling o90.save相关推荐

最新文章

热门文章