ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) java.sql.BatchUpdateException: Duplicate

sparksql把JDBC 从关系型数据库中读取数据的方式创建 DataFrame报错：

20/08/26 15:29:37 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
java.sql.BatchUpdateException: Duplicate entry '269' for key 'PRIMARY'com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '269' for key 'PRIMARY'
Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.sq

去掉数据库里的主键递增

idea中sparksql把JDBC 从关系型数据库中读取数据的方式创建 DataFrame,代码部分

import org.apache.log4j.{Level, Logger}
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSessionobject readFile {def main(args: Array[String]): Unit = {Logger.getLogger("org").setLevel(Level.ERROR)val conf: SparkConf = new SparkConf().setAppName(this.getClass.getSimpleName).setMaster("local[2]")val spark = SparkSession.builder().config(conf).getOrCreate()//      加载/保存方法
//     val frame = spark.read.json("D:/JulyMounth/spark/sparkData/people.json")
//    frame.show()
//    frame.write.format("csv").mode("append").save("D:/JulyMounth/spark/sparkData/900")//JDBC 从关系型数据库中读取数据的方式创建 DataFrameval df = spark.read.format("jdbc").option("url", "jdbc:mysql://localhost:3306/mydata").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "a_users").option("user", "root").option("password", "123456").load()df.show()df.write.format("jdbc").option("url","jdbc:mysql://localhost:3306/mydata").option("driver", "com.mysql.jdbc.Driver").option("dbtable","a_users").option("user","root").option("password","root").mode("append").save()spark.close()}
}

ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) java.sql.BatchUpdateException: Duplicate相关推荐

spark集群环境下Lost task 0.0 in stage 10.0 (TID 17, 10.28.23.202): java.io.FileNotFoundException
spark从当前目录加载文件报错,Lost task 0.0 in stage 10.0 (TID 17, 10.28.23.202): java.io.FileNotFoundException,明 ...
ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
用 bash spark-submit 在spark上跑代码的时候出现错误: ERROR executor.Executor: Exception in task 9.0 in stage 4 ...
Spark 报错：Lost task 0.0 in stage 1.0 (TID 2, xxx, executor 0): java.io.FileNotFoundExceptio
-- 昨夜西风凋碧树,独上高楼,望尽天涯路问题描述在 Spark 集群中读取本地文件出现如下错误: Lost task 0.0 in stage 1.0 (TID 2, xxxxx, execu ...
python Exception happened during processing of request from( 127.0.0.1 xxx) error [10053]
使用Django框架,删除数据时,突然提示如下错误: Exception happened during processing of request from( 127.0.0.1 xxx) er ...
.php on line 0,PHP Fatal error: Could not queue new timer in Unknown on line 0
问题 I write a simple web application for my compnay that can let user log in to arrange their work ti ...
nested exception is java.sql.SQLException: HOUR_OF_DAY: 0 -＞ 1
Cause: java.sql.SQLException: HOUR_OF_DAY: 0 -> 1 HOUR_OF_DAY: 0 -> 1; nested exception is jav ...
xx.xib: error: Illegal Configuration: Safe Area Layout Guide before iOS 9.0报错问题解决
之前是用xcode8.3.3创建的工程最近升级到Xcode9.0 遇见了这个问题在Xcode 9.0以上新建xib文件会报错 xx.xib: error: Illegal Configuratio ...
mysqldump: Got error: 2003: Can't connect to MySQL server on '127.0.0.1' (10060)
今天在用批处理进行MySQL自动备份的过程中遇到一个问题,错误提示:mysqldump: Got error: 2003: Can't connect to mysql server on '127. ...
sphinx error connection to 127.0.0.1:9312 failed (errno=0, msg=)
通过一键安装lnmp环境,运行sphinx时,提示sphinx error connection to 127.0.0.1:9312 failed (errno=0, msg=) 是因为lnmpa服务 ...

ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) java.sql.BatchUpdateException: Duplicate

ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) java.sql.BatchUpdateException: Duplicate相关推荐

最新文章

热门文章