dataFrame（DF）将数据插入ES时报错 org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect E
2024-04-24 22:28:31
SparkSql插入ES时报错解决（针对增加用户权限之后的报错解决）
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting ‘es.nodes.wan.only’
以下是报错信息
Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:340)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:97)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.spark.sql.ElasticsearchRelation.insert(DefaultSource.scala:620)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.spark.sql.DefaultSource.createRelation(DefaultSource.scala:107)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
15-11-2022 14:43:56 CST playrecordtoes INFO - at com.pukka.bigdata.dealwith.toes.SparkToEs$.main(SparkToEs.scala:112)
15-11-2022 14:43:56 CST playrecordtoes INFO - at com.pukka.bigdata.dealwith.toes.SparkToEs.main(SparkToEs.scala)
15-11-2022 14:43:56 CST playrecordtoes INFO - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
15-11-2022 14:43:56 CST playrecordtoes INFO - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
15-11-2022 14:43:56 CST playrecordtoes INFO - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
15-11-2022 14:43:56 CST playrecordtoes INFO - at java.lang.reflect.Method.invoke(Method.java:498)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15-11-2022 14:43:56 CST playrecordtoes INFO - Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: security_exception: missing authentication credentials for REST request [/]
15-11-2022 14:43:56 CST playrecordtoes INFO - null
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:469)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:426)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:388)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:392)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:168)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:735)
15-11-2022 14:43:56 CST playrecordtoes INFO - at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:330)
报错解决
在配置参数中加入 “es.net.http.auth.user”、 “es.net.http.auth.pass” 即可
val options = Map(//如果没有index,则新建"es.index.auto.create" -> "true","es.nodes.wan.only" -> "true","es.nodes" -> "192.168.20.131,192.168.20.130",//"es.nodes" -> "10.0.6.56",//"es.nodes" -> "60.167.225.130",//es的端口"es.port" -> "9200","es.net.http.auth.user" ->"elastic","es.net.http.auth.pass" ->"test@2022!")val esPath=s"play_record_$year$month/_doc"dataFrame.write.format("org.elasticsearch.spark.sql").options(options).mode(SaveMode.Append)//index为epg_itv_play_record_$year$month.save(esPath)spark.stop()
dataFrame（DF）将数据插入ES时报错 org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect E相关推荐
- Oracle数据库mybatis 插入空值时报错（with JdbcType OTHER）
Oracle数据库mybatis 插入空值时报错(with JdbcType OTHER)的两种解决方案 在使用Mybatis 3 时,发现了这个问题,当插入数据时,如果有一个字段为空值时,系统会报异 ...
- MySql插入语句时报错,错误代码1062
MySql插入语句时报错,错误代码1062 mysql 工具操作报错 程序运行报错 Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConst ...
- vue引入luckysheet插入图表时报错Uncaught TypeError: h.createChart is not a function
Luckysheet(excel)引入chartMix的问题 记录问题:离线引入Luckysheet,插入图表时报错Uncaught TypeError: h.createChart is not a ...
- ZHS16GBK字符集插入中文时报错ORA-01756
插入数据时遇到一个奇怪的问题,插入中文报错ORA-01756: quoted string not properly terminated 简单的复现测试如下: 查看客户端及数据库字符集 可以看到字符 ...
- mysql建表时插入中文_Mysql新建表，插入中文时报错“Incorrect string value: '\xE4\xBD\xA0\xE5\xA5\xBD' for column”问题...
有时候我们在往数据库中输入信息时,如果输入的内容是中文,会报错"Incorrect string value: '\xE4\xBD\xA0\xE5\xA5\xBD' for column&q ...
- 插入数据库时报错：Duplicate entry ‘AAA-BBB‘ for key ‘uk_XXX‘
原因:数据库创建了对应AAA字段名与BBB字段名的唯一索引'uk_XXX',插入数据库内已有相同数据时会报错 解决方案:插入数据前校验唯一值/删除索引
- oracle1422,Oracle错误：数据泵导出时报错ORA-1422
今天一个学习Oracle时间不长的同事在使用数据泵导出的时候碰到了这个错误. 由于前两天看到他在研究DUAL表,而且还问过我几个DUAL表相关的问题,所以看到这个错误后首先想到的就是问他是否向DUAL ...
- pytorch数据加载时报错OSError: [Errno 22] Invalid argument
将数据加载器的num_workers删去,即由 train_loader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, ...
- 将MySQL数据导入hive时报错发现如下错误
报错如下 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/shi ...
- Sqoop进行导入数据到Hive时报错：ERRORtool.ImportTool:Importfailed:java.io.IOException:java.lang.ClassNotFoundExc
项目场景: 使用版本:Sqoop1.4.7.hive3.1.2 问题描述 在使用Sqoop进行MySQL数据导入到Hive的过程中出现一下错误提示.ERRORtool.ImportTool:Impor ...
最新文章
- 剑指offer：面试题27. 二叉树的镜像
- mysql 导出gbk_把mysql 中的字符gb2312 改为gbk的方法
- 浅析网站内链优化如何营造良好的内链生态环境？
- Bootstrap File Input 真正 解决跨域问题
- 手机端适应_不轻易透露的超强技巧！详解iVX中怎样做设备自适应
- html css position,[CSS]CSS Position 详解
- ubuntu Qt Creator不能输入中文
- 4问教你搞定java中的ThreadLocal
- 翻译: 欧洲伽利略卫星-全球导航卫星系统GNSS
- 三星手机app信息推送服务器地址,三星推送服务(samsung push service)
- 交互设计理论之格式塔理论与四大法则
- 超级计算机有什么特点,计算机的特点有哪些
- read函数和write函数的使用
- 设置CFLAGS后，编译无效
- 什么叫无差别伤害_无差别伤害背后的差别
- 砸盘、销号、解散社群，Merlin Lab“跑路三连”暴露了DeFi哪些问题？
- 富康路小学毕业郭佳华
- 图文解读 Infor10 ERP SyteLine的功能
- js过滤对象中值为空的属性
- Excel工具类(详细版)