官方文档地址

  • DataX HdfsReader 插件文档
  • DataX HdfsWriter 插件文档

Reader插件文档明确说明

而配置中又有HA相关配置

没办法只能试试呗！Reader和Writer一样都支持该参数

datax_hive.json

{"job": {"setting": {"speed": {"channel": 8},"errorLimit": {"record": 0,"percentage": 1.0}},"content": [{"reader": {"name": "hdfsreader","parameter": {"path": "/user/hive/warehouse/ads.db/my_test_table/dt=${date}/*","hadoopConfig":{"dfs.nameservices": "${nameServices}","dfs.ha.namenodes.${nameServices}": "namenode1,namenode2","dfs.namenode.rpc-address.${nameServices}.namenode1": "${FS}","dfs.namenode.rpc-address.${nameServices}.namenode2": "${FSBac}","dfs.client.failover.proxy.provider.${nameServices}": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"},"defaultFS": "hdfs://${nameServices}","column": [{"index": 0,"type": "String"},{"index": 1,"type": "Long"}],"fileType": "orc","encoding": "UTF-8","fieldDelimiter": ","}},"writer":{"name": "txtfilewriter","parameter": {"path": "/home/dev/data/result","fileName": "test","writeMode": "truncate","dateFormat": "yyyy-MM-dd"}}}]}
}
# 这里我是通过shell脚本动态传参传入对应三个参数
# nameServices为cdh配置高可用时设置的nameServices1，myFS和myFSBac为对应namenode节点的8020端口服务，如： 192.168.2.123:8020
pyhon -p" -DFS=${myFS}  -DFSBac=${myFSBac}  -DnameServices=${nameServices}  -Ddate=${mydate}" datax_hive.json

添加参数后一直报错

- 经DataX智能分析,该任务最可能的错误原因是:
23-09-2019 12:44:47 CST test_hdfs_to_file INFO - com.alibaba.datax.common.exception.DataXException: Code:[HdfsWriter-06], Description:[与HDFS建立连接时出现IO异常.]. - java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:515)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:171)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.getFileSystem(HdfsHelper.java:67)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.init(HdfsWriter.java:47)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.Engine.start(Engine.java:92)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.Engine.entry(Engine.java:171)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.Engine.main(Engine.java:204)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO - Caused by: java.lang.reflect.InvocationTargetException
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:498)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    ... 18 more
23-09-2019 12:44:47 CST test_hdfs_to_file INFO - Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://nameservice1
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:93)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    ... 23 more
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -  - java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:515)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:171)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.getFileSystem(HdfsHelper.java:67)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.init(HdfsWriter.java:47)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.Engine.start(Engine.java:92)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.Engine.entry(Engine.java:171)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.Engine.main(Engine.java:204)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO - Caused by: java.lang.reflect.InvocationTargetException
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:498)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    ... 18 more
23-09-2019 12:44:47 CST test_hdfs_to_file INFO - Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://nameservice1
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:93)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    ... 23 more
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.getFileSystem(HdfsHelper.java:72)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.init(HdfsWriter.java:47)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.Engine.start(Engine.java:92)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.Engine.entry(Engine.java:171)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.core.Engine.main(Engine.java:204)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO - Caused by: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:515)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:171)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.getFileSystem(HdfsHelper.java:67)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    ... 7 more
23-09-2019 12:44:47 CST test_hdfs_to_file INFO - Caused by: java.lang.reflect.InvocationTargetException
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:498)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    ... 18 more
23-09-2019 12:44:47 CST test_hdfs_to_file INFO - Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://nameservice1
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:93)
23-09-2019 12:44:47 CST test_hdfs_to_file INFO -    ... 23 more

不能愉快的玩耍了，卡在这里好几个小时

查Issuse关键字搜索HA 有结果了

至于这三个文件怎么来，我也在此issues下给出了我的回答

具体操作

  • 下载对应三个文件
  • 备份datax安装路径下的datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar

  • 用压缩工具打开hdfsreader-0.0.1-SNAPSHOT.jar（如360压缩，右键用360打开，非解压），将上面三个文件直接拖入即可。如果是拷贝hdfsreader-0.0.1-SNAPSHOT.jar到其他路径下操作的，将操作完的jar包替换掉原来datax对应hdfsreader路径下的hdfsreader-0.0.1-SNAPSHOT.jar

接下来就可以愉快的使用了,因为在hdfs-site.xml中已经指明了

dfs.nameservices=nameservice1及其他高可用的配置

进一步发现，使用此方法配置后，datax json中连hadoopCofig参数都不需要配置了，简直是不能再赞了

hdfswriter 操作一样啊，如果对你的问题有帮助，那就点个赞吧！！！

DataX Hdfs HA(高可用)配置支持相关推荐

  1. hadoop之HDFS HA(高可用）架构安装部署

    前言 再看本文章之前,请先确保已经看过CentOS下hadoop的分布式搭建和分布式协作服务框架Zookeeper安装部署两篇文章,因为本文章是建立在它们基础之上的. 一.HA相关介绍 1.为什么出现 ...

  2. Kerberos HA高可用配置

    Kerberos 高可用配置 Kerberos 安装(主节点操作) 节点信息 data80 data81 data82 data83 备注 主Kerberos节点:data80 备Kerberos节点 ...

  3. Hadoop系列-HDFS HA高可用集群

    前言: 在HDFS集群的时候我们知道,NameNode只有一个,如果现在NameNode挂掉了,或者NameNode需要硬件或者软件的升级,那么势必就有单点问题.那么HDFS HA就是来解决这个问题的 ...

  4. hadoop的HA高可用配置(没有搞完)

    hadoop的HA主要是两部分,一部分是namenode的备份,一部分是datanode备份． namenode和datanode一般尽量不在一个节点上, 这样namenode挂掉一个,还会有另外一个 ...

  5. canal+zookeeper+mysql高可用配置

    一.部署环境 1.基础环境: 软件 版本 作用 Linux Centos7.1,8g Jdk 1.8.0_151 canal 1.1.1 canal server端,与mysql和zookeeper交 ...

  6. sparkstreaming监听hdfs目录如何终止_HDFS—HA高可用详解

    一.HA概述 1)所谓HA(high available),即高可用(7*24小时不中断服务). 2)实现高可用最关键的策略是消除单点故障.HA严格来说应该分成各个组件的HA 机制:HDFS的HA和Y ...

  7. 十二、Hadoop HA集群高可用配置

    官网地址:http://hadoop.apache.org/ 一.HA概述 1.所谓HA(High Available),即高可用(7*24小时不中断服务). 2.实现高可用最关键的策略是消除单点故障 ...

  8. HA高可用完全分布式集群 HDFS搭建

    HDFS HA 需要 设置静态IP.安装 jdk.设置ssh无密码登录.安装zookeeper.安装hadoop. 在Hadoop2.X之前,Namenode是HDFS集群中可能发生单点故障的节点,每 ...

  9. 【视频】详解HDFS的HA高可用原理

    详解HDFS的HA高可用原理

最新文章

  1. /etc/inittab
  2. postgres创建用户，修改用户密码，创建数据库
  3. 关于获取客户端Mac地址
  4. MySQL 命令大全 - 命令详解
  5. Effective Java之坚持使用Overide注解(三十六)
  6. 王道408数据结构——第八章 排序
  7. 如何 给给软件开发 添加 代理_如何与软件开发公司有效沟通
  8. Java加权负载均衡策略
  9. jmoiron sqlx mysql_mysql 一（或其他数据库）
  10. bug1-tensorflow中自定义模型的bug-input_signature
  11. Android远程过程通讯,Android系统进程间通信（IPC）机制Binder中的Client获得Server远程接口过程源代码分析(3)...
  12. 阿法狗之后的围棋世界
  13. java halt_System.exit和Runtime halt区别
  14. django 搜索功能的实现
  15. 格拉姆矩阵 Gram Matrix 简单理解
  16. 电脑售后服务人员无奈的一天-爆笑
  17. python绘制笛卡尔直角坐标系
  18. uefi+guid分区与legacy+mbr分区_硬盘分区表格式GUID和MBR知识普及
  19. Android开发中保存数据的四种方法方法
  20. 上班用“蝙蝠侠电话”沟通是一种怎样的体验？

热门文章

  1. 运行python嵌入式要求_在嵌入式IPython实例中运行配置文件启动文件
  2. 嵌入式linux下网卡设备驱动与网络协议栈的设计实现.pdf,嵌入式系统的通用网络驱动模型设计与实现.pdf...
  3. zookeeper版本更新_zookeeper介绍及运维实践
  4. centos mysql 5.5.57_centos 6.5装mysql5.7
  5. opencv方框内图像保存_opencv 图像滤波(均值,方框,高斯,中值)
  6. [模板] tarjan/联通分量/dfs树
  7. 当数据改变时，VUE是如何实现DOM更新的？
  8. druid 数据库连接池，简单配置
  9. Linux学习笔记之1——文件和目录管理（硬连接和软连接）(连结档,相当于快捷方式）...
  10. 1分钟玩转Kafka