在 aws emr 上，将 hbase table A 的数据，对 key 做 hash，写到另外一张 table B

先 scan 原表，然后 bulkload 到新表。

采坑纪录
1. bulkload 产生 hfile 前，需要先对 hash(key) 做 repartition，在 shuffle 的 read 阶段，产生了以下错误

org.apache.spark.shuffle.FetchFailedException: failed to allocate 16777216 byte(s) of direct memory (used: 3623878656, max: 3635150848)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:523)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:454)
...
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 3623878656, max: 3635150848)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594)
...

原因：在 shuffle 的 read 阶段，会申请一个跟 block（或partition）一样大小的内存，因为每个分区过大，内存不够了
相关说明：https://issues.apache.org/jira/browse/SPARK-13510

因为netty默认使用了offheap memory，所以报了这个错误。可选择加入java参数 "-Dio.netty.noUnsafe=true"，不使用 offheap 内存

2. bulkload 产生 hfile 的时候，多次发生因 executor 被 killed，导致 application 失败。通过观察，发现是 executor 往本地写文件的时候，本地空间不够了。
相关问题：https://stackoverflow.com/questions/29131449/why-does-hadoop-report-unhealthy-node-local-dirs-and-log-dirs-are-bad

于是增加 yarn 集群机器，使用 hdfs balancer 均衡数据分布。

============= yarn-nodemanager =============
2019-02-15 10:18:45,562 WARN org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection (DiskHealthMonitor-Timer): Directory /mnt/yarn error, used space above threshold of 90.0%, removing from list of valid directories
2019-02-15 10:18:45,563 WARN org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection (DiskHealthMonitor-Timer): Directory /var/log/hadoop-yarn/containers error, used space above threshold of 90.0%, removing from list of valid directories
2019-02-15 10:18:45,563 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService (DiskHealthMonitor-Timer): Disk(s) failed: 1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers
2019-02-15 10:18:45,563 ERROR org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService (DiskHealthMonitor-Timer): Most of the disks failed. 1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers
2019-02-15 10:18:45,789 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService (AsyncDispatcher event handler): Cache Size Before Clean: 589300919, Total Deleted: 0, Public Deleted: 0, Private Deleted: 0
2019-02-15 10:18:46,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl (AsyncDispatcher event handler): Container container_1549968021090_0114_01_000006 transitioned from RUNNING to KILLING
2019-02-15 10:18:46,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl (AsyncDispatcher event handler): Container container_1549968021090_0114_01_000016 transitioned from RUNNING to KILLING============= yarn-resourcemanager.log =============
019-02-15 10:18:45,664 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl (AsyncDispatcher event handler): Node ip-10-6-43-89.ap-south-1.compute.internal:8041 reported UNHEALTHY with details: 1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers
2019-02-15 10:18:45,664 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl (AsyncDispatcher event handler): ip-10-6-43-89.ap-south-1.compute.internal:8041 Node Transitioned from RUNNING to UNHEALTHY
2019-02-15 10:18:45,664 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl (ResourceManager Event Processor): container_1549968021090_0114_01_000006 Container Transitioned from RUNNING to KILLED

3. shuffle 在读取文件时，非常依赖 netty 的 offheap 堆栈，设置不使用 offheap memory 之后，会有以下错误（内存调很大也会出现）。

2019-02-17T02:56:12.949+0000: [Full GC (Ergonomics) [PSYoungGen: 465920K->465917K(931840K)] [ParOldGen: 2796146K->2796069K(2796544K)] 3262066K->3261987K(3728384K), [Metaspace: 67840K->67739K(1110016K)], 5.2891526 secs] [Times: user=18.15 sys=0.01, real=5.29 secs]
#
# java.lang.OutOfMemoryError: GC overhead limit exceeded
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 8023"...

　　或者是

2019-02-17T02:59:43.073+0000: [Full GC (Ergonomics) [PSYoungGen: 123392K->123391K(422912K)] [ParOldGen: 2796365K->2796364K(2796544K)] 2919757K->2919756K(3219456K), [Metaspace: 67051K->67051K(1107968K)], 3.3979517 secs] [Times: user=13.45 sys=0.00, real=3.39 secs]
2019-02-17T02:59:43.073+0000: [Full GC (Ergonomics) ............
2019-02-17T02:59:43.073+0000: [Full GC (Ergonomics) ............
2019-02-17T02:59:43.073+0000: [Full GC (Ergonomics) ............
2019-02-17T02:59:43.073+0000: [Full GC (Ergonomics) ............ExecutorLostFailure (executor 6 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 125095 ms

因为 shuffle 本身不占用太多内存，但产生 hfile 之前的 sort 需要很多内存，在 spark 的统一内存管理模型中，这是 other 部分的空间。推测是 spark 统一内存模型，计算出现错误，挤压了 other 部分的空间大小。于是加入下面的参数

spark.memory.fraction=0.2

4. 产生了 HFile 之后，需要导入到 hbase，遇到下面问题

Sat Feb 16 21:48:36 UTC 2019, RpcRetryingCaller{globalStartTime=1550353152797, pause=100, retries=35}, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.ipc.RemoteException): org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /apps/hbase/data/data/ap/users_v2/9a9d8ee1e23d335afb01aced349054d8/.tmp/70ad5fa3d4834fb6a47abee6101594ff could only be replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) running and no node(s) are excluded in this operation.at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1719)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3372)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3296)

因为我们 spark 与 hbase 不在一个 yarn 上，没有共享 hdfs。一开始 HFile 写在 spark 的集群上，于是产生了很多问题。之后改成 HFile 写在 hbase 的同集群中，这一步很快就过了。具体原因不详。

5. emr的配置中，spark 的本地文件缓存路径为 /mnt/yarn/usercache/hadoop/appcache/application_1549968021090_0135/blockmgr-535fd27a-4b80-4116-b855-17ab7be68f1c。与 hdfs 在一个硬盘上。

===============================================================================================

最终提交 spark 的命令为

spark-submit \--master yarn \--name UserTableFromPrimitiveToV2 \--queue default \--deploy-mode cluster \--driver-cores 2 \--driver-memory 5g \--num-executors 30 \--executor-cores 2 \--executor-memory 4g \--conf spark.driver.memoryOverhead=1g \--conf spark.executor.memoryOverhead=2g \--conf spark.dynamicAllocation.enabled=false \--conf spark.yarn.maxAppAttempts=1 \--conf spark.blacklist.enabled=false \--conf spark.memory.fraction=0.2 \--conf spark.executor.extraJavaOptions="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=0  -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseParallelGC -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p' -Dio.netty.noUnsafe=true" \--class com.hotstar.ap.ingest.batch.tool.migration.UserTableFromPrimitiveToV2 \./batch/build/libs/batch-all.jar \-e dev

转载于:https://www.cnblogs.com/keepthinking/p/10386811.html

在 aws emr 上，将 hbase table A 的数据，对 key 做 hash，写到另外一张 table B相关推荐

Apache Hudi在AWS Glue和AWS EMR上同步元数据的异同
继我们上篇文章<在AWS Glue中使用Apache Hudi>介绍了如何在Glue中同步Hudi元数据之后,本文我们再简短截说一下Hudi在EMR上的元数据同步. 首先,EMR对于元数据 ...
hadoop emr_在Amazon EMR上运行Hadoop MapReduce作业
hadoop emr 不久前,我发布了如何使用CLI设置EMR群集的信息. 在本文中,我将展示如何使用适用于AWS的Java SDK来设置集群. 展示如何使用Java AWS开发工具包执行此操作的最佳 ...
在Amazon EMR上运行Hadoop MapReduce作业
不久前,我发布了如何使用CLI设置EMR群集的信息. 在本文中,我将展示如何使用适用于AWS的Java SDK来设置集群. 展示如何使用Java AWS开发工具包执行此操作的最佳方法是展示完整的示例, ...
8、linux上安装hbase
1.基本信息版本 1.2.4 安装机器三台机器账号 hadoop 源路径 /opt/software/hbase-1.2.4-bin.tar.gz 目标路径 /opt/hbase -> / ...
aws emr 大数据分析_DataOps —使用AWS Lambda和Amazon EMR的全自动，低成本数据管道
aws emr 大数据分析 Progression is continuous. Taking a flashback journey through my 25 years career in in ...
Apache Ranger and AWS EMR Automated Installation Series (3): Windows AD + EMR-Native Ranger
文章目录 1. Solution Overview 1.1 Solution Architecture 1.2 Authentication in Detail 1.3 Authorization i ...
AWS EMR内置Ranger插件使用的IAM Role及其设计策略
AWS EMR提供三种内置的Ranger插件,分别是:S3(EMRFS),Spark,Hive,如果要启用这些插件,需要创建三个特定的IAM Role,以便相关组件能获得适当的权限.对这三种Role的 ...
Apache Ranger and AWS EMR Automated Installation Series (4): OpenLDAP + Open-Source Ranger
文章目录 1. OpenLDAP + Open-Source Ranger Solution Overview 1.1 Solution Architecture 1.2 Ranger in Deta ...
aws fargate_我如何在AWS Fargate上部署＃100DaysOfCloud Twitter Bot
aws fargate After passing my last certification, I asked myself how much time I spent studying cloud ...

在 aws emr 上，将 hbase table A 的数据，对 key 做 hash，写到另外一张 table B

在 aws emr 上，将 hbase table A 的数据，对 key 做 hash，写到另外一张 table B相关推荐

最新文章

热门文章