Hive partition prune Failed

昨天发现线上的HiveQuery：

select * from db1.t1where dt between to_char(getdate(‘variables’,’-40’),’yyyymmdd’) and ‘variables’and hour=’xxx’(其中t1 partitioned bydt,hour)

不能进行partition prune导致执行效率非常的差，问题出现在哪里呢？

把To_Char函数的代码翻出来就一目了然了：

@UDFType(deterministic= false)
@Description(name= "to_char",value = "_FUNC_(date, pattern)  converts a string with yyyy-MM-dd HH:mm:sspattern " +"to a string with givenpattern.\n"+"_FUNC_(datetime, pattern)  converts a string with yyyy-MM-dd pattern" +"to a string with givenpattern.\n"+"_FUNC_(number [,format]) convertsa number to a string\n",extended = "Example:\n"+" > SELECT to_char('2011-05-1110:00:12'.'yyyyMMdd') FROM src LIMIT 1;\n"+"20110511\n"
)

注意到这个函数是一个“非确定性”函数，Hive在做partition prune时考虑三点不进行过滤处理：

1.如果是逻辑函数的话，若所有的child节点都为null则忽略

2.非确定性函数忽略

3.其他情况，只要有child节点为null则忽略

而这里的to_char正是第二种情况，这里我们自己写了个确定性UDF来解决该问题

注：另外一个类似的Case HIVE-1173

转载于:https://blog.51cto.com/boylook/1365734

Hive partition prune Failed相关推荐

报错：org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
报错环境: CDH中集成的hive服务,启动报错,所以初始化一下元数据. 配置文件:/etc/hive/conf hive-site.xml 命令目录:/opt/cloudera/parcels/CD ...
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version, Cause:Table ‘hive.
CDH报错:org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version, Cause:Table ...
Hive初始化问题 Failed to get schema version.
Hive初始化问题 Failed to get schema version. 要细看自己的问题是什么! 引起Failed to get schema version.的原因有很多,我遇到的如下: 1 ...
hive 报错 FAILED: SemanticException Cannot find class ‘com.hadoop.mapred.DeprecatedLzoTextInputFormat’
hive 报错 FAILED: SemanticException Cannot find class 'com.hadoop.mapred.DeprecatedLzoTextInputFormat' ...
fdisk时WARNING: Re-reading the partition table failed with error 16: 设备或资源忙的问题解决方案
WARNING: Re-reading the partition table failed with error 16: 设备或资源忙现象:划分磁盘有警告, WARNING: Re-reading ...
[hive 报错]:FAILED: SemanticException Line 0:-1 Partition not found
问题:FAILED: SemanticException Line 0:-1 Partition not found ''19'' 这种情况一般是你创建的表是多级分区(比如day,hour),若操作时 ...
（详细）解决hive报错FAILED: SemanticException Cartesian products are disabled for safety的问题
在使用hive-2.3.3执行TPC-H benchmark时,遇到hive报错.而且这个错误不是以Java异常栈的形式跑出的,很可能被忽略: FAILED: SemanticException Ca ...
hive partition简介
一.背景 1.在Hive Select查询中一般会扫描整个表内容,会消耗很多时间做没必要的工作.有时候只需要扫描表中关心的一部分数据,因此建表时引入了partition概念. 2.分区表指的是在创建表 ...
解决hive报错FAILED: SemanticException Cartesian products are disabled for safety的问题
1.先看问题: FAILED: SemanticException Cartesian products are disabled for safety reasons. If you know wh ...

Hive partition prune Failed

Hive partition prune Failed相关推荐

最新文章

热门文章