hive与mysql对比之max、group by、日志分析

前期准备

mysql模型:test_max_date(id int,name varchar(255)，num int,date date)

hive模型： create table test_date_max(id int,name string,rq Date);

insert into table test_date_max values
(1,"1","2020-12-25"),
(2,"1","2020-12-28"),
(3,"2","2020-12-25"),
(4,"2","2020-12-20")
;

需求

查询每个人最新状态

计算逻辑

每个人有多条数据，日期越大，状态越新

计算过程

mysql:

SELECT id,name,date,max(date) from test_max_date group by name ORDER BY id

hive:

select name,max(rq) from  test_date_max group by name;

错误信息说明：在之前的帖子中说过hive groupby的问题。

这里hive中有id,name,日期。id是主键不重复，name是可以重复的，按照name分组，对rq使用max函数，其实是对name去重，返回name每个重复值组中的最大日期

就好比一个公司分了几个部门，部门是确定的，如果是求每个部门年龄最大的，那就是在公司全员信息表中对部门分组，对age求最大。

hive中select 字段和group by 字段必须一一匹配。

如果需要查询完整信息，一下有两种方式（附上sql、结果数据、查询时间）

方式一：
select
a.*
from
test_date_max a
join
(select name,max(rq) as rq from test_date_max group by name) b
on a.rq = b.rq and a.name = b.name

a.id   a.name   a.rq
2   1   2020-12-28
3   2   2020-12-25
Time taken: 118.387 seconds, Fetched: 2 row(s)

方式二：
select
*
from(
select
*,
row_number()over(partition by name order by rq desc) rank
from
test_date_max
)tmp
where rank=1

tmp.id   tmp.name   tmp.rq   tmp.rank
2   1   2020-12-28   1
3   2   2020-12-25   1
Time taken: 68.587 seconds, Fetched: 2 row(s)

计算日志分析——方式一3个job，方式二1个job

方式一：

hive (test)> select >     a.* > from >     test_date_max a>     join >     (select name,max(rq) as rq from  test_date_max group by name) b>     on a.rq = b.rq and a.name = b.name> ;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = admin_20210204130801_0f13ad17-7887-4a32-984d-088b5453617e
Total jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:set mapreduce.job.reduces=<number>
Starting Job = job_1611888254670_2374, Tracking URL = http://hdp6.tydic.xian:8088/proxy/application_1611888254670_2374/
Kill Command = /usr/hdp/2.6.5.0-292/hadoop/bin/hadoop job  -kill job_1611888254670_2374
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2021-02-04 13:08:27,084 Stage-2 map = 0%,  reduce = 0%
2021-02-04 13:08:44,179 Stage-2 map = 100%,  reduce = 0%, Cumulative CPU 3.78 sec
2021-02-04 13:08:59,776 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 7.68 sec
MapReduce Total cumulative CPU time: 7 seconds 680 msec
Ended Job = job_1611888254670_2374
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2021-02-04 13:09:08 Starting to launch local task to process map join;  maximum memory = 954728448
2021-02-04 13:09:09 Dump the side-table for tag: 0 with group count: 4 into file: file:/tmp/admin/a935af81-8bbe-4c2c-b2f5-d3bdaa816d9e/hive_2021-02-04_13-08-01_805_2036786040923555355-1/-local-10005/HashTable-Stage-3/MapJoin-mapfile20--.hashtable
2021-02-04 13:09:09 Uploaded 1 File to: file:/tmp/admin/a935af81-8bbe-4c2c-b2f5-d3bdaa816d9e/hive_2021-02-04_13-08-01_805_2036786040923555355-1/-local-10005/HashTable-Stage-3/MapJoin-mapfile20--.hashtable (356 bytes)
2021-02-04 13:09:09 End of local task; Time Taken: 1.24 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 2 out of 2
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1611888254670_2377, Tracking URL = http://hdp6.tydic.xian:8088/proxy/application_1611888254670_2377/
Kill Command = /usr/hdp/2.6.5.0-292/hadoop/bin/hadoop job  -kill job_1611888254670_2377
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0
2021-02-04 13:09:33,954 Stage-3 map = 0%,  reduce = 0%
2021-02-04 13:09:59,112 Stage-3 map = 100%,  reduce = 0%, Cumulative CPU 3.37 sec
MapReduce Total cumulative CPU time: 3 seconds 370 msec
Ended Job = job_1611888254670_2377
MapReduce Jobs Launched:
Stage-Stage-2: Map: 1  Reduce: 1   Cumulative CPU: 7.68 sec   HDFS Read: 7794 HDFS Write: 140 SUCCESS
Stage-Stage-3: Map: 1   Cumulative CPU: 3.37 sec   HDFS Read: 5282 HDFS Write: 141 SUCCESS
Total MapReduce CPU Time Spent: 11 seconds 50 msec
OK
a.id    a.name  a.rq
2   1   2020-12-28
3   2   2020-12-25
Time taken: 118.387 seconds, Fetched: 2 row(s)

方式二：

hive (test)> select>     *> from(>     select>         *,>         row_number()over(partition by name order by rq desc) rank>     from>         test_date_max>     )tmp> where rank=1> ;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = admin_20210204130834_f1469766-42c9-48cb-9194-2cb506a5ff6a
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:set mapreduce.job.reduces=<number>
Starting Job = job_1611888254670_2376, Tracking URL = http://hdp6.tydic.xian:8088/proxy/application_1611888254670_2376/
Kill Command = /usr/hdp/2.6.5.0-292/hadoop/bin/hadoop job  -kill job_1611888254670_2376
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2021-02-04 13:09:07,610 Stage-1 map = 0%,  reduce = 0%
2021-02-04 13:09:25,459 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 3.71 sec
2021-02-04 13:09:42,161 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.97 sec
MapReduce Total cumulative CPU time: 7 seconds 970 msec
Ended Job = job_1611888254670_2376
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1  Reduce: 1   Cumulative CPU: 7.97 sec   HDFS Read: 10327 HDFS Write: 145 SUCCESS
Total MapReduce CPU Time Spent: 7 seconds 970 msec
OK
tmp.id  tmp.name    tmp.rq  tmp.rank
2   1   2020-12-28  1
3   2   2020-12-25  1
Time taken: 68.587 seconds, Fetched: 2 row(s)

job解析

2021-02-04 16:38:57,881 Stage-1 map = 0%,  reduce = 0%
2021-02-04 16:39:13,646 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 4.81 sec
2021-02-04 16:39:20,976 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 8.78 sec

hive默认引擎是mapreduce，将sql转换成mapreduce任务，mapreduce任务分为三个阶段，map，shuffle，reduce，map阶段是读取文件，shuffle是归并排序，并将shuffle过程中的数据溢写的本地，reduce是读写shuffle过程中的文件二次计算将结果写到磁盘，从上面日志可以看出，map阶段不涉及计算，没有cpu耗时，shuffle有归并排序，有cpu计算，有cpu耗时，只不过是做简单计算，reduce阶段有读取、合并，有cpu计算，cpu耗时