hive经典面试题1--根据主播上播、下播时间戳求播出时长

现有如下数据，主播id、房间号、播出的批次号，每个批次号进出房间的时间戳、分区时间：

accid	room_id	batch_id	dateline	dt
5eb705796a8971881mFJmd	223	1629047963024	1629047965592	08-18
5eb705796a8971881mFJmd	223	1629047963024	1629047975336	08-18
8c15215f733b8e7f1m90uT	264	1629273804000	1629273815723	08-18
8c15215f733b8e7f1m90uT	264	1629273804000	1629274407664	08-18

问题：

每一次直播都有一个上播和下播，每个房间里，同一个批次号会有两条数据，分别记录了上播和下播时间，求每个主播的播出时长？

分析：

通过上面的数据，可以清晰的看出，同一个批次号里面，只要拿后一条时间戳减前一条时间戳就是每个主播的播出时长

我们来看看具体怎么实现：

方法一：

通过row_number()函数，根据主播id、房间号、播出的批次号、分区进行分组，根据时间戳进行排序得到每个主播在每个房间的上播和下播排序数据。拍完序后，怎么拿到后一条时间戳跟前一条时间戳相减呢，那只能自关联，根据排序的序号进行差1关联，取第一条数据。因为我们做了差值关联，只会有第一条数据关联上了后一条时间戳。
查看row_number()函数的语法，可以点击hive分析函数row_number()的应用

spark-sql> with test1 as > (select '5eb705796a8971881mFJmd' as accid,223 as room_id,1629047963024 as batch_id,1629047965 as dateline,'08-18' as dt> union all> select '5eb705796a8971881mFJmd' as accid,223 as room_id,1629047963024 as batch_id,1629047975 as dateline,'08-18' as dt> union all> select '8c15215f733b8e7f1m90uT' as accid,264 as room_id,1629273804000 as batch_id,1629273815 as dateline,'08-18' as dt> union all> select '8c15215f733b8e7f1m90uT' as accid,264 as room_id,1629273804000 as batch_id,1629274407 as dateline,'08-18' as dt)> select >   accid,>   room_id,>   batch_id,>   dateline_end-dateline_start as dateline,>   dt> from > (select >   t1.accid,>   t1.room_id,>   t1.batch_id,>   t1.dateline as dateline_start,>   t2.dateline as dateline_end,>   t1.rank,>   t1.dt> from > (select>   accid,>   room_id,>   batch_id,>   row_number() over(distribute by accid,room_id,batch_id,dt sort by dateline asc) as rank,>   dateline,>   dt> from test1) t1> > left join> (select>   accid,>   room_id,>   batch_id,>   row_number() over(distribute by accid,room_id,batch_id,dt sort by dateline asc) as rank,>   dateline,>   dt> from test1) t2> on t1.accid=t2.accid and t1.room_id=t2.room_id and t1.batch_id=t2.batch_id and t1.rank+1=t2.rank) t3> where t3.rank=1;
5eb705796a8971881mFJmd  223     1629047963024   10      08-18
8c15215f733b8e7f1m90uT  264     1629273804000   592     08-18
Time taken: 12.039 seconds, Fetched 2 row(s)

方法二：

上面通过row_number()函数自关联的方法，逻辑还是很清晰的，但是自关联这种方法在性能的角度来说，并不可取，也不是最优的方法。我们来看看有没有其他的方法呢？既然是取前一条和后一条时间戳，那我们可不可以通过最大值和最小值的方法去做呢？显然是可以的。只要根据主播id、房间号、播出的批次号、分区进行分组，对时间戳取最大值和最小值，那么我们就可以很清楚的知道上播和下播的时间戳了，一相减就是我们想要的播出时长。

spark-sql> with test1 as > (select '5eb705796a8971881mFJmd' as accid,223 as room_id,1629047963024 as batch_id,1629047965 as dateline,'08-18' as dt> union all> select '5eb705796a8971881mFJmd' as accid,223 as room_id,1629047963024 as batch_id,1629047975 as dateline,'08-18' as dt> union all> select '8c15215f733b8e7f1m90uT' as accid,264 as room_id,1629273804000 as batch_id,1629273815 as dateline,'08-18' as dt> union all> select '8c15215f733b8e7f1m90uT' as accid,264 as room_id,1629273804000 as batch_id,1629274407 as dateline,'08-18' as dt)> select >   accid,>   room_id,>   batch_id,>   dateline_end-dateline_start as dateline,>   dt> from > (select>   accid,>   room_id,>   batch_id,>   min(dateline) as dateline_start,>   max(dateline) as dateline_end,>   dt> from test1> group by accid,room_id,batch_id,dt) t1;
5eb705796a8971881mFJmd  223     1629047963024   10      08-18
8c15215f733b8e7f1m90uT  264     1629273804000   592     08-18
Time taken: 4.046 seconds, Fetched 2 row(s)

方法三：

以上两种方法都可以满足我们的需求，但是有没有思考过，有没有这样一个函数通过什么分组啊、排序啊，把后一条时间戳填充到前面来呢？其实是有这样函数的，只是我们不知道而已。
查看这个函数的语法，可以点击hive分析函数lead()和lag()的应用
在这里怎么用呢？lead() over()函数根会据时间排序把后一条时间戳填充到前面来形成一个新的字段。我们通过row_number()函数正序，lead() over()函数也正序，那么后一条时间戳正好会填充到第一条时间戳后面，取第一条数据就是完整的上下播了，再根据两个时间戳相减就可以得到主播的播出时长了。

spark-sql> with test1 as > (select '5eb705796a8971881mFJmd' as accid,223 as room_id,1629047963024 as batch_id,1629047965 as dateline,'08-18' as dt> union all> select '5eb705796a8971881mFJmd' as accid,223 as room_id,1629047963024 as batch_id,1629047975 as dateline,'08-18' as dt> union all> select '8c15215f733b8e7f1m90uT' as accid,264 as room_id,1629273804000 as batch_id,1629273815 as dateline,'08-18' as dt> union all> select '8c15215f733b8e7f1m90uT' as accid,264 as room_id,1629273804000 as batch_id,1629274407 as dateline,'08-18' as dt)> select >   accid,>   room_id,>   batch_id,>   dateline_end-dateline_start as dateline,>   dt> from > (select>   accid,>   room_id,>   batch_id,>   row_number() over(distribute by accid,room_id,batch_id,dt sort by dateline asc) as rank,>   dateline as dateline_start,>   lead(dateline,1) over(distribute by accid,room_id,batch_id,dt sort by dateline asc) as dateline_end,>   dt> from test1) t1> where t1.rank=1;
5eb705796a8971881mFJmd  223     1629047963024   10      08-18
8c15215f733b8e7f1m90uT  264     1629273804000   592     08-18
Time taken: 6.558 seconds, Fetched 2 row(s)

总结：

在三种方法都满足需求的时候，你会选择那种方法呢？
方法一：在我们没有什么思路的时候，只要这个方法能实现，也不失为一个好方法啊
方法二：对于这种一个头一个尾的需求，这种方法确实是一个很特殊，很有格调的方法，很简洁
方法三：运用了两个分析函数，很有逼格。这种方法对于求上播、下播的播出时长、用户的页面停留时长以及需要上一条和下一条数据做差值的时候，会更加灵活，方便

所以你会选择哪种方法呢？

关注博主，你会看到更多、更精彩的大数据知识