Spark Streaming的窗口操作

2019独角兽企业重金招聘Python工程师标准>>>

Spark Streaming的窗口操作博客分类： spark

Spark Streaming的Window Operation可以理解为定时的进行一定时间段内的数据的处理。

不要怪我语文不太好。。下面上原理图吧，一图胜千言：

滑动窗口在监控和统计应用的场景比较广泛，比如每隔一段时间(2s)统计最近3s的请求量或者异常次数，根据请求或者异常次数采取相应措施

如图：

1. 红色的矩形就是一个窗口，窗口hold的是一段时间内的数据流。

2.这里面每一个time都是时间单元，在官方的例子中，每个窗口大小(window size)是3时间单元 (time unit), 而且每隔2个单位时间，窗口会slide（滑动）一次。

所以基于窗口的操作，需要指定2个参数：

window length - The duration of the window (3 in the figure)
slide interval - The interval at which the window-based operation is performed (2 in the figure).

1.窗口大小，个人感觉是一段时间内数据的容器。

2.滑动间隔，就是我们可以理解的cron表达式吧。 - -！

举个例子吧：

还是以最著名的wordcount举例，每隔10秒，统计一下过去30秒过来的数据。

[java] view plain copy

// Reduce last 30 seconds of data, every 10 seconds
val windowedWordCounts = pairs.reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))

这里的paris就是一个MapedRDD，类似(word,1)

[java] view plain copy

reduceByKeyAndWindow // 这个类似RDD里面的reduceByKey，就是对RDD应用function

在这里是根据key，对至进行聚合，然后累加。

下面粘贴一下它的API，仅供参考：

window(windowLength, slideInterval)	Return a new DStream which is computed based on windowed batches of the source DStream.
countByWindow(windowLength,slideInterval)	Return a sliding window count of elements in the stream.
reduceByWindow(func, windowLength,slideInterval)	Return a new single-element stream, created by aggregating elements in the stream over a sliding interval using func. The function should be associative so that it can be computed correctly in parallel.
reduceByKeyAndWindow(func,windowLength, slideInterval, [numTasks])	When called on a DStream of (K, V) pairs, returns a new DStream of (K, V) pairs where the values for each key are aggregated using the given reduce function func over batches in a sliding window. Note: By default, this uses Spark's default number of parallel tasks (2 for local machine, 8 for a cluster) to do the grouping. You can pass an optional `numTasks` argument to set a different number of tasks.
reduceByKeyAndWindow(func, invFunc,windowLength, slideInterval, [numTasks])	A more efficient version of the above `reduceByKeyAndWindow()` where the reduce value of each window is calculated incrementally using the reduce values of the previous window. This is done by reducing the new data that enter the sliding window, and "inverse reducing" the old data that leave the window. An example would be that of "adding" and "subtracting" counts of keys as the window slides. However, it is applicable to only "invertible reduce functions", that is, those reduce functions which have a corresponding "inverse reduce" function (taken as parameterinvFunc. Like in `reduceByKeyAndWindow`, the number of reduce tasks is configurable through an optional argument.
countByValueAndWindow(windowLength,slideInterval, [numTasks])	When called on a DStream of (K, V) pairs, returns a new DStream of (K, Long) pairs where the value of each key is its frequency within a sliding window. Like in `reduceByKeyAndWindow`, the number of reduce tasks is configurable through an optional argument.

Output Operations

When an output operator is called, it triggers the computation of a stream. Currently the following output operators are defined:

print()	Prints first ten elements of every batch of data in a DStream on the driver.
foreachRDD(func)	The fundamental output operator. Applies a function, func, to each RDD generated from the stream. This function should have side effects, such as printing output, saving the RDD to external files, or writing it over the network to an external system.
saveAsObjectFiles(prefix, [suffix])	Save this DStream's contents as a `SequenceFile` of serialized objects. The file name at each batch interval is generated based on prefix and suffix: "prefix-TIME_IN_MS[.suffix]".
saveAsTextFiles(prefix, [suffix])	Save this DStream's contents as a text files. The file name at each batch interval is generated based on prefix and suffix: "prefix-TIME_IN_MS[.suffix]".
saveAsHadoopFiles(prefix, [suffix])	Save this DStream's contents as a Hadoop file. The file name at each batch interval is generated based on prefix and suffix: "prefix-TIME_IN_MS[.suffix]".

原创，转载请注明出处 http://blog.csdn.net/oopsoom/article/details/23776477

转载于:https://my.oschina.net/xiaominmin/blog/1599578

Spark Streaming的窗口操作相关推荐

Spark Streaming的IDEA操作在spark操作的差别和解决
Spark Streaming的IDEA操作博客https://blog.csdn.net/qq_43688472/article/details/86499291 这里就不重复操作了 [hadoo ...
Spark Streaming中的操作函数分析
参考文章:http://blog.csdn.net/dabokele/article/details/52602412 根据Spark官方文档中的描述,在Spark Streaming应用中,一个DS ...
通过Spark Streaming的window操作实战模拟热点搜索词案例实战
本博文主要内容包括: 1.在线热点搜索词实现解析 2.SparkStreaming 利用reduceByKeyAndWindow实现在线热点搜索词实战一:在线热点搜索词实现解析背景描述:在社交网络 ...
Spark Streaming简介 (三十四)
Spark Streaming简介 Spark Streaming 是 Spark 提供的对实时数据进行流式计算的组件.它是 Spark 核心 API 的一个扩展,具有吞吐量高.容错能力强的实时流数据 ...
021 Spark Streaming
1.简介 Spark Streaming抽象.架构与原理 StreamingContext 是 Spark Streaming 程序的入口,其指定sparkConf.确定DStream生成的间隔.设定 ...
实验十八 Spark实验：Spark Streaming
实验指导: 18.1 实验目的 1．了解Spark Streaming版本的WordCount和MapReduce版本的WordCount的区别: 2．理解Spark Streaming的工作流程 ...
Spark Streaming
spark streaming介绍 Spark streaming是Spark核心API的一个扩展,它对实时流式数据的处理具有可扩展性.高吞吐量.可容错性等特点.我们可以从kafka.flume.wi ...
Spark Streaming源码解读之No Receivers彻底思考
本期内容 : Direct Acess Kafka Spark Streaming接收数据现在支持的两种方式: 01. Receiver的方式来接收数据,及输入数据的控制 02. No Receive ...
Spark Streaming介绍，DStream,DStream相关操作(来自学习资料)
一. Spark Streaming介绍 1． SparkStreaming概述 1.1．什么是Spark Streaming Spark Streaming类似于Apache Storm,用于流式 ...

Spark Streaming的窗口操作

Output Operations

Spark Streaming的窗口操作相关推荐

最新文章

热门文章