SparkStreaming案例：模拟百度热搜排行榜

pom文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.cll</groupId><artifactId>day01</artifactId><version>1.0-SNAPSHOT</version><!-- 指定仓库位置，依次为aliyun、cloudera和jboss仓库 --><repositories><repository><id>aliyun</id><url>http://maven.aliyun.com/nexus/content/groups/public/</url></repository><repository><id>cloudera</id><url>https://repository.cloudera.com/artifactory/cloudera-repos/</url></repository><repository><id>jboss</id><url>http://repository.jboss.com/nexus/content/groups/public</url></repository></repositories><properties><maven.compiler.source>1.8</maven.compiler.source><maven.compiler.target>1.8</maven.compiler.target><encoding>UTF-8</encoding><scala.version>2.11.8</scala.version><scala.compat.version>2.11</scala.compat.version><hadoop.version>2.7.4</hadoop.version><spark.version>2.2.0</spark.version></properties><dependencies><dependency><groupId>org.scala-lang</groupId><artifactId>scala-library</artifactId><version>${scala.version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_2.11</artifactId><version>${spark.version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-sql_2.11</artifactId><version>${spark.version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-hive_2.11</artifactId><version>${spark.version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-hive-thriftserver_2.11</artifactId><version>${spark.version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming_2.11</artifactId><version>${spark.version}</version></dependency><!-- <dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming-kafka-0-8_2.11</artifactId><version>${spark.version}</version></dependency>--><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming-kafka-0-10_2.11</artifactId><version>${spark.version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-sql-kafka-0-10_2.11</artifactId><version>${spark.version}</version></dependency><!--<dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>2.6.0-mr1-cdh5.14.0</version></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-client</artifactId><version>1.2.0-cdh5.14.0</version></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-server</artifactId><version>1.2.0-cdh5.14.0</version></dependency>--><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>2.7.4</version></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-client</artifactId><version>1.3.1</version></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-server</artifactId><version>1.3.1</version></dependency><dependency><groupId>com.typesafe</groupId><artifactId>config</artifactId><version>1.3.3</version></dependency><dependency><groupId>mysql</groupId><artifactId>mysql-connector-java</artifactId><version>5.1.38</version></dependency></dependencies><build><sourceDirectory>src/main/scala</sourceDirectory><testSourceDirectory>src/test/scala</testSourceDirectory><plugins><!-- 指定编译java的插件 --><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><version>3.5.1</version></plugin><!-- 指定编译scala的插件 --><plugin><groupId>net.alchim31.maven</groupId><artifactId>scala-maven-plugin</artifactId><version>3.2.2</version><executions><execution><goals><goal>compile</goal><goal>testCompile</goal></goals><configuration><args><arg>-dependencyfile</arg><arg>${project.build.directory}/.scala_dependencies</arg></args></configuration></execution></executions></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.18.1</version><configuration><useFile>false</useFile><disableXmlReport>true</disableXmlReport><includes><include>**/*Test.*</include><include>**/*Suite.*</include></includes></configuration></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-shade-plugin</artifactId><version>2.3</version><executions><execution><phase>package</phase><goals><goal>shade</goal></goals><configuration><filters><filter><artifact>*:*</artifact><excludes><exclude>META-INF/*.SF</exclude><exclude>META-INF/*.DSA</exclude><exclude>META-INF/*.RSA</exclude></excludes></filter></filters><transformers><transformerimplementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"><mainClass></mainClass></transformer></transformers></configuration></execution></executions></plugin></plugins></build>
</project>

代码：

object wordcount4 {/*演示SparkStreaming的应用* 模拟百度热搜排行榜* 统计最近10s的热搜词，每隔5s计算一次----涉及到窗口，需要使用reduceBykeyAndWindow进行统计* */def main(args: Array[String]): Unit = {//创建streamingContextval conf: SparkConf = new SparkConf().setAppName("wc").setMaster("local[*]")val sc = new SparkContext(conf)val ssc = new StreamingContext(sc, Seconds(5))sc.setLogLevel("WARN")//Seconds(5)表示每个批次间隔5s//读取数据val Dstream: ReceiverInputDStream[String] = ssc.socketTextStream("node03",9999)val wordOne = Dstream.flatMap(_.split(" ")).map((_,1))//统计窗口内的数据val resDstream: DStream[(String, Int)] = wordOne.reduceByKeyAndWindow((a:Int, b:Int)=>a+b,Seconds(10),Seconds(5))//对统计出来的数据进行排序//!!!对dstream的操作，显示转化成为对rdd的操作，在操作中进行排序，因为dstream本身没有排序方法val sortedDStream = resDstream.transform(rdd => {//根据数量进行排序val sortedRDD: RDD[(String, Int)] = rdd.sortBy(_._2, false)val top3Arr: Array[(String, Int)] = sortedRDD.take(3)println("========top3===========")top3Arr.foreach(println)println("========top3===========")sortedRDD})//输出结果//注意：尽管上面的代码已经有输出打印，但是不是DStream的ouput输出操作，所以下面的output代码不能省略sortedDStream.print()//开启ssc.start()ssc.awaitTermination()}}

SparkStreaming案例：模拟百度热搜排行榜相关推荐

springboot+redis实现微博热搜排行榜
springboot+redis实现微博热搜排行榜技术模拟思路: 采用26个英文字母来实现排行,随机为每个字母生成一个随机数作为score 为了更好的体验,先做几件事: 先初始化1个月的历史数据定 ...
python爬取百度标题_Python爬取百度热搜和数据处理
一.主题式网络爬虫设计方案 1.主题式网络爬虫名称:爬取百度热搜 2.主题式网络爬虫爬取的内容与数据特征分析:百度热搜排行,标题,热度 3.主题式网络爬虫设计方案概述:先搜索网站,查找数据并比对然后再 ...
php采集百度热搜,python 爬取百度热搜
###导入模块 import requests from lxml import etree import requests,json ###网址 url="http://top.baidu ...
各大平台热搜排行榜原型
前言:腾讯.新浪微博等平台都有一个热搜的功能,目的是让用户看到最近一段时间网友们都在搜索什么,以便实时掌握热点资讯.热搜排行榜的算法其实非常简单,用C语言就可以写出来. 编程思想:每个被搜索的词条都是 ...
微信推出热搜排行榜，微博压力山大？
［微信热词功能］无论是iOS还是Android用户,在手机微信上点击上方的搜索框,然后点击［资讯］,就能看到最近热门的关键词以及对应的微信指数.点击关键词,就能看到相关的文章. 这一热词功能,不难让人 ...
关闭谷歌google右侧百度热搜（下方无偿链接），亲测有效
https://www.aliyundrive.com/s/GKs1Ahnz1vv 点击链接保存,或者复制本段内容,打开「阿里云盘」APP 1.找到下方文件夹,下载 2.打开谷歌设置---扩展程序然后 ...
一键屏蔽百度热搜，专注工作！
最近因为网络问题,用Google不稳定,只能用百度了,但是令人非常头疼的一点就是百度每次搜索时右侧都会出现[百度热搜],这个东西怎么说呢,能够实时看到时事新闻,但是这在很多时候会分散我们的注意力,为了 ...
火狐浏览器如何关闭右侧的“百度热搜”
目录问题描述: 问题解决: 问题描述: 不知道什么时候火狐浏览器新加了"百度热搜"的功能,每次使用火狐浏览器搜索时,总是会被右侧的热搜吸引,就会点进去看,而忘记了自己最初的目的. ...
AdBlock插件，在谷歌浏览器中屏蔽百度热搜或其他广告
介绍通过该插件可以将谷歌浏览器中的任何网址中的特定元素过滤掉,比如百度搜索后右侧的百度热搜,看着就很烦人,每次都忍不住看两眼,少则浪费十几秒,多则浪费个把小时.同时也可以屏蔽其他网站中的元素,让你在 ...

SparkStreaming案例：模拟百度热搜排行榜

SparkStreaming案例：模拟百度热搜排行榜相关推荐

最新文章

热门文章