SparkStreaming可视化之Wisp

-----------------------------------目录---------------------------------------

1、Wisp基础

2、Highchart函数

3、SparkStream和Wisp结合实验

-------------------------------------------------------------------------------

1、Wisp基础

Wisp的gitHub地址：

https://github.com/quantifind/wisp

Wisp的sbt路径依赖

"com.quantifind" %% "wisp" % "0.0.4"

Wisp的maven依赖

<dependency><groupId>com.quantifind</groupId><artifactId>wisp_2.10</artifactId><version>0.0.4</version>
</dependency>

注意:Wisp的开发已经在一年多前就停止了开发了，但是因为能满足一些必要的功能，而且可以把Spark处理后的数据进行Web端的显示，所以不失为一种数据展示的一种手段。

1、wisp的初次见面

第一步：注入依赖，我的sbt的build.sbt如下：

name := "hello"organization := "blog.csdn.net\legotime"version := "0.0.1-SNAPSHOT"scalaVersion := "2.10.4"libraryDependencies ++= Seq("com.quantifind" %% "wisp" % "0.0.4")

第二步：进入控制台界面（sbt console）

运行如下：

import com.quantifind.charts.Highcharts._
line((1 to 10), (1 to 10))

第三步：web端就可以直接显示刚刚我们画的图

不仅仅sbt，而且可以直接在IDEA上注入maven依赖之后，可以在IDEA上操作，数据展示依然在web端。

可以说 Wisp在数据展示也不比Zeppelin差

更多初步操作可以访问：

https://github.com/quantifind/wisp/wiki/Blog

2、 Highchart函数

case class Highchart(val series : scala.collection.Traversable[com.quantifind.charts.highcharts.Series],

val title : scala.Option[com.quantifind.charts.highcharts.Title] = { /* compiled code */ },
val chart : scala.Option[com.quantifind.charts.highcharts.Chart] = { /* compiled code */ },
val colors : scala.Option[scala.Array[com.quantifind.charts.highcharts.Color.Type]] = { /* compiled code */ },
val credits : scala.Option[com.quantifind.charts.highcharts.Credits] = { /* compiled code */ },
val exporting : scala.Option[com.quantifind.charts.highcharts.Exporting] = { /* compiled code */ },
val legend : scala.Option[com.quantifind.charts.highcharts.Legend] = { /* compiled code */ },
val plotOptions : scala.Option[com.quantifind.charts.highcharts.PlotOptions] = { /* compiled code */ },
val subtitle : scala.Option[com.quantifind.charts.highcharts.Title] = { /* compiled code */ },
val setTurboThreshold : scala.Boolean = { /* compiled code */ },
val tooltip : scala.Option[com.quantifind.charts.highcharts.ToolTip] = { /* compiled code */ },
val xAxis : scala.Option[scala.Array[com.quantifind.charts.highcharts.Axis]] = { /* compiled code */ },
val yAxis : scala.Option[scala.Array[com.quantifind.charts.highcharts.Axis]] = { /* compiled code */ }) extends scala.AnyRef with scala.Product with scala.Serializable {

第一部分：series
case class Series(val data : scala.collection.Traversable[com.quantifind.charts.highcharts.Data[_, _]],
val index : scala.Option[scala.Int] = { /* compiled code */ },
val legendIndex : scala.Option[scala.Int] = { /* compiled code */ },
val name : scala.Option[scala.Predef.String] = { /* compiled code */ },
val chart : scala.Option[com.quantifind.charts.highcharts.SeriesType.Type] = { /* compiled code */ },
val visible : scala.Option[scala.Boolean] = { /* compiled code */ },
val color : scala.Option[com.quantifind.charts.highcharts.Color.Type] = { /* compiled code */ },
val xAxis : scala.Option[scala.Predef.String] = { /* compiled code */ },
val yAxis : scala.Option[scala.Predef.String] = { /* compiled code */ },
val __name : scala.Predef.String = { /* compiled code */ }) extends com.quantifind.charts.highcharts.HighchartKey with scala.Product with scala.Serializable {

9个参数：
data:她是一个具有traversable属性的数据集合，内部形式是Data[x,y],其中 x是在坐标的位置，y的数值大小
index:目前没有看出有什么作用
legendIndex:目前没有看出有什么作用
name:这张表的名字和功能（底部和数据内都可见）
chart：使用那个表（(area, areaspline, bar, column, line, pie, scatter（上面一个一个的点）, spline)）
note:pie内其他标记名字不一样

visible：在界面可以是否可以观看到
color："#"+Integer.toHexString(color.getRGB()).substring(2)发现就几种颜色
xAxis:
yAxis:不知道怎么设置
__name：设置没图像：Ensures the object can be cast to Map[String, Any] so we can perform json serialization(_name = "Series",才有图像)

第二部：title
case class Title(
text: String = "", // Override default "Chart title"
align: Option[Alignment.Type] = None,
floating: Option[Boolean] = None,
style: Option[CSSObject] = None,
useHTML: Option[Boolean] = None,
verticalAlign: Option[VerticalAlignment.Type] = None,
x: Option[Int] = None,
y: Option[Int] = None,
var __name: String = "title"
) extends HighchartKey(__name) {
def toServiceFormat =
Map("text" -> text) ++
Map(
"align" -> align,
"floating" -> floating,
"useHTML" -> useHTML,
"verticalAlign" -> verticalAlign,
"x" -> x,
"y" -> y
).flatMap(HighchartKey.flatten) ++
HighchartKey.someStyleToServiceFormat(style)
}
参数分析：
text：title = "hello"：设置图像的title为 "hello" 不设置是没有
align: 是判断在（左，中，右）("left", "center", "right") 不设置是中
floating：true是在图内部的上面，false 是在图的外部上面
style: Option[CSSObject] = None,
useHTML：没效果
verticalAlign：("top", "middle", "bottom") title竖直位置默认是 top
x
y: x，y是自己设定位置
var __name: String = "title"（现在默认是title,恐怕以后还有其他功能）

第三部：Chart
zoomType: Option[Zoom.Type] = Some(Zoom.xy) 默认情况下是 Zoom.xy,也就是上下都放大

第四部分：colors
object Color {
type Type = String
implicit def javaColorToHex(color: java.awt.Color): Type = "#"+Integer.toHexString(color.getRGB()).substring(2)
implicit def javaColorToHexOption(color: java.awt.Color): Option[Type] = Some("#"+Integer.toHexString(color.getRGB()).substring(2))

}
通过设置java里面的颜色来进行画线，但是必须保证前面是没有设置颜色的，如果设置，那么就是取前面设置的颜色
Color.javaColorToHex(java.awt.Color.PINK) //在這里设置颜色的不好之处在于pie全部颜色为一个颜色

第五部分：credits
case class Credits(
enabled: Option[Boolean] = None,
href: String = "", // Override default Highcharts
position: Option[Position] = None,
style: Option[CSSObject] = None,
text: String = "" // Override default Highcharts
) extends HighchartKey("credits")
Credits(enabled = true,href = "good",position = Position(x = 100,y = 100),style = CSSObject(color = Color.javaColorToHex(java.awt.Color.BLUE),fontWeight = "lighter"),text = "what the ")

第六部分：exporting（保存图像配置信息）
case class Exporting(
//buttons
//chartOptions
filename: String = "chart", //保存时候的文件名字
scale: Option[Int] = None,
sourceHeight: Option[Int] = None,
sourceWidth: Option[Int] = None,
_type: Option[String] = None,
url: Option[String] = None,
width: Option[Int] = None
) extends HighchartKey("exporting") {/**/}

第七部分：legend
// No more than 22 members in a case class TODO
case class Legend(
align: Option[Alignment.Type] = None,
backgroundColor: Option[Color.Type] = None,
borderColor: Option[Color.Type] = None,
// borderRadius: Int = 5,
// borderWidth: Int = 2,
enabled: Option[Boolean] = Some(false), // override default
floating: Option[Boolean] = None,
itemDistance: Option[Int] = None,
//itemHiddenStyle
//itemHoverStyle
itemMarginBottom: Option[Int] = None,
itemMarginTop: Option[Int] = None,
//itemStyle
itemWidth: Option[Int] = None,
labelFormat: Option[String] = None, // TODO - format string helpers
//labelFormatter
layout: Option[Layout.Type] = None,
margin: Option[Int] = None,
maxHeight: Option[Int] = None,
//navigation
padding: Option[Int] = None,
reversed: Option[Boolean] = None,
rtl: Option[Boolean] = None, // right-to-left
//shadow
//style
// symbolPadding: Int = 5,
// symbolWidth: Int = 30,
title: Option[String] = None, // todo - css title
// useHTML: Boolean = false,
verticalAlign: Option[VerticalAlignment.Type] = None,
width: Option[Int] = None,
x: Option[Int] = None,
y: Option[Int] = None
) extends HighchartKey("legend") {/**/}

第八部分：plotOptions

// These are overlapping options, but are well over 22 options.
case class PlotOptions(
area: Option[PlotOptionKey] = None,
arearange: Option[PlotOptionKey] = None,
areaspline: Option[PlotOptionKey] = None,
areasplinerange: Option[PlotOptionKey] = None,
bar: Option[PlotOptionKey] = None,
boxplot: Option[PlotOptionKey] = None,
bubble: Option[PlotOptionKey] = None,
column: Option[PlotOptionKey] = None,
columnrange: Option[PlotOptionKey] = None,
errorbar: Option[PlotOptionKey] = None,
funnel: Option[PlotOptionKey] = None,
gauge: Option[PlotOptionKey] = None,
heatmap: Option[PlotOptionKey] = None,
line: Option[PlotOptionKey] = None,
pie: Option[PlotOptionKey] = None,
pyramid: Option[PlotOptionKey] = None,
scatter: Option[PlotOptionKey] = None,
series: Option[PlotOptionKey] = None,
solidgauge: Option[PlotOptionKey] = None,
spline: Option[PlotOptionKey] = None,
waterfall: Option[PlotOptionKey] = None,
var __name: String = PlotOptions.name
) extends HighchartKey(__name) {/**/}

第九个参数：subtitle
case class Title(
text: String = "", // Override default "Chart title"
align: Option[Alignment.Type] = None,
floating: Option[Boolean] = None,
style: Option[CSSObject] = None,
useHTML: Option[Boolean] = None,
verticalAlign: Option[VerticalAlignment.Type] = None,
x: Option[Int] = None,
y: Option[Int] = None,
var __name: String = "title"
)

第十个参数：setTurboThreshold: Boolean = true,
第十一个参数：tooltip: Option[ToolTip] = None,
case class ToolTip(
animation: Option[Boolean] = None,
backgroundColor: Option[Color.Type] = None,
borderColor: Option[Color.Type] = None,
borderRadius: Option[Int] = None,
borderWidth: Option[Int] = None,
// crosshairs
dateTimeLabelFormats: Option[DateTimeFormats] = None, // this has different defaults than the Axis
enabled: Option[Boolean] = None,
followPointer: Option[Boolean] = None,
followTouchMove: Option[Boolean] = None,
footerFormat: Option[String] = None,
//formatter
//headerFormat
hideDelay: Option[Int] = None,
//pointFormat
//positioner
shadow: Option[Boolean] = None,
shared: Option[Boolean] = None,
snap: Option[Int] = None,
//style
useHTML: Option[Boolean] = None,
valueDecimals: Option[Int] = None,
valuePrefix: Option[String] = None,
valueSuffix: Option[String] = None,
xDateFormat: Option[String] = None
) extends HighchartKey("ToolTip") {/**/}

第十二个参数：xAxis
case class Axis(
title: Option[AxisTitle] = Some(AxisTitle()),
allowDecimals: Option[Boolean] = None,
alternateGridColor: Option[Color.Type] = None,
categories: Option[Array[String]] = None,
dateTimeLabelFormats: Option[DateTimeFormats] = None,
endOnTick: Option[Boolean] = None,
//events
// gridLineColor: Color.Type = "#C0C0C0",
// gridLineDashStyle: String = "Solid", // TODO Bundle
// gridLineWidth: Int = 2,
var id: Option[String] = None,
labels: Option[AxisLabel] = None,
lineColor: Option[Color.Type] = None,
lineWidth: Option[Int] = None,
//linkedTo
max: Option[Int] = None,
// maxPadding: Double = 0.01,
min: Option[Int] = None,
// minPadding: Double = 0.01,
minRange: Option[Int] = None,
minTickInterval: Option[Int] = None,
//minor
offset: Option[Int] = None,
opposite: Option[Boolean] = None, // opposite side of chart, ie left / right for y-axis
//plotBands
//plotLines // TODO Kevin wants these
reversed: Option[Boolean] = None,
// showEmpty: Boolean = false,
showFirstLabel: Option[Boolean] = None,
showLastLabel: Option[Boolean] = None,
//startOfWeek
startOnTick: Option[Boolean] = None,
// tickColor: Color.Type = "#C0D0E0",
// TICK STUFF TODO
axisType: Option[AxisType.Type] = None,
var __name: String = "yAxis"
) extends HighchartKey(__name) {/**/}

第十三个参数：yAxis：同上

3、SparkStream和Wisp结合实验

import org.apache.log4j.{Level, Logger}
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}import com.quantifind.charts.Highcharts._
import com.quantifind.charts.highcharts._
import com.quantifind.charts.highcharts.Highchart._object  windowOnStreaming {def main(args: Array[String]) {/*** this is test  of SparkStreaming  and wisp*/Logger.getLogger("org.apache.spark").setLevel(Level.ERROR)Logger.getLogger("org.eclipse.jetty.Server").setLevel(Level.OFF)val conf = new SparkConf().setAppName("the Window operation of SparK Streaming").setMaster("local[2]")val sc = new SparkContext(conf)val ssc = new StreamingContext(sc,Seconds(10))//set the Checkpoint directoryssc.checkpoint("/Res")//get the socket Streaming dataval socketStreaming = ssc.socketTextStream("master",9999)val data = socketStreaming.map(x =>(x,1)).reduceByKey(_+_)data.foreachRDD( rdd =>{val topWords=rdd.top(20)(Ordering.by[(String,Int),Int](_._2))val (words,counts)=topWords.unzipvar hc=Highchart(Series(counts.zipWithIndex.map({case(x,y)=>(y,x)}), chart = SeriesType.column, name = "Word Counts", color = "#377"))/*** 设置参数 可以按如下模板* var hc=Highchart(Series(counts.zipWithIndex.map({case(x,y)=>(y,x)}), chart = SeriesType.column,name = "Word Counts", __name = "series"),Title(text = "hello",align = "center",floating = false,useHTML = true,verticalAlign = "top",x = 1,y = 1,__name = "title"),Chart(zoomType = Zoom.xy),Color.javaColorToHex(java.awt.Color.magenta),Credits(enabled = true,href = "good",position = Position(x = 100,y = 100),style = CSSObject(color = Color.javaColorToHex(java.awt.Color.BLUE),fontWeight = "lighter"),text = "what the "),Exporting(filename = "hi"),Legend(backgroundColor = Color.javaColorToHex(java.awt.Color.darkGray)),PlotOptions(area = PlotOptionKey(borderWidth = 12,groupPadding = 10,pointPadding = 14,stacking = Stacking.normal)),Title(text = "I am lego"),true,ToolTip(animation = true,backgroundColor = Color.javaColorToHex(java.awt.Color.cyan)),Axis( title = AxisTitle(text = "i am X "),lineColor = Color.javaColorToHex(java.awt.Color.BLACK)),Axis( title = AxisTitle(text = "i am Y "),lineColor = Color.javaColorToHex(java.awt.Color.gray)))*/hc = hc.copy(xAxis = hc.xAxis.map {axisArray => axisArray.map { _.copy(axisType = Option("category"), categories = Option(words.toArray)) }})plot(hc)})ssc.start()ssc.awaitTermination()}}

结果如下：

每个batchInterval 一个图

SparkStreaming可视化之Wisp相关推荐

sparkstreaming 读取mysql_SparkStreaming读取Kafka的两种方式
本文主要从以下几个方面介绍SparkStreaming读取Kafka的两种方式: 一.SparkStreaming简介二.Kafka简介三.Redis简介(可用于保存历史数据或偏移量数据) 四.S ...
通过可视化来了解你的Spark应用程序
学长推荐阅读文章,先mark到博客里,原文地址http://www.csdn.net/article/2015-07-08/2825162#rd,所有权力归原作者所有. allowtransparen ...
Kafka结合Spark-streaming 的两种连接方式(AWL与直连)
kafka结合spark-streaming的用法及说明之前博客有些,这里就不赘述了. 这篇文章说下他们结合使用的两种连接方式.(AWL与直连) 先看一张图: 这是kafka与streaming结合的 ...
基于大数据的消费者能力的数据可视化平台设计与实现
摘要大数据时代的来临,为了提高企业核心竞争力,首先需要解决电子商务大数据领域中,本设计主要为解决商家无法找到精准客户的问题,通过科学的方式对目标市场中形态各异的消费者进行精细的划分,根据每个用户不 ...
实时直播流量统计分析及可视化系统————老子明天不加班系列
实时直播流量统计分析及可视化系统系统介绍本系统主要使用了Flume.kafka.Spark.Mysql数据库.python的Django框架以及echarrts来实现整个功能本系统按照实现功能不 ...
新闻网日志实时分析可视化系统项目
本次项目是基于企业大数据经典案例项目(大数据日志分析),全方位.全流程讲解大数据项目的业务分析.技术选型.架构设计.集群规划.安装部署.整合继承与开发和web可视化交互设计. 项目代码托管于gith ...
深度阐述数据建模及可视化系统技术方案
1.系统概述数据建模及可视化系统系统是一站式全链路数据生命周期管家,帮助用户管理数据资产并挖掘价值.平台提供多源异构的数据采集模块.实时/离线计算框架,简洁易用的开发环境和平台接口,为政府机构.企业 ...
【置顶】利用 NLP 技术做简单数据可视化分析教程（实战）
置顶本人决定将过去一段时间在公司以及日常生活中关于自然语言处理的相关技术积累,将在gitbook做一个简单分享,内容应该会很丰富,希望对你有所帮助,欢迎大家支持. 内容介绍如下你是否曾经在租房时因 ...
windows安装MongoDB环境以及在pycharm中配置可视化插件
安装MongoDB数据库参考这里安装PyMongo python3 -m pip3 install pymongo指定版本 python3 -m pip3 install pymongo==3.5 ...
只要5分钟用数据可视化带你看遍11月份新闻热点事件
2017年11月份已经离我们而去,在过去的11月份我们也许经历了双十一的剁手,也可能亲眼看见了别人剁手.11月份的北京大兴区发生了"11·18"重大火灾,国内多家幼儿园也多次上了头 ...

SparkStreaming可视化之Wisp

SparkStreaming可视化之Wisp相关推荐

最新文章

热门文章