Fanatics BI和数据工程总监Amit Jain (Amit Jain, Director BI, and Data Engineering at Fanatics)

Fanatics generates a tremendous amount of data, and the best way for us to leverage this is to ensure it is accessible in an easy-to-consume format. Executives and other decision-makers may be building long-term plans based on early insights, course-correcting the performance of a particular promotion, or evaluating the results of design changes. Visualizations and reports shouldn’t be clouded by implementation details of our internal data warehouse or overloaded by the firehose of Elasticsearch and Kafka streams.

Fanatics会生成大量数据，而我们利用此数据的最佳方法是确保以易于使用的格式对其进行访问。 高管和其他决策者可能正在根据早期见识来制定长期计划，对特定促销的绩效进行课程纠正，或者评估设计变更的结果。 可视化和报告不应因我们内部数据仓库的实施细节而模糊不清，也不应因Elasticsearch和Kafka流的泛滥而过载。

To accomplish this, we created a scalable visualization framework we call “FanViz.” It allows both business users and engineering teams to develop real-time streaming dashboards without having to be experts in data visualization.

为此，我们创建了一个可扩展的可视化框架，称为“ FanViz”。 它使业务用户和工程团队都可以开发实时流式仪表板，而不必成为数据可视化方面的专家。

We believe this low code approach is the way to go, and the more we simplify the process of creating data visualizations for our users, the more we benefit from unlocking insights from the amazing data we have. FanViz architecture is entirely built on this premise.

我们相信这种低代码方法是必经之路，而且我们越为用户简化了创建数据可视化的过程，就越能从我们拥有的惊人数据中解脱出来，从而受益匪浅。 FanViz体系结构完全在此前提下构建。

To be clear, we do use some of the enterprise business intelligence tools available in the market. But they did not always suit our needs:

需要明确的是，我们确实使用了市场上一些企业商业智能工具。 但是它们并不总是适合我们的需求：

• The available software was not cost-effective, nor responsive enough. Visualizing streaming data from Kafka, Elastic search, and real-time databases was a particular problem.可用软件效率不高，响应速度也不够。 可视化来自Kafka，Elastic搜索和实时数据库的流数据是一个特殊的问题。
• It wasn’t easy to integrate data from multiple sources, such as the above streaming systems and legacy mainframes. We didn’t have one single view of our data.集成来自多个源的数据并非易事，例如上述流传输系统和旧式大型机。 我们没有一个单一的数据视图。
• We weren’t happy with the rendering performance. We want dashboards to display in less than 10 seconds, regardless of the volume of data and even on low latency networks.我们对渲染性能不满意。 我们希望仪表板在不到10秒的时间内显示，无论数据量如何，甚至在低延迟的网络上也是如此。
• Dashboards should be accessible for all users to create. Declarative metadata and simple JSON permits users to create dashboards on their own, without knowing anything about Java, JavaScript, or other web technologies.所有用户都应该可以访问仪表板。 声明性元数据和简单的JSON允许用户自行创建仪表板，而无需了解Java，JavaScript或其他Web技术。

With our visualization platform, we have full control in terms of ease of use, flexibility, and innovation to support mission-critical decisions.

通过我们的可视化平台，我们在易用性，灵活性和创新性方面拥有完全的控制权，以支持关键任务决策。

Fanviz建筑 (Fanviz Architecture)

Like most modern applications, FanViz has a component/microservices-based design. We have a clear separation between the data providers; the compute layer, the caching backend, and the visualization containers. Fanviz is a highly decoupled system with each component developed and maintained separately. The diagram below illustrates the relationship between the FanViz architecture and, the centerpiece of FanViz for end users, its declarative metadata.

像大多数现代应用程序一样，FanViz具有基于组件/微服务的设计。 我们在数据提供者之间有明确的区分； 计算层，缓存后端和可视化容器。 Fanviz是一个高度分离的系统，每个组件分别开发和维护。 下图说明了FanViz架构与它的声明性元数据之间的关系，FanViz是最终用户的核心。

FanViz元数据的详细信息 (Details of FanViz Metadata)

A FanViz user creates dashboards in pure JSON. This JSON defines the complete configuration: from how the dashboard’s HTML and CSS are set, to how different visualizations are formatted. They can define properties like how data is sorted, perform time and date computations, generate calculated metrics, or decide how the interactive menus will render. FanViz has a custom IDE with JSON validation, autocomplete, full tree node copy, and other features of standard IDEs. And just like the other FanViz components, the IDE is also continually improving.

FanViz用户使用纯JSON创建仪表板。 该JSON定义了完整的配置：从如何设置仪表板HTML和CSS，到如何设置不同的可视化格式。 他们可以定义属性，例如如何对数据进行排序，执行时间和日期计算，生成计算的度量标准或决定交互式菜单的呈现方式。 FanViz具有自定义IDE，具有JSON验证，自动完成功能，完整树节点副本以及标准IDE的其他功能。 与其他FanViz组件一样，IDE也在不断改进。

In the following diagram, you can see the IDE and the main components of the FanViz metadata:

在下图中，您可以看到IDE和FanViz元数据的主要组件：

一些元数据组件的描述 (Description of some of the metadata components)

Datasets:

数据集：

A dataset is the source of the data, such as a REST API endpoint that provides JSON data. FanViz can visualize data from any web service that offers well-formed JSON. Computations required to generate the graphs can be done by the source API, or by FanViz itself. Fanviz has a data transformation layer that aggregates data from multiple sources and combines them into a native dataset. External services can also consume this new dataset.

数据集是数据的来源，例如提供JSON数据的REST API端点。 FanViz可以可视化来自任何提供格式正确的JSON的Web服务中的数据。 生成图形所需的计算可以由源API或FanViz本身完成。 Fanviz具有数据转换层，该数据转换层可聚合来自多个来源的数据并将其合并为本地数据集。 外部服务也可以使用此新数据集。

Graphs:

图表：

The second principal component is a graph, a visualization widget defined using a standard JavaScript library. Once a FanViz graph object is created, users may use it just like they would in a tool like Microsoft Excel: all the graph formatting is defined in the JSON metadata. Graphs are standalone widgets, built using libraries like D3.js, Highcharts, and ag-Grid, to name a few. We currently support the visualizations shown in the dashboard below, such as a calendar heat map and a micro chart. We continue to add new visualization types as we improve FanViz.

第二个主要组件是图形，它是使用标准JavaScript库定义的可视化窗口小部件。 创建FanViz图形对象后，用户可以像在Microsoft Excel之类的工具中一样使用它：所有图形格式均在JSON元数据中定义。 图是独立的小部件，使用D3.js，Highcharts和ag-Grid等库构建，仅举几例。 目前，我们支持下面的信息中心中显示的可视化效果，例如日历热图和微图。 随着我们改进FanViz，我们将继续添加新的可视化类型。

Selectors and Derived Metrics :

选择器和派生指标：

Selectors and derived metrics, as is evident from their names, are the selectors used for interactive filtering and navigation in the dashboard. Derived metrics are computed on the client-side, as it can do simple computations in the JavaScript layer. The FanViz server can also perform complex calculations if needed, and the server can integrate other compute libraries from Python and Spark.

从名称中可以明显看出，选择器和派生指标是用于仪表板中的交互式筛选和导航的选择器。 派生的度量标准是在客户端进行计算的，因为它可以在JavaScript层中进行简单的计算。 如果需要，FanViz服务器还可以执行复杂的计算，并且该服务器可以集成来自Python和Spark的其他计算库。

Our selectors consume data sets and dynamically load elements based on the data provided to them. This allows us to apply filtering criteria or query a REST API to control all the elements in them dynamically. No hard coding of element data is required, which makes the end-users of FanViz productive.

我们的选择器使用数据集并根据提供给它们的数据动态加载元素。 这使我们可以应用过滤条件或查询REST API，以动态控制其中的所有元素。 无需对元素数据进行硬编码，从而使FanViz的最终用户工作高效。

Transforms :

转换：

The transform component uses Open Source libraries, like Dust.js, which helps in querying the JSON data coming from the REST API. These objects transform any JSON to the FanViz object model, which allows FanViz to consume a data service without a predefined contract.

转换组件使用Dust.js之类的开源库，该库可帮助查询来自REST API的JSON数据。 这些对象将任何JSON转换为FanViz对象模型，从而使FanViz无需预先定义的合同即可使用数据服务。

Schedulers :

调度程序：

FanViz supports both static and real-time ad hoc dashboards. The scheduler manages their refresh frequency using a 3rd party library, Later.js. There are two types of schedulers, “text” and “cron.” The text scheduler uses an easy-to-read format for less complicated refresh schedules. The cron scheduler allows finer control, using a syntax familiar to users of the Linux cron tool.

FanViz支持静态和实时即席仪表板。 调度程序使用第三方库Later.js管理其刷新频率。 调度程序有两种类型，“文本”和“ cron”。 文本计划程序使用易于读取的格式来处理不太复杂的刷新计划。 cron调度程序可以使用Linux cron工具用户熟悉的语法进行更好的控制。

FanViz仪表板流程： (FanViz Dashboard Flow:)

After the Fanviz Dashboard Metadata is loaded, we parse the JSON to predefined object models, and initialize all the objects:

加载Fanviz仪表板元数据后，我们将JSON解析为预定义的对象模型，并初始化所有对象：

• Schedulers, responsible for refreshing the data on the desired schedule.调度程序，负责按所需调度刷新数据。
• Graphs, which generate the corresponding DOM nodes (in a Thymeleaf template) and CSS styles.图形，生成相应的DOM节点(在Thymeleaf模板中)和CSS样式。
• Selectors, which can be used to target multiple graphs with filtered data.选择器，可用于使用过滤后的数据定位多个图形。
• Datasets, the source of the JSON data to be rendered.数据集，要呈现的JSON数据的来源。

With the objects initialized, we can then render the page. The diagram below describes this sequence:

初始化对象后，我们便可以渲染页面。 下图描述了此顺序：

在FanViz中进行缓存以快速呈现仪表板 (Caching in FanViz for Fast Rendering of Dashboards)

As mentioned before, we have strict performance requirements for FanViz. We gained better rendering performance by using 3rd party rendering libraries, data compression, and multiple layers of caching.

如前所述，我们对FanViz有严格的性能要求。 通过使用第三方渲染库，数据压缩和多层缓存，我们获得了更好的渲染性能。

This multi-layer, distributed, and hierarchal cache system speeds up the rendering of dashboards. A multi-level design balances performance and resource consumption, and the distributed nature make the system tolerate a single-point failure while also allowing flexible deployment.

此多层，分布式和分层缓存系统可加快仪表板的呈现速度。 多级设计平衡了性能和资源消耗，而分布式特性使系统可以承受单点故障，同时还允许灵活部署。

In the cache hierarchy, the fastest level is an intra-process, in-memory cache layer based on Guava Cache. But the capacity of Guava Cache is limited by available memory, often a problem for large dashboards. To overcome this limitation, we introduced a second cache layer, based on Redis.

在缓存层次结构中，最快的级别是基于Guava缓存的进程内内存中缓存层。 但是Guava Cache的容量受可用内存的限制，这对于大型仪表板通常是一个问题。 为了克服此限制，我们引入了基于Redis的第二个缓存层。

We deploy as many Redis servers as needed to reach the desired overall performance, dynamically changing the number as demand changes. To fulfill different user scenarios, we provide different cache policies for the Redis server: pre-cache to keep the keys in the Redis server forever, and various TTL (time to live) settings (1 minute, 1 hour, one day, 30 days, etc.), to evict the keys after the assigned period.

我们会根据需要部署尽可能多的Redis服务器，以达到所需的整体性能，并根据需求的变化动态更改数量。 为了满足不同的用户场景，我们为Redis服务器提供了不同的缓存策略：预缓存以将密钥永久保存在Redis服务器中，以及各种TTL(生存时间)设置(1分钟，1小时，1天，30天)等)，以在指定时间段后退出按键。

To illustrate performance, the following graph rendered 200,000+ points, as well as multiple tool-tips containing additional metrics, in less than 5 seconds:

为了说明性能，下图在不到5秒的时间内显示了200,000+点，以及包含其他指标的多个工具提示：

未来目标 (Future Goals)

Some of our goals for the rest of this year are:

我们今年余下的目标是：

• Continue to add new visualizations and interactivity, especially filtering one visualization from another.继续添加新的可视化和交互性，尤其是将一个可视化与另一个可视化过滤。
• Make FanViz dashboard development more accessible by adding a GUI to edit the properties metadata. Enhanced autocomplete will know which properties can be applied to a particular widget.通过添加GUI编辑属性元数据，使FanViz仪表板开发更易于访问。 增强的自动完成功能将知道哪些属性可以应用于特定的小部件。
• Incrementally fetch from the cache while scrolling, to support 500K+ data points.滚动时从缓存中增量获取数据，以支持500K +数据点。

结论 (Conclusion)

Our goal was to make a visualization framework for Fanatics that democratizes data at scale, across all of our organization. All with great ease-of-use, support for sophisticated metrics and beautiful graphics, plus fast performance. We are very proud of what we have accomplished as a team in the last few years, and are pleased to be able to share it with you.

我们的目标是为Fanatics创建一个可视化框架，以使整个组织内的数据大规模民主化。 所有功能都非常易于使用，支持复杂的指标和精美的图形，并具有快速的性能。 我们为过去几年中团队取得的成就感到自豪，并很高兴能与您分享。

致谢 (Acknowledgments)

We would like to extend our appreciation to Andrea Longo, Sophie Shang, and the Data Engineering teams for contributing to this article.

我们要感谢 Andrea Longo ， Sophie Shang 和数据工程团队为本文做出的贡献。