This article is a group effort, with contributions from the authors of our InfoVis’19 paper.

本文是我们集体努力的成果，我们的 InfoVis'19论文 的作者 对此 做出了贡献 。

TLDR: A new generation of visualization authoring systems has emerged in the past few years. Designed to support a common goal, these systems vary in terms of their visualization models, system architectures, and user interfaces. What are the strengths and weaknesses of these systems? How do we choose the right tools to build our visualization? We propose to use critical reflections as a method to compare these systems.

TLDR：过去几年中出现了新一代的可视化创作系统。 为了支持一个共同目标而设计，这些系统在可视化模型，系统架构和用户界面方面有所不同。 这些系统的优点和缺点是什么？ 我们如何选择正确的工具来构建可视化？ 我们建议使用临界反射作为比较这些系统的方法。

Chris is a visualization enthusiast, with a background in graphic design and some basic knowledge of data processing in Excel. An avid follower of numerous visualization design blogs and podcasts, Chris likes to apply the design knowledge and practices learned from these sources when making data visualizations. Template based tools like Excel charts are not flexible enough for novel visualization designs. Programming toolkits like D3 are powerful, but they also seem quite daunting to someone like Chris who has never coded before. Drawing tools like Adobe Illustrator are great for sketching, but lack functionality for encoding data as visual properties of graphical elements.

克里斯(Chris)是一位可视化爱好者，具有图形设计背景和一些Excel中数据处理的基本知识。 克里斯是众多可视化设计博客和播客的忠实拥护者，克里斯喜欢在进行数据可视化时运用从这些资源中学到的设计知识和实践。 Excel图表等基于模板的工具不够灵活，无法进行新颖的可视化设计。 像D3这样的编程工具包功能强大，但是对于像Chris这样从未使用过编码的人来说，它们似乎也令人望而生畏。 诸如Adobe Illustrator之类的绘图工具非常适合草绘，但是缺少将数据编码为图形元素的可视属性的功能。

What Chris needs is a tool that provides the flexibility to go beyond chart templates, does not require programming expertise, yet supports automated mapping from data to visuals in a WYSIWYG (What You See Is What You Get) interface. Such a tool would be useful in many scenarios: sketching mockups for brainstorming design ideas, step-by-step explanation of how a visualization works, quick figure production for an article, to name a few. To address this need, a new generation of interactive visualization authoring systems has emerged over the past few years, which includes Lyra, Data Illustrator, Charticulator, Data-Driven Guides, DataInk, Datylon, and more.

克里斯需要的是一种工具，该工具可提供超越图表模板的灵活性，不需要编程专家，但支持在所见即所得(所见即所得)界面中将数据自动映射到视觉效果。 这样的工具在许多情况下都将很有用：草绘用于集思广益的设计思想的模型，逐步说明可视化的工作原理，快速生成文章的图形，仅举几例。 为了满足这种需求，在过去几年中出现了新一代的交互式可视化创作系统，其中包括Lyra ， Data Illustrator ， Charticulator ， Data-Driven Guides ， DataInk ， Datylon等。

At first glance, these systems appear quite similar: they all try to enable the creation of expressive visualizations in a graphical user interface without programming. Upon closer inspection and hands-on experimentation, however, these systems vary significantly in terms of functionality and user interface. For example, some provide a Pen Tool for drawing free-from shapes, which is absent in other systems; data binding is done through drag-and-drop in some systems, and through drop-down menus in others. As these systems were released, people on social media began to wonder about how they compare to each other. Andy Kirk, for example, called this an arms race of datavis tools, and later talked about these tools on a Data Stories podcast. It is our goal to shed some insights on how to judge this “arms race”.

乍一看，这些系统看起来非常相似：它们都试图在无需编程的情况下在图形用户界面中创建富有表现力的可视化图像。 然而，经过仔细检查和动手实验，这些系统在功能和用户界面方面有很大不同。 例如，有些提供了用于绘制没有形状的钢笔工具，而在其他系统中则没有这种工具。 数据绑定是通过某些系统中的拖放操作以及其他系统中的下拉菜单完成的。 随着这些系统的发布，社交媒体上的人们开始怀疑他们之间如何进行比较。 例如，安迪·柯克(Andy Kirk)称这是Datavis工具的军备竞赛 ，后来在Data Stories播客中谈到了这些工具。 我们的目标是对如何判断“军备竞赛”发表一些见识。

Evaluating These Authoring Systems

评估这些创作系统

How do these systems compare to each other? Which systems are better for which visualizations or tasks? These questions are important for users who are considering which system to choose. They are also of interest to researchers and developers who are trying to understand the strengths and weaknesses of these systems. Such understanding can inform the design and development of similar systems in the future.

这些系统之间如何比较？ 哪些系统更适合哪些可视化或任务？ 这些问题对于正在考虑选择哪种系统的用户而言非常重要。 试图了解这些系统的优点和缺点的研究人员和开发人员也对它们很感兴趣。 这种了解可以为将来类似系统的设计和开发提供参考。

Evaluating these complex systems is no easy task though. Several traditional evaluation methods are available, but each of them can only tell us part of the story. For example, many of the tools include an example gallery, showcasing the visualizations that can be created, but these examples tell us little about the user experience of interacting with these tools. Usability studies can reveal potential difficulties encountered by users, but these typically involve predefined authoring tasks that may not reflect what users do in the real-world. Longitudinal deployment can gather valuable feedback on how these systems are used in real-world settings over a period of time, but this approach requires a long time and an active user community.

但是，评估这些复杂的系统并非易事。 有几种传统的评估方法，但每种方法只能告诉我们部分故事。 例如，许多工具都包括一个示例库，其中展示了可以创建的可视化效果，但是这些示例几乎没有告诉我们与这些工具进行交互的用户体验。 可用性研究可以揭示用户遇到的潜在困难，但是这些通常涉及预定义的创作任务，这些任务可能无法反映用户在现实世界中的行为。 纵向部署可以收集有关一段时间内在实际环境中如何使用这些系统的宝贵反馈，但是这种方法需要很长时间并且需要活跃的用户社区。

Furthermore, these methods do not offer a direct comparison between these systems. We can, of course, design lab studies to compare a few systems by asking participants to perform a set of predefined tasks, but the outcomes of such studies can be potentially biased by a number of factors. First, it is hard to ensure comparable training since the amount and quality of training materials vary greatly among these systems. Second, the choice of tasks used in such studies will influence the fairness of comparisons.

此外，这些方法没有提供这些系统之间的直接比较。 当然，我们可以通过要求参与者执行一组预定义的任务来设计实验室研究，以比较几个系统，但是此类研究的结果可能会受到许多因素的影响。 首先，由于这些系统之间培训材料的数量和质量差异很大，因此难以确保可比的培训。 其次，此类研究中所用任务的选择将影响比较的公平性。

Critical Reflections as an Evaluation Method

批判性反思作为一种评估方法

Given the limitations of the above evaluation methods, is there any other way to compare these systems? As the creators of three of these systems (Lyra, Data Illustrator, and Charticulator), we know every detail about our tools, better than anyone else. We were wondering about the feasibility of using critical reflections as an evaluation method. Critical reflections are modeled after design critiques — a popular process in the design community for analyzing how effectively design choices meet goals. Unlike critiques, which occur throughout an iterative design process, critical reflections are retrospective after the systems have been built and allow us to understand different trade-offs when making design choices in these systems.

鉴于上述评估方法的局限性，还有其他方法可以比较这些系统吗？ 作为这三个系统(Lyra，Data Illustrator和Charticulator)的创建者，我们比其他人更了解我们工具的每个细节。 我们想知道使用临界反射作为评估方法的可行性。 批判性思考是根据设计评论建模的，这是设计界中流行的过程，用于分析设计选择如何有效地实现目标。 与在整个迭代设计过程中进行的批判不同，批判性思考是在系统构建后进行回顾的，这使我们能够在这些系统中进行设计选择时了解不同的取舍。

The process of critical reflection went as follows: the creators of all three systems met weekly for 1 to 2 hour video conference meetings over the course of 3 months. During these meetings we directly compared these systems: commenting on our design and implementation, reflecting on practical feedback from the user community, and addressing missed or unexplored research directions. In critical reflections, a candid attitude is essential. We decided to leave no stone unturned by evaluating every possible visualization authoring task: mark instantiation and customization, glyph composition, path points and segments generation, linking glyphs, data scoping for glyphs, mapping data values to visual properties, scale management, axes and legend creation, layout items, nesting layout and applying coordinate systems.

进行批判性反思的过程如下：所有三个系统的创建者在3个月的时间里每周开会1至2个小时的视频会议。 在这些会议期间，我们直接比较了这些系统：评论我们的设计和实现，反思用户社区的实际反馈以及解决遗漏或未探索的研究方向。 在批判性反思中，坦率的态度至关重要。 我们决定通过评估所有可能的可视化创作任务来坚定不移：标记实例化和自定义，字形组成，路径点和段生成，链接字形，字形的数据作用域，将数据值映射到视觉属性，比例管理，轴和图例创建，布局项目，嵌套布局和应用坐标系。

To structure these conversations, we began by considering the eight evaluative dimensions proposed by Ren et al. We documented these discussions by recording meeting notes in a shared online folder. At times, comparing the systems required isolated, preliminary reflection on the individual authoring systems, which we documented in the shared folder. Each team carried out these isolated reflections as “take-home” tasks before the next weekly meeting. These isolated activities provided the necessary time to exhaustively consider the ways in which each system met or fell short of our defined objectives.

为了构建这些对话，我们首先考虑Ren等人提出的八个评估维度 。 我们通过在共享的在线文件夹中记录会议记录来记录这些讨论。 有时，将需要比较的系统进行比较，然后对各个创作系统进行初步思考，我们将它们记录在共享文件夹中。 每个团队在下一次每周会议之前将这些孤立的思考作为“带回家”的任务进行。 这些孤立的活动提供了必要的时间来详尽地考虑每个系统达到或未达到我们定义的目标的方式。

Expressivity and Learnability

表现力和学习能力

Through this process, we collectively identified expressivity and learnability as two primary dimensions in the design of visualization authoring systems. Expressivity refers to the array of possible visualizations a tool can create, i.e., “Can I create the visualizations using this system?” Learnability refers to how easy it is to learn and use the system.

通过这一过程，我们将可视化创作系统设计中的表现力和可学习性共同确定为两个主要方面。 表现力是指工具可以创建的一系列可视化效果，即“我可以使用该系统创建可视化效果吗？” 可学习性是指学习和使用系统有多么容易。

Comparison of the three systems begins at the level of system components along these two dimensions. We find instances where systems have incorporated each other’s approach (e.g., Charticulator offers two distinct data binding mechanisms drawn from Lyra and Data Illustrator, respectively) and areas of broad overlap (e.g., in the set of marks available for use), radically different approaches (e.g., the degree to which scales are exposed, and how complex layouts are interactively specified). These differences result in significant trade-offs between expressivity and learnability. For example, at one end of the spectrum, in Lyra, authors are able to manually construct a scale independent of any data binding interaction; on the other end, Charticulator does not distinguish a scale from its axis or legend; Data Illustrator lies in-between. When an author clicks the bind icon they are prompted to create a new scale, or reuse an existing scale if the field has been previously used. Such differences have clear implications for expressivity. In Data Illustrator, merging the scale functions enables authoring visualizations such as Gantt Charts. Authors using Lyra have greater control over scale functions, and reuse or merge scales in a more fine-grained fashion than in Data Illustrator. However, this expressivity comes with a non-trivial complexity cost. On-going feedback for Data Illustrator and Lyra indicate that authors struggle to understand the role that scales play.

这三个系统的比较从这两个维度的系统组件级别开始。 我们发现系统相互融合的方法(例如，Charticulator提供了两种分别来自Lyra和Data Illustrator的数据绑定机制)和广泛重叠的区域(例如，在可供使用的标记集中)的实例，这是完全不同的方法(例如，比例尺的暴露程度以及如何以交互方式指定复杂的布局)。 这些差异导致表现力和可学习性之间的重大权衡。 例如，在天琴座谱系的一端，作者能够手动构建一个独立于任何数据绑定交互作用的量表。 另一方面，Charticulator不能将刻度与其轴或图例区分开来； Data Illustrator位于两者之间。 当作者单击绑定图标时，将提示他们创建一个新的比例尺，或者如果以前使用过该字段，则重复使用现有的比例尺。 这种差异对表达具有明显的含义。 在Data Illustrator中，合并比例尺功能可以创作可视化内容，例如甘特图。 使用Lyra的作者可以更好地控制比例尺功能，并且可以比Data Illustrator更细粒度地重用或合并比例尺。 但是，这种表达方式带来了不小的复杂性成本。 对Data Illustrator和Lyra的持续反馈表明，作者难以理解规模作用。

A summary of the system components and the video demonstrations are available at https://vis-tools-reflections.github.io/

有关系统组件和视频演示的摘要，请访问https://vis-tools-reflections.github.io/。

Author, Data, and Task: Shared Assumptions and Future Directions

作者，数据和任务：共同的假设和未来方向

Through these reflections, we also distill three fundamental assumptions that underlie and cut-across all three systems. In terms of the author, the systems assume that the users need to be comfortable with structuring input data and exhibiting a level of computational thinking. In addition, some systems assume familiarity with other systems. Given Data Illustrator and Charticulator’s ties to Adobe and Microsoft, respectively, they may attract authors who are already familiar with other systems produced by these organizations. Attaining a better understanding of skill transfer for learning our systems is an important direction for future research.

通过这些思考，我们还提炼出了这三个系统的基础和切入点的三个基本假设。 在 笔者来看，系统假设用户需要的舒适与结构的输入数据并显示出计算思维的水平。 另外，某些系统假定与其他系统熟悉。 鉴于Data Illustrator和Charticulator分别与Adobe和Microsoft的联系，它们可能会吸引已经熟悉这些组织生产的其他系统的作者。 更好地理解技能转移以学习我们的系统是未来研究的重要方向。

In terms of data, all three systems assume that the data is formatted as a stand-alone file, is static and relatively small (typically hundreds of rows). More importantly, the dataset needs to be formatted in a particular way that may not be clear to authors a priori. In particular, all three systems expect datasets to be structured in a long (often referred to as “tidy,” and as opposed to wide) format. These systems should consider adding higher-level scaffolding that automatically infers or suggests appropriate transformations when necessary, analogous to their existing mechanisms for data binding which automatically infer definitions for scales and guides. Alternatively, users may leverage other systems (e.g., Tableau Prep) that provide such capabilities already.

就数据而言 ，所有三个系统均假定数据格式化为独立文件，并且是静态的并且相对较小(通常为数百行)。 更重要的是，该数据集需要以一种特殊的方式可能不是很清楚的作者事先进行格式化。 特别是，所有这三个系统都希望数据集以长格式(通常称为“整齐”，而不是宽格式)进行结构化。 这些系统应考虑添加更高级别的脚手架，该脚手架可在必要时自动推断或建议适当的转换，类似于其现有的数据绑定机制，该机制可自动推断比例尺和指南的定义。 可替代地，用户可以利用已经提供了这种能力的其他系统(例如，Tableau Prep)。

Finally, these systems are authoring tools, not design tools. In other words, if users want to explore the visualization design space, our systems are of little help as they begin with a blank canvas, a tabula rasa. Our systems are better suited for tasks where users have a specific chart design in mind. How might future versions of these systems allow authors to explore and combine divergent design ideas? Or how might our systems incorporate visualization design recommendation?

最后，这些系统是创作工具，而不是设计工具。 换句话说，如果用户想探索可视化设计空间，那么我们的系统几乎没有帮助，因为它们始于一块空白的画布，即tabula rasa 。 我们的系统更适合于用户考虑特定图表设计的任务。 这些系统的未来版本将如何允许作者探索和组合各种设计思想？ 还是我们的系统如何纳入可视化设计建议？

Conclusion

结论

Using critical reflections as an evaluation method, we perform a thorough analysis of the components in three visualization authoring systems (Lyra, Data Illustrator and Charticulator). Potential users like Chris will be able to use the analysis results to inform their selection of authoring tools. Visualization researchers may also find critical reflections as a useful method in the evaluation of similar systems designed to support a common goal.

使用批判性反射作为一种评估方法，我们对三个可视化创作系统(Lyra，Data Illustrator和Charticulator)中的组件进行了全面的分析。 像Chris这样的潜在用户将能够使用分析结果来告知他们选择的创作工具。 可视化研究人员也可能会发现，批判性反思是评估旨在支持共同目标的类似系统的有用方法。