HelloFresh values a strong user experience, so proper testing is at the core of how we build great software. With a growing customer base and added code complexity, our Platform team realized our existing testing infrastructure was not growing with the needs of our customers and internal stakeholders. In this post, I will share our journey to building an Elastic selenium infrastructure using aerokube stack on and more various tools around this stack.

HelloFresh重视强大的用户体验，因此正确的测试是我们构建出色软件的核心。 随着客户群的增加和代码复杂性的增加，我们的平台团队意识到我们现有的测试基础架构并未随着客户和内部利益相关者的需求而增长。 在本文中，我将分享使用aerokube堆栈以及围绕该堆栈的更多各种工具来构建ElasticSelenium基础架构的过程 。

In HelloTech, we design software according to a services oriented architecture, which means hundreds of different services have their own set of tests. Before releasing key components of the system, we want to check that all our services work together as a single system and deliver the best user experience to our customers.

在HelloTech中，我们根据面向服务的体系结构设计软件，这意味着数百种不同的服务都有自己的测试集。 在发布系统的关键组件之前，我们希望检查我们所有的服务是否可以作为一个系统一起工作，并为我们的客户提供最佳的用户体验。

In our case the tests themselves were not a pain point, we write great tests ;). The problem was our infrastructure. We evaluated solutions based on the following criteria: ease of use, reliability, cost effectiveness, and security.

在我们的案例中，测试本身并不是一个痛点，我们编写了出色的测试;)。 问题是我们的基础设施。 我们根据以下标准评估了解决方案：易用性，可靠性，成本效益和安全性。

我们的选择和实施细节： (Our choice and implementation details:)

为错误添加更多上下文 (Adding more context to the errors)

Usually selenium tests expose only UI base errors (e.g NoSuchElementException and more as described here: link). In the scope of reasons why such errors happened we would like to add more tools that could help in detection why an element is not visible or why something is not loaded, because raw selenium errors are not reflecting the real reasons of why something is not loading as it was expected. Platform team decided to add more tools to our selenium stack, and the first choice was to add a proxy for recording http(s) calls being made during the autotest session. To achieve this, used a fork of browsermob_proxy which gave us access to the new testing artifact and the Har file.

通常，Selenium测试仅暴露UI基本错误(例如NoSuchElementException以及此处所述的更多链接 ： link )。 在发生此类错误的原因范围内，我们希望添加更多工具，以帮助检测为何元素不可见或为何未加载某些内容，因为原始Selenium错误并未反映出为何未加载某些内容的真实原因。如预期的那样。 平台团队决定向我们的Selenium堆栈中添加更多工具，并且第一选择是添加一个代理，以记录在自动测试会话期间进行的http(s)调用。 为了实现这一目标，使用了一个浏览器mob_proxy的分支，它使我们可以访问新的测试工件和Har文件。

报告中 (Reporting)

This also enabled us to build some reporting around it. Proxy is reporting 50x status codes to the log collector where we parse them and compose the report based on status codes and end points. We are sending this report daily to the slack channel where all parties involved in development can review and analyse the failures. Example of daily report available on screenshot below:

这也使我们能够围绕它构建一些报告。 代理正在向日志收集器报告50倍的状态代码，我们在其中解析它们并根据状态代码和端点编写报告。 我们每天都会将此报告发送到备用渠道，开发中涉及的所有各方都可以在该渠道中查看和分析故障。 下面的屏幕快照提供了每日报告的示例：

Besides this, we integrate reason of failures to our automated test report, so it’s now clearly visible where and why error happened.

除此之外，我们还将失败原因整合到我们的自动化测试报告中，因此现在可以清楚地看到发生错误的位置和原因。

扩展我们的Selenium基础设施 (Scaling our selenium infrastructure)

Initially, we had one EC2 instance (“m5.2xlarge” with 8 vCPU cores and 32 GB of memory) handling all testing requests and aspects of our infrastructure. Additionally, we’re using selenoid, provided by aerokube, to run our tests. Unfortunately, we encountered some scalability issues with this solution; as more services were added, the load on instance kept increasing and soon it was not enough on a company level. We needed to investigate potential alternatives to improve our testing infrastructure’s scalability.

最初，我们有一个EC2实例(“ m5.2xlarge”，带有8个vCPU内核和32 GB内存)，用于处理所有测试请求和基础架构的各个方面。 此外，我们使用aerokube提供的selenoid来运行测试。 不幸的是，我们在此解决方案中遇到了一些可伸缩性问题。 随着更多服务的添加，实例上的负载不断增加，很快在公司级别上还不够。 我们需要研究潜在的替代方案，以改善测试基础架构的可扩展性。

As an initial step, we decided to use small machines to run selenoid nodes with available browser types (chrome/firefox/opera etc.) instead of one big machine. We selected “t3a.small” instances. To build automation for these instances we used terraform + packer, both tools helped us to provision instances with preinstalled software and configure in a predictable way. We created an autoscaling group which controls the number of selenoid nodes inside this group. Auto Scaling group having scheduled triggers for a cost efficient way of using resources, it means that during business hours we add more nodes to the auto scaling group and during night we decrease this number. In addition we also built a small lambda function which is calling status end point of our main orchestrator for selenium based requests (grid router). The lambda function is only adding nodes to the autoscaling group and after the business hours, they are removed by scheduled autoscaling actions.

作为第一步，我们决定使用小型计算机来运行具有可用浏览器类型(chrome / firefox / opera等)的类Selenium节点，而不是一台大型计算机。 我们选择了“ t3a.small”实例。 为了为这些实例构建自动化，我们使用了terraform + packer ，这两种工具都可以帮助我们为实例配备预安装的软件并以可预测的方式进行配置。 我们创建了一个自动缩放组，该组控制该组内的类神经节的数量。 Auto Scaling组已安排了触发器，以节省成本的方式使用资源，这意味着在工作时间内，我们向Auto Scaling组添加了更多节点，而在晚上，我们减少了此数量。 此外，我们还构建了一个小的lambda函数，该函数为基于Selenium的请求(网格路由器)调用主协调器的状态端点。 lambda功能仅将节点添加到自动伸缩组，并且在营业时间之后，将通过计划的自动伸缩操作将其删除。

For orchestration of selenium requests we are using a solution based on aerokube ggr. The only difference from the original ggr is that we have added a flag called “uniformDistribution”. This flag is used when ggr is not able to find a free selenoid host to schedule a test run, in this case our forked version of ggr is calling status end point for underlying selenoid nodes and looking for the least loaded node to schedule the session. When scaling activity is happening we are using consul for rendering ggr template, discovery of new selenoid nodes and reload configuration of ggr to add or remove nodes. Consul is running on both ggr and selenoid nodes and responsible to register nodes and build the quota file on ggr hosts. Example of consul template:

对于Selenium的编排要求，我们使用的是基于aerokube的解决方案GGR 。 与原始ggr的唯一区别是我们添加了一个名为“ uniformDistribution”的标志。 当GGR是不是能找到一个免费的selenoid主机调度试运行，在这种情况下，我们的分叉的版本，使用该标志GGR呼吁地位终点底层selenoid节点，寻找最小负载的节点来安排会议。 当发生扩展活动时，我们将使用领事来渲染ggr模板，发现新的类瘤节点并重新加载ggr的配置以添加或删除节点。 Consul在ggr和selenoid节点上都运行，并负责在ggr主机上注册节点并构建配额文件。 领事模板示例：

费用 (Costs)

After adding the above steps of using consul and auto scaling groups we were able to reduce the costs of selenoid infrastructure by using spot instances to run the selenoid nodes. Auto scaling group sends notifications about upcoming spot instance interruption (2 mins ahead), thus consul removes instance from pool and the testing session gets cancelled. For notifications, we used sns topic to wait for messages about interruption (more information about sns and how to setup it available in official docs). The screenshot below illustrates the cost of running the selenium infrastructure; usually we are limiting our autoscale group to 40 selenoid nodes. The costs of previously used to run selenoid “m5.2xlarge”-machine were about 313 usd per month for single machine which were able to launch less number of parallel sessions, and now with new setup costs decreased in average to 135usd per month.

添加了使用领事和自动伸缩组的上述步骤后，我们能够通过使用竞价型实例运行Selenium素节点来降低Selenium素基础设施的成本。 自动伸缩组会发送有关即将发生的竞价型实例中断的通知(提前2分钟)，因此领事将从池中删除实例，并取消测试会话。 对于通知，我们使用sns主题等待有关中断的消息(有关sns以及如何在官方文档中进行设置的更多信息)。 下面的屏幕快照说明了运行Selenium基础架构的成本； 通常，我们将自动缩放组限制为40个类神经节。 以前用于运行Selenium鼓“ m5.2xlarge”机器的成本约为单台机器每月313美元，能够启动较少数量的并行会话，而现在新设备的平均成本降低至每月135美元。

监控方式 (Monitoring)

On every selenoid node and every ggr we have installed a telegraf agent which is producing metrics in prometheus format. Those are scrapped by prometheus, and based on this data, we build a grafana dashboard and alerting for selenoid:

在每个Selenium状体节点和每个ggr上，我们都安装了Telegraf代理，该代理以Prometheus格式生成度量。 那些被普罗米修斯报废了，并基于此数据，我们构建了一个grafana仪表板并警告类Selenium素：

Abstract schema of services used to build solution

用于构建解决方案的服务的抽象架构

结论 (Conclusion)

As a Platform team, we have built a scalable and efficient infrastructure based on Aerokube’s widely known Selenoid. With this solution, we are now able to deliver better end to end products to our customers and provide a scalable and reliable platform for end-to-end test execution. We hope this article can help you to build a strong and cost efficient solution on top of well known tools and help your developers trace failures and deliver quality features.

作为平台团队，我们基于Aerokube广为人知的Selenoid建立了可扩展且高效的基础架构。 通过此解决方案，我们现在能够为客户提供更好的端到端产品，并为端到端测试执行提供可扩展且可靠的平台。 我们希望本文可以帮助您在众所周知的工具之上构建强大且经济高效的解决方案，并帮助您的开发人员跟踪故障并提供高质量的功能。