Too good to be true?

难以置信？

背景 (Background)

Sometime in 2017 I set about doing a survey of the application of AI in testing and DevOps. That led to a series of talks I gave about the subject (see slideshare). That led me to strongly believe that, while the adoption of such technologies was still nascent then, it held the promise to be a lynchpin for helping us address the testing of our increasingly complex software systems. This led to the starting up of a Meetup group in the Bay Area called the “AI in Testing and DevOps” group (we have since renamed it to AI in Testing and Testing AI).

在2017年的某个时候，我开始对AI在测试和DevOps中的应用进行调查。 这导致了我就该主题进行了一系列讨论(请参阅slideshare )。 这使我坚信，尽管那时仍只是采用这种技术，但它有望成为帮助我们应对日益复杂的软件系统的测试的关键。 这导致在湾区成立了一个名为“测试和DevOps的AI”小组的Meetup小组(此后我们将其重命名为“ Testing and Testing AI的AI” )。

问题是什么？ (What is the problem?)

The following is a typical Agile / Lean Development process:

以下是典型的敏捷/精益开发流程：

1. Product team dreams of features that constitute a product产品团队梦想着构成产品的功能
2. Product team defines an MVP产品团队定义了MVP
3. Dev team builds the MVP开发团队建立MVP
4. The product is dog-fed internally产品内部由狗喂食
5. The product is sent to beta testers for feedback该产品已发送给Beta测试人员以提供反馈
6. The product is shipped to production (with accompanying crash management, telemetry, etc.)产品已交付生产(附带崩溃管理，遥测等)
7. Product team gathers metrics, real user feedback to refine the product features产品团队收集指标，真实的用户反馈以完善产品功能
8. Product team designs A/B tests to test hypotheses on refinements产品团队设计A / B测试以测试细化假设
9. Go To 3转到3

Now in order to execute this flow with velocity, sufficient quality and not frustrate or annoy your users, you need Continuous Testing. This means to:

现在，为了以高速度，足够的质量执行此流程，并且不让您的用户感到沮丧或烦恼，您需要进行持续测试。 这意味着：

• Test early and keep testing尽早测试并继续测试
• Test at each code commit在每次提交代码时进行测试

To achieve this, you need:

为此，您需要：

1. Automated, frequent integration tests自动化，频繁的集成测试
2. Full regression runs. If necessary have nightly regression test runs.完全回归运行。 如有必要，每晚进行回归测试运行。
3. 24/7 test cycle24/7测试周期
4. If a bug is found, fix it fast (if not immediately), don’t let it fester in the code base如果发现错误，请尽快修复(如果不是立即解决)，不要让它在代码库中恶化
5. Every line of code checked in must include tests检入的每一行代码都必须包含测试

The common theme here is Test Automation, lots and lots of it.

这里的共同主题是测试自动化，很多很多。

测试自动化的现实是什么？ (What is the reality of Test Automation?)

Unit Tests

单元测试

Unless there are code coverage gates in Continuous Integration (CI) pipelines to block code merges, often due to time pressure to deliver, teams may skip this and incur technical debt.

除非在持续集成(CI)管道中没有代码覆盖范围的门来阻止代码合并，否则通常由于交付时间的压力，团队可能会跳过此步骤并招致技术债务。

Module Level Tests

模块级测试

Often this depends on whether there is sufficient developer resources to spare.

通常，这取决于是否有足够的开发人员资源可供使用。

API Level Tests

API级别测试

Depends on whether there are test resources.

取决于是否有测试资源。

UI / Client Level Tests

用户界面/客户端级别测试

• Often 0.5 or 1 sprint behind feature通常落后功能0.5或1冲刺
• UI tests are often “flaky” or unreliable (for various reasons)UI测试通常“不稳定”或不可靠(由于各种原因)
• Frustrating for test engineers when UI is rapidly changing用户界面快速变化时，测试工程师感到沮丧
• Many teams resort to using manual testing as stop gap but ends up being the permanent solution许多团队求助于使用手动测试来解决问题，但最终却成为了永久解决方案
• Some UI frameworks do not lend themselves well to test automation, e.g. new UI identifiers (eg. CSS selectors) with each build一些UI框架不能很好地测试自动化，例如，每次构建时都有新的UI标识符(例如CSS选择器)

In a survey conducted by Codefresh, 4 out of 5 developers cite lack of test automation as an obstacle to code delivery [1]. In another survey, 43% of developers in financial services in the US and UK say that they have found it difficult to meet unit test coverage set by their managers [2]. Another data point is that organizations waste about $300bil a year fixing bad code [3]. In that same report, it says that only 6% of respondents of the survey automate all tests.

在Codefresh进行的一项调查中，五分之四的开发人员认为缺乏测试自动化是代码交付的障碍[1]。 在另一项调查中，美国和英国43％的金融服务开发人员表示，他们发现很难达到其经理设定的单元测试范围[2]。 另一个数据点是，组织每年浪费约3000亿美元来修复错误代码[3]。 在同一份报告中，它说只有6％的调查受访者将所有测试自动化。

In a previous article, I also mentioned that most developers either do not have the interest or are not sufficiently incentivized to write tests. Add the fact that most server-side software architectures now are becoming micro-service heavy with, you have a compounded problem of complexity.

在上一篇文章中，我还提到大多数开发人员要么没有兴趣，要么没有足够的动力去编写测试。 加上大多数服务器端软件体系结构现在正变得越来越繁琐的微服务这一事实，您将面临复杂的复杂性问题。

AI / ML如何提供帮助？ Roomba类比 (How can AI/ML Help? The Roomba Analogy)

So how can AI help? Well, I like to use what I call the “Roomba Analogy.” How many of you love vacuuming your home? I really doubt many of you do. However it is absolutely necessary for hygiene and overall cleanliness. With a robot vacuum cleaner, you could just let it run around sucking up the dust bunnies without much human intervention. Now that doesn’t mean you do not need to vacuum at all, it just means you can do it less often and focus on the hard to reach areas.

那么AI如何提供帮助呢？ 好吧，我喜欢使用所谓的“ Roomba类比”。 你们中有多少人喜欢给家里打扫灰尘？ 我真的很怀疑你们中的许多人。 但是，对于卫生和整体清洁绝对必要。 使用机器人真空吸尘器，您可以让它在没有太多人为干预的情况下四处吸吮兔子的灰尘。 现在，这并不意味着您根本不需要清理，只是意味着您可以减少清理次数，而将精力集中在难以到达的地方。

This is similar to AI in testing ie. Let the AI or ML engine do a large part of the work of isolating bugs so that your test team or engineers are able to focus on the kinds of tests that require more effort to design the test. This is just one use case. There are several others as well.

这类似于测试中的AI。 让AI或ML引擎承担隔离错误的大部分工作，以便您的测试团队或工程师能够专注于需要更多精力来设计测试的测试类型。 这只是一个用例。 还有其他几个。

如何在测试过程中应用AI的一些示例 (Some examples of how AI can be applied in the testing process)

识别和隔离移动应用程序崩溃(NimbleApp)(Identifying and isolating mobile app crashes (NimbleApp))

In a previous article I mentioned that tools like NimbleApp from Headspin could be used to automatically crawl your app after every PR to identify crashes so that test engineers are able to focus on more valuable tests. This is like the Roomba I mentioned previously tirelessly plugging away finding crashes. This frees up engineers to work on more challenging tests such as automating memory leak detection (iOS) or flushing out JNI integration issues (Android).

在前面的文章中，我提到的从喜欢NimbleApp该工具街机可以用来自动抓取您的应用后，每PR查明崩溃，使测试工程师能够专注于更有价值的测试。 这就像我之前提到的Roomba孜孜不倦地寻找崩溃信息一样。 这使工程师可以腾出时间从事更具挑战性的测试，例如自动执行内存泄漏检测(iOS)或清除JNI集成问题(Android)。

静态代码分析中的AI(deepcode.ai) (AI in static code analysis (deepcode.ai))

Symbolic AI is a technique of using rules to guide decision making. In the 80s, there was great hope placed on such rule-based expert systems to be the basis of Japan’s 5th Generation Computing effort. Unfortunately, the goals were too ambitious (they tried to encode all the rules in certain domains) and computing was not as cheap as in today’s cloud centered, commodity computing model.

符号AI是一种使用规则指导决策的技术。 在80年代，寄希望于这种基于规则的专家系统成为日本第五代计算工作的基础。 不幸的是，目标太过雄心勃勃(他们试图在某些领域对所有规则进行编码)，并且计算不像今天的以云为中心的商品计算模型便宜。

However things have changed over the years: (a) computing became cheaper and commoditized and (b) a better understanding that such tools are best suited to well defined and structured domains.

但是，这些年来，情况发生了变化：(a)计算变得更加便宜和商品化；(b)更好地了解到此类工具最适合定义明确和结构化的领域。

One example is deepcode.ai. They use Symbolic AI to perform static code analysis that not only applies rules on what good coding practices are but also overcomes a number of shortcomings of taint analysis to isolate possible attack vectors in the code.

一个示例是deepcode.ai 。 他们使用Symbolic AI进行静态代码分析，该分析不仅适用于良好编码实践的规则，而且克服了污点分析的许多缺点，无法隔离代码中可能的攻击向量。

使用用户流量和ML计算覆盖率并自动生成测试自动化(ProdPerfect) (Using user traffic and ML to calculate coverage and auto generate test automation (ProdPerfect))

One of the problems with test management is this: we treat every single test with equal importance. That of course is not true, some tests are more likely to cause a user bug than others simply because some functionality is hardly used. You can imagine if your regression suite runs in the thousands of test cases and each test case takes a few minutes to run. ProdPerfect’s approach is to instrument your Web site, obtain clickstream data and use data analytics and machine learning to decide which are the key user flows that are most important to test. Then generate test code to automatically generate an end-to-end test suite.

测试管理的问题之一是：我们对每个测试都同样重视。 当然这是不正确的，因为某些测试很少使用，因此某些测试比其他测试更可能导致用户错误。 您可以想象，如果您的回归套件可以在成千上万的测试用例中运行，并且每个测试用例需要花费几分钟的时间才能运行。 ProdPerfect的方法是检测您的网站，获取点击流数据并使用数据分析和机器学习来确定哪些是最重要的测试用户流。 然后生成测试代码以自动生成端到端测试套件。

使用AI / ML决定是否在部署错误的情况下执行回滚(Harness.io) (Using AI/ML to decide on whether to perform a rollback in the event of a bad deploy (Harness.io))

One of the problems with Continuously Deployment (CD) is this: how do you know if you performed a bad deploy and need to immediately perform a rollback? Harness.io aims to solve that problem by using observability data from APM tools eg. Sumo Logic, New Relic, AppDynamics to ascertain if there is a need for auto-rollback.

连续部署(CD)的问题之一是：您如何知道是否执行了错误的部署并需要立即执行回滚？ Harness.io旨在通过使用APM工具(例如)的可观察性数据来解决该问题。 Sumo Logic，New Relic，AppDynamics确定是否需要自动回滚。

可观察性和预测(Splunk) (Observability and prediction (Splunk))

Splunk is one of the more popular system observability tools out there. The basic use case is log aggregation and analytics. However, they also provide a Machine Learning Toolkit (MLTK) that allows a user to apply standard ML algorithms eg. Linear and Logistic Regression, K-means clustering etc to the log data collected by Splunk. Recently, Splunk also launched MLTK Smart Assistants to make it even easier for you to perform standard ML operations such as Forecasting, Outlier Detection and Clustering on your log data.

Splunk是目前比较流行的系统可观察性工具之一。 基本用例是日志聚合和分析。 但是，他们还提供了机器学习工具包(MLTK)，该工具包允许用户应用标准的ML算法。 对Splunk收集的日志数据进行线性和Logistic回归，K均值聚类等。 最近，Splunk还启动了MLTK智能助手，使您更轻松地对日志数据执行标准ML操作，例如预测，异常值检测和聚类。

错误分类 (Bug Triage)

In a separate series of articles, I explored the possibility of using Deep Neural Networks and other traditional Machine Learning algorithms to classify Jira bugs to bug priority (eg. Major, Minor). These articles are the following:

在另一系列文章中，我探讨了使用深度神经网络和其他传统机器学习算法将Jira错误分类为错误优先级(例如，主要，次要)的可能性。 这些文章如下：

• Building a Deployable Jira Bug Classification Engine using Tensorflow

  使用Tensorflow构建可部署的Jira Bug分类引擎

• Building a Deployable Jira Bug Classification Engine Using Amazon Sagemaker

  使用Amazon Sagemaker构建可部署的Jira Bug分类引擎

• Building a Deployable Jira Bug Classification Engine using Google AutoML

  使用Google AutoML构建可部署的Jira Bug分类引擎

所有这些听起来很花哨，但是有人真的在用吗？(All of this sounds fancy but is anyone really using this?)

The answer is yes. In a survey conducted with the members of the Meetup group in July this year:

答案是肯定的。 在今年7月与Meetup组成员进行的一项调查中：

• 55% of respondents said they were using some form of AI in testing55％的受访者表示他们在测试中使用某种形式的AI
• 23% were using AI for Test Generation23％的人使用AI进行测试生成
• 46% were using it for anomaly detection, fault assignment, refactoring46％的人将其用于异常检测，故障分配，重构
• 23% were using it for Code Analysis23％的人将其用于代码分析
• 30% were using it for Test Optimization30％的人将其用于测试优化

在哪里可以找到更多信息(Where to find more information)

Have I piqued your interest? Some ways you can find more information are (a) join the AI Software Testing Association (AISTA) at attesting.org or (b) join our Meetup group at AI in Testing and Testing AI.

我是否引起了您的兴趣？ 你可以找到更多信息的一些方法是：(a)加入AI软件测试协会(AISTA)在attesting.org或(b)参加我们的聚会组AI测试和测试AI 。

Looking forward to seeing you at our next Meetup!

期待在下一次聚会上见到您！