The purpose of this article is to provide some guide on how to conduct analysis of a sample scenario A/B test results using R, evaluate the results and draw conclusions based on the analysis.

本文的目的是提供一些指南，说明如何使用R对示例场景A / B测试结果进行分析，评估结果并根据分析得出结论。

Before we begin, let’s review what an A/B Test means and what it is typically used for and what are some of its limitations.

在开始之前，让我们回顾一下A / B测试的含义，通常的用途以及它的一些局限性。

定义 (Definition)

A/B Testing also known as split testing is a general methodology used online when you want to test a new product or feature. The goal is to design an experiment that is robust and gives repeatable results so as to make an informed decision to launch or not.

A / B测试(也称为拆分测试)是您要测试新产品或功能时在网上使用的一种通用方法。 目标是设计一个健壮的实验，并给出可重复的结果，以便做出是否启动的明智决定。

Typically, this test involves comparing two web pages by showing two variants A and B, to a similar number of visitors and the variant which gives better conversion rate wins.

通常，此测试涉及通过将两个变体A和B展示给相似数量的访问者来比较两个网页，并且变体可以带来更好的转化率。

It is mainly an experiment where two or more variations of the same webpage are compared against each other by displaying them to real-time visitors to determine which one performs better for a given goal. A/B testing is not limited by web pages only, you can A/B test your emails, popups, sign up forms, apps and more.

这主要是一项实验，通过向实时访问者显示同一网页的两个或多个变体进行比较，以确定哪个变体在给定目标下效果更好。 A / B测试不仅限于网页，您可以A / B测试电子邮件，弹出窗口，注册表单，应用程序等。

Typically one of many reasons why an A/B test is done is to ensure new features planned to be introduced do have a measurable (positive) impact on the intended objective for wanting to introduce such feature(s).

通常，进行A / B测试的许多原因之一就是要确保计划引入的新功能确实对想要引入此类功能的目标具有可衡量的(积极的)影响。

局限性 (Limitations)

A/B test is one of many tools for conversion optimization. It is not an independent solution. It won’t fix all your conversion issues. It can’t fix typical issues you get with messy data. You need to do more than just an A/B test to really improve on conversions

A / B测试是许多用于转化优化的工具之一。 这不是一个独立的解决方案。 它不能解决您所有的转换问题。 它无法解决因凌乱的数据而引起的典型问题。 您要做的不只是A / B测试，还可以真正改善转换率

Now, let’s take a dive into a practical example of a case study. The datasets and full analysis details can be found on my github If you need some help understanding and using Github, there is a useful link for beginners here.

现在，让我们深入研究一个案例研究的实际示例。 该数据集和全面分析的细节上可以找到我的GitHub如果你需要一些帮助了解和使用Github上，有初学者有用的链接在这里 。

案例分析 (Case Study)

Imagine we have results of A/B tests from two hotel booking websites. (Note: data is made up to look like the real one.) We need to conduct A/B test analysis of the data, draw conclusions from the data and make recommendations to management or product teams.

想象一下，我们有两个酒店预订网站的A / B测试结果 。 (注意：数据组成看起来像真实的一样。)我们需要对数据进行A / B测试分析，从数据中得出结论，并向管理层或产品团队提出建议。

Note: The analysis below assumes some little knowledge of statistical concepts so as to make it concise and to the point.

注意：下面的分析假设您对统计概念有些了解，以使其简洁明了。

数据集摘要 (Dataset Summary)

1. Variant A is the control group which depicts the existing products or features on the website.变体A是一个对照组，描述了网站上的现有产品或功能。
2. Variant B is the experimental group to experiment the new version of product or feature to see if users like it or if it increases bookings (conversions).变体B是一个实验小组，用于测试产品或功能的新版本，以查看用户是否喜欢它或它是否增加了预订(转化)。
3. Converted — Based on the given dataset, there are two categories defined by logical values (True or false):已转换-根据给定的数据集，由逻辑值定义了两个类别(“真”或“假”)：
4. Converted = True when customer successfully makes a booking当客户成功进行预订时，已转换= True
5. Converted = False when customer visits the sites but does not make a booking当客户访问网站但未进行预订时，Converted = False

检验假设 (Test Hypothesis)

零假设 (Null Hypothesis)

Both version A and B have the same probability of driving customer booking or conversion. In other words, there is no effect or no difference between version A and B.

版本A和版本B具有相同的驱动客户预订或转化的可能性。 换句话说，版本A和B之间没有影响或没有区别。

替代假设 (Alternative Hypothesis)

Both version A and B have a different probability of driving customer booking or conversion. There is a difference between version A and B. Version B is better than A in driving customer bookings. PExp_B != Pcont_A

版本A和版本B具有不同的驱动客户预订或转化的可能性。 版本A和版本B之间存在差异。在推动客户预订方面，版本B优于版本B。 PExp_B！= Pcont_A

分析 (Analysis)

Let’s do the analysis together. Download the dataset here. Please note you must have RStudio installed to do this.

让我们一起做分析。 在此处下载数据集。 请注意，您必须安装RStudio才能执行此操作。

1. Prepare the dataset and load the tidyverse library which contains the relevant packages used for the analysis.准备数据集并加载tidyverse库，该库包含用于分析的相关软件包。

library(tidyverse)setwd(“~egot_\\Projects\\ABTest”) #set up your own directoryABTest <- read.csv("Website Results.csv", header = TRUE)#Using read.csv base R file import function so the data can be imported into a dataframe in RStudio.save(ABTest, file = "~rda\\ABTest.rda")  #save in your own directory

2. Let’s filter conversions for variants A & B and compute their corresponding conversion rates.

2.让我们过滤变体A和B的转化并计算其相应的转化率。

3. Let’s compute the relative uplift using conversion rates A & B. The uplift is a percentage of the increase.

3.让我们使用转化率A和B计算相对提升。提升是增量的百分比。

uplift <- (conv_rate_B - conv_rate_A)/ conv_rate_A * 100uplift #82.72%#B is better than A by 83%. This is high enough to decide a winner.

4. Let’s compute pooled probability, standard error , margin of error and difference in proportion (point estimate) for variants A & B.

4.让我们计算变体A和B的合并概率，标准误差，误差容限和比例差异(点估计)。

5. Let’s compute the z-score.

5.让我们计算z得分。

#5a. Compute the Z-score so we can determine the p-valuez_score <- d_hat/SE_poolprint(z_score) #2.249546

6. Using this z-score, we can quickly determine the p-value via a look-up table, or using the code below:

6.使用此z分数，我们可以通过查找表或以下代码快速确定p值 ：

#5b.Let's compute p_value using the z_score valuep_value <- pnorm(q = -z_score, mean = 0, sd = 1) * 2print(p_value) #0.02447777

7. Let’s compute the confidence interval for the pool

7.让我们计算池的置信区间

8. Let’s visualize the results computed so far in a dataframe (table):

8.让我们在数据框(表)中可视化到目前为止所计算的结果：

模型 (Model)

Based on the computations made so far, i created a model for computing ABTests. You can simply throw in your file with corresponding ABTest results in exactly same format as the file i used and once you run this model your ABTest result will be computed and returned to you within seconds.

基于到目前为止的计算，我创建了一个用于计算ABTests的模型。 您可以简单地将文件和相应的ABTest结果以与我使用的文件完全相同的格式放入文件中，一旦运行此模型，ABTest结果将被计算并在几秒钟内返回给您。

10. Finally let’s visualize the results of A & B using a normal distribution.

10.最后，让我们使用正态分布可视化A和B的结果。

结论与建议 (Conclusions & Recommendation)

1. There are 721 hits and 20 conversions for variant A and 730 hits and 37 conversions for variant B.

1.变体A有721个匹配和20个转换，变体B有730个匹配和37个转换。

2. Relative uplift of 82.72% based on a Conversion rate for A = 2.77%, Conversion rate for B = 5.07%. Hence variant B is better than A by 82.72%.

2.基于A的转化率= 2.77％，B的转化率= 5.07％，相对提升为82.72％。 因此，变体B比A好82.72％。

3. P-value computed for this analysis was 0.02448 (p < 0.05 or p lower than 5% significance level). Hence, the tests results show strong statistical significance. You can be 95% confident that this result is a consequence of the changes made and not a result of random chance.

3.为此分析计算的P值为0.02448(P <0.05或P低于5％的显着性水平)。 因此，测试结果显示出很强的统计学意义。 您可以95％确信此结果是所做更改的结果，而不是随机机会的结果。

4. Based on the aforementioned results that show strong statistical significance. you should reject the null hypothesis and proceed with launch.

4.基于上述结果，具有很强的统计学意义。 您应该拒绝零假设并继续进行发布。

5. Hence, Accept variant “B” and roll it out for 100% of the users.

5.因此，接受变体“ B”并将其推广给100％的用户。

Note: Please check for the entire source code including the visualization on my Github repository.

注意：请检查完整的源代码，包括我Github存储库中的可视化文件。

Thanks for reading.

谢谢阅读。