How do we evaluate how well a machine learning classifier or test model performs? How do we know if a medical test is reliable enough to use in a clinical setting?

^ h流量做我们评估如何以及机器学习分类或测试模型执行？ 我们如何知道医学测试是否足够可靠以在临床环境中使用？

While a highly accurate coronavirus test may be useful where there is a higher incidence, why is it less informative in populations with lower rates of disease? This sounds counterintuitive and confusing, but does have applications for determining the utility of your own binary classifiers!

尽管在发生率较高的地方使用高精度的冠状病毒检测可能很有用，但为什么在疾病发生率较低的人群中它的信息量较少？ 这听起来违反直觉和令人困惑，但是确实有一些应用程序可以确定您自己的二进制分类器的实用程序！

We define the validity of a test by measuring its specificity and sensitivity. Quite simply, we want to know how often the test identifies true positives and true negatives.

我们通过测量其特异性和敏感性来定义测试的有效性。 简而言之，我们想知道测试多久会发现真实的阳性和阴性的。

Our sensitivity describes how well our test catches all of our positive cases. Sensitivity is calculated by dividing the number of true-positive results by the total number of positives (which include false positives).

我们的敏感性描述了我们的测试对所有阳性病例的捕捉程度。 灵敏度是通过将真实阳性结果的数目除以阳性总数(包括假阳性)而得出的。

Our specificity describes how well our test classifies negative cases as negatives. Specificity is calculated by dividing the number of true-negative results by the total number of negatives (which include false negatives).

我们的特异性描述了我们的测试将阴性案例分类为阴性的效果。 特异性是通过将阴性结果为真阴性结果的总数除以阴性结果总数(包括假阴性结果)而得出的。

The important question is whether a model is meaningful? Simply going by sensitivity and specificity rates won’t cut it! To determine how meaningful or clinically useful a test might be for a population, we need underlying information about the expected incidence or prevalence of a disease. We use Bayes’ Theorem to understand this:

重要的问题是模型是否有意义？ 仅凭敏感性和特异性率就无法解决问题！ 为了确定测试对人群的意义或临床意义，我们需要有关疾病的预期发病率或患病率的基础信息。 我们使用贝叶斯定理来理解这一点：

We take a population of 1 000 000 people, where 10% of them have a certain disease. We use a very reliable test with 98% specificity and sensitivity. Here Event A describes the unconditional probability of this disease in the population. P(A) = 0.10.

我们的人口为100万人，其中10％患有某种疾病。 我们使用具有98％特异性和敏感性的非常可靠的测试。 在这里， 事件A描述了该疾病在人群中的无条件概率。 P(A)= 0.10 。

Event B is the unconditional probability of our test coming up positive. We can calculate P(B) by looking at how many total positives we would get. In this population, we expect 98 000 true positives, calculated by multiplying the disease rate by the total population and sensitivity. For false positives, we take the probability of not having this disease (0.90) and multiple it by the population and (1-Specificity). So we get 18 000 false positives in this scenario. Our P(B) or total positives is then 11.6%.

事件B是我们的测试呈阳性的无条件概率。 我们可以通过查看获得的总阳性数来计算P(B) 。 在该人群中，我们期望通过将疾病发生率乘以总人群和敏感性得出的98 000个真实阳性。 对于误报，我们将不患这种疾病的概率设为(0.90) ，然后乘以总体(1-Specificity)并将其乘以。 因此，在这种情况下，我们得到18000个误报。 那么我们的P(B)或总正值就是11.6％。

Now things get a little bit more complicated. All of these values describe how accurate this test is for this population. But it doesn’t tell us the chances that one person who tests positive has the disease. We need to apply Bayes’ Theorem, using these unconditional values as our prior assumption.

现在事情变得更加复杂了。 所有这些值都说明此测试对该人群的准确性。 但这并没有告诉我们测试阳性的人患上这种疾病的机会。 我们需要应用贝叶斯定理，将这些无条件值用作我们的先前假设。

Then if someone takes a test that comes out positive, what is the probability that the disease is present?

然后，如果有人进行了阳性检测，那么该疾病存在的可能性是多少？

Here we can start to define our variables.

在这里，我们可以开始定义变量。

• P(A) = 0.10

  P(A)= 0.10

• P(B) = 0.116

  P(B)= 0.116

• P(B|A) describes the probability of getting a positive result regardless of whether it’s a true-positive or not, while P(A) is the presence of the disease. Thus P(B|A) is our sensitivity. P(B|A) = 0.98.

  P(B | A)描述获得阳性结果的可能性，而不管其是否为真阳性，而P(A)是疾病的存在。 因此，P(B | A)是我们的灵敏度。 P(B | A)= 0.98。

• P(A|B) = 0.98 * 0.1 / 0.116 = 84.5%

  P(A | B) = 0.98 * 0.1 / 0.116 = 84.5％

So here we see that even with high sensitivity and specificity, the test may not be as accurate in some populations. Using Bayes’ Theorem, we can calculate this quite easily.

因此，在这里我们看到，即使具有高灵敏度和特异性，该检测在某些人群中也可能不那么准确。 使用贝叶斯定理，我们可以很容易地计算出这一点。

What would happen though if the disease was less common in our population? Recall that sensitivity and specificity remain at 98%.

但是，如果这种疾病在我们的人群中不那么普遍，会发生什么呢？ 回想一下，敏感性和特异性仍保持在98％。

• P(A) = 0.01

  P(A) = 0.01

• P(B) = (True Positives + True Negatives)/Total Population = (0.01*0.98 + 0.02*0.99)/10000000 = (9800 + 19800)/1000000 = 0.0296

  P(B) =(真阳性+真阴性)/总人口=(0.01 * 0.98 + 0.02 * 0.99)/ 10000000 =(9800 + 19800)/ 1000000 = 0.0296

• P(A|B) = 0.98 * 0.01 / 0.296 = 33.1%

  P(A | B) = 0.98 * 0.01 / 0.296 = 33.1％

Since the disease is now rarer, the posterior probability of receiving a positive test result when you have the disease is lower. As the prevalence of disease decreases within the population, so does our positive predictive value!

由于这种疾病现在比较罕见，因此患这种疾病时获得阳性检测结果的后继可能性较低。 随着人口中疾病流行率的降低，我们的积极预测价值也将下降 ！

When generating a biomedical test or another model with binary classification, keep in mind when it might be useful. Looking at how well our test works for our specific population because with a low enough prevalence of disease — it might not be very useful!

在生成生物医学测试或具有二进制分类的其他模型时，请记住可能有用。 看看我们的测试对特定人群的效果如何，因为疾病的患病率很低-可能不是很有用！

It follows that if you’re generating a classifier for something that might be rare in the general population, you need very high sensitivity and specificity for a high positive predictive value!

因此，如果要针对一般人群中罕见的事物生成分类器，则需要非常高的敏感性和特异性才能获得较高的阳性预测值！