Usually, when I tell you a student has got 90 marks, you would think this is a very good student. Instead, if I say the marks are 75, that probably means the student might be average. However, as a Data Scientist/Analyst, we need at least ask two questions immediately:

通常，当我告诉您一个学生获得90分时，您会认为这是一个非常好的学生。 相反，如果我说分数是75，则可能意味着该学生可能是平均水平。 但是，作为数据科学家/分析师，我们至少需要立即提出两个问题：

1. Is the full mark 100?满分是100？
2. What’s the distribution of the marks in the class?班上分数的分布是什么？

The first question is obviously important and perhaps everyone would ask because 90/150 is definitely not better than 75/100. The second question is a little bit subtle and possibly only a “data person” will have this sensitivity.

第一个问题显然很重要，也许每个人都会问，因为90/150绝对不比75/100好。 第二个问题有些微妙，可能只有“数据人”才具有这种敏感性。

In fact, in order to make sure an exam having its results normally distributed in the class, it is quite common to select exam questions as follows:

实际上，为了确保考试的成绩能在班级中正常分布，选择考试问题非常普遍，如下所示：

1. Basic and easy questions — 70%基本问题和简单问题-70％
2. Extended questions that will need a deep understanding of the knowledge — 20%需要深入了解知识的扩展问题-20％
3. Difficult questions that will need to be solved with not only adequate knowledge but also some insights — 10%不仅需要足够的知识而且还需要一些洞察力来解决困难的问题-10％

What if we have 100% easy questions or 100% difficult questions? If so, we’re very likely to have results that are not normally distributed in a class.

如果我们有100％的简单问题或100％的困难问题怎么办？ 如果是这样，我们很可能会获得不在类中正常分布的结果。

为什么我们需要规范化/标准化数据？ (Why Do We Need to Normalise/Standardise Data?)

Then, we have our main topic now. I have been a tutor at a University for 5 years. It sometimes cannot be guaranteed that the exam questions are precisely followed the above proportions. To make sure it is fair to all the students, in other words, not too many students failed or too many students got A grades, sometimes we need to normalise the marks to make sure it follows the normal distribution.

然后，我们现在有了我们的主要主题。 我已经在大学任教5年了。 有时不能保证考试题准确地遵循上述比例。 为确保对所有学生公平，换句话说，没有太多的学生不及格或没有太多的学生获得A级，有时我们需要对分数进行归一化以确保其服从正态分布。

Also, when we want to compare students from different universities, or we want to aggregate the results of multiple exams, normalisation is also very important.

另外，当我们想比较来自不同大学的学生，或者想要汇总多次考试的结果时，标准化也很重要。

For demonstration purposes, Let’s suppose we have 5 different exams. Because we need to randomly generate the expected distribution, the following imports are needed.

为了演示的目的，我们假设我们有5种不同的考试。 因为我们需要随机生成期望的分布，所以需要以下导入。

import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom scipy.stats import skewnorm  # used to generate skewed dist

1.满分是100。基本问题太多 (1. Full mark is 100. Too many basic questions)

ex1 = np.array(skewnorm.rvs(a=-10, loc=95, scale=20, size=200)).astype(int)

2.满分为100。难题太多 (2. Full mark is 100. Too many difficult questions)

ex2 = np.array(skewnorm.rvs(a=5, loc=30, scale=20, size=200)).astype(int)

3.满分是100。正态分布 (3. Full mark is 100. Normally distributed)

ex3 = np.random.normal(70, 15, 200).astype(int)ex3 = ex3[ex3 <= 100]

4.满分是50。正态分布 (4. Full mark is 50. Normally distributed)

ex4 = np.random.normal(25, 7, 200).astype(int)

5.满分是200。正态分布 (5. Full mark is 200. Normally distributed)

ex5 = np.random.normal(120, 30, 200).astype(int)

Let’s plot them together using Seaborn distplot.

让我们使用Seaborn distplot它们绘制在一起。

plt.figure(figsize=(16,10))sns.distplot(ex1)sns.distplot(ex2)sns.distplot(ex3)sns.distplot(ex4)sns.distplot(ex5)plt.show()

It is very obvious that these 5 different exams have completely different distributions. When we get such a dataset, we can’t compare them directly.

很明显，这5种不同的考试分布完全不同。 当我们获得这样的数据集时，我们不能直接比较它们。

最小-最大归一化 (Min-Max Normalisation)

The basic idea of Min-Max Normalisation is to normalise all the values into the interval [0,1]. It is fairly easy to do this.

最小-最大归一化的基本思想是将所有值归一化为间隔[0,1]。 这很容易做到。

from sklearn import preprocessingmin_max_scaler = preprocessing.MinMaxScaler()ex1_norm_min_max = min_max_scaler.fit_transform(ex1.reshape(-1,1))ex2_norm_min_max = min_max_scaler.fit_transform(ex2.reshape(-1,1))ex3_norm_min_max = min_max_scaler.fit_transform(ex3.reshape(-1,1))ex4_norm_min_max = min_max_scaler.fit_transform(ex4.reshape(-1,1))ex5_norm_min_max = min_max_scaler.fit_transform(ex5.reshape(-1,1))

Please be noticed that we need to convert our NumPy arrays into vectors before they can be normalised. So, the easiest way of doing this is to reshape them into column vectors reshape(-1, 1).

请注意，我们需要先将NumPy数组转换为向量，然后才能对其进行归一化。 因此，最简单的方法是将它们整形为列向量reshape(-1, 1) 。

After they are normalised, we don’t have to convert it back to a 1-D array to visualise. Below is the histogram after normalising. It is more confident now to put the 5 different exam results together.

将它们标准化后，我们不必将其转换回一维数组即可进行可视化。 以下是归一化后的直方图。 现在将5种不同的考试结果放在一起更有信心。

Z分数标准化 (Z-Score Standardisation)

Z-Score is another commonly used technique. It is called standardisation rather than normalisation because it “standardises” the data in two aspects:

Z分数是另一种常用的技术。 之所以称为标准化而不是标准化是因为它从两个方面“标准化”数据：

1. De-mean the data so that all the standardised data will have their mean equals to zero.对数据进行去均值，以便所有标准化数据的均值等于零。
2. Re-scale the data by dividing the standard deviation so that the data distribution will be either “expanded” or “shrank” based on the extent they deviate from the mean.通过划分标准偏差来重新缩放数据，以便根据数据偏离均值的程度来“扩大”或“缩小”数据分布。

Therefore, we can calculate it as follows.

因此，我们可以如下计算。

from sklearn import preprocessingex1_scaled = preprocessing.scale(ex1)ex2_scaled = preprocessing.scale(ex2)ex3_scaled = preprocessing.scale(ex3)ex4_scaled = preprocessing.scale(ex4)ex5_scaled = preprocessing.scale(ex5)

It can be seen that the Z-Score Standardisation not only normalised the exam results, but also re-scaled them.

可以看出，Z-Score标准化不仅规范了考试结果，而且还对它们进行了重新定标。

摘要 (Summary)

In this article, the exams are used as examples to explain why we need to normalise or standardise datasets. In fact, I have seen many learners and Data Science students who are really enthusiasts of algorithms. They may know many different types of data mining and machine learning algorithms. However, I would say that data transformation is more important than selecting an algorithm in most of the time.

在本文中，以考试为例来说明为什么我们需要标准化或标准化数据集。 实际上，我见过很多真正喜欢算法的学习者和数据科学专业的学生。 他们可能知道许多不同类型的数据挖掘和机器学习算法。 但是，我要说的是，在大多数情况下，数据转换比选择算法更重要。

Therefore, I have also demonstrated how to use the Python Sci-kit Learn library to easily normalise/standardise the data. Hope it helps someone who has just entered in the Data Science and Data Analytics area.

因此，我还演示了如何使用Python Sci-kit Learn库轻松地对数据进行标准化/标准化。 希望它对刚进入“数据科学和数据分析”区域的人有所帮助。

All the code used in this article can be found in this Google Colab Notebook.

本文中使用的所有代码都可以在此Google Colab笔记本中找到。