By Michael Chau, Anthony Yu, Richard Liaw

由 迈克尔洲 ， 安东尼宇 ， 理查德·廖

Everyone knows about Scikit-Learn — it’s a staple for data scientists, offering dozens of easy-to-use machine learning algorithms. It also provides two out-of-the-box techniques to address hyperparameter tuning: Grid Search (GridSearchCV) and Random Search (RandomizedSearchCV).

每个人都知道Scikit-Learn，它是数据科学家的必备品，它提供了数十种易于使用的机器学习算法。 它还提供了两种开箱即用的技术来解决超参数调整：网格搜索(GridSearchCV)和随机搜索(RandomizedSearchCV)。

Though effective, both techniques are brute-force approaches to finding the right hyperparameter configurations, which is an expensive and time-consuming process!

尽管有效，但这两种技术都是寻找正确的超参数配置的蛮力方法，这是一个昂贵且耗时的过程！

如果您想加快此过程怎么办？ (What if you wanted to speed up this process?)

In this blog post, we introduce tune-sklearn, which makes it easier to leverage these new algorithms while staying in the Scikit-Learn API. Tune-sklearn is a drop-in replacement for Scikit-Learn’s model selection module with cutting edge hyperparameter tuning techniques (bayesian optimization, early stopping, distributed execution) — these techniques provide significant speedups over grid search and random search!

在此博客文章中，我们介绍tune-sklearn ，这使得在保留Scikit-Learn API的同时更容易利用这些新算法。 Tune-sklearn使用尖端的超参数调整技术(贝叶斯优化，提前停止，分布式执行) 替代了Scikit-Learn的模型选择模块，这些技术大大提高了网格搜索和随机搜索的速度！

Here’s what tune-sklearn has to offer:

以下是tune-sklearn提供的功能：

• Consistency with Scikit-Learn API: tune-sklearn is a drop-in replacement for GridSearchCV and RandomizedSearchCV, so you only need to change less than 5 lines in a standard Scikit-Learn script to use the API.

  与Scikit-Learn API的一致性： tune-sklearn是GridSearchCV和RandomizedSearchCV的直接替代，因此您只需在标准Scikit-Learn脚本中更改少于5行即可使用该API。

• Modern hyperparameter tuning techniques: tune-sklearn allows you to easily leverage Bayesian Optimization, HyperBand, and other optimization techniques by simply toggling a few parameters.

  现代超参数调整技术： tune-sklearn使您可以通过简单地切换几个参数来轻松利用贝叶斯优化，HyperBand和其他优化技术。

• Framework support: tune-sklearn is used primarily for tuning Scikit-Learn models, but it also supports and provides examples for many other frameworks with Scikit-Learn wrappers such as Skorch (Pytorch), KerasClassifiers (Keras), and XGBoostClassifiers (XGBoost).

  框架支持： tune-sklearn主要用于调整Scikit-Learn模型，但它也支持许多Scikit-Learn包装器，例如Skorch(Pytorch)，KerasClassifiers(Keras)和XGBoostClassifiers(XGBoost)。

• Scale up: Tune-sklearn leverages Ray Tune, a library for distributed hyperparameter tuning, to efficiently and transparently parallelize cross validation on multiple cores and even multiple machines.

  扩大规模： Tune-sklearn利用Ray Tune (一个用于分布式超参数调整的库)来高效透明地并行化多核甚至多台机器上的交叉验证。

Tune-sklearn is also fast. To see this, we benchmark tune-sklearn (with early stopping enabled) against native Scikit-Learn on a standard hyperparameter sweep. In our benchmarks we can see significant performance differences on both an average laptop and a large workstation of 48 CPU cores.

Tune-sklearn也很快 。 为此，我们在标准超参数扫描中将tune-sklearn (启用了提前停止)与本机Scikit-Learn进行了基准测试。 在我们的基准测试中，我们可以看到普通笔记本电脑和具有48个CPU内核的大型工作站在性能上都存在显着差异。

For the larger benchmark 48-core computer, Scikit-Learn took 20 minutes for a 40,000-size dataset searching over 75 hyperparameter sets. Tune-sklearn took a mere 3 and a half minutes — sacrificing minimal accuracy.*

对于更大的基准48核计算机，Scikit-Learn用了20分钟的时间来搜索40,000个数据集，并搜索了75个超参数集。 Tune-sklearn仅花费了3分半钟-牺牲了最低的准确性。*

* Note: For smaller datasets (10,000 or fewer data points), there may be a sacrifice in accuracy when attempting to fit with early stopping. We don’t anticipate this to make a difference for users as the library is intended to speed up large training tasks with large datasets.

*注意：对于较小的数据集(10,000个或更少的数据点)，在尝试适应早期停止时可能会牺牲准确性。 我们预计这不会对用户产生任何影响，因为该库旨在加快使用大型数据集的大型培训任务的速度。

简单的60秒演练 (Simple 60 second Walkthrough)

Let’s take a look at how it all works.

让我们看一下它们的工作原理。

Run pip install tune-sklearn ray[tune] or pip install tune-sklearn "ray[tune]" to get started with our example code below.

运行pip install tune-sklearn ray[tune]或pip install tune-sklearn "ray[tune]"以开始下面的示例代码。

TuneGridSearchCV示例 (TuneGridSearchCV Example)

To start out, it’s as easy as changing our import statement to get Tune’s grid search cross validation interface:

首先，就像更改导入语句以获取Tune的网格搜索交叉验证界面一样简单：

And from there, we would proceed just like how we would in Scikit-Learn’s interface! Let’s use a “dummy” custom classification dataset and an SGDClassifier to classify the data.

从那里开始，我们将像在Scikit-Learn界面中一样进行操作！ 让我们使用“虚拟”自定义分类数据集和SGDClassifier对数据进行分类。

We choose the SGDClassifier because it has a partial_fit API, which enables it to stop fitting to the data for a certain hyperparameter configuration. If the estimator does not support early stopping, we would fall back to a parallel grid search.

我们选择SGDClassifier是因为它具有partial_fit API，这使它可以停止拟合特定超参数配置的数据。 如果估算器不支持提早停止，我们将退回到并行网格搜索。

As you can see, the setup here is exactly how you would do it for Scikit-Learn! Now, let’s try fitting a model.

如您所见，此处的设置正是您为Scikit-Learn所做的设置！ 现在，让我们尝试拟合模型。

Note the slight differences we introduced above:

请注意我们上面介绍的细微差别：

1. a new early_stopping variable, and

   一个新的early_stopping变量，以及

2. a specification of max_iters parameter

   max_iters参数的规范

The early_stopping determines when to stop early — MedianStoppingRule is a great default but see Tune’s documentation on schedulers here for a full list to choose from. max_iters is the maximum number of iterations a given hyperparameter set could run for; it may run for fewer iterations if it is early stopped.

early_stopping决定了何时提前停止-MedianStoppingRule是一个很好的默认设置，但是请参阅此处有关调度程序的Tune文档，以获取完整列表。 max_iters是给定超参数集可以运行的最大迭代次数； 如果它提前停止，它可能会运行较少的迭代。

Try running this compared to the GridSearchCV equivalent.

尝试将其与GridSearchCV等效运行。

TuneSearchCV贝叶斯优化示例 (TuneSearchCV Bayesian Optimization Example)

Other than the grid search interface, tune-sklearn also provides an interface, TuneSearchCV, for sampling from distributions of hyperparameters.

除了网格搜索界面之外， tune-sklearn还提供了一个接口TuneSearchCV，用于从超参数分布中进行采样。

In addition, you can easily enable Bayesian optimization over the distributions in TuneSearchCV in only a few lines of code changes.

此外，您只需更改几行代码即可轻松地对TuneSearchCV中的发行版启用贝叶斯优化。

Run pip install scikit-optimize to try out this example:

运行pip install scikit-optimize尝试以下示例：

As you can see, it’s very simple to integrate tune-sklearn into existing code. Check out more detailed examples and get started with tune-sklearn here and let us know what you think! Also take a look at Ray’s replacement for joblib, which allows users to parallelize training over multiple nodes, not just one node, further speeding up training.

如您所见，将tune-sklearn集成到现有代码中非常简单。 在此处查看更详细的示例并开始使用tune-sklearn ，让我们知道您的想法！ 还可以看看Ray 替代 joblib的方法，它可以使用户在多个节点(而不仅仅是一个节点)上并行进行训练，从而进一步加快了训练速度。

文档和示例 (Documentation and Examples)

• Documentation*

  文档 *

• Example: Skorch with tune-sklearn

  示例： 带有tune-sklearn的Skorch

• Example: Scikit-Learn Pipelines with tune-sklearn

  示例： 使用tune-sklearn的Scikit-Learn管道

• Example: XGBoost with tune-sklearn

  示例： 带有tune-sklearn的XGBoost

• Example: KerasClassifier with tune-sklearn

  示例： 带有tune-sklearn的KerasClassifier

• Example: LightGBM with tune-sklearn

  示例： LightGBM和tune-sklearn

*Note: importing from ray.tune as shown in the linked documentation is available only on the nightly Ray wheels and will be available on pip soon

*注意： 如链接文档中所示， 从 ray.tune 导入 仅在每晚的Ray轮上可用，并且很快将在pip上可用