1.与您的数据库相比，Python非常非常慢 (1. Python is Very , Very Slow compared to your database)

Update – redditor Riddlerforce found valid issues with this section, in that I was not testing over a network connection. Results here are updated. The conclusion is the same, but not as hyperbolically amusing as it was before.

更新 – redditor Riddlerforce在本节中发现了有效的问题，因为我没有通过网络连接进行测试。 此处的结果已更新。 结论是相同的，但不像以前那样夸张有趣。

Let’s first review asynchronous programming’s sweet spot, the I/O Bound application:

让我们首先回顾异步编程的甜蜜点 ，在I / O密集型应用程序：

I/O Bound refers to a condition in which the time it takes to complete a computation is determined principally by the period spent waiting for input/output operations to be completed. This circumstance arises when the rate at which data is requested is slower than the rate it is consumed or, in other words, more time is spent requesting data than processing it.

I / O界限是指一种条件，其中完成计算所需的时间主要由等待输入/输出操作完成所花费的时间确定。 当请求数据的速度比消耗数据的速度慢，或者换句话说， 请求数据花费的时间多于处理数据的时间时，就会出现这种情况。

A great misconception I seem to encounter often is the notion that communication with the database takes up a majority of the time spent in a database-centric Python application. This perhaps is a common wisdom in compiled languages such as C or maybe even Java, but generally not in Python. Python is very slow, compared to such systems; and while Pypy is certainly a big help, the speed of Python is not nearly as fast as your database, when dealing in terms of standard CRUD-style applications (meaning: not running large OLAP-style queries, and of course assuming relatively low network latencies). As I worked up in my PyMySQL Evaluation for Openstack, whether a database driver (DBAPI) is written in pure Python or in C will incur significant additional Python-level overhead. For just the DBAPI alone, this can be as much as an order of magnitude slower. While network overhead will cause more balanced proportions between CPU and IO, just the CPU time spent by Python driver itself still takes up twice the time as the network IO, and that is without any additional database abstraction libraries, business logic, or presentation logic in place.

我似乎经常遇到的一个很大的误解是，与数据库的通信占用了以数据库为中心的Python应用程序所花费的大部分时间。 这在C甚至Java之类的编译语言中可能是一种常识，但在Python中通常不是。 与此类系统相比，Python 非常慢。 尽管Pypy无疑是一个很大的帮助，但是在处理标准CRUD风格的应用程序时（意味着：不运行大型OLAP风格的查询，并且当然假设网络相对较低），Python的速度几乎不及您数据库的速度。延迟）。 正如我在PyMySQL Openstack评估中所做的那样，无论数据库驱动程序（DBAPI）是用纯Python还是用C编写的，都会产生大量的Python级别的额外开销。 仅对于DBAPI，这可能会慢一个数量级。 尽管网络开销会导致CPU和IO之间的分配比例更加均衡，但Python驱动程序本身所花费的CPU时间仍然是网络IO的两倍，并且在其中没有任何其他数据库抽象库，业务逻辑或表示逻辑。地点。

This script, adapted from the Openstack entry, illustrates a pretty straightforward set of INSERT and SELECT statements, and virtually no Python code other than the barebones explicit calls into the DBAPI.

这个脚本是从Openstack条目改编而来的，它说明了一套非常简单的INSERT和SELECT语句，除了对DBAPI的准系统显式调用外，几乎没有Python代码。

MySQL-Python, a pure C DBAPI, runs it like the following over a network:

MySQL-Python是纯C DBAPI，可通过以下网络运行它：

DBAPI (cProfile):  <module 'MySQLdb'>47503 function calls in 14.863 seconds
DBAPI (straight time):  <module 'MySQLdb'>, total seconds 12.962214

With PyMySQL, a pure-Python DBAPI,and a network connection we’re about 30% slower:

使用PyMySQL，纯Python DBAPI和网络连接时，速度要慢30％：

DBAPI (cProfile):  <module 'pymysql'>23807673 function calls in 21.269 seconds
DBAPI (straight time):  <module 'pymysql'>, total seconds 17.699732

Running against a local database, PyMySQL is an order of magnitude slower than MySQLdb:

针对本地数据库运行，PyMySQL比MySQLdb慢一个数量级：

DBAPI:  <module 'pymysql'>, total seconds 9.121727DBAPI:  <module 'MySQLdb'>, total seconds 1.025674

To highlight the actual proportion of these runs that’s spent in IO, the following two RunSnakeRun displays illustrate how much time is actually for IO within the PyMySQL run, both for local database as well as over a network connection. The proportion is not as dramatic over a network connection, but in that case network calls still only take 1/3rd of the total time; the other 2/3rds is spent in Python crunching the results. Keep in mind this is just the DBAPI alone; a real world application would have database abstraction layers, business and presentation logic surrounding these calls as well:

为了突出显示这些运行在IO中所花费的实际比例，以下两个RunSnakeRun显示说明了PyMySQL运行中IO在本地数据库以及通过网络连接上实际花费的时间。 在网络连接上，这个比例并不那么大，但是在那种情况下，网络呼叫仍然只占总时间的1/3。 其他2 / 3rd花费在Python处理结果上。 请记住，这仅仅是DBAPI本身 ； 现实世界中的应用程序还将具有围绕这些调用的数据库抽象层，业务和表示逻辑：

Local connection – clearly not IO bound.

本地连接–显然不受IO约束。

Network connection – not as dramatic, but still not IO bound (8.7 sec of socket time vs. 24 sec for the overall execute)

网络连接–不那么剧烈，但仍然不受IO限制（套接字时间8.7秒，而整体执行时间为24秒）

Let’s be clear here, that when using Python, calls to your database, unless you’re trying to make lots of complex analytical calls with enormous result sets that you would normally not be doing in a high performing application, or unless you have a very slow network, do not typically produce an IO bound effect. When we talk to databases, we are almost always using some form of connection pooling, so the overhead of connecting is already mitigated to a large extent; the database itself can select and insert small numbers of rows very fast on a reasonable network. The overhead of Python itself, just to marshal messages over the wire and produce result sets, gives the CPU plenty of work to do which removes any unique throughput advantages to be had with non-blocking IO. With real-world activities based around database operations, the proportion spent in CPU only increases.

让我们在这里明确一点，当使用Python时，对数据库的调用，除非您尝试使用大量结果集进行许多复杂的分析调用，而这些结果集通常是在高性能应用程序中不会执行的，或者除非您拥有网络速度较慢，通常不会产生IO绑定效应。 当我们与数据库对话时，我们几乎总是使用某种形式的连接池，因此连接的开销已在很大程度上得到减轻。 数据库本身可以在合理的网络上非常快速地选择并插入少量行。 Python本身的开销只是为了通过网络封送消息并生成结果集，这给CPU带来了很多工作要做，从而消除了非阻塞IO所具有的任何独特的吞吐量优势。 使用围绕数据库操作的实际活动，仅花费在CPU上的比例就会增加。

2. AsyncIO使用吸引人的但效率相对较低的Python范例 (2. AsyncIO uses appealing, but relatively inefficient Python paradigms)

At the core of asyncio is that we are using the @asyncio.coroutine decorator, which does some generator tricks in order to have your otherwise synchronous looking function defer to other coroutines. Central to this is the yield from technique, which causes the function to stop its execution at that point, while other things go on until the event loop comes back to that point. This is a great idea, and it can also be done using the more common yield statement as well. However, using yield from, we are able to maintain at least the appearance of the presence of return values:

asyncio的核心是我们使用@ asyncio.coroutine装饰器，该装饰器会执行一些生成器技巧，以使您本来同步的外观函数遵循其他协程。 技术的产量是技术的核心，它使函数在该点停止执行，而其他事情继续进行，直到事件循环回到该点为止。 这是一个好主意，也可以使用更常见的yield语句来完成。 但是，使用yield from ，我们至少可以保留返回值的外观：

@asyncio.coroutine
@asyncio.coroutine
def def some_coroutinesome_coroutine ():():conn conn = = yield from yield from dbdb .. connectconnect ()()return return conn
conn

That syntax is fantastic, I like it a lot, but unfortunately, the mechanism of that return conn statement is necessarily that it raises a StopIteration exception. This, combined with the fact that each yield from call more or less adds up to the overhead of an individual function call separately. I tweeted a simple demonstration of this, which I include here in abbreviated form:

该语法很棒，我非常喜欢它，但是不幸的是，该return conn语句的机制必定会引发StopIteration异常。 这与以下事实结合在一起：每次调用产生的收益或多或少会分别加总单个函数调用的开销。 我在推特上发布了一个简单的演示，并以缩写形式包含在此处：

The results we get are that the do-nothing yield from + StopIteration take about six times longer:

我们得到的结果是，+ StopIteration 产生的 无效操作花费的时间大约是原来的六倍：

yield from: 12.52761328802444
normal: 2.110536064952612

To which many people said to me, “so what? Your database call is much more of the time spent”. Never minding that we’re not talking here about an approach to optimize existing code, but to prevent making perfectly. The PyMySQL example should illustrate that Python overhead adds up very fast, even just within a pure Python driver, and in the overall profile dwarfs the time spent within the database itself. However, this argument still may not be convincing enough.

许多人对我说：“那又怎样？ 您的数据库调用花费的时间更多。” 没关系，我们在这里不是在讨论一种优化现有代码的方法，而是要防止代码的完美实现。 PyMySQL示例应说明，即使只是在纯Python驱动程序中，Python的开销也会很快增加，并且在总体配置文件中，花费在数据库本身上的时间相形见war。 但是，这种说法可能仍不足以令人信服。

So, I will here present a comprehensive test suite which illustrates traditional threads in Python against asyncio, as well as gevent style nonblocking IO. We will use psycopg2 which is currently the only production DBAPI that even supports async, in conjunction with aiopg which adapts psycopg2’s async support to asyncio and psycogreen which adapts it to gevent.

因此，在这里，我将介绍一个全面的测试套件，该套件说明了针对asyncio的Python中的传统线程以及gevent样式的非阻塞IO。 我们将使用psycopg2（它是当前唯一一个甚至支持async的生产DBAPI）与aiopg结合使用， aiopg将psycopg2的异步支持调整为asyncio，而psycogreen使其适应于gevent。

The purpose of the test suite is to load a few million rows into a Postgresql database as fast as possible, while using the same general set of SQL instructions, such that we can see if in fact the GIL slows us down so much that asyncio blows right past us with ease. The suite can use any number of connections simultaneously; at the highest I boosted it up to using 350 concurrent connections, which trust me, will not make your DBA happy at all.

测试套件的目的是在使用相同的通用SQL指令集的同时，尽快将几百万行加载到Postgresql数据库中，这样我们就可以了解到，实际上GIL是否使我们的速度减慢了这么多，以至于异步被打击轻松地经过我们。 该套件可以同时使用任意数量的连接。 最高提升我它使用350个并发连接，这相信我，不会让所有的DBA高兴。

The results of several runs on different machines under different conditions are summarized at the bottom of the README. The best performance I could get was running the Python code on one laptop interfacing to the Postgresql database on another, but in virtually every test I ran, whether I ran just 15 threads/coroutines on my Mac, or 350 (!) threads/coroutines on my Linux laptop, threaded code got the job done much faster than asyncio in every case (including the 350 threads case, to my surprise), and usually faster than gevent as well. Below are the results from running 120 threads/processes/connections on the Linux laptop networked to the Postgresql database on a Mac laptop:

自述文件的底部总结了在不同条件下在不同机器上进行的几次运行的结果。 我可以获得的最佳性能是在一台笔记本电脑上运行Python代码，而另一台笔记本电脑与Postgresql数据库接口，但是在我运行的几乎所有测试中，无论我在Mac上仅运行15个线程/协程，还是运行350（！）个线程/协程在我的Linux笔记本电脑上，在每种情况下（包括350个线程的情况，令我惊讶的是），线程代码都能比asyncio更快地完成工作，并且通常也比gevent快。 以下是在联网到Mac笔记本电脑上的Postgresql数据库的Linux笔记本电脑上运行120个线程/进程/连接的结果：

Python2.7.8 threads (22k r/sec, 22k r/sec)
Python3.4.1 threads (10k r/sec, 21k r/sec)
Python2.7.8 gevent (18k r/sec, 19k r/sec)
Python3.4.1 asyncio (8k r/sec, 10k r/sec)

Above, we see asyncio significantly slower for the first part of the run (Python 3.4 seemed to have some issue here in both threaded and asyncio), and for the second part, fully twice as slow compared to both Python2.7 and Python3.4 interpreters using threads. Even running 350 concurrent connections, which is way more than you’d usually ever want a single process to run, asyncio could hardly approach the efficiency of threads. Even with the very fast and pure C-code psycopg2 driver, just the overhead of the aiopg library on top combined with the need for in-Python receipt of polling results with psycopg2’s asynchronous library added more than enough Python overhead to slow the script right down.

上面，我们看到运行的第一部分的asyncio明显慢得多（Python 3.4在线程和asyncio中似乎都存在一些问题），而在第二部分，与Python2.7和Python3.4相比，它慢了两倍解释器使用线程。 即使运行350个并发连接，这比您通常希望运行单个进程的方式要多得多，但asyncio几乎无法达到线程效率。 即使使用非常快速且纯净的C代码psycopg2驱动程序，仅aiopg库的顶部开销以及使用psycopg2的异步库在Python中接收轮询结果的需求，都增加了足够的Python开销来减慢脚本运行速度。

Remember, I wasn’t even trying to prove that asyncio is significantly slower than threads; only that it wasn’t any faster. The results I got were more dramatic than I expected. We see also that an extremely low-latency async approach, e.g. that of gevent, is also slower than threads, but not by much, which confirms first that async IO is definitely not faster in this scenario, but also because asyncio is so much slower than gevent, that it is in fact the in-Python overhead of asyncio’s coroutines and other Python constructs that are likely adding up to very significant additional latency on top of the latency of less efficient IO-based context switching.

记住，我什至没有试图证明asyncio比线程慢得多。 只是没有更快。 我得到的结果比我预期的要生动得多。 我们还看到，极低延迟的异步方法（例如gevent）也比线程慢，但幅度不大，这首先证实了异步IO在这种情况下绝对不是更快，而且还因为asyncio如此慢与gevent相比，实际上是asyncio的协程和其他Python构造在Python中的开销，除了基于IO的上下文切换效率较低的延迟之外，可能还会增加非常多的附加延迟。

def def transfertransfer (( amountamount , , payerpayer , , payeepayee , , serverserver ):):if if not not payerpayer .. sufficient_funds_for_withdrawalsufficient_funds_for_withdrawal (( amountamount ):):raise raise InsufficientFundsInsufficientFunds ()()loglog (( "{payer} has sufficient funds.""{payer} has sufficient funds." , , payerpayer == payerpayer ))payeepayee .. depositdeposit (( amountamount ))loglog (( "{payee} received payment""{payee} received payment" , , payeepayee == payeepayee ))payerpayer .. withdrawwithdraw (( amountamount ))loglog (( "{payer} made payment""{payer} made payment" , , payerpayer == payerpayer ))serverserver .. update_balancesupdate_balances ([([ payerpayer , , payeepayee ])
])

cursorcursor .. executeexecute (("select id from geo_record where fileid="select id from geo_record where fileid= %s%s  and logrecno= and logrecno= %s%s "" ,,(( itemitem [[ 'fileid''fileid' ], ], itemitem [[ 'logrecno''logrecno' ])
])
)
)
row row = = cursorcursor .. fetchonefetchone ()
()
geo_record_id geo_record_id = = rowrow [[ 00 ]]cursorcursor .. executeexecute (("select d.id, d.index from dictionary_item as d ""select d.id, d.index from dictionary_item as d ""join matrix as m on d.matrix_id=m.id where m.segment_id="join matrix as m on d.matrix_id=m.id where m.segment_id= %s%s  " ""order by m.sortkey, d.index""order by m.sortkey, d.index" ,,(( itemitem [[ 'cifsn''cifsn' ],)
],)
)
)
dictionary_ids dictionary_ids = = [[rowrow [[ 00 ] ] for for row row in in cursor
cursor
]
]
assert assert lenlen (( dictionary_idsdictionary_ids ) ) == == lenlen (( itemitem [[ 'items''items' ])])for for dictionary_iddictionary_id , , element element in in zipzip (( dictionary_idsdictionary_ids , , itemitem [[ 'items''items' ]):]):cursorcursor .. executeexecute (("insert into data_element ""insert into data_element ""(geo_record_id, dictionary_item_id, value) ""(geo_record_id, dictionary_item_id, value) ""values ("values ( %s%s , ,  %s%s , ,  %s%s )")" ,,(( geo_record_idgeo_record_id , , dictionary_iddictionary_id , , elementelement )))
)

数据库代码通过ACID处理并发，而不是进程内同步 (Database Code Handles Concurrency through ACID, Not In-Process Synchronization)

Whether or not we’ve managed to use threaded code or coroutines with implicit or explicit IO and find all the race conditions that would occur in our process, that matters not at all if the thing we’re talking to is a relational database, especially in today’s world where everything runs in clustered / horizontal / distributed ways – the handwringing of academic theorists regarding the non-deterministic nature of threads is just the tip of the iceberg; we need to deal with entirely distinct processes, and regardless of what’s said, non-determinism is here to stay.

无论我们是否设法使用带有隐式或显式IO的线程代码或协程，并找到在过程中将发生的所有竞争条件，如果我们要谈论的是关系数据库，那都根本不重要在当今世界，一切都以集群/横向/分布式方式运行的情况下，学术理论家对线程的非确定性的nature之以鼻只是冰山一角。 我们需要处理完全不同的流程，无论怎么说，不确定性都会保留下来。

For database code, you have exactly one technique to use in order to assure correct concurrency, and that is by using ACID-oriented constructs and techniques. These unfortunately don’t come magically or via any known silver bullet, though there are great tools that are designed to help steer you in the right direction.

对于数据库代码，您可以使用一种确切的技术来确保正确的并发性，即使用面向ACID的构造和技术 。 不幸的是，尽管有许多出色的工具可以帮助您指引正确的方向，但是这些方法并不能神奇地通过任何已知的灵丹妙药来解决。

All of the example transfer() functions above are incorrect from a database perspective. Here is the correct one:

从数据库角度来看，上述所有示例transfer（）函数都不正确。 这是正确的：

def def transfertransfer (( amountamount , , payerpayer , , payeepayee , , serverserver ):):with with transactiontransaction .. beginbegin ():():if if not not payerpayer .. sufficient_funds_for_withdrawalsufficient_funds_for_withdrawal (( amountamount , , locklock == TrueTrue ):):raise raise InsufficientFundsInsufficientFunds ()()loglog (( "{payer} has sufficient funds.""{payer} has sufficient funds." , , payerpayer == payerpayer ))payeepayee .. depositdeposit (( amountamount ))loglog (( "{payee} received payment""{payee} received payment" , , payeepayee == payeepayee ))payerpayer .. withdrawwithdraw (( amountamount ))loglog (( "{payer} made payment""{payer} made payment" , , payerpayer == payerpayer ))serverserver .. update_balancesupdate_balances ([([ payerpayer , , payeepayee ])
])

See the difference? Above, we use a transaction. To call upon the SELECT of the payer funds and then modify them using autocommit would be totally wrong. We then must ensure that we retrieve this value using some appropriate system of locking, so that from the time that we read it, to the time that we write it, it is not possible to change the value based on a stale assumption. We’d probably use a SELECT .. FOR UPDATE to lock the row we intend to update. Or, we might use “read committed” isolation in conjunction with a version counter for an optimistic approach, so that our function fails if a race condition occurs. But in no way does the fact that we’re using threads, greenlets, or whatever concurrency mechanism in our single process have any impact on what strategy we use here; our concurrency concerns involve the interaction of entirely separate processes.

看到不同？ 上面，我们使用一个事务。 要求选择付款人的资金然后使用自动提交修改它们是完全错误的。 然后，我们必须确保使用某种适当的锁定系统来检索此值，以便从读取它到编写它的时间， 不可能基于过时的假设来更改该值。 我们可能会使用SELECT .. FOR UPDATE锁定要更新的行。 或者，我们可以将“读提交”隔离与版本计数器结合使用，以获得更乐观的方法，这样，如果出现竞争条件，我们的功能就会失败。 但是，在单个流程中使用线程，greenlet或任何并发机制这一事实绝对不会对我们在此处使用的策略产生任何影响； 我们的并发问题涉及完全独立的流程之间的相互作用。