php执行sql内存溢出

This article explores SQL Sort, Spill, Memory and Adaptive Memory Grant Feedback mechanism in SQL Server.

本文探讨了SQL Server中SQL排序，溢出，内存和自适应内存授予反馈机制。

Sorting is one of the key operations in query processing. SQL Server can achieve sorting by either reading data in an ordered fashion, for example, performing ordered Rowstore index scan or performing an explicit sort. If we want to get sorted data from a Columnstore index, the only option is to perform a sort explicitly with a Sort operator in a query plan, because a Columnstore index has no particular order, at least at the moment of writing this post.

排序是查询处理中的关键操作之一。 SQL Server可以通过以有序方式读取数据来实现排序，例如，执行有序Rowstore索引扫描或执行显式排序。 如果我们要从Columnstore索引中获取排序的数据，则唯一的选择是在查询计划中使用Sort运算符显式执行排序，因为至少在撰写本文时，Columnstore索引没有特定的顺序。

Columnstore indexes were first introduced in SQL Server 2012, and starting from this version, we got a new query execution mode, called Batch Mode. Batch Mode was originally designed for column store indexes as an execution technique optimized to deal with a large number of rows. Operators, running in a traditional Row Mode, process one row at a time, operators in a Batch Mode process one batch of rows at a time, where a batch is a portion of approximately 900 rows.

列存储索引是在SQL Server 2012中首次引入的，从该版本开始，我们获得了一种新的查询执行模式，称为批处理模式。 批处理模式最初是为列存储索引设计的，是一种优化的处理技术，可处理大量行。 在传统的行模式下运行的操作员一次只能处理一行，而在批处理模式下的操作员一次只能处理一批行，其中一批大约是900行的一部分。

In 2012 and 2014 a SQL Sort operator used to run in a Row Mode only, however, starting from SQL Server 2016 the SQL Sort for a Batch Mode was also implemented. In this post, we are going to look at some Batch Mode Sort peculiarities.

在2012年和2014年，SQL Sort运算符以前只能在行模式下运行，但是，从SQL Server 2016开始，还实现了批处理模式SQL Sort。 在这篇文章中，我们将看一些批处理模式排序的特性。

内存授予 (Memory Grant)

Sort is a memory consuming operator, it allocates memory for the internal structures that are used to store intermediate sorted results before it starts producing a sorted output. For that purpose, there should be some amount of memory, allocated for the query.

Sort是一个消耗内存的运算符，它在开始产生排序输出之前，会为用于存储中间排序结果的内部结构分配内存。 为此，应该为查询分配一些内存。

This memory is often called a workspace memory, the amount of the workspace memory for a query is called a memory grant. A memory grant is calculated during the query compilation and then, when the execution should start, this amount is requested and, depending on the available memory, granted to a query. There are a gateway and a resource semaphore mechanism that regulates the memory allocation process, but we will not delve into it here because it is not the topic of this post.

该内存通常称为工作空间内存，用于查询的工作空间内存量称为内存授权。 在查询编译期间计算内存授权，然后在执行开始时请求该数量，并根据可用内存将其授权给查询。 有一个网关和一个资源信号量机制可以调节内存分配过程，但是我们在这里不再赘述，因为这不是本文的主题。

The important thing here is that a reserved memory grant could not change during the query execution (with a few exceptions, like Sort during index builds). This grant reservation and its immutability make a query execution more reliable, because if the memory is reserved, the query has less chances to run out of memory under high load. This also prevents the situation when one query consumes the entire server memory, because it needs to.

这里重要的是， 保留的内存授予在查询执行期间不能更改 （有一些例外，例如在索引构建期间进行排序）。 这种授权保留及其不变性使查询执行更加可靠，因为如果保留了内存，则查询在高负载下用完内存的机会较小。 这也可以防止一种查询消耗全部服务器内存的情况，因为它需要这样做。

We may see memory grant information in a query plan if we look at the properties of the root plan element. Let’s run a query against a clustered Columnstore index with an ORDER BY clause and took in the memory grant information. I will use SQL Server 2017 CTP 1.3 (though all the examples work also in SQL Server 2017 CTP 2.0 with slightly different numbers) and a sample AdventureworksDW2016CTP3 database under compatibility level 130 (compatibility level of SQL Server 2016). Let’s turn on Include Actual Execution Plan in SSMS and run the query.

如果我们查看根计划元素的属性，则可能会在查询计划中看到内存授予信息。 让我们使用ORDER BY子句对群集的Columnstore索引进行查询，并获取内存授予信息。 我将使用SQL Server 2017 CTP 1.3（尽管所有示例在数字稍有不同SQL Server 2017 CTP 2.0中也可以使用）和示例兼容级别130（SQL Server 2016的兼容级别）下的AdventureworksDW2016CTP3数据库示例。 让我们打开“在SSMS中包括实际执行计划”并运行查询。

use AdventureworksDW2016CTP3;
go
alter database AdventureworksDW2016CTP3 set compatibility_level = 130;
go
select CarrierTrackingNumber
from dbo.FactResellerSalesXL_CCI c
where c.DueDate between '20140101' and '20150101'
order by c.CarrierTrackingNumber, c.CustomerPONumber;
go

We see a simple plan with a Columnstore index scan and a Sort, if we look at the SELECT element’s properties we’ll also see the memory grant information:

我们看到一个带有Columnstore索引扫描和Sort的简单计划，如果我们查看SELECT元素的属性，我们还将看到内存授予信息：

Let’s quickly recall, what do they mean:

让我们快速回顾一下，它们是什么意思：

• SerialRequiredMemory – the minimum amount of memory for the query to start execution. This memory is needed to create internal data structures to handle memory consuming operators, without this memory the query will not start. Calculated at the compile time. SerialRequiredMemory –查询开始执行的最小内存量。 需要此内存才能创建内部数据结构来处理消耗内存的运算符，如果没有此内存，查询将不会启动。 在编译时计算。
• SerialDesiredMemory – the required memory, plus the ideal memory that a query should have for all the data to be processed in memory, this depends on the cardinality and size estimate. Calculated at the compile time. SerialDesiredMemory –所需的内存，加上查询对于所有要在内存中处理的数据应具有的理想内存，这取决于基数和大小估计。 在编译时计算。
• RequiredMemory – if a query plan goes parallel and runs under a certain degree of parallelism (DOP), it should create as many operators as the DOP, and hence as many internal structures for memory consuming operators as the DOP. So, this value is SerialDesiredMemory*DOP + Exchange operators’ memory. In our case the DOP is one so it equals SerialRequiredMemory. RequiredMemory –如果查询计划是并行的并在一定程度的并行度（DOP）下运行，则它应创建与DOP一样多的运算符，因此要为消耗内存的运算符创建与DOP一样多的内部结构。 因此，该值为SerialDesiredMemory * DOP + Exchange操作员的内存。 在我们的例子中，DOP是1，因此它等于SerialRequiredMemory。
• DesiredMemory – this is an additional memory with RequiredMemory, in our case the plan is serial and this property value equals SerialDesiredMemory property value. DesiredMemory –这是带有RequiredMemory的附加内存，在我们的情况下，计划是串行的，并且此属性值等于SerialDesiredMemory属性值。
• MaxQueryMemory – Maximum amount of memory available for individual query grant. This was MaxQueryMemory –可用于单个查询授权的最大内存量。 在2014 SP2和2016中added in 2014 SP2 and 2016. This value may change depending on the overall memory consumption by SQL Server components. 添加了此值。此值可能会更改，具体取决于SQL Server组件的总体内存消耗。
• RequestedMemory – The server checks if the desired memory exceeds maximum memory available for the query, if it exceeds, that it is reduced until it fits the limit. This amount of memory is asked from the resource semaphore and called a requested memory. RequestedMemory –服务器检查所需的内存是否超过了查询可用的最大内存，如果超过，则将其减少直到满足限制。 此数量的内存是从资源信号量中请求的，称为请求的内存。
• GrantedMemory – This is the amount of memory that was actually granted to the query. GrantedMemory –这是实际授予查询的内存量。
• MaxUsedMemory – This shows how much memory the query actually used during the execution. MaxUsedMemory –显示执行期间查询实际使用的内存量。

Now when we recalled the meaning of the main properties, let’s talk about the situation when the granted memory is not enough.

现在，当我们回顾主要属性的含义时，让我们讨论授予的内存不足的情况。

行模式下SQL排序溢出 (SQL Sort Spill in a Row Mode)

As it was mentioned earlier a memory grant is fixed and could not change even if it is found that a query needs more during the execution. You may ask, why not to request all the memory it needs during the compilation? Well, a server actually does it, but, first there might be not enough memory at the moment (we’ll not discuss this situation now) and second – there might be a mistake in a memory calculation because it calculates memory before the actual execution starts.

如前所述，内存授权是固定的，即使发现执行期间查询需要更多，也无法更改。 您可能会问，为什么不在编译过程中请求它需要的所有内存？ 嗯，服务器实际上是这样做的，但是，首先，目前可能没有足够的内存（我们现在不讨论这种情况），其次-内存计算中可能会出错，因为它会在实际执行之前计算内存开始。

The memory amount calculation is based on the estimated number of rows (cardinality) and estimated row size, if one of those parameters differs from the actual values (for a number of reasons, like outdated statistics, for example), the calculation is wrong and the desired memory is not enough. To be able to proceed a query execution, the Query Processor should either – request some more memory (which is forbidden in most cases) or free some used memory temporary spilling the data somewhere, in case of SQL Server, spilling to tempdb.

内存量计算基于估计的行数（基数）和估计的行大小，如果这些参数之一与实际值不同（例如由于多种原因，例如过时的统计信息），则计算错误，并且所需的内存不足。 为了能够继续执行查询，查询处理器应该–请求更多的内存（在大多数情况下是禁止的），或者释放一些用过的内存以将数据临时溢出到某些地方（如果是SQL Server，则溢出到tempdb）。

Let’s look at the spill example, that happens because of the cardinality estimation mistake. While in a real-world cardinality estimation mistakes might be introduced by different factors like, inaccurate or outdated statistics, skewed derived statistics in plan operators, guesses and so on, for a simple demo (I want to keep the plan shape as simple as possible), let’s introduce a cardinality estimation mistake by tricking the optimizer. There are about 10 000 000 rows in the table in reality, but we will pretend there are twice less – 5 000 000 rows.

让我们看一下由于基数估计错误而发生的溢出示例。 在现实世界中，基数估计错误可能是由不同的因素引起的，例如不准确或过时的统计信息，计划算子中的偏斜派生统计信息，猜测等等，对于一个简单的演示（我想使计划的形状尽可能简单） ），让我们通过诱骗优化器来引入基数估计错误。 实际上，该表中大约有1千万行，但是我们假装少了两倍-5千万行。

I will use the same query as above, but I also will use an undocumented trace flag 9453 to disable a Batch Mode execution, and a MAXDOP hint to keep a simple serial plan shape.

我将使用与上述相同的查询，但是我还将使用未记录的跟踪标志9453禁用批处理模式执行，并使用MAXDOP提示来保持简单的串行计划形状。

update statistics dbo.FactResellerSalesXL_CCI with rowcount = 5000000 --11669600
go
select CarrierTrackingNumber
from dbo.FactResellerSalesXL_CCI c
where c.DueDate between '20140101' and '20150101'
order by c.CarrierTrackingNumber, c.CustomerPONumber
option(querytraceon 9453, maxdop 1);

The query plan looks like this:

查询计划如下所示：

You may notice a yellow triangle sign on the Sort operator and the warning telling us, that there is a SQL sort spill. Sort starts an iterative process of spilling some data portions to tempdb, sorting, loading spilled data, sorting again, merging sorted data and so on. Spill, because of the tempdb interaction, can make a query very slow, especially if the spill level is high.

您可能会注意到Sort运算符上的黄色三角形符号，并且警告告诉我们，SQL排序溢出。 排序开始一个迭代过程，该过程将一些数据部分溢出到tempdb，排序，加载溢出的数据，再次排序，合并排序的数据等。 由于tempdb的交互作用，溢出会使查询非常慢，尤其是在溢出级别较高的情况下。

Note, that in the query plan requested memory (1) equals granted memory (2) and equals 68 232 KB, which is not enough and 8163 extra pages were written and read to and from tempdb (3).

请注意，在查询计划中，请求的内存（1）等于授权的内存（2）等于68232 KB，这还不够，并且向tempdb（3）读写了8163个额外的页面。

批处理模式下SQL排序溢出 (SQL Sort Spill in a Batch Mode)

Let’s now remove the undocumented trace flag and MAXDOP and run our query in a Batch Mode.

现在，让我们删除未记录的跟踪标志和MAXDOP并以批处理模式运行查询。

select CarrierTrackingNumber
from dbo.FactResellerSalesXL_CCI c
where c.DueDate between '20140101' and '20150101'
order by c.CarrierTrackingNumber, c.CustomerPONumber;

We may see the following plan:

我们可能会看到以下计划：

Unfortunately, for some reason, if short runs in a batch mode, there are no spill details in a query plan, so we can’t see, how much data was spilled. But if you look at the memory grant information you may notice something interesting.

不幸的是，由于某种原因，如果以批处理方式短暂运行，则查询计划中没有溢出详细信息，因此我们看不到有多少数据被溢出。 但是，如果查看内存授予信息，您可能会注意到一些有趣的事情。

The granted memory (2) is more than the requested memory (1)! That contradicts with the statement that a query cannot request more memory during the execution. And indeed, in case of a batch mode SQL sort it can request more memory to maintain a spill. Let’s look at this process in more details with the help of extended events.

授予的内存（2）大于请求的内存（1）！ 这与查询在执行期间不能请求更多内存的说法相矛盾。 实际上，在批处理模式下，SQL排序可以请求更多内存来保持溢出。 让我们借助扩展事件来更详细地了解此过程。

We enable the following extended events.

我们启用以下扩展事件。

• additional_memory_grant – occurs when a query tries to get more memory grant during execution. Failure to get this additional memory grant may cause the query slowdown. Additional_memory_grant –当查询在执行过程中尝试获取更多内存授权时发生。 无法获得此额外的内存授权可能会导致查询变慢。
  • requested_memory_kb – The size of requested memory grant in KB request_memory_kb –请求的内存授予的大小，以KB为单位
  • granted_memory_kb – The actual granted memory in KB. This can be zero if this additional grant request fails. grant_memory_kb –实际授予的内存，以KB为单位。 如果此附加授予请求失败，则该值为零。
  • resource_pool_id – ID of resource pool where this grant request belongs to resource_pool_id –此授权请求所属的资源池的ID
• batchmode_sort_spill_file – Record the spill file read/write information for batch mode sort. batchmode_sort_spill_file –记录溢出文件读/写信息以进行批处理模式排序。
  • file_io_mode – Read or write a spill file 0 means read, 1 means write file_io_mode –读取或写入溢出文件0表示读取，1表示写入
  • query_dop – DOP of current query query_dop –当前查询的DOP
  • spill_stage – Stage of SQL sort, when spill happens, 1 means retrieve data from lower iterator, 4 means parallel external merge, 8 means return final sorted data splash_stage – SQL排序的阶段，发生溢出时，1表示从较低的迭代器检索数据，4表示并行外部合并，8表示返回最终排序的数据
  • node_id – Numa Node ID node_id – Numa节点ID
  • thread_id – Thread ID thread_id –线程ID
  • query_id – Unique ID for the current sort query query_id –当前排序查询的唯一ID
  • row_file_id – Global row file id row_file_id –全局行文件ID
  • deepdata_file_id – Global deepdata file id deepdata_file_id –全局深度数据文件ID
  • start_row_id – Row file first row ID start_row_id –行文件的第一行ID
  • row_counts – Number of rows in row file row_counts –行文件中的行数
  • deepdata_size_in_pages – Number of pages to store deep data deepdata_size_in_pages –存储深度数据的页面数
• batchmode_sort_status – Record batch mode sort status batchmode_sort_status –记录批处理方式的排序状态
  • query_dop – DOP of current query query_dop –当前查询的DOP
  • thread_id – Thread ID thread_id –线程ID
  • query_id – Unique ID for the current sort query query_id –当前排序查询的唯一ID
  • time_stamp – Time this status change is recorded time_stamp –记录此状态更改的时间
  • status – In what status, current batch mode sort is status –在什么状态下，当前批处理模式排序为
  • description – Description of current status description –当前状态的描述

Let’s run the trace session filtered by the SPID, click Watch Live Data, run the previous query, stop the session and look into the Live Data grid (I have added two extra columns granted_memory_kb and description, the rest of the data are available in the Details section below the grid).

让我们运行由SPID过滤的跟踪会话，单击“观看实时数据”，运行上一个查询，停止会话，然后查看“实时数据”网格（我添加了两个额外的列Grant_memory_kb和description ，其余数据可在网格下方的详细信息部分）。

Like a Row Mode Sort, Batch Mode SQL Sort is also a stop-and-go iterator, when the parent iterator asks for the data, a Sort calls the method Open and execution is stopped until the method is finished. This method creates internal structures, uses them to sort data and as soon as the data is sorted the execution goes further, sort produces the sorted output from the internal structures previously sorted on the open phase.

与行模式排序类似，批处理模式SQL排序也是一个走走停停的迭代器，当父迭代器要求数据时，排序将调用方法Open，并且执行将停止，直到该方法完成为止。 此方法创建内部结构，使用它们对数据进行排序，对数据进行排序后，执行就会进一步进行，sort从先前在开放阶段进行排序的内部结构中生成排序后的输出。

We see 2622 events, most of them fired when a batch mode sort produces batches, you may see BpGetNextBatch start and BpGetNextBatch returned in the description column. These events are not so interesting for us here, more interesting is a sort open phase, where the additional memory grant happens, you see BpOpen start and BpOpen end description. To save space I have aggregated the results.

我们看到了2622个事件，其中大多数是在批处理方式排序产生批处理时触发的，您可能会在描述列中看到BpGetNextBatch开始和BpGetNextBatch返回 。 这些事件对我们来说不是那么有趣，更有趣的是排序开放阶段，在此阶段发生额外的内存授予，您会看到BpOpen开始和BpOpen结束描述。 为了节省空间，我汇总了结果。

Then I used the event’s description above to decode file_io_mode and spill_stage column values.

然后，我使用上面的事件描述来解码file_io_mode和spill_stage列值。

Then I merged all that to one description and made a sum calculation of the memory grant.

然后，我将所有内容合并为一个描述，并对内存授权进行了总和计算。

If you recall that the requested memory was 66 720 KBs, and then calculate 66 720 + 20 480 = 87 200. Which equals granted memory – 87 200 KBs. So, we have found all the additional memory grants.

如果您还记得所请求的内存为66 720 KB，然后计算66 720 + 20 480 = 87200。这等于已授予的内存– 87 200 KB。 因此，我们找到了所有其他的内存授予。

From the extended event sequence, you may notice that all the memory grants are done before the actual parallel external sort merge operations are done. First memory chunk (4096 KB) before writing down the data from lower iterator, second, bigger chunk (16 384 KB) before performing sort runs.

从扩展事件序列中，您可能会注意到，所有内存授予都在完成实际的并行外部排序合并操作之前完成。 在执行排序运行之前，先从较低的迭代器记下数据，然后再写入较大的块（16 384 KB），然后再写入第一个内存块（4096 KB）。

The reason for the additional memory grant is that a server needs additional memory for the spill structures. The difference between a row mode SQL Sort and a batch mode SQL Sort is, that the last one was designed with an ability of dynamic memory grant and uses an optimistic strategy for memory grabbing. This means, that a server creates spill structures during the runtime and only when it detects a low memory condition, while a row mode Sort is more conservative and creates all the necessary structures beforehand. I would like to thank Pedro Lopes (b|t) from MSSQL Tiger Team for clarifying this behavior.

授予额外内存的原因是服务器需要用于溢出结构的额外内存。 行模式SQL Sort和批处理模式SQL Sort之间的区别在于，最后一个设计为具有动态内存授予功能，并且使用乐观策略进行内存获取。 这意味着，服务器仅在运行时且仅在检测到内存不足的情况下才创建溢出结构，而行模式Sort更保守，并事先创建了所有必要的结构。 我要感谢MSSQL Tiger Team的Pedro Lopes（ b | t ）阐明了此行为。

避免SQL排序溢出 (Avoiding SQL Sort Spill)

We just have looked at some spill and memory grant details in different execution modes. In this section, we’ll talk about how to avoid spilling, because it slows down the execution when interacts with tempdb.

我们只是研究了不同执行模式下的一些溢出和内存授予详细信息。 在本节中，我们将讨论如何避免溢出，因为它在与tempdb交互时会减慢执行速度。

Avoid SQL Sort

避免SQL排序

The best way to avoid a SQL sort spill is to avoid sort at all. We may achieve this by adding a proper Rowstore index that may be used to perform an ordered scan to get the data sorted directly from an index. Unfortunately, it is not always possible because of the query plan shape and the intermediate plan operators that don’t preserve order.

避免SQL排序溢出的最佳方法是完全避免排序。 我们可以通过添加适当的行存储索引来实现此目的，该索引可用于执行有序扫描以直接从索引中获取排序的数据。 不幸的是，由于查询计划的形状以及不保留顺序的中间计划运算符，因此并非总是可能的。

If we are talking about a Columnstore index, which has no keys and no particular order in the current version, we can’t get sorted data at all. However, starting from SQL Server 2016 you may have additional Rowstore indexes on a table and probably use them to get sorted data in some scenarios.

如果我们要谈论的是当前版本中没有键且没有特定顺序的Columnstore索引，那么我们根本无法获得排序的数据。 但是，从SQL Server 2016开始，您可能在表上具有其他行存储索引，并且在某些情况下可能会使用它们来获取排序的数据。

Fix Estimates

修正估算

In this example, we deliberately skewed the estimates to get a cardinality mistake while keeping the plan shape simple. In a real life, there might be a lot of real problems with estimates and a lot of ways how to deal with them. Deeper discussion of this topic deserves a separate post, so I’ll quickly enlist the most popular reasons for the incorrect estimates and brief descriptions of the solutions.

在此示例中，我们故意使估计值产生歪斜，以得到基数错误，同时使计划形状保持简单。 在现实生活中，估计可能存在很多实际问题，以及如何处理这些问题。 对该主题进行更深入的讨论时，应单独撰写一篇文章，因此，我将Swift列出导致错误估计和解决方案简短描述的最受欢迎原因。

Statistics

统计

• Missing – create manually or turn on auto creation if it is turned off, switch to a temporary table instead of a table variable if a query contains a table variable, read only DB – move to SQL Server 2012 or create statistics and then go back to read only, remote server – provide necessary permission or update to SQL Server 2012 SP1. 丢失–手动创建或打开自动创建（如果已关闭）；如果查询包含表变量，则切换到临时表而不是表变量；只读DB –移至SQL Server 2012或创建统计信息，然后返回只读，远程服务器–提供必要的权限或更新SQL Server 2012 SP1。
• 2371 if the SQL Server version is less than 2016. 2371 。
• Inaccurate – update with higher sample rates or with a FULLSCAN option. 不准确–使用更高的采样率或FULLSCAN选项进行更新。
• Skewed (atypical distribution or too much data to fit in 200 histogram steps) – create filtered stats. 歪斜（非典型分布或太多数据以至于不能以200个直方图步长容纳）–创建过滤的统计信息。
• Skewed by a number of plan operators further in a query plan – break complex query into several simpler queries, materializing intermediate results into temporary tables. 许多计划运营商在查询计划中进一步倾斜–将复杂查询分解为几个简单查询，将中间结果具体化为临时表。
• Can’t be used by the optimizer – avoid type conversions in predicates, create computed column, use inline UDF instead of scalar UDF, use option RECOMPILE/OPTIMIZE FOR/Dynamic SQL or break code into multiple branches in case of the unknown local variable. 优化器无法使用–避免在谓词中进行类型转换，创建计算列，使用内联UDF代替标量UDF，使用选项RECOMPILE / OPTIMIZE FOR / Dynamic SQL或在未知局部变量的情况下将代码分成多个分支。

Estimation model

估算模型

• • database compatibility level (this also changes the other optimizer related things); 数据库兼容性级别 （这也会更改其他与优化器有关的内容）；
  • flags 9481 and 2312; 标志 9481和2312;
  • hints FORCE_LEGACY_CARDINALITY_ESTIMATION and FORCE_DEFAULT_CARDINALITY_ESTIMATION; 提示 FORCE_LEGACY_CARDINALITY_ESTIMATION和FORCE_DEFAULT_CARDINALITY_ESTIMATION;
  • scoped configuration option LEGACY_CARDINALITY_ESTIMATION. 范围的配置选项LEGACY_CARDINALITY_ESTIMATION。
• • create multicolumn statistics (it helps in some cases); 创建多列统计信息（在某些情况下有帮助）；
  • use trace flag 4137 for SQL Server 2008 – 2012; 对SQL Server 2008 – 2012使用跟踪标志4137；
  • undocumented trace flags 9471 and 9472 for SQL Server 2014-2016; SQL Server 2014-2016的未记录跟踪标志9471和9472;
  • query hint ASSUME_MIN_SELECTIVITY_FOR_FILTER_ESTIMATES for SQL Server 2016. SQL Server 2016的查询提示ASSUME_MIN_SELECTIVITY_FOR_FILTER_ESTIMATES。
• flags 2389 and 2390 for SQL Server 2012 and below, 标志 2389和2390，对于SQL Server 2016， hint ENABLE_HIST_AMENDMENT_FOR_ASC_KEYS for SQL Server 2016. 提示 ENABLE_HIST_AMENDMENT_FOR_ASC_KEYS。
• described earlier in my blog, but those are not documented and not intended for production usage). 描述了许多TF，但这些TF没有记录，因此没有供生产使用）。
• Row Goals – trace flag 4138 or query hint DISABLE_OPTIMIZER_ROWGOAL for SQL Server 2016. 行目标–跟踪标志4138或SQL Server 2016的查询提示DISABLE_OPTIMIZER_ROWGOAL。
• 4199 or query hint ENABLE_QUERY_OPTIMIZER_HOTFIXES for SQL Server 2016. 4199或查询提示ENABLE_QUERY_OPTIMIZER_HOTFIXES。

There are some other, more exotic reasons and solutions, but the enlisted above work for the most cases.

还有其他一些更奇特的原因和解决方案，但是上面列出的大多数情况下都是可行的。

行模式SQL排序– TF 7470 (Row Mode SQL Sort – TF 7470)

There might be a situation when a row mode sort spills to tempdb even if cardinality estimates are correct. In that case, you may use a trace flag 7470, that was introduced in the fix “FIX: Sort operator spills to tempdb in SQL Server 2012 or SQL Server 2014 when estimated number of rows and row size are correct”.

即使基数估计正确，也可能存在行模式排序溢出到tempdb的情况。 在这种情况下，您可以使用修复程序中引入的跟踪标志7470“ FIX：当估计的行数和行大小正确时，排序运算符溢出到SQL Server 2012或SQL Server 2014中的tempdb ”。

批处理模式SQL排序– TF 9389 (Batch Mode SQL Sort – TF 9389)

For a batch mode sort, in case you cannot fix the estimates there is a trace flag 9389, which turns on dynamic memory grants for batch mode operators (not only a Sort) in SQL Server 2016 and upwards.

对于批处理模式排序，如果无法固定估计值，则有一个跟踪标志9389，它将在SQL Server 2016及更高版本中为批处理模式运算符（不仅是排序）打开动态内存授予。

dbcc traceon (9389);
go
select CarrierTrackingNumber
from dbo.FactResellerSalesXL_CCI c
where c.DueDate between '20140101' and '20150101'
order by c.CarrierTrackingNumber, c.CustomerPONumber
;
go
dbcc traceoff (9389);
go

If we enable this TF on a session level and run our query we will see the following query plan.

如果我们在会话级别启用此TF并运行查询，我们将看到以下查询计划。

When this TF is enabled, you see that a memory grant in now about 93 MBs, which is enough to avoid a SQL sort spill.

启用此TF后，您会看到现在有大约93 MB的内存授予，足以避免SQL排序溢出。

This trace flag may not help if you have not enough memory, for example, when I set an option ‘max server memory (MB)’ to 480 MBs, I saw the following plan.

如果没有足够的内存，此跟踪标志可能无济于事，例如，当我将选项“最大服务器内存（MB）”设置为480 MB时，我看到了以下计划。

You may notice that I even ran the query with a TF 9389, the Max Query Memory was too low to reach 93 MBs and as soon as a memory grant exceeded this threshold no more memory was granted.

您可能会注意到，我什至使用TF 9389运行查询，“最大查询内存”太低，无法达到93 MB，并且一旦内存授予超出此阈值，便不再授予更多内存。

Fix Memory Grant

修复内存授予

Starting from SQL Server 2016 there are two query hints MIN_GRANT_PERCENT and MAX_GRANT_PERCENT to control the memory grant. These hints set minimum and maximum memory grant in the percent of a query default memory limit. We may use the first one, which guarantees that a query will get the maximum of required memory or min grant.

从SQL Server 2016开始，有两个查询提示MIN_GRANT_PERCENT和MAX_GRANT_PERCENT来控制内存授予。 这些提示以查询默认内存限制的百分比设置最小和最大内存授权。 我们可以使用第一个，它保证查询将获得所需的最大内存或最小授权。

select CarrierTrackingNumber
from dbo.FactResellerSalesXL_CCI c
where c.DueDate between '20140101' and '20150101'
order by c.CarrierTrackingNumber, c.CustomerPONumber
option(min_grant_percent = 10.0);

We may see that there is no more spill in the query plan.

我们可能会看到查询计划中不再有溢出。

You may also see that Granted Memory is 157 400 KB which is 10% of Max Query Memory 1 574 024 KB as we have specified with the hint, which is more than necessary to avoid spill. So, be careful with the hint not to waste memory.

您可能还会看到，Granted Memory为157 400 KB，这是我们用提示指定的最大查询内存1 574 024 KB的10％，这比避免溢出要多得多。 因此，请小心不要浪费内存。

2017: Adaptive Memory Grant Feedback

2017年：自适应内存授权反馈

SQL Server 2017 may address a memory problem in the other way. A feature that is called Adaptive Memory Grant Feedback, which is one of the features from the Adaptive Query Processing family of features will help us to address the spill.

SQL Server 2017可能以其他方式解决内存问题。 称为“自适应内存授权反馈”的功能是“自适应查询处理”功能家族的功能之一，将帮助我们解决溢出问题。

This feature was first introduced in the CTP 1 for SQL Server 2017 and demands database compatibility level 140. The idea of this feature is to use repeating queries and adjust memory from one execution to another. In case of inaccurate memory grant, SQL Server updates a cached plan with an actual value from the previous execution, so the next execution will have the accurate memory grant. I recommend to read a blog post by Joe Sack – Introducing Batch Mode Adaptive Memory Grant Feedback.

此功能首次在SQL Server 2017的CTP 1中引入，要求数据库兼容级别140。此功能的目的是使用重复查询并将内存从一次执行调整为另一次执行。 如果内存分配不正确，SQL Server将使用上一次执行的实际值更新缓存的计划，因此下一次执行将具有准确的内存分配。 我建议阅读Joe Sack的博客文章- 批处理模式自适应内存授予反馈简介 。

Before we switch to a new compatibility level and run our query, I’d like to make an important notice.

在我们切换到新的兼容性级别并运行我们的查询之前，我想发出一个重要的通知。

Note: Adaptive memory grant feedback works with cached plans. Many people report about a strange issue with plan cache flushing in a developer’s environment due to a memory pressure while there are no signs of memory pressure. To repro the example below, make sure that a plan cache is not flushed.

注意：自适应内存授权反馈适用于缓存的计划。 许多人报告说，由于没有内存压力的迹象，在内存不足的情况下，开发人员环境中的计划缓存刷新出现了一个奇怪的问题。 要复制下面的示例，请确保未刷新计划缓存。

Now, let’s switch to the CL 140, clear cache and run the query three times.

现在，让我们切换到CL 140，清除缓存并运行查询3次。

alter database AdventureworksDW2016CTP3 set compatibility_level = 140;
go
alter database scoped configuration clear procedure_cache;
go
select CarrierTrackingNumber
from dbo.FactResellerSalesXL_CCI c
where c.DueDate between '20140101' and '20150101'
order by c.CarrierTrackingNumber, c.CustomerPONumber
;
go 3

We may observe the following plans.

我们可能会遵守以下计划。

You may notice that during the first execution there was a spill (yellow triangle on a Sort operator), during the second and the third one, memory was adjusted by adaptive memory grant feedback.

您可能会注意到，在第一次执行过程中发生了溢出（Sort运算符上的黄色三角形），在第二次和第三个过程中，通过自适应内存授权反馈来调整内存。

You may observe the memory grant details in the query plans:

您可能会在查询计划中看到内存授予详细信息：

The Desired Memory changes from one execution to another. During the first execution, the Desired Memory property was set to 66 720 KBs, however, that was not enough and spill to tempdb has occurred. During the second execution, SQL Server uses Adaptive Memory Grant Feedback feature and adjusts memory to fit the sort data size, it sets the Desired Memory to 211 192 KBs and spill doesn’t happen. However, Max Used Memory property tells us that only 94 208 KBs was really used by the query so the memory grant of 211 192 KBs is excessive. Because of that during the third execution Adaptive Memory Grant Feedback feature adjusts the memory grant once again, this time, it lowers memory grant to 124 416 KBs, which is closer to the Max Used Memory, so there is almost no memory waste.

所需的内存从一个执行更改为另一个执行。 在第一次执行过程中，“所需的内存”属性设置为66720 KB，但这还不够，并且已溢出到tempdb。 在第二次执行期间，SQL Server使用“自适应内存授予反馈”功能并调整内存以适合排序数据的大小，它将“所需内存”设置为211192 KB，并且不会发生溢出。 但是，“最大已用内存”属性告诉我们，查询实际上只使用了94208 KB，因此211192 KB的内存授权过多。 因此，在第三次执行期间，“自适应内存授权反馈”功能再次调整了内存授权，这次，它将内存授权降低到124 416 KB，这接近于“最大已用内存”，因此几乎没有内存浪费。

If we enable an extended events session with two extended events: spilling_report_to_memory_grant_feedback and memory_grant_updated_by_feedback – we will see adaptive memory grant feedback in more details.

如果我们启用具有两个扩展事件的扩展事件会话： spilling_report_to_memory_grant_feedback和memory_grant_updated_by_feedback-我们将看到更多详细的自适应内存授予反馈。

There are three events, let’s look at their properties.

有三个事件，让我们看一下它们的属性。

The first one is fired because of the spill and tells how much data was spilled.

第一个因泄漏而被解雇，并告知泄漏了多少数据。

The memory problem is detected and the server decides to use adaptive memory grant feedback.

检测到内存问题，服务器决定使用自适应内存授权反馈。

Notice the property ideal_additional_memory_before_kb which is 66 568 KBs, this value we have seen in the first execution plan before (66 720 KBs in the plan is 152 KBs of Required Memory with 66 568 KBs of additional memory). Then memory grant is adjusted and after the adjustment we see the property ideal_additional_memory_after_kb which is 211 040 KBs, this value together with 152 KBs of Required Memory gives us 211 192 KBs which we have seen in the second execution plan.

请注意，属性Ideal_additional_memory_before_kb为66568 KB，这是我们之前在第一个执行计划中看到的值（计划中的66720 KB为152KB的所需内存，其中66568KB为额外的内存）。 然后调整内存授权，并在调整后看到属性Ideal_additional_memory_after_kb为211 040 KB，此值加上152 KB的“必需内存”使我们在第二个执行计划中看到了211 192 KB。

During the next execution the memory is lowered, we see 211 040 KBs that was used during the second execution, and after adjustment we see 124 264 KBs, together with 152 KBs of Required memory we get 124 416 KBs total, which we have seen in the third query plan.

在下一次执行期间，内存被降低，我们看到第二次执行期间使用了211040 KB，经过调整后，我们看到了124264 KB，与152KB的必需内存一起，我们总共获得了124416KB，我们在第三查询计划。

The adaptive memory grant feedback feature is one of the Adaptive Query Processing features available in SQL Server 2017 under Compatibility Level 140, remember that a plan should be cached to make this feature work.

自适应内存授予反馈功能是SQL Server 2017在兼容级别140下可用的自适应查询处理功能之一，请记住应缓存一个计划以使此功能起作用。

2017: SQL Sort Compression

2017年：SQL排序压缩

At the end of this post, I’d like to share with you one more feature that is related to a batch sort – SQL sort compression. This is an undocumented feature, at least in CTP 1.3 – 2.0, and I have no idea, will it stay in SQL Server 2017 RTM or it will be removed, but still it is in the current build and we will try it.

在本文的最后，我想与您分享与批处理排序相关的另一个功能-SQL排序压缩。 至少在CTP 1.3 – 2.0中，这是一个未记录的功能，我不知道，它将保留在SQL Server 2017 RTM中还是将其删除，但仍在当前版本中，我们将尝试使用。

To enable it, we should use an undocumented trace flag 10810.

要启用它，我们应该使用未记录的跟踪标志10810。

In the script below we make a snapshot of the DMV dm_io_virtual_file_stats and its columns num_of_bytes_read, num_of_bytes_written before the execution and after the execution, then will calculate the difference “before” and “after”, to see how much data was written to the tempdb during the SQL Sort spill. Then we enable a trace flag and do exactly the same thing. After that we will compare data size with and without compression. I’ll also use WinDbg with public symbols and set a break point on the method CBpSortPageCompression::CompressOnePage, to prove that it is actually executed when the trace flag is enabled.

在下面的脚本中，我们对DMV dm_io_virtual_file_stats及其列num_of_bytes_read，num_of_bytes_write在执行之前和执行之后进行快照，然后将计算“ before”和“ after”之差，以查看在执行过程中向tempdb写入了多少数据SQL排序溢出。 然后，我们启用跟踪标记并执行完全相同的操作。 之后，我们将比较压缩和不压缩的数据大小。 我还将使用带有公共符号的WinDbg，并在方法CBpSortPageCompression :: CompressOnePage上设置一个断点，以证明在启用跟踪标志时它实际上已执行。

Let’s run the script:

让我们运行脚本：

use AdventureworksDW2016CTP3;
go
alter database scoped configuration clear procedure_cache;
go
drop table if exists #b;
drop table if exists #a;
select num_of_bytes_read, num_of_bytes_written into #b from sys.dm_io_virtual_file_stats(db_id('tempdb'),1);
go
-- No Sort Compression
select CarrierTrackingNumber from dbo.FactResellerSalesXL_CCI c
where c.DueDate between '20140101' and '20150101'
order by c.CarrierTrackingNumber, c.CustomerPONumber
go
select num_of_bytes_read, num_of_bytes_written into #a from sys.dm_io_virtual_file_stats(db_id('tempdb'),1);
select num_of_KBs_read = (a.num_of_bytes_read - b.num_of_bytes_read)/1024.,num_of_KBs_written = (a.num_of_bytes_written - b.num_of_bytes_written)/1024.
from #b b cross join #a a;
go
alter database scoped configuration clear procedure_cache;
go
drop table if exists #b;
drop table if exists #a;
select num_of_bytes_read, num_of_bytes_written into #b from sys.dm_io_virtual_file_stats(db_id('tempdb'),1);
go
dbcc traceon(10810);
go
-- With Sort Compression
select CarrierTrackingNumber from dbo.FactResellerSalesXL_CCI c
where c.DueDate between '20140101' and '20150101'
order by c.CarrierTrackingNumber, c.CustomerPONumber
go
dbcc traceoff(10810);
go
select num_of_bytes_read, num_of_bytes_written into #a from sys.dm_io_virtual_file_stats(db_id('tempdb'),1);
select num_of_KBs_read = (a.num_of_bytes_read - b.num_of_bytes_read)/1024.,num_of_KBs_written = (a.num_of_bytes_written - b.num_of_bytes_written)/1024.
from #b b cross join #a a;
go

Both of the test queries has spilled to tempdb. During the execution of the first part the breakpoint was not hit. However, when the trace flag is enabled we see the following picture in the debugger call stack window.

这两个测试查询都已溢出到tempdb。 在执行第一部分期间，未命中断点。 但是，启用跟踪标志后，我们将在调试器调用堆栈窗口中看到以下图片。

And if we look at the amount of data we will see that during the first execution, there was about 49 MBs written and read from tempdb, while during the second execution there was about 46 MBs.

而且，如果我们查看数据量，我们将看到在第一次执行期间，从tempdb读写大约49 MB，而在第二次执行期间，大约有46 MB。

About 7% less data size because it was compressed, not too much, but the compression rate highly depends on the data and in the other case we may observe higher compression. I ran this script in SQL Server 2017 CTP 1.3 and in SQL Server 2017 CTP 2.0 and it works for both but with slightly different numbers.

由于压缩后的数据大小大约减少了7％，但压缩率很大程度上取决于数据，在其他情况下，我们可能会观察到更高的压缩率。 我在SQL Server 2017 CTP 1.3和SQL Server 2017 CTP 2.0中运行了此脚本，它适用于两个脚本，但数字略有不同。

结论 (Conclusion)

In this post, we observed SQL sort and memory grant details and some new features of the SQL Server 2017, like adaptive memory grant feedback and sort compression. In the next post, we will look at the next feature of the Adaptive Query Processing called Adaptive Join.

在本文中，我们观察到了SQL排序和内存授予的详细信息以及SQL Server 2017的一些新功能，例如自适应内存授予反馈和排序压缩。 在下一篇文章中，我们将介绍Adaptive Query Processing的下一个功能，即Adaptive Join。

Stay tuned and thank you for reading!

请继续关注，并感谢您的阅读！

目录 (Table of contents)

SQL Server 2017: Columnstore Indexes and Trivial Plan

SQL Server 2017: Columnstore in-place updates

SQL Server 2017: Scalar Subquery Simplification

SQL Server 2017: Interleaved Execution for mTVF

SQL Server 2017: Sort, Spill, Memory and Adaptive Memory Grant Feedback

SQL Server 2017: Statistics to Compile a Query Plan

SQL Server 2017: How to Get a Parallel Plan

SQL Server 2017: Adaptive Join Internals

SQL Server 2017：栏目索引和简单计划

SQL Server 2017：列存储就地更新

SQL Server 2017：标量子查询简化

SQL Server 2017：mTVF的交错执行

SQL Server 2017：排序，溢出，内存和自适应内存授予反馈

SQL Server 2017：编译查询计划的统计信息

SQL Server 2017：如何取得平行计画

SQL Server 2017：自适应联接内部

翻译自: https://www.sqlshack.com/sql-server-2017-sort-spill-memory-and-adaptive-memory-grant-feedback/

php执行sql内存溢出