tcp码流中查找rtp头

We’re facing a challenge with several of our data flows that use more time than they have in the past and we’re not sure when this trend started. We know in the past month, our reports have been delayed by over a day from the start to the finish. For some of our data flows we use SQL Server Agent that calls SSIS packages or procedures, while some of them use a custom data import and reporting application we’ve created. How can we track the length of time for these data flows, since we’re using a combination of tools for importing data?

我们面临着一些数据流所面临的挑战，这些数据流所花费的时间比过去更多，我们不确定这种趋势何时开始。 我们知道在过去的一个月中，我们的报告从开始到完成都被延迟了一天以上。 对于我们的某些数据流，我们使用SQL Server代理调用SSIS包或过程，而其中一些使用我们创建的自定义数据导入和报告应用程序。 由于我们使用多种工具来导入数据，我们如何跟踪这些数据流的时间长度？

挑战 (The Challenge)

Since we have a multi-step process, we want to identify which of these steps may be the issue of delay. In this tip, we’ll assume that this “process” is an ETL flow with 3 steps – importing data, validating the data, and comparing the data to a base set. We could apply the same logic if we’re dealing with another process that has a different arrangement in its flow, such as validating first, importing the data, then comparing the data. While the steps may differ, how we determine what is causing delay and where can optimize (or change) will not change.

由于我们有一个多步骤流程，因此我们想确定其中哪些步骤可能是延迟问题。 在本技巧中，我们将假定此“过程”是一个ETL流，它包含3个步骤-导入数据，验证数据并将数据与基本集进行比较。 如果我们要处理另一个流程具有不同安排的流程，则可以应用相同的逻辑，例如首先进行验证，导入数据然后比较数据。 尽管步骤可能有所不同，但我们如何确定导致延迟的原因以及可以在何处进行优化（或更改）的方法不会改变。

测量速度 (Measure the Speed)

We’ll use an example of an ETL flow that uses a SQL Server Job with three steps:

我们将使用一个ETL流程示例，该示例使用三个步骤来使用SQL Server作业：

• An import step, where data are imported into a staging table 导入步骤，将数据导入到临时表中
• A cleaning step, where the data are cleaned before entering into a destination table 清除步骤，其中在输入目标表之前先清除数据
• A verification step, which for the SQL Server Job Agent comes last 验证步骤，最后一步是SQL Server Job Agent

We see the three steps in the example SQL Server Job which imports, cleans and verifies data.

我们在示例SQL Server作业中看到了三个步骤，该步骤可以导入，清除和验证数据。

We can query the msdb database to get the time information about the steps in the job, which will help us determine where we can work on improving performance. In the below query, we query three system tables from the msdb database – sysjobs, sysjobsteps and sysjobhistory. To get the date and time information formatted using a year, month, day and time format, we’ll use the system scalar value function dbo.agent_datetime – a function which Microsoft does not provide information on, but which we can see accepts two parameters – a date and time (the below image shows this).

我们可以查询msdb数据库以获取有关作业步骤的时间信息，这将有助于我们确定可以在何处改善性能。 在下面的查询中，我们从msdb数据库中查询三个系统表-sysjobs，sysjobsteps和sysjobhistory。 为了获得使用年，月，日和时间格式格式化的日期和时间信息，我们将使用系统标量值函数dbo.agent_datetime –该函数Microsoft不提供信息，但是我们可以看到它接受两个参数–日期和时间（下图显示了这一点）。

The system scalar value function dbo.agent_datetime in the msdb database.

msdb数据库中的系统标量值函数dbo.agent_datetime。

USE [msdb]
GOSELECT t2.step_name CompareStepName, t3.run_date CompareRunDate, t3.run_duration CompareRunTotalTime, dbo.agent_datetime(t3.run_date,t3.run_time)
FROM dbo.sysjobs tINNER JOIN dbo.sysjobsteps t2 ON t.job_id = t2.job_idINNER JOIN dbo.sysjobhistory t3 ON t2.job_id = t3.job_idAND t2.step_id = t3.step_id
WHERE t.[name] = 'ETLOvernight4'

Example result from our query showing the run times of each job step of ETLOvernight4.

我们查询的示例结果显示了ETLOvernight4每个作业步骤的运行时间。

Our output shows the run time of each step with the total duration (according to Microsoft, run_duration reports in the HHMMSS format). In some cases, msdb is treated like most system databases – restricted. Rather than query msdb for time information, we’ll create our own tracking table and insert times when our ETL steps pass. Either msdb or our own tracking table will function for getting time information, however, the latter will be more useful if we enter an environment where we’re not allowed to access higher level databases.

我们的输出显示每个步骤的运行时间以及总持续时间（根据Microsoft，HHMMSS格式的run_duration报告）。 在某些情况下，msdb被视为与大多数系统数据库一样–受限制。 我们将创建自己的跟踪表并在ETL步骤通过时插入时间，而不是向msdb查询时间信息。 msdb或我们自己的跟踪表都可以获取时间信息，但是，如果我们进入不允许访问更高级别数据库的环境，则后者将更加有用。

In the below code, we create our table along with the steps for this ETL job – ETLOvernight4. We could create a fourth step that only inserts a start time for the job, but instead we’ll add the start time of the job to step 1 with a step of 0 and then at the end of every step with an incrementing value, add a finish time for the step. The below image shows this for the first step – the second and third step also match with the below ending times shown in the code:

在下面的代码中，我们将创建表以及此ETL作业– ETLOvernight4的步骤。 我们可以创建第四步，仅插入作业的开始时间，但是相反，我们将作业的开始时间添加到第1步（步长为0），然后在每步结束时以递增值添加，步骤的完成时间。 下图显示了第一步的步骤–第二步和第三步也与代码中显示的以下结束时间匹配：

CREATE TABLE etlJobTracking(JobName VARCHAR(100) NOT NULL,JobStepNumber TINYINT NOT NULL,JobTime DATETIME DEFAULT GETDATE(),JobNotes VARCHAR(25) NULL
)-- Step 1 - start:
INSERT INTO etlJobTracking (JobName,JobStepNumber,JobNotes)
VALUES ('ETLOvernight4',0,'Start')-- Step 1 - end:
INSERT INTO etlJobTracking (JobName,JobStepNumber)
VALUES ('ETLOvernight4',1)-- Step 2 - end:
INSERT INTO etlJobTracking (JobName,JobStepNumber)
VALUES ('ETLOvernight4',2)-- Step 3 - end:
INSERT INTO etlJobTracking (JobName,JobStepNumber)
VALUES ('ETLOvernight4',3)

Example of us adding our code to step 1 – this step will have both the start of the job and end of the step, while other steps will have the end.

我们将代码添加到步骤1的示例-此步骤将同时具有工作的开始和步骤的结束，而其他步骤将具有结束。

We can now query this table with the job name and the below result shows an example of what we’d see.

现在，我们可以使用工作名称查询该表，以下结果显示了我们看到的示例。

SELECT *
FROM etlJobTracking
WHERE JobName = 'ETLOverNight4'
ORDER BY JobTime DESC

Output of query on the second run from the next day using timestamps at the end of each completed step.

从第二天开始，在第二步运行时从每个完成的步骤结束时使用时间戳输出查询的结果。

Relative to the ETL or data flow tools we’re using, we can use the above within the tool – from a PowerShell Invoke-SqlCmd line to an Execute SQL Task in SSIS throughout our SSIS flow wherever the context is appropriate. The reason we look at the above method for measuring a flow’s time is because SQL Server Agent isn’t used in all environments.

相对于我们使用的ETL或数据流工具，我们可以在工具中使用以上内容–在合适的上下文中，从PowerShell Invoke-SqlCmd行到SSIS中的SSIS中的执行SQL任务。 我们之所以看上面的方法来测量流的时间，是因为未在所有环境中都使用SQL Server代理。

One final development technique we may use is a timestamp on the changing tables throughout the data flow. If our first step involves a staging table during the import, our second step involves a final table for the validation and comparison, both of these tables would have a timestamp tracking on their schema, which is either created or updated as a step in our process finishes. In the below image, we see the result of a query from the final table that returns the date and time information of each step.

我们可能使用的最后一种开发技术是在整个数据流中更改表上的时间戳。 如果我们的第一步在导入过程中涉及一个临时表，而我们的第二步涉及用于验证和比较的最终表，则这两个表都将对其模式进行时间戳跟踪，该跟踪可在我们流程中的一个步骤中创建或更新完成。 在下图中，我们看到了来自最终表的查询结果，该查询返回了每个步骤的日期和时间信息。

For the record id of 1964387, we see the import time, validation time and compare time.

对于记录ID 1964387，我们看到了导入时间，验证时间和比较时间。

From a storage perspective, the data type DATETIME use 8 bytes, which can add costs. However, we don’t have to keep these columns in our final presentation, so once the data have been recorded on these columns, we can save the aggregate and remove them (or in the presentation table structure, avoid keeping them). This technique is a popular approach to storing timestamp information in ETL processes because we can compare the time on the row level, if we need granular information. The downside is that it does add costs – though we can optimize this by saving a summary of the information.

从存储角度来看，数据类型DATETIME使用8个字节，这会增加成本。 但是，我们不必在最终演示文稿中保留这些列，因此，一旦将数据记录在这些列中，我们就可以保存汇总并将其删除（或在演示文稿表结构中，避免保留它们）。 该技术是在ETL流程中存储时间戳信息的一种流行方法，因为如果需要粒度信息，我们可以在行级别比较时间。 缺点是确实增加了成本，尽管我们可以通过保存信息摘要来优化成本。

比较速度的增长（如果适用） (Compare the Speed’s Growth (If Applicable))

After we have a measure of the speed of our process (in this case, an ETL job), we can identify the following:

在衡量了流程速度（在本例中为ETL作业）之后，我们可以确定以下内容：

1. Which step uses the most time? Can this step be optimized without vertical or horizontal scale, such as data cleaning and query optimization? 哪一步使用最多时间？ 是否可以在没有垂直或水平范围的情况下优化此步骤，例如数据清理和查询优化？
2. Is the time for each step execution growing along with the data size? If so, this predicts a future scaling problem. The below query shows us a comparison of average times based on the month for this job using the etlJobTracking table we created in the previous step – though we could apply similar logic to other time tracking techniques 每个步骤执行的时间是否随着数据大小而增加？ 如果是这样，这预示着将来的扩展问题。 下面的查询显示了使用上一步中创建的etlJobTracking表对基于该工作的月份的平均时间的比较-尽管我们可以将类似的逻辑应用于其他时间跟踪技术

DECLARE @jobname VARCHAR(100) = 'ETLOverNight4'
;WITH MeasureGrowth AS(SELECT ROW_NUMBER() OVER (ORDER BY JobName, JobTime ASC) OrderId, CAST(JobTime AS DATE) JobIdDate, *FROM etlJobTrackingWHERE JobName = @jobnameAND JobTime BETWEEN '2018-01-01' AND '2018-02-28'
), GroupAll AS(
SELECTt2.JobStepNumber, t.JobIdDate, DATEDIFF(MINUTE,t.JobTime,t2.JobTime) StepMinutes
FROM MeasureGrowth tINNER JOIN MeasureGrowth t2 ON t.JobIdDate = t2.JobIdDateAND t.OrderId = (t2.OrderId - 1)
)
SELECTMONTH(JobIdDate) JobMonth, JobStepNumber, AVG(StepMinutes) JobMonthAvgMinutes
FROM GroupAll
GROUP BY MONTH(JobIdDate), JobStepNumber

We see growth in the length of time from January to February of the first and third step.

我们看到第一步和第三步从1月到2月的时间长度有所增长。

The second point is one to consider from our measurements: as an example, if we see that our cleaning data step grows each month (especially geometric growth), scaling our design will be the only long-term solution – whether we scale the resources, such as more memory, CPU, etc or scaling out data horizontally where we clean batches of data at a time. One of the major reasons why scale becomes a problem for many companies is because they’re not measuring, comparing, and seeing the predicted growth patterns: they tend to do something only when something fails.

第二点是我们需要从测量中考虑的一点：例如，如果我们看到清洁数据步骤每个月都在增长（尤其是几何增长），那么缩放设计将是唯一的长期解决方案–是否缩放资源，例如更多的内存，CPU等，或水平扩展数据，一次可以清理一批数据。 对许多公司而言，规模成为问题的主要原因之一是因为他们没有测量，比较和查看预期的增长模式：它们倾向于仅在失败时才采取行动。

As our example in the above image shows, in two months, we’ve seen growth in two steps – the third step which grew from an average of one minute to an average of 38 minutes. This is an example of a problem sign and the earlier we scale, the faster we can avoid a major problem in the future. Scaling will cost, but scaling early costs less than scaling later, if our data volume is growing (we’ll have to scale more than less). It’s also possible that our data volume isn’t growing and we’re only seeing issues because of other processes that may be running in parallel that could be disrupting our job. The above query helps us measure the job side of our challenge – our data side and parallel queries and processes may indicate another challenge that doesn’t involve scale.

如上图中的示例所示，在两个月内，我们看到了两步增长–第三步从平均一分钟增长到平均38分钟。 这是一个问题征兆的示例，我们扩展的越早，将来就可以更快地避免出现重大问题。 如果我们的数据量在增长（我们将不得不进行更多扩展），那么进行扩展将需要一定的成本，但是早期进行扩展的成本要小于随后进行扩展的成本。 还有可能我们的数据量没有增长，而我们只是看到问题，因为可能并行运行的其他进程可能会干扰我们的工作。 上面的查询可以帮助我们衡量挑战的工作方面–数据方面以及并行的查询和流程可能表明另一个不涉及规模的挑战。

We should also consider that we may be able to combine steps. For an example, if we have to compare data to a data set range and we have another step for validating our data, can we combine these steps? Also, should be doing both with T-SQL or .NET? For an example, if we have a step that uses .NET which works with validating a row-based object, it may be most efficient to add all validation at that point, rather than do the validation in steps. The same applies to T-SQL: if we have to read a set of data for validation, can we combine all the validation into one SELECT statement versus several validation steps (think of multiple SELECT statements and the reads cost versus one on the same set of data). With the information about the growth, we can take the next steps of possibly combining, changing or scaling our design.

我们还应该考虑到我们可以组合步骤。 例如，如果我们必须将数据与数据集范围进行比较，而我们又有另一步骤来验证数据，我们可以将这些步骤组合在一起吗？ 另外，应该同时使用T-SQL还是.NET？ 例如，如果我们有一个使用.NET的步骤来验证基于行的对象，那么在该点添加所有验证可能比在步骤中进行验证更有效。 同样适用于T-SQL：如果我们必须读取一组数据进行验证，是否可以将所有验证合并到一个SELECT语句中，而不是几个验证步骤（考虑多个SELECT语句，并且读取成本与同一组中的一个相比）数据的）。 有了有关增长的信息，我们就可以采取可能的组合，更改或扩展设计的下一步。

摘要 (Summary)

Anytime we have a multi-step process, such as an ETL or application flow, we want to monitor the time of each step and use what we know about our time to prepare for our future design, scale, or change that it may require. We’ve look at how to get this information from the SQL Server Job Agent, if we’re using it, or how we can construct our own custom design, if we use other tools from SSIS to a custom application.

每当我们有一个多步骤的流程（例如ETL或应用程序流程）时，我们都希望监视每个步骤的时间，并利用我们所了解的时间来准备我们将来可能需要的设计，规模或更改。 我们正在研究如何使用SQL Server Job Agent获取这些信息（如果正在使用的话），或者如果我们使用从SSIS到自定义应用程序的其他工具，那么如何构建自己的自定义设计。

翻译自: https://www.sqlshack.com/tracking-times-in-data-flows-for-finding-performance-issues/

tcp码流中查找rtp头