如何使用SQL Server主数据服务替换硬编码的查找

介绍 (Introduction)

A big part of the technical debt in my organization’s data warehouse (DW) and business intelligence (BI) environments relates to hardcoded lookup data. This is data required by the business to make sense of transactional data but was never planned for in the underlying source system and consequently get injected into DW and BI solutions. Inevitably, it is only a matter of time before DW and BI team lose track of the places wherein the hardcoded data reside thus making it difficult to maintain. Furthermore, due to lack of documentation or staff retention, anyone who subsequently takes over these DW/BI solutions can unknowingly create duplicate lookup data. In this article, I explain how we reduced such technical debt in my organization by moving most of the hardcoded lookups into SQL Server Master Data Services (MDS).

我组织的数据仓库（DW）和商业智能（BI）环境中的技术债务很大一部分与硬编码的查找数据有关。业务需要使用这些数据来理解交易数据，但从未在底层源系统中计划这些数据，因此将其注入DW和BI解决方案中。不可避免地，DW和BI团队失去对硬编码数据所在位置的跟踪只是时间问题，因此难以维护。此外，由于缺乏文档或人员保留，任何随后接管这些DW / BI解决方案的人都可能在不知不觉中创建重复的查找数据。在本文中，我将说明如何通过将大多数硬编码的查询移至SQL Server主数据服务（MDS）中来减少组织中的此类技术负担。

对您的查询进行硬编码的挑战 (The Challenges of Hardcoding Your Lookups)

Figure 1 shows a sample QlikView dashboard that contains a breakdown of a total number of policy claims received by claimant’s gender. The problem with the information on this dashboard is that whilst it looks like gender ID 3 has submitted more policy claims compared to 1 and 2, consumers of this information may not readily understand what gender IDs 1, 2 and 3 actually stand for.

图1显示了一个示例QlikView仪表板，其中包含按索赔人性别收到的保单索赔总数的细分。此信息中心上的信息存在的问题是，尽管性别ID 3似乎比1和2提交了更多的保单声明，但是此信息的消费者可能不容易理解性别ID 1、2和3的实际含义。

In fact, at first glance, you might easily accuse the BI team of delivering a poorly designed dashboard, but actually the representation of claimant’s gender as 1, 2 and 3 in the dashboard can be traced back to the input dataset received from source system, which is shown in Table 1.

实际上，乍看之下，您可能会轻易地指责BI团队提供了一个设计不当的仪表板，但实际上，仪表板上索赔人的性别表示为1、2和3 可以追溯到从源系统接收的输入数据集，如表1所示。

Ideally, one would prefer that a data lookup for gender 1, 2 and 3 values should be handled by the source system development team but there could be circumstances that prevent or delay the implementation of such a lookup in the source system. So, reluctantly, the DW and BI teams could be asked (sometimes instructed) to temporarily implement a lookup solution within their platforms. The DW and BI teams could implement the lookup in the following manner:

理想情况下，希望由源系统开发团队处理针对性别1、2和3值的数据查找，但是在某些情况下可能会阻止或延迟在源系统中执行此类查找。因此，很不情愿地，可能会要求DW和BI团队（有时会被指示）在其平台上临时实施查找解决方案。 DW和BI团队可以通过以下方式实现查找：

DW Team’s Implementation of Hardcoded Lookup

The data warehouse team can create a derived dimension in a form of a SQL Server view that employs the UNION function to join several hardcoded SELECT statements as shown in Script 1.
```
CREATE VIEW [dbo].[DimGender]
ASSELECT 1 AS [ID],'Male' AS [Description]UNIONSELECT 2 AS [ID],'Female' AS [Description]UNIONSELECT 3 AS [ID],'Other' AS [Description];
GO
```
Script 1: Hardcoded Lookup Dimension

The BI team can then simply reference the newly created view as a lookup for gender in their dashboard. The updated dashboard which now includes gender description instead id gender ID is shown in Figure 2.

Figure 2: Updated QlikView Dashboard with Gender description

DW团队的硬编码查找实现

数据仓库团队可以以SQL Server视图的形式创建派生维，该视图采用UNION函数来联接几个硬编码的SELECT语句，如脚本1所示。
```
CREATE VIEW [dbo].[DimGender]ASSELECT 1 AS [ID],'Male' AS [Description]UNIONSELECT 2 AS [ID],'Female' AS [Description]UNIONSELECT 3 AS [ID],'Other' AS [Description];GO
```
脚本1：硬编码查找维度

然后，BI团队可以简单地引用新创建的视图作为其仪表板中性别的查找。更新后的仪表板现在包括性别描述而不是id性别ID， 如图2所示。

图2：更新后的带有性别描述的QlikView仪表板
BI Team’s Implementation of Hardcoded Lookup

Instead of relying on the DW team to first add a lookup dimension before it can be available for QlikView, the BI team can apply the hardcoding of gender values in their QlikView data model. This can be done by firstly capturing the hardcoded values in some excel document and then loading that excel document into QlikView as shown in Figure 3.

Figure 3: Loading of excel file into QlikView

BI团队的硬编码查找实现

BI团队可以在QlikView数据模型中应用性别值的硬编码，而不必依赖DW团队先添加查找维，然后才能将其用于QlikView。这可以通过首先捕获某些excel文档中的硬编码值，然后将该excel文档加载到QlikView中来完成， 如图3所示。

图3：将excel文件加载到QlikView中

As mentioned above, the main challenge with relying on hardcoded lookups is that whenever new lookup IDs are introduced in the transactional data, DW and BI solutions will remain outdated until you manually edit the lookup dataset again. To demonstrate such a scenario, I have altered the source dataset presented in Table 1 to include a 10^th transaction that now includes gender ID 4 as highlighted in yellow in Table 2.

如上所述，依赖硬编码查找的主要挑战是，每当在事务数据中引入新的查找ID时，DW和BI解决方案将保持过时，直到您再次手动编辑查找数据集为止。为了演示这种情况，我对表1中显示的源数据集进行了更改，使其包括第十^个事务，该事务现在包括性别ID 4，如表2中黄色突出显示的。

Once the data in QlikView is reloaded, you will immediately notice that the last vertical bar in the dashboard is missing gender description as shown Figure 4.

重新加载QlikView中的数据后，您将立即注意到，仪表板上的最后一个竖线缺少性别描述，如图4所示。

The gender description for the 4^th bar is missing because the excel document used as a lookup for gender IDs hasn’t been updated to include the description for gender ID 4. Once gender ID 4 is captured into the excel lookup document and QlikView data model is reloaded, the missing gender description will then be resolved. The challenge with this kind of manual intervention depends on the number of files needed to be updated. Therefore, while the updating of a single lookup document could seem trivial, having to update several multiple lookup files that are stored in various locations can become very tedious. Thus, a more robust and dynamic mechanism is required to address the issue of dealing with hardcoded lookup data.

对于^第 4栏中的性别说明缺少因为用作查找性别ID的excel文件还没有被更新为包括用于性别ID 4.描述一旦性别ID 4被捕获到的excel文件查找和QlikView的数据模型重新加载后，缺失的性别描述将得到解决。这种手动干预的挑战取决于需要更新的文件数量。因此，尽管单个查找文档的更新看起来很简单，但是必须更新存储在各个位置的多个多个查找文件却非常繁琐。因此，需要一种更健壮和动态的机制来解决处理硬编码查找数据的问题。

解 (Solution)

One such robust and dynamic mechanism involves implementing a master data management solution using SQL Server Master Data Services (MDS). MDS provides several features such as the ability to enable business users to define, capture and maintain master data (or lookups). It also reduces unnecessary time spent by IT resources (i.e. DW and BI team members) conducting data fixes. MDS ultimately provides a consolidated single view of domain data that can be referenced by different systems within an organization.

一种这样的健壮和动态的机制涉及使用SQL Server主数据服务（MDS）实现主数据管理解决方案。 MDS提供了多种功能，例如使业务用户能够定义，捕获和维护主数据（或查找）的功能。它还减少了IT资源（即DW和BI团队成员）进行数据修复所花费的不必要时间。 MDS最终提供了域数据的统一的单一视图，可以由组织内的不同系统引用。

To demonstrate the power of MDS, we have imported the gender lookup data from an excel document into an MDS entity called Gender Entity which is shown in Figure 5. Through this web-based interface (or using an excel add-in for MDS), business users can manage descriptions for all gender IDs.

为了演示MDS的功能，我们已经将性别查找数据从excel文档导入了一个名为Gender Entity的MDS实体， 如图5所示。通过此基于Web的界面（或使用MDS的excel加载项），业务用户可以管理所有性别ID的描述。

MDS further provides subscription views, which can be used by systems such as QlikView and Data Warehouse to access MDS data via the backend. Figure 6 shows a sample subscription view – GenderLookUp – that has been created to export Gender Entity data.

MDS进一步提供了订阅视图，QlikView和Data Warehouse等系统可以使用该订阅视图通过后端访问MDS数据。图6显示了一个示例订阅视图GenderLookUp ，它已创建为导出Gender Entity数据。

As soon as the subscription view is created, it will appear under the mdm schema in the Views sub-node of a SQL Server database configured to store MDS information, as shown in Figure 7.

创建预订视图后，它将立即显示在配置为存储MDS信息SQL Server数据库的Views子节点的mdm模式下， 如图7所示。

Finally, just remember that any structural changes to MDS entities will not be propagated into subscription views unless the subscription view is regenerated.

最后，请记住，除非重新生成订阅视图，否则对MDS实体的任何结构更改都不会传播到订阅视图中。

摘要 (Summary)

Hardcoded lookup data can contribute to an organization’s technical debt if not handled correctly. Using applications such as SQL Server Master Data Services both data ware and business intelligence teams can take advantage of moving the responsibility for maintaining the data back to business users. The DW and BI teams can use subscription views to access data captured in the back-end.

如果处理不当，硬编码的查找数据可能会增加组织的技术负担。使用诸如SQL Server主数据服务之类的应用程序，数据件和商业智能团队都可以利用将维护数据的责任转移回业务用户的优势。 DW和BI团队可以使用订阅视图来访问后端捕获的数据。

看更多 (See more)

For BI documentation, consider ApexSQL Doc, a tool that documents SQL Server instances, databases, objects, SSIS packages, SSAS cubes, SSRS reports, Tableau server sites and SharePoint Server farms.

对于BI文档，请考虑ApexSQL Doc （该工具记录SQL Server实例，数据库，对象，SSIS包，SSAS多维数据集，SSRS报告，Tableau服务器站点和SharePoint Server服务器场）。

参考资料 (References)

Master Data Services Overview (MDS) 主数据服务概述（MDS）
Create a Subscription View to Export Data (Master Data Services) 创建订阅视图以导出数据（主数据服务）
SQL UNION Operator SQL UNION运算符

翻译自: https://www.sqlshack.com/replace-hardcoded-lookups-using-sql-server-master-data-services/