AUDITMAP.AI文本分析平台 (AUDITMAP.AI TEXT ANALYSIS PLATFORM)

Table of Contents1. Abstract: Why AI for Internal Audit and Risk Management?2. Introduction3. Contemporary Internal Audit Challenges4. AuditMap.ai: A Platform for Audit Enhancement5. Limitations and the Way Forward6.

目录 1. 摘要：为什么要使用AI进行内部审计和风险管理？ 2. 简介 3. 当代内部审计面临的挑战 4. AuditMap.ai：增强审计的平台 5. 局限性和前进方向 6。

1.摘要：为什么要使用AI进行内部审计和风险管理？ (1. Abstract: Why AI for Internal Audit and Risk Management?)

Internal audit tasks within large organizations are slowed by the volume of documentation. Slow audit response time, sampling-based audit planning, and reliance on keyword searches are all indicators that automation is required to accelerate internal audit tasks. Audit quality also suffers when relevant gaps or risks are not disclosed to stakeholders in a timely manner. This work outlines a workflow automation solution called AuditMap.ai. The solution contains several artificial intelligence models that read in thousands of audit reports in various languages to continuously identify and organize the relevant text within. Rather than replacing the auditor, AuditMap.ai assists in the human-centered audit planning and execution process.

大量文档会减慢大型组织内的内部审核任务。 缓慢的审核响应时间，基于采样的审核计划以及对关键字搜索的依赖都表明自动化是加速内部审核任务所必需的。 如果没有及时向利益相关者披露相关的差距或风险，审计质量也会受到影响。 这项工作概述了名为AuditMap.ai的工作流自动化解决方案。 该解决方案包含多个人工智能模型，这些模型以各种语言读取成千上万的审核报告，以不断识别和组织其中的相关文本。 而不是更换审核员， AuditMap.ai可以协助以人为中心的审核计划和执行过程。

2.简介 (2. Introduction)

The internal audit function of organizations is under pressure to deliver results to meet the assurance demands of stakeholders and protect the organization from emerging threats. Keeping the big picture in perspective is difficult given the volume of reports that exist within an organization. This work outlines a solution that reads thousands of audit reports to categorize and organize the relevant text within the reports.

组织的内部审计职能正承受着交付成果以满足利益相关者的保证要求并保护组织免受新出现威胁的压力。 鉴于组织内存在大量报告，因此很难全面了解大局。 这项工作概述了一种解决方案，该解决方案可以读取数千个审核报告，以对报告中的相关文本进行分类和组织。

Internal auditors provide a line of defense against preventable errors and omissions that may lower quality, tarnish a firm’s reputation and trustworthiness, miss opportunities, or lead to direct financial losses. Internal audit’s main role is the detection of inefficiencies, noncompliances, and the prevention of losses. These activities are delivered by means of risk-based assessment and communication with board committees, whereas financial audit is focused on the detection of potentially material issues in recordings of transactions, and at times, corrections. In other words, losses detected in a financial audit have already occurred, while the results of internal audit’s work highlight gaps in compliance, quality, and other areas. To identify these gaps, internal audit operations require the tracking of outcomes in each risk area that the organization’s programs are exposed to. This tracking requires contextualization and fast results, rather than an annual summary report to the audit committee.

内部审计师提供了可预防的错误和疏漏的防护措施，这些错误和疏漏可能会降低质量，损害公司的声誉和信誉，错失机会或导致直接的财务损失。 内部审计的主要作用是发现效率低下，不合规以及防止损失。 这些活动是通过基于风险的评估和与董事会委员会的沟通来进行的，而财务审计则侧重于检测交易记录中有时可能存在的重大问题，有时还会进行更正。 换句话说，在财务审计中发现的损失已经发生，而内部审计的工作结果则突出了合规性，质量和其他领域的差距 。 为了识别这些差距，内部审计操作需要跟踪组织计划所面临的每个风险领域的结果。 这种跟踪需要情境化和快速的结果，而不是向审计委员会提交年度总结报告。

Internal audit holds a unique assurance position within the corporate structure. It has historical roots in financial auditing and has since evolved to provide a much greater range of assurance. Internal audit is quite different in practice from financial auditing. Whereas modern financial auditing assesses transactions and their recordings in support of financial statement accuracy (at times using double-entry accounting software); internal audit’s performance-based assessments serve the goal of reporting to the audit committee and senior management on the state of their organization’s governance, processes, procedures, risks, controls, case reports, and much more. Furthermore, at public corporations, and within specific sectors such as healthcare and finance, internal audit activities and risk disclosures are required by law [1] [2].

内部审计在公司结构中占有独特的保证地位。 它在财务审计中具有悠久的历史，并已发展成为提供范围更大的保证。 内部审计在实践上与财务审计完全不同。 鉴于现代财务审计会评估交易及其记录以支持财务报表的准确性(有时会使用重复输入的会计软件)； 内部审计基于绩效的评估的目标是向审计委员会和高级管理层报告其组织的治理，流程，程序，风险，控制，案例报告等的状态。 此外，在法律法规 [1] [2]中 ，在公共公司以及特定部门(例如医疗保健和金融)内，内部审计活动和风险披露是必需的 。

3.当代内部审计挑战 (3. Contemporary Internal Audit Challenges)

Internal reporting has crossed into the era of big data, and this is leading to information overload across the corporate landscape. As a result of the high volumes of report data, a Data Rich Information Poor situation is creeping across the audit landscape [3]. This situation is characterized by an organization tracking many indicators under the assumption that they assure quality, but missing the emerging risks because the information at hand was not measured by any of the tracked indicators. The answer must be better technology for converting data into actionable information. Technology is a critical driver of efficiency and productivity in the internal audit function [4]. Quoting from the 2017 International Standards for the Professional Practice of Internal Auditing: “Internal auditors must have sufficient knowledge of key information technology risks and controls and available technology-based audit techniques to perform their assigned work” [5]. As outlined in the 2019 Brydon report [6], the landscape within the auditing field is also shifting toward increased enforcement of the separation between auditors and their clients, driving companies to rotate their assurance providers at increased rates. Furthermore, the report identifies that “There appears to be a widespread consensus that automating existing data related audit tasks is underway and its extension inevitable.”

内部报告已进入大数据时代，这导致整个企业范围内的信息过载。 由于报告数据量很大，整个审计环境中都出现了“ 数据贫乏”信息不足的情况[3] 。 这种情况的特点是，组织在保证质量的前提下跟踪许多指标，但由于手头的信息没有用任何跟踪的指标来衡量，因此错过了新出现的风险。 答案必须是将数据转换为可操作信息的更好技术。 在内部审计职能中，技术是效率和生产率的关键驱动力 [4] 。 引用《 2017年内部审计专业实践国际标准》：“内部审计师必须对关键信息技术风险和控制以及可用的基于技术的审计技术有足够的了解，以执行其分配的工作” [5] 。 正如2019年Brydon报告[6]所概述的那样，审计领域的局面也正在朝着加强对审计师与其客户之间的隔离的执法方向发展，从而促使公司以更高的比率轮换其担保人。 此外，该报告还指出：“似乎已经达成了广泛的共识，即与现有数据相关的审计任务正在实现自动化，并且其扩展不可避免。”

Vast private textual datasets of internal reports have overwhelmed the traditional role of auditors. For example. an organization with 100,000 employees in a highly regulated field (e.g., aircraft manufacturing), can generate millions of documents over a 10 year period. Forming this carefully recorded report data into theories and assessment plans is time-consuming, and yet the results of these assessments can be extremely time-sensitive. There is a dire need for management to know where the risks are on a Tuesday, but the audit function tends to produce reports quarterly and annually [7]. Furthermore, risk management is under ever more pressure to look outwards for emerging risks. In addition to the pressure to deliver quickly, the assessments discussed within reports are routinely sample-based, resulting in a lack of full coverage. It is common to select reports to sample using keywords, which can lead to missing critical documents that state similar concepts without using the specified keywords. It is also common to miss the connection between reports across time, such as repeated risks, growing numbers of controls, and other time-based phenomena.

内部报告的大量私人文本数据集已经淹没了审计师的传统角色。 例如。 在高度管制的领域(例如飞机制造)中拥有100,000名员工的组织，可以在10年内生成数百万个文档。 将这些经过仔细记录的报告数据形成为理论和评估计划非常耗时，但是这些评估的结果可能非常耗时 。 管理层迫切需要知道星期二的风险在哪里，但是审计职能往往每季度和每年产生一次报告[7] 。 此外，风险管理承受着越来越大的压力，需要向外寻找新出现的风险。 除了快速交付的压力外，报告中讨论的评估通常以样本为基础，从而导致缺乏全面覆盖 。 通常使用关键字选择要采样的报告，这可能会导致缺少陈述相似概念而不使用指定关键字的关键文档。 通常会错过跨时间的报告之间的联系，例如重复风险，控件数量的增加以及其他基于时间的现象。

In addition to the need for automation when assessing large datasets, the human factor also calls for additional automation of audit processes. Human auditors experience pressure to understate material weaknesses [8]. The integration of algorithms into the analysis process may insulate human auditors from these pressures to some degree. In addition, humans are limited in their capacity to aggregate and make sense of large datasets. Unfortunately, operating with spreadsheets and word processing programs as the engines of team-based work product is limited by the human capacity to make sense of vast datasets. Text-based work product and reporting frustrate the creation of governance metrics and the delivery of planning activities. Planning a contemporary internal audit is, at its root, managing information overload. Reviewing large enterprise control environments is expensive and time-consuming. Even though reports are mostly digitized, the task remains daunting for any human team to read and understand in full. Program coverage tends to thin out in audit areas that are less tightly tied to perceived financial risk. Furthermore, non-obvious connections between topics can be overlooked, as low-risk areas receive fewer audit resources. Internal audit and risk management functions find themselves over-evaluating some areas of operations while missing other rarely audited ones. In response to these challenges, significant artificial intelligence and data analytics adoption initiatives have been undertaken by the major audit organizations in recent years [9] [10] [11]. The goal to obtain a quantifiable overview of core governance programs remains out of reach for most enterprises, as artificial intelligence technologies have not yet reached significant adoption within audit firms. Several applications of artificial intelligence within audit processes are proposed in the literature, but few are applied by the major audit firms [12] [13].

除了在评估大型数据集时需要自动化之外， 人为因素还要求对审核流程进行额外的自动化。 人类审计师承受着低估实质性弱点的压力[8] 。 将算法集成到分析过程中可以使审计人员在一定程度上免受这些压力的影响。 另外，人类聚集和理解大型数据集的能力有限。 不幸的是，使用电子表格和文字处理程序作为基于团队的工作产品的引擎受到人类理解大量数据集的能力的限制。 基于文本的工作产品和报告阻碍了治理指标的创建和计划活动的交付。 计划当代内部审计的根本是管理信息过载 。 审查大型企业控制环境既昂贵又耗时。 尽管报告大部分都是数字化的，但对于任何人类团队来说，要全面阅读和理解该任务仍然是艰巨的。 计划的覆盖范围往往在与感知的财务风险不太紧密相关的审计领域中变薄。 此外，由于低风险区域收到的审核资源较少，因此可以忽略主题之间的明显联系。 内部审计和风险管理职能发现自己对某些运营领域进行了过度评估，而忽略了其他很少审计的领域。 为应对这些挑战，主要审计组织近年来已采取了重要的人工智能和数据分析采用计划[9] [10] [11] 。 由于人工智能技术尚未在审计公司中得到广泛采用，因此对于大多数企业而言，获得可量化的核心治理计划概述的目标仍然遥不可及。 文献中提出了人工智能在审计过程中的几种应用，但主要审计公司却很少应用[12] [13] 。

Predictive models such as [14] have been put forward in the financial auditing academic literature, and this work leads naturally to similar predictive and automated innovations within audit workflows. Artificial intelligence is coming to internal audit and risk management functions and will present new opportunities for the transformation of corporate governance.

诸如[14]的预测模型已经在财务审计的学术文献中提出，并且这项工作自然导致了审计工作流程中类似的预测和自动化创新。 人工智能将用于内部审计和风险管理功能 ，并将为公司治理的变革提供新的机会。

Public disclosure is an area where correct and timely identification of risks is critical, and often mandated by law. In a public relations crisis, identifying relevant information in reports for subsequent public disclosure is important and time-sensitive. Often this information is not tracked within a risk register or quality management system, as the risk in question may be new or unexpected. Regulatory risk disclosures can also be time-critical, as filing dates can be at times inflexible. Risk disclosures in corporate filings increase risk perceptions among investors [15], and so, perhaps unsurprisingly, useful risk disclosures in corporate filings are rare. The legal requirements to disclose risks are subjective and therefore not difficult to circumvent with generic statements [15] [16] [17]. However, research reveals a relationship between annual filings and SEC comment letters [18], whereby corporations are more likely to disclose risks if (1) they perceive that non-disclosure may lead to a finding by the SEC, or (2) after an SEC finding was issued to the corporation by the SEC. Given the importance of risk disclosure in quarterly and annual filings, it is clear that there is a strong need for a solution that can detect risks in a timely manner to facilitate the disclosure process, especially in time-critical situations. More generally, assessing the strength of quality management is an important capability for internal auditors to have [19] [20].

公开披露是正确，及时地识别风险至关重要的领域 ，通常是法律规定的。 在公共关系危机中，识别报告中的相关信息以进行后续的公开披露非常重要且对时间敏感。 通常，在风险记录或质量管理系统中不会跟踪此信息，因为所讨论的风险可能是新风险，也可能是意料之外的风险。 监管风险披露也可能是时间紧迫的，因为申请日期有时可能不够灵活。 公司备案中的风险披露增加了投资者对风险的认识[15] ，因此，毫不奇怪，公司备案中有用的风险披露很少。 披露风险的法律要求是主观的，因此不难用一般性陈述来规避[15] [16] [17] 。 但是，研究表明，年度备案与SEC评论信之间存在一种关系[18] ，如果公司(1)认为未披露可能导致SEC做出发现，或者(2)在发现并披露之后，公司更有可能披露风险。 SEC的调查结果是由SEC向公司发布的。 鉴于风险披露在季度和年度申报中的重要性，很显然， 强烈需要一种能够及时发现风险以促进披露过程的解决方案 ，尤其是在时间紧迫的情况下。 更一般而言，评估质量管理的强度是内部审核员拥有[19] [20]的重要能力。

Prior to the release of AuditMap.ai, machine learning has been applied to risk disclosure documents for various applications such as annual report analysis to assess similarity [21], internal financial controls [22], and IT security [23] [24]. These and similar applications of machine learning within the audit sphere represent initiatives moving toward a larger goal of digital transformation and predictive auditing. The lag in adoption of natural language processing and machine learning in internal audit, relative to other fields such as law and accounting, could be explained by institutional inertia, a lack of training datasets, the reimbursement model for consultants, the requirement to understand documents in multiple languages, and differing standards for reporting. These various factors holding back the field are now shifting, leading to a major opportunity for audit automation with machine learning [11]. Assessing these factors in more detail, the hourly pay structure for professional services firms may discourage innovations that reduce billable hours. In addition, the data required to model audit processes is also a closely guarded corporate secret, and so labeled report data must be painstakingly collected and labeled by subject matter experts. The type of text data in the reports varies widely between teams within an organization. Assessments such as country risk can focus entirely on external documentation, while internal controls can focus entirely on internal documentation. Another factor holding back the adoption of artificial intelligence in audit is the lack of data in multiple languages such as English, French, German, and Arabic. Furthermore, the features of audit reports are unusual as compared to standard text corpora such as news reports and books. Specifically, audit reports express a higher language level than typical documents, because of their requirement to abstract complex problem patterns. In addition, a number of internal audit standards and risk management frameworks exist, and their adoption varies by geography. For example, ISO 31000 [25] is more prevalent in Europe, whereas COSO [26] is more prevalent in the United States. Other important frameworks include COBIT [27], TSC [28], and NIST [29]. Internal auditors use these frameworks to ensure best practices, and these frameworks are key to the reproducibility of high-quality audits.

在AuditMap.ai发布之前，机器学习已应用于各种应用程序的风险披露文档，例如用于评估相似性的年度报告分析[21] ，内部财务控制[22]和IT安全[23] [24] 。 审计领域内机器学习的这些以及类似的应用代表着朝着数字转换和预测性审计的更大目标迈进的举措。 相对于法律和会计等其他领域，内部审计在采用自然语言处理和机器学习方面的滞后现象可以用制度惯性，缺乏培训数据集，顾问的报销模型，理解法律文件的要求来解释。多种语言和不同的报告标准 。 这些阻碍该领域发展的各种因素正在发生变化，从而为利用机器学习进行审计自动化带来了重大机遇[11] 。 更详细地评估这些因素， 专业服务公司的时薪结构可能会阻碍减少可计费时间的创新 。 此外，为审计流程建模所需的数据也是一个严密保护的公司机密，因此必须由主题专家精心收集和标记带有标签的报告数据。 报告中的文本数据类型在组织内的各个团队之间差异很大。 国家风险之类的评估可以完全侧重于外部文档，而内部控制可以完全侧重于内部文档。 阻碍在审计中采用人工智能的另一个因素是缺乏英语，法语，德语和阿拉伯语等多种语言的数据 。 此外，与诸如新闻报道和书籍之类的标准文本语料库相比，审计报告的功能是不寻常的。 具体而言，由于审计报告要求抽象复杂的问题模式，因此其表达的语言水平高于典型文档。 此外，存在许多内部审计标准和风险管理框架 ，其采用情况因地理位置而异。 例如，ISO 31000 [25]在欧洲更为普遍，而COSO [26]在美国更为普遍。 其他重要框架包括COBIT [27] ，TSC [28]和NIST [29] 。 内部审计师使用这些框架来确保最佳实践，而这些框架对于高质量审计的可重复性至关重要 。

Audit quality, speed, and competition for efficiency are drivers of artificial intelligence adoption. For example, the need for timely identification of gaps or risks and their disclosure to stakeholders has a tight connection with corporate performance. Adoption will also involve auditor education and scientific testing regimes for monitoring artificial intelligence performance. Using publicly disclosed corporate reports, benchmarks for this performance evaluation should be developed in future work, in multiple languages. The recall, precision, and bias in each model should be tested with these benchmarks.

审计质量，速度和效率竞争是人工智能应用的驱动力。 例如，需要及时发现差距或风险并将其披露给利益相关者，这与公司绩效紧密相关。 采用还将涉及审计师教育和科学测试制度，以监测人工智能的绩效。 使用公开披露的公司报告，应在将来的工作中以多种语言制定绩效评估的基准。 应使用这些基准测试每个模型的召回率，精度和偏差。

There is not broad agreement in the assurance industry or the academic literature regarding the nature of the coming changes. Some assessments conclude that auditors will be replaced by artificial intelligence innovations [10]. This is likely incorrect. Instead, the future is likely one where auditors work with artificial intelligence in the same way that they have adopted spreadsheets and word processing to enhance their workflows with digital automation. This work takes the position that incremental improvement through the application of many specialized models will provide the initial boost of automation to audit teams. On a longer-term basis, the broad replacement of auditing with technology is very unlikely.

在保险行业或学术文献中，关于即将发生的变化的性质尚未达成广泛共识。 一些评估得出结论认为，审计师将被人工智能创新所取代[10] 。 这可能是不正确的。 取而代之的是，审计师可能会采用人工智能，就像采用电子表格和文字处理来增强数字自动化的工作流程一样，是一种未来。 这项工作的立场是， 通过应用许多专业模型进行的逐步改进将为审计团队提供自动化的初步推动。 从长远来看，用技术广泛取代审计是极不可能的。

Auditors are likely to keep their existing processes in place while working to execute them more often, and with higher coverage as a result of artificial intelligence solutions. The future is more assistive than prescriptive. In this view, artificial intelligence does not replace auditor decision making, judgment, or assessment interviews. Instead, innovations accelerate planning and execution activities related to corrective and preventive actions. The key outcomes should be increased audit quality and speed, moving in the direction of continuous auditing.

审计师可能会保留他们现有的流程，同时努力更频繁地执行它们，并且由于人工智能解决方案而覆盖面更广。 未来比规定更具辅助性 。 按照这种观点，人工智能不能代替审核员的决策，判断或评估访谈。 相反，创新会加快与纠正和预防措施有关的计划和执行活动。 关键结果应该是提高审计质量和速度，朝着持续审计的方向发展。

4. AuditMap.ai：增强审计的平台 (4. AuditMap.ai: A Platform for Audit Enhancement)

AuditMap.ai is a solution for audit teams that can help them to make sense of large amounts of documentation. It can also be used by risk managers to discover emerging risks. The solution enables audit teams to quickly retrieve and action text within uploaded documents. The activities performed with AuditMap.ai are required as part of the strategic and tactical planning activities of the internal audit function. The solution automatically performs activities in support of the internal auditors’ information-intensive tasks. Figure 1 below summarizes the process through which auditors make use of the solution.

AuditMap.ai是面向审核团队的解决方案，可以帮助他们理解大量文档。 风险管理人员也可以使用它来发现新出现的风险。 该解决方案使审核团队能够快速检索上载文档中的文本并采取措施。 内部审计职能的战略和战术计划活动需要使用AuditMap.ai执行的活动。 该解决方案自动执行活动以支持内部审核员的信息密集型任务。 下面的图1总结了审核员使用解决方案的过程。

The audit team begins using the platform by defining business goals. They then proceed to define the organization’s preferred audit topics. The team also uploads their audit reports and other documents to the platform via manual upload or an Extract, Transform, and Load (ETL) task (Fig 1 (a)). The platform includes a dataset concept for managing document sets across clients. During ingestion, a machine learning model within the platform classifies the uploaded documents against the defined audit topics. The model architectures are based upon state-of-the-art machine learning models [30] [31] [32], trained on proprietary training datasets. Additional machine learning models perform automated extraction of linguistic entities, extract entity relationships, a cross-document analysis of statement similarity, and classification of key statements — those indicative of corporate risk, mitigations, and those indicative of key insights. Further processing is performed in order to assess the relevance of document segments to generally accepted enterprise risk management frameworks (Fig 1 (b)). The findings resulting from document ingestion and automated analysis are made available through the system’s user portal, a web application allowing auditors to perform a technology-assisted review of the contents with role-based access control. When exploring the results of the machine learning processes, auditors can observe trends over time within programs or topics, and can flag specific risks or controls for deeper analysis and paragraph or document-level context, or for relabeling (Fig 1 (c)). Lastly, the solution includes an interactive workbench for the rapid creation and export of working papers.

审核团队通过定义业务目标开始使用平台。 然后，他们继续定义组织的首选审核主题。 该团队还通过手动上传或提取，转换和加载(ETL)任务(图1(a))将其审计报告和其他文档上传到平台。 该平台包括一个数据集概念，用于跨客户端管理文档集。 在摄取期间，平台内的机器学习模型会根据定义的审核主题对上传的文档进行分类。 模型架构基于在专有训练数据集上进行训练的最新机器学习模型[30] [31] [32] 。 其他机器学习模型执行语言实体的自动提取，提取实体关系，对语句相似性的跨文档分析以及关键语句的分类-那些表示企业风险，缓解措施和那些关键见解。 为了评估文档段与公认的企业风险管理框架之间的相关性，需要执行进一步的处理(图1(b))。 通过系统的用户门户(通过Web应用程序，审核员可以使用基于角色的访问控制对内容进行技术辅助的审核)可以获取来自文档提取和自动分析的结果。 在探索机器学习过程的结果时，审核员可以观察计划或主题内一段时间内的趋势，并可以标记特定的风险或控件以进行更深入的分析和段落或文档级上下文，或进行重新标记(图1(c))。 最后，该解决方案包括一个交互式工作台，用于快速创建和导出工作文件。

The platform provides auditors and risk managers with a simplified, self-directed capacity to manually include information discovered during research by reducing the steps between the identification of information in a dataset of documents, and its addition to work artifacts (Fig 1 (d)). The delivery of work items to stakeholders is accomplished via export (Fig 1 (e)).

该平台为审核员和风险管理人员提供了简化的自我指导能力，可通过减少在文档数据集中识别信息与将其添加到工作工件之间的步骤来手动包括研究期间发现的信息(图1(d)) 。 通过出口完成向利益相关者交付工作项目(图1(e))。

Figure 2 shows some of the user interface components used by auditors. The platform enables the narrowing down of focus. For example, in a selected dataset with 17,571 sentences from 35 reports, only 418 sentences were highlighted as being indicative of risk. Some were not “real” sentences, as they may be sentence fragments such as table of contents entries, or table data. With that in mind, AuditMap was able to provide a 97.6% reduction in data to be analyzed. 9,800 entities were identified. Some examples of interesting sentences identified in publicly available reports as indicative of risk are the following (numbers in round braces indicate classification confidence):

图2显示了审核员使用的一些用户界面组件。 该平台可以缩小焦点范围。 例如，在来自35个报告的具有17571个句子的选定数据集中，只有418个句子被突出显示为风险指示。 有些不是“真实”的句子，因为它们可能是句子片段，例如目录条目或表数据。 考虑到这一点，AuditMap能够减少97.6％的数据进行分析。 确定了9,800个实体。 以下是一些在公共报告中标识的一些有趣的句子，它们表示风险：(大括号中的数字表示分类的置信度)：

• (98.4%) “We noted that no prioritization exercise was documented to determine which JHAs were to be conducted first nor did we see evidence that priority was given to the development of JHAs based on recent events incidents or operational risks.” [33]

  (98.4％) “我们注意到，没有记录优先级的活动来确定首先要进行哪些JHA，也没有证据表明基于最近发生的事件或操作风险优先开发JHA。” [33]

• (52.8%) “Given that similar findings have been identified in past audits, we would suggest that [Entity] require all regional SCCs to use printer codes to retrieve printed [Identification Data] letters from shared network printers.” [34]

  (52.8％) “鉴于在过去的审计中发现了类似的发现，我们建议[实体]要求所有区域性SCC使用打印机代码从共享的网络打印机中检索打印的[标识数据]字母。” [34]

• (99.3%) “Based on interviews conducted, it was found that [Department1] used backups in the past to selectively fix problems with regards to the three application systems; however, it has not had to perform a full database recovery.” [35]

  (99.3％) “根据进行的访谈，发现[部门1]过去使用备份来有选择地解决有关三个应用程序系统的问题； 但是，它不必执行完整的数据库恢复。” [35]

• (86.1%) “Les dossiers relatifs aux installations et à l’administration de l’approvisionnement ne semblaient pas être appuyés par des documents étayant une piste d’audit uniforme” [36]

  (86.1％) “在类似的文件申请审批过程中获得辅助安装和管理的文件” [36]

• (73.7%) “Die EFK vermisst in den Abläufen und bei den Kontrollhandlungen im Prozess die angemessene Nachvollziehbarkeit und Transparenz” [37]

  (73.7％) “在EFAkäufen和ben den Kontrollhandlungen im Prozess死于Angemessene Nachvollziehbarkeit和Transparenz中的EFK失败” [37]

5.局限性和前进的方向 (5. Limitations and the Way Forward)

Adoption of AuditMap.ai artificial intelligence into the audit and risk management industry is likely to change outcomes. It is likely to change the nature of assurance itself. However, artificial intelligence adoption within audit must be paired with a quantitative assessment of the limitations of the technology, and staff training that emphasizes the limitations of the technology. Blind adoption could lead to reputational risk in the event of artificial intelligence failures. It is therefore prudent to be aware of the functional limitations of machine learning in relation to assurance and assess the acceptability of these limitations.

在审计和风险管理行业中采用AuditMap.ai人工智能很可能会改变结果。 这很可能会改变担保本身的性质。 但是，审计中采用人工智能必须与对技术局限性的定量评估以及强调技术局限性的人员培训相结合。 如果出现人工智能故障，盲目采用可能会导致声誉风险 。 因此，应谨慎考虑与保证有关的机器学习的功能限制，并评估这些限制的可接受性。

The two types of machine learning applied in AuditMap.ai are supervised learning for classification, and unsupervised learning for contextual representation and similarity assessment. Supervised learning models applied to proprietary client data is unlikely to have perfect recall and precision. This means that some risks and controls will be missed by the algorithm, and some statements will be incorrectly classified. It is critical for the human auditor to understand these limitations, and to have easy access to a corrective capability within the workflow that can relabel statements on the fly. AuditMap.ai does have this capability.

AuditMap.ai中应用的两种类型的机器学习是用于分类的监督学习和用于上下文表示和相似度评估的无监督学习。 应用于专有客户数据的监督学习模型不太可能具有完美的召回率和准确性。 这意味着该算法将遗漏某些风险和控制，并且某些语句将被错误地分类 。 对于人工审核员而言，了解这些限制并在工作流程中轻松获得可以即时重新标记报表的纠正功能至关重要。 AuditMap.ai确实具有此功能。

Supervised learning is also susceptible to learning bias from data, if it is trained on arbitrary client data, and therefore AuditMap.ai models are trained on a proprietary primary dataset addressing this issue, prior to deployment into an auditor’s environment. Although bias may be addressed in the initial deployment, it is an issue that needs to be measured and assessed, especially when model retraining takes place.

如果对有监督的学习进行了任意客户数据训练，则还容易受到来自数据的学习偏见的影响，因此，在部署到审计员的环境之前，AuditMap.ai模型是在解决该问题的专有主数据集上进行训练的。 尽管在初始部署中可能会解决偏差，但这是一个需要度量和评估的问题 ，尤其是在进行模型重新训练时。

Unsupervised learning is similarly limited to the contexts it has been exposed to. The technology is susceptible to errors when faced with a radically new context. In some cases, a supervised model relies on the representation created using unsupervised learning, and changing the distribution that the unsupervised model was trained on can ruin the predictive power of the supervised model. For example, the models in AuditMap.ai are trained to classify text from audit reports, and have never been exposed to email messages or text messages. Feeding such data into the models results in poor similarity understanding because their writing style and vocabulary are radically different from the training data. It is therefore important to consider the scope of the data that is included in the technology adoption prior to deployment.

类似地， 无监督学习仅限于其所接触的环境。 面对全新的环境时，该技术容易出错。 在某些情况下，监督模型依赖于使用无监督学习创建的表示形式，并且更改无监督模型所受训练的分布可能会破坏监督模型的预测能力。 例如，对AuditMap.ai中的模型进行了训练，可以对审核报告中的文本进行分类，并且从未暴露给电子邮件或文本消息。 将此类数据输入模型中会导致对相似性的理解不佳，因为它们的写作风格和词汇与培训数据根本不同。 因此，重要的是在部署之前考虑技术采用中包含的数据范围 。

Missing information is another key issue to consider. There is often information that is outside the dataset of audit reports and working papers, that is only obtainable by going out into the real world and collecting data through the process of internal audit. The assumption that information extracted from an internal audit dataset (e.g., relationship graphs, risks, mitigations, insights) fully covers the state of the organization is surely false. Auditors need to remain curious and ask tough questions about missing risks, missing procedures, and generally understand where internal audit has poor coverage in terms of internal assessments. AuditMap.ai helps the audit team to identify where information is likely missing. However, the initiative to fill in the blanks still remains with the human internal audit team. Having access to the big-picture view enables the audit team to think about what information may be missing by topic, or through time.

信息丢失是要考虑的另一个关键问题。 通常，审计报告和工作文件的数据集之外的信息只能通过进入真实世界并通过内部审计过程来收集数据来获得。 从内部审计数据集中提取的信息(例如，关系图，风险，缓解措施，洞察力)完全覆盖组织状态的假设肯定是错误的。 审计师需要保持好奇心，并就缺失的风险，缺失的程序提出棘手的问题，并通常了解内部审计在内部评估方面覆盖面较弱的地方。 AuditMap.ai帮助审核团队确定可能丢失信息的位置。 但是，填补内部空白的主动权仍然由内部人员审核小组负责。 可以访问全局视图，使审核团队可以按主题或时间考虑可能丢失的信息。

Auditors have to ask whether adoption of this imperfect and approximate technology is better than the status quo, and if it improves the quality and speed of audits. Auditors should evaluate AuditMap.ai adoption quantitatively and dispassionately as they consider adopting the technology. We are running a series of webinars to engage with audit and risk management professionals, demonstrate the platform, and line up pilots.

审核员必须询问采用这种不完善和近似的技术是否比现状更好 ，以及是否可以提高审核的质量和速度 。 审核员在考虑采用该技术时，应定量且无差别地评估AuditMap.ai的采用。 我们正在举办一系列网络研讨会，与审计和风险管理专业人士互动，演示平台并安排试点。

The website will soon post a link to the upcoming webinar. If you liked this article, then have a look at some of my past articles in AI for internal audit, “How AuditMap.ai Improves Internal Audit” and “Better Internal Audits with Artificial Intelligence.” I also want to thank professor Miodrag Bolic from the University of Ottawa for his feedback on this work. Have you noticed that AuditMap.ai has a new website? And hey, join the newsletter via the site!

该网站将很快发布到即将举行的网络研讨会的链接 。 如果您喜欢这篇文章，请查看我过去在AI中用于内部审核的一些文章，“ AuditMap.ai如何改善内部审核 ”和“ 使用人工智能更好地进行内部审核 ”。 我还要感谢渥太华大学的Miodrag Bolic教授对这项工作的反馈。 您是否注意到AuditMap.ai有一个新网站？ 嘿，通过该网站加入新闻通讯！

Until next time!

直到下一次！

-Daniel

-丹尼尔

6.参考 (6. References)

[1] United States Public Law: Quality System Regulation. 21 CFR part 820 (1996)[2] United States Public Law: Prospectus summary, risk factors, and ratio of earnings to fixed charges (Item 503). 17 CFR part 229.503 (2011)[3] Goodwin, S.: Data rich, information poor (drip) syndrome: is there a treatment? Radiology management 18(3) (1996) 45–49[4] Eulerich, M., Masli, A.: The use of technology based audit techniques in the internal audit function–is there an improvement in efficiency and effectiveness? Available at SSRN 3444119 (2019)[5] Institute of Internal Auditors: International standards for the professional practice of internal auditing. Institute of Internal Auditors (2017)[6] Sir Donald Brydon, CBE: Assess, Assure And Inform: Improving Audit Quality And Effectiveness; Report Of The Independent Review Into The Quality And Effectiveness Of Audit. The Crown (2019) Accessed on Jan 2, 2020 https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment%5Fdata/file/852960/brydon-review-final-report.pdf.[7] Chan, D.Y., Vasarhelyi, M.A.: Innovation and practice of continuous auditing. International Journal of Accounting Information Systems 12(2) (2011) 152–160 [8] Cowle, E., Rowe, S.P.: Don’t make me look bad: How the audit market penalizes auditors for doing their job. (September 2019) Available at SSRN: https://ssrn.com/abstract=3228321.[9] Kokina, J., Davenport, T.H.: The emergence of artificial intelligence: How automation is changing auditing. Journal of Emerging Technologies in Accounting 14(1)(2017) 115–122[10] Alina, C.M., Cerasela, S.E., Gabriela, G., et al.: Internal audit role in artificial intelligence. Ovidius University Annals, Economic Sciences Series 18(1) (2018) 441–445[11] Sun, T., Vasarhelyi, M.A., et al.: Embracing textual data analytics in auditing with deep learning. (2018) Universidad de Huelva.[12] Sun, T., Vasarhelyi, M.A.: Deep learning and the future of auditing: How an evolving technology could transform analysis and improve judgment. CPA Journal 87(6) (2017)[13] Appelbaum, D.A., Kogan, A., Vasarhelyi, M.A.: Analytical procedures in external auditing: A comprehensive literature survey and framework for external audit analytics. Journal of Accounting Literature 40 (2018) 83–101[14] Kuenkaikaew, S., Vasarhelyi, M.A.: The predictive audit framework. The International Journal of Digital Accounting Research 13(19) (2013) 37–71[15] Kravet, T., Muslu, V.: Textual risk disclosures and investors’ risk perceptions. Review of Accounting Studies 18(4) (2013) 1088–1122[16] Schrand, C.M., Elliott, J.A.: Risk and financial reporting: A summary of the discussion at the 1997 aaa/fasb conference. Accounting Horizons 12(3) (1998) 271[17] Jorgensen, B.N., Kirschenheiter, M.T.: Discretionary risk disclosures. The Accounting Review 78(2) (2003) 449–469[18] Brown, S.V., Tian, X., Wu Tucker, J.: The spillover effect of sec comment letters on qualitative corporate disclosure: Evidence from the risk factor disclosure. Contemporary Accounting Research 35(2) (2018) 622–656[19] Bhattacharya, U., Rahut, A., De, S.: Audit maturity model. Computer Science Information Technology 4 (12 2013)[20] Thabit, T.: Determining the effectiveness of internal controls in enterprise risk management based on COSO recommendations. In: International Conference on Accounting, Business Economics and Politics. (2019)[21] Fan, J., Cohen, K., Shekhtman, L.M., Liu, S., Meng, J., Louzoun, Y., Havlin, S.: A combined network and machine learning approaches for product market forecasting. arXiv preprint arXiv:1811.10273 (2018)[22] Boskou, G., Kirkos, E., Spathis, C.: Assessing internal audit with text mining. Journal of Information & Knowledge Management 17(02) (2018) 1850020[23] Boxwala, A.A., Kim, J., Grillo, J.M., Ohno-Machado, L.: Using statistical and machine learning to help institutions detect suspicious access to electronic health records. Journal of the American Medical Informatics Association 18(4) (2011) 498–505[24] Endler, D.: Intrusion detection. applying machine learning to Solaris audit data. In: Proceedings 14th Annual Computer Security Applications Conference (Cat. №98EX217), IEEE (1998) 268–279[25] International Organization for Standardization: Risk management — Guidelines. Standard, ISO 31000:2018, Geneva, CH (February 2018)[26] Committee of Sponsoring Organizations of the Treadway Commission and others: Internal Control — Integrated Framework. (2013)[27] Information Systems Audit and Control Association: Cobit 5: Implementation. ISACA (2012)[28] American Institute of Certified Public Accountants: Trust Services Criteria. AICPA (2017) Accessed on Jan 15, 2020https://www.aicpa.org/content/dam/aicpa/interestareas/frc/assuranceadvisoryservices/downloadabledocuments/trust-services-criteria.pdf.[29] Bowen, P., Hash, J., Wilson, M.: Information security handbook: a guide for managers. In: NIST Special Publication 800–100, National Institute of Standards and Technology. (2007)[30] Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018)[31] Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 (2019)[32] Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)[33] Internal Audit and Program Evaluation Directorate: Audit of Occupational Health and Safety, March 2019. Technical report, Canada Border Services Agency, Ottawa, CA (March 2019)[34] Internal Audit Services Branch: Audit of the Management and Delivery of the Social Insurance Number Program, December 2015. Technical report, Employment and Social Development Canada, Ottawa, CA (December 2015)[35] Internal Audit Services Branch: Audit of the Departmental Information System and Technology Controls — Phase 1– Application Controls, 2014. Technical report, Employment and Social Development Canada, Ottawa, CA (November 2014)[36] Audit interne: Achats et marchés, Novembre 2018 Rapport d’audit interne. Technical report, Bureau du surintendant des institutions financieres, Ottawa, CA (November 2018)[37] Swiss Federal Audit Office: Prüfung der IT-Plattform NOVA für den öffentlichen Verkehr — Schweizerische Bundesbahnen. Technical report, Switzerland, Bern, Switzerland (July 2019)

[1]美国公法：质量体系法规。 21 CFR第820部分(1996) [2]美国公法：招股说明书摘要，风险因素以及收益与固定费用的比率(项目503)。 17 CFR part 229.503(2011) [3] Goodwin，S .：数据丰富，信息贫乏(滴灌)综合征：有治疗方法吗？ 放射学管理18(3)(1996)45-49 [4] Eulerich，M.，Masli，A .：在内部审计职能中使用基于技术的审计技术–效率和效力是否有所改善？ 可在SSRN 3444119(2019)中获得。 [5]内部审计师协会：内部审计专业惯例的国际标准。 内部审计师协会(2017) [6] CBE Donald Brydon爵士：评估，保证和告知：提高审计质量和有效性； 关于审核质量和有效性的独立审核报告。 The Crown(2019)于2020年1月2日访问https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment%5Fdata/file/852/brydon-review-final-report.pdf 。 [7] Chan，DY，Vasarhelyi，MA：持续审核的创新与实践。 国际会计信息系统杂志12(2)(2011)152–160 [8] Cowle，E.，Rowe，SP：别让我看上去不好：审计市场如何惩罚审计师从事工作。 (2019年9月)在SSRN上可用： https ://ssrn.com/abstract=3228321。 [9] Kokina，J。，达文波特，TH：人工智能的出现：自动化如何改变审计。 会计新兴技术杂志14(1)(2017)115-122 [10] Alina，CM，Cerasela，SE，Gabriela，G.等人：内部审计在人工智能中的作用。 Ovidius大学年鉴，经济科学丛书18(1)(2018)441–445 [11] Sun，T.，Vasarhelyi，MA等：在深度学习审计中拥抱文本数据分析。 (2018)韦尔瓦大学。 [12] Sun，T.，Vasarhelyi，MA：深度学习和审计的未来：不断发展的技术如何能够改变分析并改善判断力。 CPA Journal 87(6)(2017) [13] Appelbaum，DA，Kogan，A.，Vasarhelyi，MA：外部审计中的分析程序：全面的文献调查和外部审计分析框架。 会计文学杂志40(2018)83–101 [14] Kuenkaikaew，S.，Vasarhelyi，MA：预测性审计框架。 国际数字会计研究杂志13(19)(2013)37-71 [15] Kravet，T.，Muslu，V .：文本风险披露和投资者的风险感知。 会计研究评论18(4)(2013)1088–1122 [16] Schrand，CM，Elliott，JA：风险与财务报告：1997年Aaa / fasb会议的讨论摘要。 Accounting Horizo​​ns 12(3)(1998)271 [17] Jorgensen，BN，Kirschenheiter，MT：全权风险披露。 《会计评论》 78(2)(2003)449–469 [18] Brown，SV，Tian，X.，Wu Tucker，J.：sec评论信对定性公司披露的溢出效应：来自风险因素披露的证据。 当代会计研究35(2)(2018)622–656 [19] Bhattacharya，U.，Rahut，A.，De，S .：审计成熟度模型。 计算机科学信息技术4(12 2013) [20] Thabit，T .：基于COSO建议确定内部控制在企业风险管理中的有效性。 在：国际会计，商业经济学和政治学会议。 (2019) [21] Fan，J.，Cohen，K.，Shekhtman，LM，Liu，S.，Meng，J.，Louzoun，Y.，Havlin，S .:结合网络和机器学习方法的产品市场预测。 arXiv预印本arXiv：1811.10273(2018) [22] Boskou，G.，Kirkos，E.，Spathis，C .：使用文本挖掘评估内部审核。 信息与知识管理杂志17(02)(2018)1850020 [23] Boxwala，AA，Kim，J.，Grillo，JM，Ohno-Machado，L .：使用统计和机器学习来帮助机构检测对电子的可疑访问健康记录。 美国医学信息学协会杂志18(4)(2011)498–505 [24] Endler，D .：入侵检测。 将机器学习应用于Solaris审核数据。 在：会议记录第14届年度计算机安全应用程序会议(目录编号98EX217)，IEEE(1998)268-279 [25]国际标准化组织：风险管理—准则。 标准，ISO 31000：2018，日内瓦，CH(2018年2月) [26]特雷德韦委员会和其他组织的赞助组织委员会：内部控制—集成框架。 (2013) [27]信息系统审计与控制协会：Cobit 5：实施。 ISACA(2012) [28]美国注册会计师协会：信托服务标准。 AICPA(2017)，于2020年1月15日访问https://www.aicpa.org/content/dam/aicpa/interestareas/frc/assuranceadvisoryservices/downloadabledocuments/trust-services-criteria.pdf 。 [29] Bowen，P.，Hash，J.，Wilson，M .：信息安全手册：管理者指南。 在：美国国家标准与技术研究院NIST特别出版物800–100中。 (2007) [30] Devlin，J.，Chang，M.，Lee，K.，Toutanova，K .: BERT：为理解语言而对深度双向转换器进行的预训练。 CoRR abs / 1810.04805(2018) [31] Yang，Z.，Dai，Z.，Yang，Y.，Carbonell，J.，Salakhutdinov，R.，Le，QV：Xlnet：语言理解的广义自回归预训练。 arXiv预印本arXiv：1906.08237(2019) [32] Kim，Y .：用于句子分类的卷积神经网络。 arXiv预印本arXiv：1408.5882(2014) [33]内部审计与计划评估局：职业健康与安全审计，2019年3月。技术报告，加拿大边境服务局，加拿大渥太华(2019年3月) [34]内部审计服务处：《社会保险号码计划》的管理和交付审核，2015年12月。技术报告，加拿大就业与社会发展部，加拿大渥太华(2015年12月) [35]内部审计服务部：部门信息系统和技术控制的审计—第1阶段–应用程序控制，2014年。技术报告，加拿大就业与社会发展部，加拿大渥太华(2014年11月) [36]审计内部：Achats etmarchés，2018年11月，审计内部审计。 技术报告，金融机构监管局，加利福尼亚州渥太华(2018年11月)。 [37]瑞士联邦审计署：IT平台研究与发展基金会– Schweizerische Bundesbahnen。 技术报告，瑞士，伯尔尼，瑞士(2019年7月)