AZURE 日志分析自动告警

小伙伴们好久不见，今天我们来聊聊中国 AZURE 的日志分析告警。为什么是中国 AZURE，目前中国 AZURE 的 Monitor 服务和运维相关周围服务和 Global 是有所不同的，所以有些功能和设计不能复制和套用全球版 AZURE 的架构。我们先看一下中国 AZURE 运维管理方面一些平台原生功能的缺失：
1. Azure Monitor 支持新的 Metric 指标分析服务，但不支持基于新的 Metric 指标分析的告警设置，简而言之能看不能告警；
2. 中国 AZURE 目前不支持 Azure Log Analytics 服务，平台原生不支持日志分析服务，无法通过原生服务进行日志分析和告警。所以对于平台原生支持的一些 Metric 或者 Log 无法通过 Azure Monitor 或 Azure Log Analytics 分析并发送告警。

今天的 Demo 中以一个例子，通过 EventHub + Stream Analytics + Function 来实现流式分析实时告警。

架构图：

日志源：EventHub 原生已经支持对于 Azure 平台服务的日志消息采集能力，VM 的日志可以通过 Azure VM Diagnotics Extension 进行聚集并传入 EventHub, 对于平台的原生服务可以直接与 EventHub 集成。对于非 Azure 原生服务，比如客户自己的一些日志系统等可以通过 Logstash，Fluentd 的方式将日志注入到 EventHub, Azure 已经有相关的插件来支持和 Logstash 这种日志服务进行集成。

日志聚集：EventHub 来做日志的聚集，可以将多个日志源聚集到同一个 EventHub 下来实现日志消费下游服务的统一分发。

日志实时分析：Stream Analytics 来对 EventHub 聚集的日志进行消费，完成日志的流式实时分析，在此 Demo 中，Sream Analytics 进行应用网关（Application Gateway）的后端服务节点的健康状态情况，当可用节点小于一个时，触发告警事件。

日志告警：通过 Function 服务，以事件驱动的方式获得 Stream Analytics 的告警，执行 Function 代码推送告警。此 Demo，以邮件告警为例，如果客户有短消息推送等其它推送需求，可以类同方式调取集成。

配置方法：

1. 配置日志源

此次 Demo 中以应用网关的 Metric 日志为例，此处忽略应用网关的相关创建动作和配置动作，下面是开启日志推送到 EventHub 的配置方法，此步执行前需要创建好 EventHub

2. 配置 EventHub

EventHub 配置方法比较简单，创建 EventHub，然后为了方便区分后端消费者，在创建好的 EventHub 下创建消费组

3. 配置 Stream Analytics 服务

此处跳过 Steam Analytics 的创建过程，直接在创建好的 Stream Analytics 服务上进行配置，分别配置 Input 和 Output，Stream Analytics 作为 EventHub 消息的消费者，首先我们需要在 Stream Analytics 中将 EventHub 配置为 Input，反之 Function 服务是 Stream Analytics 的数据消费者，所以把 Function 服务配置为 Ouput。

此 Demo 中有架构有一些微调，EventHub 和 Stream Analytics 分别对消息事件做了两次处理，流程如下：

原因是因为 Application Gateway 推送出来的 Metric 日志是以5分钟为间隔一条消息，每个消息中包含5分钟内每分钟的消息，是以 Json 嵌套数组的方式来做的，我们通过第一层的 Stream Analytics 来完成将嵌套的 Metric 日志序列化，将每分钟的 Metric 日志以独立消息的方式注入到 EventHub，然后重新在第二层的 Stream Analytics 中来进行流式分析，以5分钟为间隔来分析5分钟内性能指标的平均值，当平均触碰阈值后生成告警事件，将告警事件通知 Function 服务，通知事件内包含监控指标类型名称和现有指标5分钟平均值，Function 服务以事件驱动执行通知分发程序将告警以相应方式推送到相关责任人。

从 Application Gateway 的生成的 RAW Date 日志格式参考如下：

{"records":[{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:27:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:28:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:29:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:25:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:26:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:27:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:28:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:29:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:25:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:26:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:27:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:28:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:29:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:27:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:28:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:29:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:27:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:28:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:29:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:25:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:26:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:27:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:28:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:29:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"}],"EventProcessedUtcTime":"2018-09-10T07:38:52.6261568Z","PartitionId":0,"EventEnqueuedUtcTime":"2018-09-10T07:35:52.4790000Z"}
{"records":[{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:30:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:31:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:32:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:33:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:34:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:30:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:31:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:32:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:33:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:34:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:30:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:31:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:32:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:33:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:34:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:30:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:31:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:32:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:33:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:34:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:30:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:31:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:32:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:33:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:34:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:30:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:31:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:32:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:33:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:34:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"}],"EventProcessedUtcTime":"2018-09-10T07:42:19.8630447Z","PartitionId":0,"EventEnqueuedUtcTime":"2018-09-10T07:42:18.9160000Z"}
{"records":[{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:35:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:36:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:38:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":869,"minimum":869,"maximum":869,"average":869,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:39:00.0000000Z","metricName":"Throughput","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:35:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:36:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:38:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:39:00.0000000Z","metricName":"UnhealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:35:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:36:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:38:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:39:00.0000000Z","metricName":"HealthyHostCount","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:35:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:36:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:38:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":12,"minimum":12,"maximum":12,"average":12,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:39:00.0000000Z","metricName":"TotalRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:35:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:36:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:38:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:39:00.0000000Z","metricName":"FailedRequests","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:38:00.0000000Z","metricName":"ResponseStatus","timeGrain":"PT1M"},{"count":1,"total":12,"minimum":12,"maximum":12,"average":12,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:39:00.0000000Z","metricName":"ResponseStatus","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:35:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":0,"minimum":0,"maximum":0,"average":0,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:36:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:38:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"},{"count":1,"total":1,"minimum":1,"maximum":1,"average":1,"resourceId":"/SUBSCRIPTIONS/4507938F-A0AC-4571-978E-7CC741A60AF8/RESOURCEGROUPS/ALERTDEMO/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/ALERTDEMO","time":"2018-09-10T07:39:00.0000000Z","metricName":"CurrentConnections","timeGrain":"PT1M"}],"EventProcessedUtcTime":"2018-09-10T07:45:55.9598069Z","PartitionId":0,"EventEnqueuedUtcTime":"2018-09-10T07:45:55.8810000Z"}

第一层 StreamAnalytics 配置：

Input：第一层 EventHub (alertdemo)，Ouput：第二层 EventHub (EventhubStream)

查询语句

WITH
Metric AS
(SELECT   arrayElement.ArrayIndex,  arrayElement.ArrayValue  FROM alertdemo as event  CROSS APPLY GetArrayElements(event.records) AS arrayElement
),
TransformedInput AS (SELECTMetric.arrayvalue.*FROM Metric
)
SELECT*
INTO EventhubStream
FROM TransformedInput

第二层 StreamAnalytics 配置：

Input：第二层 EventHub (EventhubStream)，Ouput：Function (FuncOutput)

SELECTmetricName,AVG(average) as avg
INTO FuncOutput
FROM EventhubStream TIMESTAMP BY time
GROUP BYmetricName,TumblingWindow(minute, 5)
HAVING(avg(average) <= 1 and metricName = 'HealthyHostCount')

4. 配置 Function 服务

这里在配置过程中有个地方需要注意：需要在 Function 服务的 SSL 部分将 TLS 版本设置为 1.0，这个是 Function 和 Stream Analytics 服务集成的要求。

本例中以 Python Runtime 为例，创建一个 Http Trigger 触发的 Function 函数，代码如下：

import os
import json
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMETextfromaddr = "******@***.com"
toaddr = "******@***.com"postreqdata = json.loads(open(os.environ['req']).read())
if postreqdata:#Create Alert Messagemsg = MIMEMultipart()msg['From'] = fromaddrmsg['To'] = toaddrmsg['Subject'] = "Alert Fire"body = postreqdata[0]['metricname'] + " fire the alert"msg.attach(MIMEText(body,'plain'))#Send Alert Messages = smtplib.SMTP('smtp.***.com')s.ehlo()s.login("*******@***.com", '******')s.sendmail(fromaddr, toaddr, msg.as_string())#Prepare Success Code
returnData = {#HTTP Status Code:"status": 200,#Response Body:"body": "<h1>Azure Works :)</h1>",# Send any number of HTTP headers"headers": {"Content-Type": "text/html","X-Awesome-Header": "YesItIs"}
}# Output the response to the client
output = open(os.environ['res'], 'w')
output.write(json.dumps(returnData))

检查邮件告警

本文中的 Demo 只是一个简单的示例，大家可以根据自己实际的业务场景需求对流分析部分的告警策略自行定义，流式分析服务内置了很多分析能力可以满足我们不同的分析需求。

参考阅读：

1. Stream Analytics 常用语法：https://msdn.microsoft.com/zh-cn/azure/stream-analytics/reference/stream-analytics-query-language-reference

2. Stream Analytics 分析场景示例：https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-stream-analytics-query-patterns

3. Azure 平台服务日志参考：https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-supported-metrics

AZURE 日志分析自动告警相关推荐

Android debuglogger日志分析-自动重启
大家有没有遇到和我一样的问题,android设备(我这里android 平板)用着用着突然就黑屏自动重启了,重启后一切正常,这个问题还是概率性的,复现都不好复现... 本人公司是做平板定制的,主要针对 ...
软件定义数据中心(SDDC)的日志分析
现代化基础设施不断生成日志数据的速度已远远超过人类分析的速度.而且,现在的数据中心可以在脚本控制下建立或拆除,其活动数量和数据量都在呈指数增长. 传统的数据分析法是每周或每天依照列表审查日志文件,这种 ...
shell脚本：Dos 攻击防范、系统发送告警、MySQL 数据库备份单、MySQL 数据库备份多、Nginx 日志分析、网卡实时流量、服务器磁盘利用率
系统配置初始化脚本 #/bin/bash # 设置时区并同步时间 ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime if ! crontab ...
Expect的安装与应用，及实现自动检测另外一台服务器运行状态并重启，和使用expect脚本远程批量管理服务器与日志分析
学习Expect Expect是什么? Expect是一个免费的编程工具语言,用来实现自动和交互式任务进行通信,而无需人的干预. Expect是不断发展的,随着时间的流逝,其功能越来越强大,已经成为 ...
Kubernetes Ingress 日志分析与监控的最佳实践
2019独角兽企业重金招聘Python工程师标准>>> 前言目前Kubernetes(K8s)已经真正地占领了容器编排市场,是默认的云无关计算抽象,越来越多的企业开始将服务构建在K ...
Kubernetes日志分析利器：Elassandra部署使用指南
Elassandra是一个基于Apache Cassandra的Elasticsearch实现,有效结合了两者的优势,弥补了Elasticsearch的一些使用限制(单点故障.在线升级等).结合Flu ...
调用链路_全链路日志分析解决方案介绍
为何需要链路分析? 您是否了解运维环境的网络架构和业务系统架构?当业务升级或变更时,是否对已有的架构图及时作出更新?当发生故障时,是否能够快速判断哪个业务系统模块或接口出现了问题,是否能够快速判断故障 ...
大数据主题分享第三期 | 基于ELK的亿级实时日志分析平台实践
猫友会希望建立更多高质量垂直细分社群,本次是"大数据学习交流付费群"的第三期分享. "大数据学习交流付费群"由猫友会联合,斗鱼数据平台总监吴瑞诚,卷皮BI技术总 ...
ELK实时日志分析平台环境部署--完整记录
在日常运维工作中,对于系统和业务日志的处理尤为重要.今天,在这里分享一下自己部署的ELK(+Redis)-开源实时日志分析平台的记录过程(仅依据本人的实际操作为例说明,如有误述,敬请指出)~ ==== ...

AZURE 日志分析自动告警

AZURE 日志分析自动告警相关推荐

最新文章

热门文章