http://blog.csdn.net/pipisorry/article/details/44783647

机器学习Machine Learning - Andrew NG courses学习笔记

Anomaly Detection异常检测

Problem Motivation问题的动机

Anomaly detection example

Applycation of anomaly detection

Note:for Frauddetection: users behavior examples of features of a users activity may be on the website it'd be things like,maybe x1 is how often does this user log in, x2,the number of what pages visited, or the number of transactions, maybe x3 is the number of posts of the users on the forum, feature x4 could be the typing speed of the user.And so you can model p of x based on this sort of data.

Gaussian Distribution高斯分布(正态分布)

高斯分布

Note:

1. the width of this bell-shaped curve,sigma, is also called one standard deviation.sigma代表的是钟形的宽度。
2. p of x semicolon Mu comma sigma squared denotes that the probability of x is parametrized by the two parameters Mu and sigma squared.
3. p of x plotted as a function of x,for a fixed value of Mu and of sigma squared sigma squared, that's called the variance.

Parameter estimation参数估计

Note:

1. suspect that each of these examples was distributed according to a normal or Gaussian distribution with some parameter Mu and some parameter sigma squared.

2. estimate Mu is going to be just the average of my example.So Mu is the mean parameter,
3. these estimates are actually the maximum likelihood estimates of the parameters of Mu and sigma squared.
4. this first term becomes 1 over m minus 1, instead of 1 over m. In machine learning, people tend to use this 1 over m formula.But in practice, whether it is 1 over m or 1 over m minus one, makes essentially no difference, assuming m is reasonably large, it's a large training set size.

Algorithm异常检测算法

density estimation密度估计

Note:

1. model p of x from the data sets.we are going to try to figure out what are high probability features, what are lower probability types of features.
2. this equation actually corresponds to an independence assumption on the values of the features x1 through xn.But in practice it turns out that the algorithm of this fragment, it works just fine,whether or not these features are anywhere close to independent and even if independence assumption doesn't hold true.

3. the problem of estimating this distribution p of x, they're sometimes called the problem of density estimation.

4. 不同的features有不同的mu和mean.
异常检测算法

Note:

1. to choose features that describe general properties of the things that you're collecting data on.
2. mu J just take the mean over my training set of the values of the j feature.

3. 异常的就是说产生新example的features总的概率相当低,发生了就是异常的。

Anomaly detection example

{为什么这里不用supervised learning, e.g. svm,而是用的anomaly detection: 在后两节会讲到}

Developing and Evaluating an Anomaly Detection System开发和评价异常检测系统

Note:

1. Training set is unlabled, cross validation & test set is labled.

2. 对于异常检测问题,要检测出的是anomalous的,所以anomalous对应y = 1

Note:

1. we call training set an unlabeled training set but all of these examples are really ones that correspond to y equals 0.so that's our training set of all,good, or the vast majority of good examples.

2. So, we will use these 6000 engines to fit p of x. And so we would these 6,000 examples to estimate the parameters Mu 1, sigma squared 1, up to Mu N, sigma squared N.
3. someone put the same 4000 in the cross validation set and the test set.but we like to think of the cross validation set and the test set as being completely different data sets to each other,it is not considered a good machine learning practice.

Algorithm evaluation算法评估

Note:

1. these labels are will be very skewed because y equals zero, that is normal examples, usually be much more common than y equals 1 than anomalous examples.所以要用precion/recall评估,而不能使用classification accuracy。

2. to set epsilon, evaluate the algorithm on the cross validation set, and then when we've picked the set of features, when we've found the value of epsilon, do the final evaluation of the algorithm on the test sets.

Anomaly Detection vs. Supervised Learning异常检测vs.监督学习

{if we have this labeled data,why don't we just use a supervised learning algorithm logistic regression or a neural network,to try to learn directly from our labeled data, to predict whether y equals one or y equals zero}

分别在哪种情况下使用the properties of a learning problem that cause to treat it as an anomaly detention verses a supervised learning

Note:

1. Anomaly Detection:when we are doing the process of estimating p of x, of fitting all those Gaussian parameters,we need only negative examples to do that.So if you have a lot of negative data,we can still fit to p of x pretty well.
2. Anomaly Detection:for anomaly detection applications often there are many different types of anomalies that could go wrong that could break an aircraft engine.it can be difficult for an algorithm to learn from your small set of positive examples what the anomalies look like.And in particular,future anomalies may look nothing like the ones you've seen so far.new way for an aircraft engine to be broken that you have just never seen before,then it might be more promising to just model the negative examples, with a sort of a Gaussian model P of X. Rather than try too hard to model the positive examples.
3. for the SPAM problem, we usually have enough examples of spam email to see,most of these different types of SPAM email, because we have a large set of examples of SPAM, and that's why we usually think of SPAM as asupervised learningsetting, even though, there may be many different types of SPAM.
some applications of anomaly detection versus supervised learning应用上的区别

Note: if you are very a major online retailer, and have had a lot of people try to commit fraud on your website,sometimes fraud detection could actually shift over to the supervised learning column.for some manufacturing processes, if you're manufacturing very large volumes and you've seen a lot of bad examples, maybe manufacturing could shift to the supervised learning column as well.

Choosing What Features to Use选择使用哪些features

Non-gaussian features转换成gaussian features

Note:

1. in case your data looks non-Gaussian, the algorithms will often work just find.
2. play with different transformations of the data in order to make it look more Gaussian.
3. more generally with log x with x2 and some constant c and this constant could be something to try to make it look as Gaussian as possible.
4. new feature x_new (0.05) looks more Gaussian than my previous one and then I might instead use this new feature to feed into my anomaly detection algorithm.

5. You could also have hist of log of x, that's another example of a transformation you can use.that also look pretty Gaussian.So, I can also define x_new equals log of x.
come up with features for an anomaly detection algorithm

Note:

1. Look at the anomaly that the algorithm is failing to flag, and see if that inspires you tocreate some new feature.so that with this new feature it becomes easier to distinguish the anomalies from your good examples.
2. 绿色x代表anomaly example, 只有一个feature时会区分错误,加一个feature x2时就可以正确区分。

Note:

1. I have a very high CPU load, and have a very high network traffic.suspect the failure cases is one of my computers has a job that gets stuck in some infinite loop.and so the CPU load grows,but the network traffic doesn't because it's just spinning it's wheels and doing a lot of CPU work,stuck in some infinite loop.create a new feature, X5,which might be CPU load divided by network traffic.
2. And by creating features like these, you can start to capture anomalies that correspond to unusual combinations of values of the features.

Multivariate Gaussian Distribution (Optional)多变量高斯分布

{sometimes catch some anomalies that the earlier algorithm didn't}

Note:

1. most of the data data lies in this region(对应蓝色区域内), and so thatgreen cross is pretty far away from any of the data I've seen.It looks like that should be raised as an anomaly.
2. 但对于green cross,p(x1)和p(x2)分别都相对正常,就不会将其判定为anomaly.对应在洋红色区域内。

Multivariate Gaussian (Normal)distribution多变量高斯分布

Multivariate Gaussian (Normal)examples

——

——

Note:

1. Sigma is a covariance matrix and measures the variance or the variability of the features X1 X2.Sigma 对角线是方差,两边就是协方差,协方差means两个features的线性相关度。
2. X1 and X2 tend to be highly correlated with each other for example.to change the off diagonal entries of this covariance matrix.就会出现斜着的椭圆。so increase the off-diagonal entries from .5 to .8, it is more andmore thinly peaked along this sort of x equals y line.

Anomaly Detection using the Multivariate Gaussian Distribution (Optional)用多变量的高斯分布进行异常检测

Note:And you set sigmato be equal to this.And this is actually just like the sigma, when we were using the PCA.

Relationship to original model单变量和多变量的联系和区别

Note:

1. the original model actually corresponds to a special case of a multivariate Gaussian distribution.this special case is defined by constraining the distribution of p of x, the multivariate a Gaussian distribution of p of x,so that the contours of the probability density function are axis aligned(轴对齐).
2. the multivariate Gaussian distribution,corresponds exactly to the old model, if the covariance matrix sigma, has only 0 elements off the diagonals.

Note:

1. the multivariate Gaussian model has a lot of parameters, so this covariance matrix sigma is an n by n matrix,has roughly n squared parameters, because it's a symmetric matrix,it's actually closer to n squared over 2 parameters, but this is a lot of parameters, so you need make sure you have a fairly large value for m, make sure you have enough data to fit all these parameters.
2. m greater than or equal to 10 n would be a reasonable rule of thumb to make sure that you can estimate this covariance matrix sigma reasonably well.

3. in problems where you have a very large training set or m is very large and n is not too large, then themultivariate Gaussian model is well worth considering and may work better as well, and can save you from having to spend your time to manually create extra features in case the anomalies turn out to be captured by unusual combinations of values of the features.

4. covariance matrix sigma non-invertible, they're usually 2 cases for this.Oneis if it's failing to satisfy this m greater than n condition;secondcase is if you have redundant features.if you have 2 features that are the same.if your x1 is just equal to x2. Or if you have redundant features like maybe your features X3 is equal to feature X4, plus feature X5,well X3 doesn't contain any extra information.

Reviews复习

from:http://blog.csdn.net/pipisorry/article/details/44783647

ref:《Anomaly Detection with Apache Spark》Spark上的异常检测

如何利用参数和非参数方法来检测异常值(I)

如何利用参数和非参数方法来检测异常值(II)

论文:基于夹角的高维数据异常检测方法(ABOD)《Angle-based outlier detection in high-dimensional data》HP Kriegel, A Zimek (SIGKDD 2008)

异常检测用于社区关键言论发现[Beyond Trending Topics: identifying important conversations in communities]

异常检测在Netflix的应用[Netflix使用的异常服务器侦测技术]

拓扑异常监测《Topological Anomaly Detection》

开源(R):基于马氏距离/Cerioli方法的多元异常监测CerioliOutlierDetection

Machine Learning - XV. Anomaly Detection异常检测 (Week 9)相关推荐

  1. Anomaly Detection异常检测基础

    什么是 Anomaly(异常)? 虽然说是 异常,但其实是以训练集为核心,判断输入数据是否与训练集中的数据 "类似".在不同的领域可以有不同的叫法,比如:outlier Detec ...

  2. 异常检测(Anomaly detection): 异常检测算法(应用高斯分布)

    估计P(x)的分布--密度估计 我们有m个样本,每个样本有n个特征值,每个特征都分别服从不同的高斯分布,上图中的公式是在假设每个特征都独立的情况下,实际无论每个特征是否独立,这个公式的效果都不错.连乘 ...

  3. Anomaly Detection 异常检测之 CSI:Contrasting shifted instances论文解读

    1. 背景 异常检测目标 给定数据集 { x m } m = 1 M \{x_m\}^M_{m=1} {xm​}m=1M​,分布外(OOD)检测的目标是对 { x m } \{x_m\} {xm​}上 ...

  4. CutPaste: Self-Supervised Learning for Anomaly Detection and Localization 全文翻译+详细解读

    CutPaste: Self-Supervised Learning for Anomaly Detection and Localization 全文翻译+详细解读 文章速览 全文翻译及详细解释 0 ...

  5. CutPaste:Self-Supervised Learning for Anomaly Detection and Localization 论文解读(缺陷检测)

    CutPaste:Self-Supervised Learning for Anomaly Detection and Localization CutPaste:用于异常检测和定位的自监督学习 文章 ...

  6. (ch9) Deep Learning for Anomaly Detection: A Survey

    Deep Learning for Anomaly Detection: A Survey https://www.researchgate.net/publication/330357393_Dee ...

  7. 【Paper】Deep Learning for Anomaly Detection:A survey

    论文原文:PDF 论文年份:2019 论文被引:253(2020/10/05) 922(2022/03/26) 文章目录 ABSTRACT 1 Introduction 2 What are anom ...

  8. Machine Learning for Malware Detection

    Machine Learning for Malware Detection 序 一.恶意软件检测知识基础 数据获取的两个阶段 安卓操作系统架构 安卓应用程序的关键组成部分 特征选择 原始特征数据预处 ...

  9. CutPaste Self-Supervised Learning for Anomaly Detection and Localization

    CutPaste: Self-Supervised Learning for Anomaly Detection and Localization 用于异常检测和定位的自监督学习 简洁版 问题:异常的 ...

  10. Deep Learning for Anomaly Detection: A Review

    本文是对<Deep Learning for Anomaly Detection: A Review>的文字部分翻译,图表请参看原文. 深度学习进行异常检测:综述 摘要 1 引言 2 异常 ...

最新文章

  1. [云炬ThinkPython阅读笔记]3.3 构建
  2. 计算机专业专业课代号408,计算机专业考研你一定要知道的事情!
  3. 【翻译】Prometheus 2.4.0 新特性
  4. 我爱我家:旗下APP已全面兼容鸿蒙系统
  5. 3-7 DNA序列(DNA Consensus String, ACM/ICPC Seoul 2006, UVa1368)
  6. Centos7开放及查看端口
  7. [转载] python下 import Matplotlib.pyplot as plt 的使用
  8. 媒体管理工具Media Encoder 2022 for Mac
  9. linux逻辑卷管理(LVM)
  10. attachEvent报错原因
  11. 计算机里怎么添加桌面,电脑上怎么把我的电脑添加到桌面上
  12. SQL语句基础-多表连接查询
  13. JavaScript数组求和
  14. 赏析角度有哪些_从哪些角度赏析句子
  15. 惊爆Oblog4.6 的0day
  16. 天池竞赛——工业蒸汽量预测(完整代码详细解析)
  17. python大数据入门书籍推荐书目_python 大数据入门书籍
  18. Zuul1与Spring Cloud Gateway的区别
  19. w7电脑蓝屏怎么解决_教您电脑蓝屏怎么办
  20. 产品与解决方案的问题。

热门文章

  1. JScrollPane恢复正常滚动量
  2. PaddlePaddle(6)—— 基于Hub Serving的微信小程序模型部署
  3. CentOS 7操作系统中设置系统时间/时区的方法
  4. [最小割][Kruskal] Luogu P5039 最小生成树
  5. 软件工程第二次作业 软件评价
  6. iOS 8 Xcode6 设置Launch Image 启动图片转
  7. 如何使用四个语句来提高 SQL Server 的伸缩性
  8. python源码中的学习笔记_第11章_模块与包
  9. python源码中的学习笔记_第4章_列表
  10. pymongo基本方法使用笔记