降维-基于RDD的API

降维-基于RDD的API
• Singular value decomposition (SVD)
o Performance
o SVD Example
• Principal component analysis (PCA)
Dimensionality reduction is the process of reducing the number of variables under consideration. It can be used to extract latent features from raw and noisy features or compress data while maintaining the structure. spark.mllib provides support for dimensionality reduction on the RowMatrix class.
• 奇异值分解（SVD）
o 性能
o SVD示例
• 主成分分析（PCA）
降维是减少所考虑的变量数量的过程。可用于从原始和嘈杂的特征中提取潜在特征，或者在保持结构的同时压缩数据。 spark.mllib提供对RowMatrix类降维的支持。
奇异值分解Singular value decomposition (SVD)
Singular value decomposition (SVD) factorizes a matrix into three matrices奇异值分解（SVD）将矩阵分解为三个矩阵: UU, ΣΣ, and VV such that
A=UΣVT,A=UΣVT,
where
• UU is an orthonormal matrix, whose columns are called left singular vectors, 正交矩阵，其列称为左奇异矢量
• ΣΣ is a diagonal matrix with non-negative diagonals in descending order, whose diagonals are called singular values, 具有非负对角线降序的对角矩阵，其对角线称为奇异值
• VV is an orthonormal matrix, whose columns are called right singular vectors. 正交矩阵，其列称为右奇异向量。
For large matrices, usually we don’t need the complete factorization but only the top singular values and its associated singular vectors. This can save storage, de-noise and recover the low-rank structure of the matrix. 对于大型矩阵，通常不需要完整的因式分解，而仅需要顶部top奇异值及其关联的奇异矢量。可以节省存储空间，降低噪声并恢复矩阵的低阶结构。
If we keep the top k singular values, then the dimensions of the resulting low-rank matrix will be如果保持领先 ķ 奇异值，则所得低秩矩阵的维将为：
• UU: m×km×k,
• ΣΣ: k×kk×k,
• VV: n×kn×k.
Performance
We assume n is smaller than m假设n小于m. The singular values and the right singular vectors are derived from the eigenvalues and the eigenvectors of the Gramian matrix 奇异值和右奇异向量是从Gramian矩阵的特征值和特征向量得出。The matrix storing the left singular vectors UU, is computed via matrix multiplication as U=A(VS−1)U=A(VS−1), if requested by the user via the computeU parameter. The actual method to use is determined automatically based on the computational cost存储左奇异向量的矩阵ü通过矩阵乘法计算为 ü= A ，如果用户通过computeU参数请求。实际使用的方法是根据计算成本自动确定的:
• If nn is small (n<100n<100) or k is large compared with nn (k>n/2), we compute the Gramian matrix first and then compute its top eigenvalues and eigenvectors locally on the driver. This requires a single pass with O(n2) storage on each executor and on the driver, and O(n2k) time on the driver.
• Otherwise, we compute (ATA)v in a distributive way and send it to ARPACK to compute (ATA)(ATA)’s top eigenvalues and eigenvectors on the driver node. This requires O(k) passes, O(n) storage on each executor, and O(nk) storage on the driver.
• 如果 ñ 是小（n < 100）或者 ķ 与 ñ （k > n / 2个），首先计算Gramian矩阵，然后在驱动程序上局部计算其最高特征值和特征向量。需要一次通过O （ñ2个）存储在每个执行器和驱动程序上，以及 O （ñ2个k ）在驱动程序上的时间。
• 否则，计算（一个ŤA ）v以分布式方式将其发送到 ARPACK以进行计算（一个ŤA ），驱动程序节点上的最大特征值和特征向量。需要O （k ）通过， O （n ）存储在每个执行器上，以及 Ø（ñķ）存储在驱动程序上。
SVD Example
spark.mllib provides SVD functionality to row-oriented matrices, provided in the RowMatrix class. spark.mllib向RowMatrix类提供的面向行的矩阵，提供SVD功能。
Scala
• Java
• Python
Refer to the SingularValueDecomposition Scala docs for details on the API. 有关该API的详细信息，请参考SingularValueDecompositionScala文档。
import org.apache.spark.mllib.linalg.Matrix
import org.apache.spark.mllib.linalg.SingularValueDecomposition
import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.linalg.distributed.RowMatrix

val data = Array(
Vectors.sparse(5, Seq((1, 1.0), (3, 7.0))),
Vectors.dense(2.0, 0.0, 3.0, 4.0, 5.0),
Vectors.dense(4.0, 0.0, 0.0, 6.0, 7.0))

val rows = sc.parallelize(data)

val mat: RowMatrix = new RowMatrix(rows)

// Compute the top 5 singular values and corresponding singular vectors.
val svd: SingularValueDecomposition[RowMatrix, Matrix] = mat.computeSVD(5, computeU = true)
val U: RowMatrix = svd.U // The U factor is a RowMatrix.
val s: Vector = svd.s // The singular values are stored in a local dense vector.
val V: Matrix = svd.V // The V factor is a local dense matrix.
Find full example code at “examples/src/main/scala/org/apache/spark/examples/mllib/SVDExample.scala” in the Spark repo.
The same code applies to IndexedRowMatrix if U is defined as an IndexedRowMatrix.
Principal component analysis (PCA) 主成分分析
Principal component analysis (PCA) is a statistical method to find a rotation such that the first coordinate has the largest variance possible, and each succeeding coordinate, in turn, has the largest variance possible. The columns of the rotation matrix are called principal components. PCA is used widely in dimensionality reduction.
spark.mllib supports PCA for tall-and-skinny matrices stored in row-oriented format and any Vectors.
主成分分析（PCA）是一种统计方法，用于查找旋转，以使第一个坐标具有最大的方差，而每个后续坐标又具有最大的方差。旋转矩阵的列称为主成分。PCA被广泛用于降维。
spark.mllib 支持PCA用于以行格式和任何向量存储的高和稀疏矩阵。
• Scala
• Java
• Python
The following code demonstrates how to compute principal components on a RowMatrix and use them to project the vectors into a low-dimensional space.
Refer to the RowMatrix Scala docs for details on the API.
以下代码演示了如何在 RowMatrix 上计算主成分，将向量投影到低维空间中。
有关该API的详细信息，请参考RowMatrixScala文档。
import org.apache.spark.mllib.linalg.Matrix
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.linalg.distributed.RowMatrix

val data = Array(
Vectors.sparse(5, Seq((1, 1.0), (3, 7.0))),
Vectors.dense(2.0, 0.0, 3.0, 4.0, 5.0),
Vectors.dense(4.0, 0.0, 0.0, 6.0, 7.0))

val rows = sc.parallelize(data)

val mat: RowMatrix = new RowMatrix(rows)

// Compute the top 4 principal components.
// Principal components are stored in a local dense matrix.
val pc: Matrix = mat.computePrincipalComponents(4)

// Project the rows to the linear space spanned by the top 4 principal components.
val projected: RowMatrix = mat.multiply(pc)
Find full example code at “examples/src/main/scala/org/apache/spark/examples/mllib/PCAOnRowMatrixExample.scala” in the Spark repo.
The following code demonstrates how to compute principal components on source vectors and use them to project the vectors into a low-dimensional space while keeping associated labels:
Refer to the PCA Scala docs for details on the API.
在Spark存储库中找到完整的示例代码。
以下代码演示了如何在源向量上计算主成分，将向量投影到低维空间中，同时保留关联的标签：
有关该API的详细信息，请参考PCAScala文档。
import org.apache.spark.mllib.feature.PCA
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.rdd.RDD

val data: RDD[LabeledPoint] = sc.parallelize(Seq(
new LabeledPoint(0, Vectors.dense(1, 0, 0, 0, 1)),
new LabeledPoint(1, Vectors.dense(1, 1, 0, 1, 0)),
new LabeledPoint(1, Vectors.dense(1, 1, 0, 0, 0)),
new LabeledPoint(0, Vectors.dense(1, 0, 0, 0, 0)),
new LabeledPoint(1, Vectors.dense(1, 1, 0, 0, 0))))

// Compute the top 5 principal components.
val pca = new PCA(5).fit(data.map(_.features))

// Project vectors to the linear space spanned by the top 5 principal
// components, keeping the label
val projected = data.map(p => p.copy(features = pca.transform(p.features)))
Find full example code at “examples/src/main/scala/org/apache/spark/examples/mllib/PCAOnSourceVectorExample.scala” in the Spark repo.

降维-基于RDD的API相关推荐

Hive数据分析——Spark是一种基于rdd（弹性数据集）的内存分布式并行处理框架，比于Hadoop将大量的中间结果写入HDFS，Spark避免了中间结果的持久化...
转自:http://blog.csdn.net/wh_springer/article/details/51842496 近十年来,随着Hadoop生态系统的不断完善,Hadoop早已成为大数据事实上 ...
Spark论文思想之-基于RDD构建的模型（Shark的来龙去脉）
3.1 介绍首先RDD提供以下功能: 跨集群的不可变存储(在Spark中,记录是指Java Object) 使用键对数据进行分区控制考虑分区的粗粒度运算符由于是内存计算,所以低延迟 3.2 在R ...
soapui工具_基于开源的API测试工具！不再为web服务负载测试而发愁
通过一个可视化.拖拽式的界面,LoadUI允许您实时.交互式地创建.配置和重分配负载测试.在单一测试环境下,LoadUI提供完整的测试覆盖,支持所有标准的协议和技术.它功能强大,能从任意数量的本地和远 ...
.NET Core微服务之基于Ocelot实现API网关服务（续）
Tip: 此篇已加入.NET Core微服务基础系列文章索引一.负载均衡与请求缓存 1.1 负载均衡为了验证负载均衡,这里我们配置了两个Consul Client节点,其中ClientServic ...
基于ArcGIS JS API实现垂直滑动缩放条
文章目录需求背景需求分析效果图完整代码注意事项严格来说并不是基于ArcGIS JS API,应该是基于Dojo的dijit里面的VerticalSlider和VerticalRule,但是 ...
Spark笔记：复杂RDD的API的理解（上）
本篇接着讲解RDD的API,讲解那些不是很容易理解的API,同时本篇文章还将展示如何将外部的函数引入到RDD的API里使用,最后通过对RDD的API深入学习,我们还讲讲一些和RDD开发相关的scala ...
为支持两个语言版本，我基于谷歌翻译API写了一款自动翻译的 webpack 插件
大家好,我是若川.持续组织了6个月源码共读活动,感兴趣的可以点此加我微信 ruochuan12 参与,每周大家一起学习200行左右的源码,共同进步.同时极力推荐订阅我写的<学习源码整体架构系列& ...
.NET Core微服务之路：基于Ocelot的API网关实现--http/https协议篇
前言最近一直在忙公司和私下的兼职,白天十个小时,晚上四个小时,感觉每天都是打了鸡血似的,精神满满的,连自己那已经学打酱油的娃都很少关心,也有很长一段时间没有更新博客了,特别抱歉,小伙伴们都等得想取关 ...
api签名_使用签名保护基于HTTP的API
api签名我在EMC上的一个平台上可以构建SaaS解决方案. 与越来越多的其他应用程序一样,该平台具有基于RESTful HTTP的API. 使用JAX-RS之类的开发框架,构建这样的API相对容易 ...

降维-基于RDD的API

降维-基于RDD的API相关推荐

最新文章

热门文章