网络嵌入算法-Network Embedding-LINE/LANE/M-NMF

本文结构安排

M-NMF
LANE
LINE

什么是Network Embedding？

LINE

一阶二阶相似度衡量.png

[Information Network]
An information network is defined as $G = (V, E)$ , where $V$ is the set
of vertices, each representing a data object and $E$ is the
set of edges between the vertices, each representing a relationship between two data objects. Each edge $e∈Ee\in E$ is an ordered pair $e = (u, v)$ and is associated with a weight $w_{uv} > 0$ , which indicates the strength of the relation. If $G$ is undirected, we have $(u, v) \equiv (v, u)$ and $wuv≡wvuw_{uv} \equiv w_{vu}$ ; if G is directed, we have $\neq (v,u)$ and $\neq w vu$
[First-order Proximity] The first-order proximity in a network is the local pairwise proximity between two vertices. For each pair of vertices linked by an edge $(u, v)$ , the weight on that edge, $w_{uv}$ , indicates the first-order proximity between u and v. If no edge is observed between u and v, their first-order proximity is 0. The first-order proximity usually implies the similarity of two nodes in a real-world network.

LINE with First-order Proximity:The first-order proximity refers to the local pairwise proximity between the vertices in the network. For each undirected edge $(i, j)$ , the joint probability between vertex $v_{i}$ and $v_{j}$ as follows:
$p1(vi,vj)=11+exp⁡(−u⃗iT⋅u⃗j)p_{1}(v_{i},v_{j})=\frac{1}{1+\exp(-\vec{u}_{i}^{T} \cdot \vec{u}_{j})}$
iT⋅u
j)1
where $u_{i} \in R^{d} $ is the low-dimensional vector representation of vertex $v_{i}$ . $p^1(i,j)=wijW\hat{p}_{1}(i,j) = \frac{w_{ij}}{W}$ ,where $\sum_{(i,j) \in E}^{ }w_{ij}$ .
And its empirical probability can be defined as $p^1(i,j)=wijW\hat{p}_{1}(i,j)=\frac{w_{ij}}{W}$ ,where $W=∑(i,j)∈EwijW=\sum_{(i,j)\in E}^{ }w_{ij}$ .

To preserve the first-order proximity we can minimize the following objective function:
$O1=d(p^1(⋅,⋅),p1(⋅,⋅))O_{1}=d(\hat{p}_{1}(\cdot,\cdot),p_{1}(\cdot,\cdot))$
where $d(⋅,⋅)d(\cdot,\cdot)$ is the distance between two distributions. We choose to minimize the KL-divergence of two probability distributions. Replacing $d(⋅,⋅)d(\cdot,\cdot)$ with KL-divergence and omitting some constants, we have:
$O1=−∑(i,j)∈Ewijlog⁡p1(vi,vj)O_{1}=-\sum_{(i,j)\in E}^{ }w_{ij}\log p_{1}(v_{i},v_{j})$
[Second-order Proximity] The second-order proximity between a pair of vertices (u,v) in a network is the similarity between their neighborhood network structures. Mathematically, let $p_{u} = (w_{u,1} ,...,w_{u,|V|})$ denote the first-order proximity of u with all the other vertices,then the second-order proximity between u and v is determined by the similarity between p u and p v . If no vertex is linked from/to both u and v, the second-order proximity between u and v is 0.

The second-order proximity assumes that vertices sharing many connections to other vertices are similar to each other. In this case, each vertex is also treated as a specific “context” and vertices with similar distributions over the “contexts” are assumed to be similar.
Therefore, each vertex plays two roles: the vertex itself and a specific “context” of other vertices.We introduce two vectors $u⃗i\vec{u}_{i}$
i and $u⃗i′\vec{u}_{i}^{'}$ , where $u⃗i\vec{u}_{i}$ is the representation of $v_{i}$ when it is treated as a vertex while $u⃗i′\vec{u}_{i}^{'}$ is the representation of $v_{i}$ when it is treated as a specific “context”. For each directed edge $(i, j)$ ,we first define the probability of “context” $v_{j}$ generated by vertex $v_{i}$ as:

$p2(vj,vi)=exp⁡(u⃗i′T⋅u⃗i)∑k=1∣V∣exp⁡(u⃗k′T⋅u⃗i)p_{2}(v_{j},v_{i})=\frac{\exp(\vec{u}_{i}^{'T} \cdot \vec{u}_{i}) }{\sum_{k=1}^{|V|}\exp(\vec{u}_{k}^{'T} \cdot \vec{u}_{i})}$
k′T⋅u
i)exp(u
i′T⋅u
i)
where $∣ V ∣$ is the number of vertices or “contexts”. $p^2(vi,vj)=wijd\hat{p}_{2}(v_{i},v_{j}) = \frac{w_{ij}}{d}$ ,where $\sum_{k \in Neibour(i)}^{ }w_{ik}$ .

The second-order proximity assumes that vertices with similar distributions over the contexts are similar to each other. To preserve the second-order proximity, we should make the conditional distribution of the contexts $p2(⋅∣vi)p_{2}(\cdot|v_{i})$ specified by the low-dimensional representation be close to the empirical distribution $p^2(⋅∣vi)\hat{p}_{2}(\cdot |v_{i})$ .Therefore, we minimize the following objective function:

$O2=∑i∈Vλid(p^2(⋅∣vi),p2(⋅∣vi))O_{2}=\sum_{i \in V}^{ }\lambda_{i}d(\hat{p}_{2}(\cdot | v_{i}),p_{2}(\cdot | v_{i}))$

where $d(⋅,⋅)d(\cdot,\cdot)$ is the distance between two distributions.
$\lambda_{i} $ in the objective function is to represent the prestige of vertex i in the network,which can be measured by the degree or estimated through algorithms.

The empirical distribution $p^2(⋅∣vi)\hat{p}_{2}(\cdot |v_{i})$ is defined as
$p^2(vj∣vi)=wijdi\hat{p}_{2}(v_{j} |v_{i})=\frac{w_{ij}}{d_{i}}$ ,where $w_{ij}$ is the weight of the edge $(i, j)$ and $d_{i}$ is the out-degree of vertex i. Here we adopt KL-divergence as the distance function:

$O2=−∑(i,j)∈Ewijlog⁡p2(vj∣vi)O_{2}=-\sum_{(i,j)\in E}^{ }w_{ij}\log p_{2}(v_{j}|v_{i})$

minimize this objective $O_{2}$ , we are able to represent every vertex $v{i}$ with a d-dimensional vector $u⃗i\vec{u}_{i}$
i
[Large-scale Information Network Embedding] Given a large network $G = (V, E)$ , the problem of Large-scale Information Network Embedding aims to represent each vertex $\in V$ into a low-dimensional space $R^{d}$ ,learning a function $fG:V→Rdf_{G}:V \rightarrow R^{d}$ , where $\gg d$ . In the space $R^{d}$ , both the first-order proximity and the second-order proximity between the vertices are preserved.

We adopt the asynchronous stochastic gradient algorithm (ASGD) for optimizing $O_{2}$ ,In each step, the ASGD algorithm samples a mini-batch of edges and then updates the model parameters. If an edge $(i, j)$ is sampled, the gradient the embedding vector $u⃗i\vec{u}_{i}$
i of vertex i will be calculated as:

$∂O2∂u⃗i=wij∂log⁡p2(vj∣vi)∂u⃗i\frac{\partial O_{2}}{\partial \vec{u}_{i}}=w_{ij}\frac{\partial \log p_{2}(v_{j}|v_{i})}{\partial \vec{u}_{i}}$
i∂O2=wij∂u
i∂logp2(vj∣vi)

Optimizing objectives are computationally expensive,which requires the summation over the entire set of vertices when calculating the conditional probability $p2(⋅∣vi)p_{2}(\cdot |v_{i})$ . To address this problem, we adopt the approach of\textbf{ negative sampling }proposed.

$\min_{U,U'} O_{2} = \sum_{(i,j)\in E}^{ }w_{ij}[\log \sigma(\vec{u}_{j}^{'T}\cdot \vec{u}_{i}) + \sum_{i=1}^{K}E_{v_{n}}\sim P_{n}(v)[\log \sigma(-\vec{u}_{k}^{'T}\cdot \vec{u}_{i})]]$
j′T⋅u
i)+i=1∑KEvn∼Pn(v)[logσ(−u
k′T⋅u
i)]]

$∂O2∂u⃗i=−wij[u⃗j′(1−σ(u⃗j′T⋅u⃗i))−∑k=1Ku⃗k′(u⃗k′T⋅u⃗i)]\frac{\partial O_{2}}{\partial \vec{u}_{i}} = -w_{ij}[\vec{u}_{j}^{'}(1-\sigma(\vec{u}_{j}^{'T}\cdot \vec{u}_{i})) -\sum_{k=1}^{K}\vec{u}_{k}^{'}(\vec{u}_{k}^{'T}\cdot \vec{u}_{i})]$

i∂O2=−wij[u

j′(1−σ(u

j′T⋅u

i))−k=1∑Ku

k′(u

k′T⋅u

i)]

$∂O2∂u⃗j′=−wiju⃗i[1−σ(u⃗j′T⋅u⃗i)]\frac{\partial O_{2}}{\partial \vec{u}_{j}^{'}} = -w_{ij}\vec{u}_{i}[1-\sigma(\vec{u}_{j}^{'T}\cdot \vec{u}_{i})]$

j′∂O2=−wiju

i[1−σ(u

j′T⋅u

i)]

$∂O2∂u⃗k′=wiju⃗iσ(u⃗k′T⋅u⃗i)\frac{\partial O_{2}}{\partial \vec{u}_{k}^{'}} = w_{ij}\vec{u}_{i}\sigma(\vec{u}_{k}^{'T}\cdot \vec{u}_{i})$

k′∂O2=wiju

iσ(u

k′T⋅u

Update parameter $u⃗i,u⃗j′,u⃗k′\vec{u}_{i},\vec{u}_{j}^{'},\vec{u}_{k}^{'}$

i,u

j′,u

k′:

$u⃗i=u⃗i−ρ∂O2∂u⃗i\vec{u}_{i} = \vec{u}_{i} - \rho \frac{\partial O_{2}}{\partial \vec{u}_{i}}$

i=u

i−ρ∂u

i∂O2

$u⃗j′=u⃗j′ρ∂O2∂u⃗j′\vec{u}_{j}^{'} = \vec{u}_{j}^{'} \rho \frac{\partial O_{2}}{\partial \vec{u}_{j}^{'}}$

j′=u

j′ρ∂u

j′∂O2

$u⃗k′=u⃗k′−ρ∂O2∂u⃗k′\vec{u}_{k}^{'} = \vec{u}_{k}^{'} - \rho \frac{\partial O_{2}}{\partial \vec{u}_{k}^{'}}$

k′=u

k′−ρ∂u

k′∂O2

The above is the result of optimizing $O_{2}$ , and the obtained $U$ is the result of the second-order similarity. The optimization of $O_{1}$ is similar to optimization of $O_{2}$ , only one variable U needs to be updated. Just change $\vec{u}_{j}^{’} $ to

M-NMF

The objective function is not convex, and we separate the
optimization to four subproblems and iteratively optimize them, which guarantees each subproblem converges to the local minima.

objective function:
$min⁡M,U,H,C=∣∣S−MU∣∣F2+α∣∣H−UCT∣∣F2−βtr(HTBH)\min_{M,U,H,C}=||S-MU||_{F}^{2}+\alpha||H-UC^{T}||_{F}^{2}-\beta tr(H^{T}BH)$

$s.t.,M≥0,U≥0,H≥,C≥,tr(HTH)=ns.t.,M\geq 0,U\geq0,H\geq,C\geq,tr(H^{T}H)=n$

M-subproblem: With other parameters in objective function fixed leads to a standard NMF formulation,the updating rule for M is:

$\leftarrow M \odot \frac{SU}{MU^{T}U}$

U-subproblem: Updating U with other parameters in objective function
fixed leads to a joint NMF problem,the updating rule is:

$\leftarrow U \odot \frac{S^{T}M+\alpha HC}{U(M^{T}M+\alpha C^{T}C)}$

C-subproblem: Updating C with other parameters in objective function
fixed also leads to a standard NMF formulation,the updating rule of C is:

$\leftarrow C \odot \frac{H^{T}U}{CU^{T}U}$

H-subproblem: This is the fixed point equation that the solution must satisfy at convergence. Given an initial value of H, the successive updating rule of H is:

$\leftarrow H \odot \sqrt{ \frac{-w\beta B_{1}H+\sqrt{ \bigtriangleup}}{8\lambda HH^{T}H}}$

where $△=2β(B1H)⊙2β(B1H)+16λ(HHTH)⊙(2βAH+2αUCT+(4λ−2α)H)\bigtriangleup = 2\beta(B_{1}H) \odot 2\beta(B_{1}H) + 16\lambda(HH^{T}H)\odot(2\beta AH+2\alpha UC^{T}+(4\lambda - 2\alpha)H)$

LANE

see as another article of my blog
[论文阅读——LANE-Label Informed Attributed Network Embedding原理即实现]https://www.jianshu.com/p/1abb24bb8a04

LINE-O2 Spark实现

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import breeze.linalg._
import breeze.numerics._
import breeze.stats.distributions.Rand
import scala.math._object LINE {//生成一个随机数序列List，Range是范围，num是随机序列个数def RandList(Range:Int,num:Int) : List[Int] = {var resultList:List[Int]=Nilwhile (resultList.length < num){val randomNum = (new util.Random).nextInt(Range)if(!resultList.exists(s => s==randomNum )){resultList=resultList:::List(randomNum)}}return resultList}def RandNumber(Range:Int) : Int = {val randomNum = (new util.Random).nextInt(Range)return randomNum}def Sigmoid(In:Double): Double = {var Out:Double = 1.0/(math.exp(-1.0*In)+1)return Out}def main(args: Array[String]) {if (args.length < 4) {System.err.println("Usage: LINE <Adjacent Matrix> <Adjacent Edge> <Negative Sample> <dimension>")System.exit(1)}//负采样个数val NS = args(2).toIntprintln("Negative Sample: "+NS)//图嵌入的维度val Dim = args(3).toIntprintln("Embedding dimension: "+Dim)//spark配置和上下文val conf = new SparkConf().setAppName("LINE")val sc = new SparkContext(conf)//输入邻接矩阵val InputFile = sc.textFile(args(0),3)//输入邻接表文件val EgdeFile = sc.textFile(args(1),3)//输出输入的文件行数val InputFileCount = InputFile.count().toIntprintln("InputFileCount(number of lines): "+InputFileCount)//随机采样率val sample_rate : Double = 0.1//负采样哈希表的映射长度val HashTableSize: Int = 50000println("HashTableSize: "+HashTableSize)//LINE O_2 的二阶相似度变量 var U_vertex = DenseMatrix.rand(InputFileCount, Dim, Rand.uniform)var U_context = DenseMatrix.rand(InputFileCount, Dim, Rand.uniform)//邻接矩阵RDDval Adjacent = InputFile.map(line => line.split(",")).map(splitline => splitline.map(word => word.toDouble))val EgdeSet = EgdeFile.map(line => line.split(",")).map(splitline => splitline.map(word => word.toDouble))//当数据量变大，collect操作将会有崩溃 待优化点1val AdjacentCollect = Adjacent.collect()//邻接矩阵的行和列val rows = AdjacentCollect.lengthval cols = AdjacentCollect(0).length//邻接矩阵拉长为一维向量val flattenAdjacent = AdjacentCollect.flatten//邻接矩阵转为 breeze 矩阵val AdjacentMatrix = new DenseMatrix(cols,rows,flattenAdjacent).t//println(Adjacent.take(10).toList)// Adjacent.foreach{//    rdd => println(rdd.toList)// }//每个点的度RDDval VertexDegree = Adjacent.map(line => line.reduce((x,y) => x+y))//所有点的度求和var SumOfDegree = VertexDegree.reduce((x,y)=>x+y)//var SumOfDegree = sc.accumulator(0)//VertexDegree.foreach(x => SumOfDegree += x)  //对点的概率进行平滑，3/4次幂val SmoothProbability = VertexDegree.map(degree => degree/SumOfDegree).map(math.pow(_,0.75))//求SmoothProbability的累积概率CumulativeProbabilityval p : Array[Double] = SmoothProbability.collect()val CumulativeProbability : Array[Double] = new Array[Double](InputFileCount)for(i <- 0 to InputFileCount-1) {var inner_sum : Double = 0.0for(j <- 0 to i){inner_sum = inner_sum + p(j)}CumulativeProbability(i) = inner_sum}//归一化后的累积概率后，乘以HashTableSize并取整，可以得到0~HashTableSize之内的整数val HashProbability : Array[Int] = new Array[Int](InputFileCount)//累积概率的最大值var max_cpro = CumulativeProbability(InputFileCount-1)for(i <- 0  to InputFileCount-1){HashProbability(i) = ((CumulativeProbability(i)/max_cpro)*HashTableSize).toInt}//点的id的哈希表val HashTable : Array[Int] = new Array[Int](HashTableSize+1)//循环生成哈希映射，HashTableSize大小的数组，数组内存储的是点的id标识for(i <- 0 to InputFileCount-1) {if (i==0) {var start : Int = 0var end : Int = HashProbability(1)for(j <- start to end) {HashTable(j) = i}}else {var start : Int = HashProbability(i-1)var end : Int = HashProbability(i)for(j <- start to end) {HashTable(j) = i}}}println("HashTable(HashTableSize):"+HashTable(HashTableSize))val sample_num = (sample_rate*InputFileCount).toIntprintln("sample_num "+sample_num)var O2_Array: Array[Double] = new Array[Double](100)for(iterator <- 0 to 99){//println("the iterator is "+iterator)var learningrate = 0.1var O_2 = 0.0//false表示无放回采样 选取预先选定的采样数量var sampling = EgdeSet.takeSample(false,sample_num)for(i <- 0 to sample_num-1){var objective = 0.0//println("i is " + i)var row:Int = sampling(i)(0).toIntvar col:Int = sampling(i)(1).toInt//println("row:"+row)//println("col:"+col)var u_j_context = U_context(col,::).tvar u_j_context_t = U_context(col,::)var u_i_vertex = U_vertex(row,::).tvar part1=(-1)*sampling(i)(2)*u_j_context*(1-Sigmoid((u_j_context_t*u_i_vertex).toDouble))//println("part1: "+part1)//生成0~50000的NS个随机数，用于挑选负采样样本var negativeSampleSum = DenseVector.zeros[Double](Dim)var RandomSet : List[Int] = RandList(50000,NS)//println("RandomSet is:"+RandomSet)for(j <- 0 to RandomSet.length-1){//println(RandomSet(j))var u_k_context = U_context(HashTable(RandomSet(j)),::).tvar u_k_context_t = U_context(HashTable(RandomSet(j)),::)negativeSampleSum = negativeSampleSum + u_k_context*Sigmoid((u_k_context_t*u_i_vertex).toDouble)}//println("negativeSampleSum: "+negativeSampleSum)var part2 = sampling(i)(2)*negativeSampleSum//println("part2: "+part2)var d_O2_ui = part1-part2//println("d_O2_ui: "+d_O2_ui)//更新u_ivar tmp1 = u_i_vertex - learningrate*(d_O2_ui)//println(tmp1(0)+" "+tmp1(1))// println("previous U_context(row,::): "+U_context(row,::))for(k1 <- 0 to Dim-1){U_vertex(row,k1) = tmp1(k1)}//println("after U_context(row,::): "+U_context(row,::))var d_O2_uj_context = (-1)*sampling(i)(2)*u_i_vertex*(1-Sigmoid((u_j_context_t*u_i_vertex).toDouble))//更新u_j'var tmp2 = u_j_context - learningrate*(d_O2_uj_context)for(k2 <- 0 to Dim-1){U_context(row,k2) = tmp2(k2)}//更新u_k'var negative_cal = 0.0for(j <- 0 to RandomSet.length-1){var u_k_context = U_context(HashTable(RandomSet(j)),::).tvar u_k_context_t = U_context(HashTable(RandomSet(j)),::)//这两行用于计算目标函数的值var sigmoid_uk_ui = Sigmoid((u_k_context_t*u_i_vertex).toDouble)negative_cal = negative_cal + math.log(sigmoid_uk_ui)//对u_k'求导var d_O2_uk_context = sampling(i)(2)*u_i_vertex*sigmoid_uk_uivar tmp3 = u_k_context - learningrate*d_O2_uk_contextfor(k3 <- 0 to Dim-1){U_context(HashTable(RandomSet(j)),k3) = tmp2(k3)}}//计算误差的变化objective = (-1)*sampling(i)(2)*(math.log(Sigmoid((u_j_context_t*u_i_vertex).toDouble)) + negative_cal)O_2 = O_2 + objective}O2_Array(iterator) = O_2}val U2_HDFS = sc.parallelize(U_vertex.toArray,3)val O2_HDFS = sc.parallelize(O2_Array,3)//a(::, 2)   println("======================")//println(formZeroToOneRandomMatrix)//VertexDegree.saveAsTextFile("file:///usr/local/data/line")//IndexSmoothProbability.saveAsTextFile("file:///usr/local/data/line")//HashProbability.saveAsTextFile("file:///usr/local/data/line")U2_HDFS.saveAsTextFile("file:///usr/local/data/U2")O2_HDFS.saveAsTextFile("file:///usr/local/data/O2")println("======================")sc.stop()}
}