我正在使用大型稀疏二进制矩阵.我使用Scipy稀疏矩阵实现来压缩它们.从scipy.spatial.distance计算Jaccard距离不支持对稀疏矩阵的直接操作,因此：

>将整个稀疏矩阵转换为密集,然后在每一行上作为内存饥饿的向量进行操作

要么

>遍历稀疏,使用getrow()抓住每一行并运行.

要么

>编写我们自己的实现来处理稀疏矩阵.

为了正确看待,这里是示例代码：

import scipy.spatial.distance as d

import numpy as np

from scipy.sparse import csr_matrix

# benchmark performance

X = np.random.random((3000, 3000))

# binarize

X[X > 0.3] = 0

X[X>0] = 1

mat = csr_matrix(X)

a = np.zeros(3000)

a[4] = a[100] = a[22] =1

a = csr_matrix(a)

def jaccard_fast(v1,v2):

common = v1.dot(v2.T)

dis = (v1 != v2).getnnz()

if common[0,0]:

return 1.0-float(common[0,0])/float(common[0,0]+dis)

else:

return 0.0

def benchmark_jaccard_fast():

for i in range(mat.shape[0]):

jaccard_fast(mat.getrow(i),a)

def benchmark_jaccard_internal_todense():

for v1,v2 in zip(mat.todense(),a.todense()):

d.jaccard(v1,v2)

def benchmark_jaccard_internal_getrow():

for i in range(mat.shape[0]):

d.jaccard(mat.getrow(i),a)

print "Jaccard Fast:"

%time benchmark_jaccard_fast()

print "Jaccard Scipy (expanding to dense):"

%time benchmark_jaccard_internal_todense()

print "Jaccard Scipy (using getrow):"

%time benchmark_jaccard_internal_getrow()

其中jaccard_fast是我自己的实现.看起来我的实现比scipy稀疏矩阵上的内部更快,但是getrow()似乎会减慢我的实现速度.当我对scipy.spatial.distance.jaccard对jaccard_fast进行基准测试时,结果如下：

Jaccard Fast:

CPU times: user 1.28 s, sys: 0 ns, total: 1.28 s

Wall time: 1.28 s

Jaccard Scipy (expanding to dense):

CPU times: user 28 ms, sys: 8 ms, total: 36 ms

Wall time: 37.2 ms

Jaccard Scipy (using getrow):

CPU times: user 1.82 s, sys: 0 ns, total: 1.82 s

Wall time: 1.81 s

任何有关如何避免getrow瓶颈的帮助将不胜感激.由于内存限制,我无法使用todense()扩展我的稀疏矩阵.

解决方法:

In [33]: timeit for row in mat: x=row # sparse iteration

1 loops, best of 3: 510 ms per loop

In [35]: timeit for row in mat.todense(): x=row # dense iteration

10 loops, best of 3: 175 ms per loop

但我发现使用稀疏矩阵时你的d.jacard也会变慢

In [36]: ad=a.todense()

In [37]: timeit for row in mat.todense(): d.jaccard(row,ad) # all dense

1 loops, best of 3: 734 ms per loop

In [38]: timeit for row in mat: d.jaccard(row.todense(),ad) # inner dense

1 loops, best of 3: 1.69 s per loop

In [39]: timeit for row in mat: d.jaccard(row,a) # all sparse

1 loops, best of 3: 4.61 s per loop

消除地理因素

In [40]: mrow=mat.getrow(0)

In [41]: mrowd=mrow.todense()

In [42]: timeit d.jaccard(mrow, a) # one sparse row

1000 loops, best of 3: 1.32 ms per loop

In [43]: timeit d.jaccard(mrow.todense(), a.todense()) # with conversion

1000 loops, best of 3: 539 µs per loop

In [44]: timeit d.jaccard(mrowd, ad) # dense

10000 loops, best of 3: 173 µs per loop

======================

我需要重新运行这些测试,因为d.jaccard不能用于稀疏(并且jaccard_fast不能用于密集).因此,将稀疏行迭代问题与jaccard计算分开将需要更多工作.

我重写了jaccard_fast：

def my_jaccard(mat, a):

common = mat*a.T # sparse does the large matrix product well

dis=np.array([(row!=a).getnnz() for row in mat]) # iterative

cA = common.A.ravel()

return 1 - cA/(cA + dis)

它返回与密集行上运行的d.jaccard匹配的值.对于common为0的行,d.jaccard返回1.我似乎不需要cA测试(除非cA和dis在同一个插槽中都可能为0).

In [141]: r=np.array([d.jaccard(row,ad) for row in mat.todense()])

In [142]: r1=my_jaccard(mat,a)

In [143]: np.allclose(r,r1)

Out[143]: True

速度只有一半.如果我可以返工,那么dis calc应该具有相似的速度.

In [144]: timeit r=np.array([d.jaccard(row,ad) for row in mat.todense()])

1 loops, best of 3: 783 ms per loop

In [145]: timeit r1=my_jaccard(mat,a)

1 loops, best of 3: 1.29 s per loop

进一步调整计算结果.我屏蔽了0的常用值.这有两个目的 – 它确保我们没有除以0的问题,并且它减少了迭代次数,从而提高了速度.

def my_jaccard(mat, a):

common=mat*a.T

cA = common.A.ravel()

mask = cA!=0

cA = cA[mask]

dis = np.array([(row!=a).getnnz() for row, b in zip(mat,mask) if b])

ret = np.ones(mat.shape[0])

ret[mask] = 1 - cA/(cA+dis)

return ret

有了这个,时间有点下降.

In [188]: timeit my_jaccard(mat,a)

1 loops, best of 3: 1.04 s per loop

==================

在那个问题中,我研究了将稀疏矩阵与1行矩阵进行比较,并发现使用sparse.kron来复制行,是复制numpy广播的最快方法.

在jaccard中使用该想法来计算dis数组

def my_jaccard1(mat, a):

common = mat*a.T

cA = common.A.ravel()

aM = sparse.kron(a,np.ones((mat.shape[0],1),int))

dis = (mat!=aM).sum(1)

ret = 1-cA/(cA+dis.A1)

return ret

随着这个时间显着改善(10倍)：

In [318]: timeit my_jaccard1(mat,a)

1 loops, best of 3: 97.1 ms per loop

我可以像以前一样应用掩蔽来防止零除;但它实际上减慢了计算速度(到140ms).

def my_jaccard3(mat, a):

common = mat*a.T

cA = common.A.ravel()

mask = cA!=0

cA = cA[mask]

aM = sparse.kron(a,np.ones((len(cA),1),int))

dis = (mat[mask,:]!=aM).sum(1)

ret = np.ones(mat.shape[0])

ret[mask] = 1 - cA/(cA+dis.A1)

return ret

========================

编辑 – 测试疑似病例

In [75]: x,y= np.array([1,1,0,0,1,0]), np.array([0,0,1,0,1,0])

In [76]: d.jaccard(x,y)

Out[76]: 0.75

In [78]: jaccard_fast(sparse.csr_matrix(x),sparse.csr_matrix(y))

Out[78]: 0.75

我的版本：

In [79]: my_jaccard(sparse.csr_matrix(x),sparse.csr_matrix(y))

Out[79]: array([ 0.75])

...

In [82]: my_jaccard3(sparse.csr_matrix(x),sparse.csr_matrix(y))

Out[82]: array([ 0.75])

(编辑 – 显式使用sparse.kron)

标签：python,scipy,sparse-matrix

来源： https://codeday.me/bug/20190628/1311561.html