【Scikit-Learn】使用k-均值对文档进行聚类分析

本文使用的数据集来自mlcomp.org上的20news-18828，下载地址为：mlcomp.org/datasets/379

我们只对sci.crypt、sci.electronices、sci.med和sci.space这4个子目录的数据进行k-均值聚类分析。

1. 读入数据

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as npfrom time import time
from sklearn.datasets import load_filesprint("loading documents ...")
t = time()
docs = load_files('datasets/clustering/clustering')
print("summary: {0} documents in {1} categories.".format(len(docs.data), len(docs.target_names)))
print("done in {0} seconds".format(time() - t))
'''
loading documents ...
summary: 3949 documents in 4 categories.
done in 0.3440744876861572 seconds
'''

2. 对语料文件进行TF-IDF向量化

from sklearn.feature_extraction.text import TfidfVectorizermax_features = 20000
print("vectorizing documents ...")
t = time()
vectorizer = TfidfVectorizer(max_df=0.4, # 过滤词频大于 0.4 的单词min_df=2, # 如果一个单词只在两个以下（包含两个）的文档里出现，则也把这个单词过滤掉max_features=max_features, encoding='latin-1')
X = vectorizer.fit_transform((d for d in docs.data))
print("n_samples: %d, n_features: %d" % X.shape)
print("number of non-zero features in sample [{0}]: {1}".format(docs.filenames[0], X[0].getnnz()))
print("done in {0} seconds".format(time() - t))
'''
vectorizing documents ...
n_samples: 3949, n_features: 20000
number of non-zero features in sample [datasets/clustering/clustering\sci.electronics\11902-54322]: 56
done in 0.9584255218505859 seconds
'''

3. 运行k-均值算法进行聚类

from sklearn.cluster import KMeansprint("clustering documents ...")
t = time()
n_clusters = 4
kmean = KMeans(n_clusters=n_clusters, # 聚类个数为 4max_iter=100, # 最多进行 100 次 k-均值迭代tol=0.01, # 当中心点移动距离小于 0.1 时就认为算法已经收敛，停止迭代verbose=1, # 输出迭代过程信息n_init=3) # 进行 3 次 k-均值运算后求平均值
kmean.fit(X);
print("kmean: k={}, cost={}".format(n_clusters, int(kmean.inertia_)))
print("done in {0} seconds".format(time() - t))
'''
clustering documents ...
Initialization complete
Iteration  0, inertia 7502.039
Iteration  1, inertia 3849.399
Iteration  2, inertia 3839.072
Iteration  3, inertia 3834.151
Iteration  4, inertia 3830.283
Iteration  5, inertia 3827.674
Iteration  6, inertia 3825.856
Iteration  7, inertia 3824.488
Iteration  8, inertia 3823.273
Iteration  9, inertia 3822.468
Iteration 10, inertia 3821.969
Iteration 11, inertia 3821.726
Iteration 12, inertia 3821.545
Iteration 13, inertia 3821.433
Iteration 14, inertia 3821.361
Iteration 15, inertia 3821.316
Iteration 16, inertia 3821.280
Iteration 17, inertia 3821.241
Iteration 18, inertia 3821.210
Iteration 19, inertia 3821.192
Iteration 20, inertia 3821.180
Iteration 21, inertia 3821.178
Converged at iteration 21: center shift 0.000000e+00 within tolerance 4.896692e-07
Initialization complete
Iteration  0, inertia 7521.309
Iteration  1, inertia 3839.890
Iteration  2, inertia 3826.370
Iteration  3, inertia 3823.083
Iteration  4, inertia 3821.392
Iteration  5, inertia 3820.446
Iteration  6, inertia 3819.945
Iteration  7, inertia 3819.539
Iteration  8, inertia 3819.154
Iteration  9, inertia 3818.839
Iteration 10, inertia 3818.604
Iteration 11, inertia 3818.401
Iteration 12, inertia 3818.286
Iteration 13, inertia 3818.179
Iteration 14, inertia 3818.103
Iteration 15, inertia 3818.068
Iteration 16, inertia 3818.032
Iteration 17, inertia 3817.988
Iteration 18, inertia 3817.947
Iteration 19, inertia 3817.903
Iteration 20, inertia 3817.863
Iteration 21, inertia 3817.808
Iteration 22, inertia 3817.777
Iteration 23, inertia 3817.760
Iteration 24, inertia 3817.745
Iteration 25, inertia 3817.727
Iteration 26, inertia 3817.723
Converged at iteration 26: center shift 0.000000e+00 within tolerance 4.896692e-07
Initialization complete
Iteration  0, inertia 7511.412
Iteration  1, inertia 3845.646
Iteration  2, inertia 3834.184
Iteration  3, inertia 3828.370
Iteration  4, inertia 3825.227
Iteration  5, inertia 3822.804
Iteration  6, inertia 3821.556
Iteration  7, inertia 3821.051
Iteration  8, inertia 3820.743
Iteration  9, inertia 3820.438
Iteration 10, inertia 3820.183
Iteration 11, inertia 3819.960
Iteration 12, inertia 3819.704
Iteration 13, inertia 3819.417
Iteration 14, inertia 3819.133
Iteration 15, inertia 3818.751
Iteration 16, inertia 3818.352
Iteration 17, inertia 3818.065
Iteration 18, inertia 3817.937
Iteration 19, inertia 3817.877
Iteration 20, inertia 3817.848
Iteration 21, inertia 3817.816
Iteration 22, inertia 3817.793
Iteration 23, inertia 3817.764
Iteration 24, inertia 3817.738
Iteration 25, inertia 3817.727
Iteration 26, inertia 3817.719
Iteration 27, inertia 3817.715
Iteration 28, inertia 3817.709
Iteration 29, inertia 3817.700
Iteration 30, inertia 3817.698
Iteration 31, inertia 3817.693
Iteration 32, inertia 3817.682
Iteration 33, inertia 3817.673
Iteration 34, inertia 3817.664
Iteration 35, inertia 3817.648
Iteration 36, inertia 3817.641
Iteration 37, inertia 3817.623
Iteration 38, inertia 3817.605
Iteration 39, inertia 3817.590
Iteration 40, inertia 3817.576
Iteration 41, inertia 3817.559
Iteration 42, inertia 3817.552
Iteration 43, inertia 3817.550
Converged at iteration 43: center shift 0.000000e+00 within tolerance 4.896692e-07
kmean: k=4, cost=3817
done in 25.344247817993164 seconds
'''

4. 观察聚类结果

# kmean.labels_ 保存了这些文档的类别信息
len(kmean.labels_)
'''3949
'''# 查看第1000到1010号文档的真实标签，及聚类预测出的标签，聚类结果大致准确
kmean.labels_[1000:1010]
'''array([0, 0, 0, 3, 1, 3, 3, 0, 3, 3])
'''docs.filenames[1000:1010]
'''
array(['datasets/clustering/clustering\\sci.crypt\\10888-15289','datasets/clustering/clustering\\sci.crypt\\11490-15880','datasets/clustering/clustering\\sci.crypt\\11270-15346','datasets/clustering/clustering\\sci.electronics\\12383-53525','datasets/clustering/clustering\\sci.space\\13826-60862','datasets/clustering/clustering\\sci.electronics\\11631-54106','datasets/clustering/clustering\\sci.space\\14235-61437','datasets/clustering/clustering\\sci.crypt\\11508-15928','datasets/clustering/clustering\\sci.space\\13593-60824','datasets/clustering/clustering\\sci.electronics\\12304-52801'],dtype='<U58')
'''# 由于每个样本都是 20000 维的向量，因此 4 个聚类中心 也都是 20000维的向量
kmean.cluster_centers_.shape
'''(4, 20000)
'''# 查看每个聚类中心 词频最高的单词
from __future__ import print_functionprint("Top terms per cluster:")order_centroids = kmean.cluster_centers_.argsort()[:, ::-1]
"""# argsort()函数是将 Numpy 数组 进行升序排列 ，并返回排序后的索引# [::-1]运算是将升序变成降序
a = np.array([[20, 10, 30, 40], [100, 300, 200, 400], [1, 5, 3, 2]])
a.argsort()[:, ::-1]array([[3, 2, 0, 1],[3, 1, 2, 0],[1, 2, 3, 0]])
"""terms = vectorizer.get_feature_names() # 返回词典
for i in range(n_clusters):print("Cluster %d:" % i, end='')for ind in order_centroids[i, :10]:print(' %s' % terms[ind], end='')print()
'''
Top terms per cluster:
Cluster 0: key clipper encryption chip government will keys escrow we nsa
Cluster 1: space henry nasa toronto pat zoo moon shuttle spencer orbit
Cluster 2: msg she my pitt gordon geb banks her he has
Cluster 3: any my by know me your like anyone will out
'''

【Scikit-Learn】使用k-均值对文档进行聚类分析相关推荐

python机器学习案例系列教程——k均值聚类、k中心点聚类
全栈工程师开发手册 (作者:栾鹏) python数据挖掘系列教程上一篇我们学习了层次聚类.层次聚类只是迭代的把最相近的两个聚类匹配起来.并没有给出能给出多少的分组.今天我们来研究一个K均值聚类.就是 ...
hanlp 词频统计_10.HanLP实现k均值--文本聚类
AI 人工智能 10.HanLP实现k均值--文本聚类 10. 文本聚类正所谓物以类聚,人以群分.人们在获取数据时需要整理,将相似的数据归档到一起,自动发现大量样本之间的相似性,这种根据相似性归档的 ...
k均值聚类算法考试例题_KMeans (K均值)算法讲解及实现
算法原理 KMeans算法是典型的基于距离的聚类算法,采用距离作为相似性的评价指标,即认为两个对象的距离越近,其相似度就越大.该算法认为簇是由距离靠近的对象组成的,因此把得到紧凑且独立的簇作为最终目标 ...
python docx 合并文档图片_不再为处理PDF烦恼，python处理操作PDF全攻略
本篇聊下Python对pdf的各种操作,包含pdf转word,pdf转图片,pdf翻转,加密,加水印等. pdf转换word文档保留格式 pdf转换为word文档,被大众经常使用的是纯Python库 ...
文档词频矩阵_论文理解：从词嵌入到文档距离
论文作者简介本论文第一作者Matt J. Kusner是牛津大学的副教授,致力于设计适应现实世界问题需求的新机器学习模型(例如,fair algorithms, discrete generativ ...
python docx 合并文档图片_python玩转pdf全攻略
本篇说点轻松的,聊下Python对pdf的各种操作,包含pdf转word,pdf转图片等. pdf转换word文档保留格式 pdf转换为word文档,被大众经常使用的是纯Python库pdfmine ...
mongodb中带条件的文档--$gte、 $gt、$lte、$lt、 $eq、 $ne、distinct、与、或（$or），$in、$exists
带条件的文档的格式为:db.数据集.find({ " " : " " }) $gte:大于等于 $gt:大于 $lte:小于等于 $lt:小于 $eq:等于 $ ...
原创 | 一文读懂K均值（K-Means）聚类算法
作者:王佳鑫审校:陈之炎本文约5800字,建议阅读10+分钟本文为你介绍经典的K-Means聚类算法. 概述众所周知,机器学习算法可分为监督学习(Supervised learning)和无监督学习 ...
一文读懂K均值（K-Means）聚类算法
作者:王佳鑫审校:陈之炎本文约5800字,建议阅读10+分钟本文为你介绍经典的K-Means聚类算法. 概述众所周知,机器学习算法可分为监督学习(Supervised learning)和无监督 ...

【Scikit-Learn】使用k-均值对文档进行聚类分析

1. 读入数据

2. 对语料文件进行TF-IDF向量化

3. 运行k-均值算法进行聚类

4. 观察聚类结果

【Scikit-Learn】使用k-均值对文档进行聚类分析相关推荐

最新文章

热门文章