学术前沿趋势分析Task01

一、任务说明

任务主题：论文数量统计，即统计2019年全年计算机各个方向论文数量；
任务内容：赛题的理解、使用 Pandas 读取数据并进行统计；
任务内容：任务成果：学习 Pandas 的基础操作；

二、数据集介绍

数据集的格式如下：

id：arXiv ID，可用于访问论文；
submitter：论文提交者；
authors：论文作者；
title：论文标题；
comments：论文页数和图表等其他信息；
journal-ref：论文发表的期刊的信息；
doi：数字对象标识符，https://www.doi.org；
report-no：报告编号；
categories：论文在 arXiv 系统的所属类别或标签；
license：文章的许可证；
abstract：论文摘要；
versions：论文版本；
authors_parsed：作者的信息。

三、arxiv论文类别介绍

‘astro-ph’: ‘Astrophysics’,
‘astro-ph.CO’: ‘Cosmology and Nongalactic Astrophysics’,
‘astro-ph.EP’: ‘Earth and Planetary Astrophysics’,
‘astro-ph.GA’: ‘Astrophysics of Galaxies’,
‘cs.AI’: ‘Artificial Intelligence’,
‘cs.AR’: ‘Hardware Architecture’,
‘cs.CC’: ‘Computational Complexity’,
‘cs.CE’: ‘Computational Engineering, Finance, and Science’,
‘cs.CV’: ‘Computer Vision and Pattern Recognition’,
‘cs.CY’: ‘Computers and Society’,
‘cs.DB’: ‘Databases’,
‘cs.DC’: ‘Distributed, Parallel, and Cluster Computing’,
‘cs.DL’: ‘Digital Libraries’,
‘cs.NA’: ‘Numerical Analysis’,
‘cs.NE’: ‘Neural and Evolutionary Computing’,
‘cs.NI’: ‘Networking and Internet Architecture’,
‘cs.OH’: ‘Other Computer Science’,
‘cs.OS’: ‘Operating Systems’,

四、具体代码实现以及讲解

4.1 导入package并读取原始数据

# 导入所需的package
import seaborn as sns #用于画图
from bs4 import BeautifulSoup #用于爬取arxiv的数据
import re #用于正则表达式，匹配字符串的模式
import requests #用于网络连接，发送网络请求，使用域名获取对应信息
import json #读取数据，我们的数据为json格式的
import pandas as pd #数据处理，数据分析
import matplotlib.pyplot as plt #画图工具

# 读入数据
data  = [] #初始化
#使用with语句优势：1.自动关闭文件句柄；2.自动显示（处理）文件读取数据异常
with open("arxiv-metadata-oai-snapshot.json", 'r') as f: for line in f: data.append(json.loads(line))data = pd.DataFrame(data) #将list变为dataframe格式，方便使用pandas进行分析
data.shape #显示数据大小
>>> (1796911, 14)

(1796911, 14) 其中的1796911表示数据总量，14表示特征数，对应我们1.2节说明的论文的14种信息。

data.head()

	id	submitter	authors	title	comments	journal-ref	doi	report-no	categories	license	abstract	versions	update_date	authors_parsed
0	0704.0001	Pavel Nadolsky	C. Bal\'azs, E. L. Berger, P. M. Nadolsky, C.-...	Calculation of prompt diphoton production cros...	37 pages, 15 figures; published version	Phys.Rev.D76:013009,2007	10.1103/PhysRevD.76.013009	ANL-HEP-PR-07-12	hep-ph	None	A fully differential calculation in perturba...	[{'version': 'v1', 'created': 'Mon, 2 Apr 2007...	2008-11-26	[[Balázs, C., ], [Berger, E. L., ], [Nadolsky,...
1	0704.0002	Louis Theran	Ileana Streinu and Louis Theran	Sparsity-certifying Graph Decompositions	To appear in Graphs and Combinatorics	None	None	None	math.CO cs.CG	http://arxiv.org/licenses/nonexclusive-distrib...	We describe a new algorithm, the $(k,\ell)$-...	[{'version': 'v1', 'created': 'Sat, 31 Mar 200...	2008-12-13	[[Streinu, Ileana, ], [Theran, Louis, ]]
2	0704.0003	Hongjun Pan	Hongjun Pan	The evolution of the Earth-Moon system based o...	23 pages, 3 figures	None	None	None	physics.gen-ph	None	The evolution of Earth-Moon system is descri...	[{'version': 'v1', 'created': 'Sun, 1 Apr 2007...	2008-01-13	[[Pan, Hongjun, ]]
3	0704.0004	David Callan	David Callan	A determinant of Stirling cycle numbers counts...	11 pages	None	None	None	math.CO	None	We show that a determinant of Stirling cycle...	[{'version': 'v1', 'created': 'Sat, 31 Mar 200...	2007-05-23	[[Callan, David, ]]
4	0704.0005	Alberto Torchinsky	Wael Abu-Shammala and Alberto Torchinsky	From dyadic $\Lambda_{\alpha}$ to $\Lambda_{\a...	None	Illinois J. Math. 52 (2008) no.2, 681-689	None	None	math.CA math.FA	None	In this paper we show how to compute the $\L...	[{'version': 'v1', 'created': 'Mon, 2 Apr 2007...	2013-10-15	[[Abu-Shammala, Wael, ], [Torchinsky, Alberto, ]]

4.2 数据预处理

count：一列数据的元素个数；
unique：一列数据中元素的种类；
top：一列数据中出现频率最高的元素；
freq：一列数据中出现频率最高的元素的个数；

data['categories'].describe()count      1796911
unique       62055
top       astro-ph
freq         86914
Name: categories, dtype: object

以上的结果表明：共有1,796,911个数据，有62,055个子类，其中最多的种类是astro-ph，即Astrophysics（天体物理学），共出现了86914次。

由于部分论文的类别不止一种，所以下面我们判断在本数据集中共出现了多少种独立的数据集。

# 所有的种类(独立的)
unique_categories = set([i for l in [x.split(' ') for x in data['categories']] for i in l])

len(unique_categories)
>>> 176

unique_categories{'acc-phys','adap-org','alg-geom',...'stat.OT','stat.TH','supr-con'}

这里使用了 split 函数将多类别使用 “ ”（空格）分开，组成list，并使用 for 循环将独立出现的类别找出来，并使用 set 类别，将重复项去除得到最终所有的独立paper种类。

从以上结果发现，共有176种论文种类，而官网上是153种论文种类，这说明存在一些官网上没有的类别，不过对于我们的计算机方向的论文没有影响。

我们的任务要求对于2019年以后的paper进行分析，所以首先对于时间特征进行预处理，从而得到2019年以后的所有种类的论文：

data['update_date'].astype # object类型<bound method NDFrame.astype of 0          2008-11-26
1          2008-12-13
2          2008-01-13
3          2007-05-23
4          2013-10-15...
1796906    2009-10-30
1796907    2016-11-18
1796908    2009-10-30
1796909    2009-10-30
1796910    2009-10-30
Name: update_date, Length: 1796911, dtype: object>

#将update_date从例如2019-02-20的str变为datetime格式，并提取处year
data['year'] = pd.to_datetime(data['update_date']).dt.year
#找出 year 中2019年以后的数据
data = data[data['year'] >= 2019]
data.shape # 还剩395123条数据
>>> (395123, 15)

data.reset_index(drop=True, inplace=True) #重新编号
data.head() #查看结果

	id	submitter	authors	title	comments	journal-ref	doi	report-no	categories	license	abstract	versions	update_date	authors_parsed	year
0	0704.0297	Sung-Chul Yoon	Sung-Chul Yoon, Philipp Podsiadlowski and Step...	Remnant evolution after a carbon-oxygen white ...	15 pages, 15 figures, 3 tables, submitted to M...	None	10.1111/j.1365-2966.2007.12161.x	None	astro-ph	None	We systematically explore the evolution of t...	[{'version': 'v1', 'created': 'Tue, 3 Apr 2007...	2019-08-19	[[Yoon, Sung-Chul, ], [Podsiadlowski, Philipp,...	2019
1	0704.0342	Patrice Ntumba Pungu	B. Dugmore and PP. Ntumba	Cofibrations in the Category of Frolicher Spac...	27 pages	None	None	None	math.AT	None	Cofibrations are defined in the category of ...	[{'version': 'v1', 'created': 'Tue, 3 Apr 2007...	2019-08-19	[[Dugmore, B., ], [Ntumba, PP., ]]	2019
2	0704.0360	Zaqarashvili	T.V. Zaqarashvili and K Murawski	Torsional oscillations of longitudinally inhom...	6 pages, 3 figures, accepted in A&A	None	10.1051/0004-6361:20077246	None	astro-ph	None	We explore the effect of an inhomogeneous ma...	[{'version': 'v1', 'created': 'Tue, 3 Apr 2007...	2019-08-19	[[Zaqarashvili, T. V., ], [Murawski, K, ]]	2019
3	0704.0525	Sezgin Ayg\"un	Sezgin Aygun, Ismail Tarhan, Husnu Baysal	On the Energy-Momentum Problem in Static Einst...	This submission has been withdrawn by arXiv ad...	Chin.Phys.Lett.24:355-358,2007	10.1088/0256-307X/24/2/015	None	gr-qc	None	This paper has been removed by arXiv adminis...	[{'version': 'v1', 'created': 'Wed, 4 Apr 2007...	2019-10-21	[[Aygun, Sezgin, ], [Tarhan, Ismail, ], [Baysa...	2019
4	0704.0535	Antonio Pipino	Antonio Pipino (1,3), Thomas H. Puzia (2,4), a...	The Formation of Globular Cluster Systems in M...	32 pages (referee format), 9 figures, ApJ acce...	Astrophys.J.665:295-305,2007	10.1086/519546	None	astro-ph	None	The most massive elliptical galaxies show a ...	[{'version': 'v1', 'created': 'Wed, 4 Apr 2007...	2019-08-19	[[Pipino, Antonio, ], [Puzia, Thomas H., ], [M...	2019

这里我们就已经得到了所有2019年以后的论文，下面我们挑选出计算机领域内的所有文章：

#爬取所有的类别
website_url = requests.get('https://arxiv.org/category_taxonomy').text #获取网页的文本数据
soup = BeautifulSoup(website_url,'lxml') #爬取数据，这里使用lxml的解析器，加速
root = soup.find('div',{'id':'category_taxonomy_list'}) #找出 BeautifulSoup 对应的标签入口
tags = root.find_all(["h2","h3","h4","p"], recursive=True) #读取 tags

https://www.jianshu.com/p/fb6ee6cc5c1c

import requests
requests.get(url) #get请求核心代码是requests.get(url)
requests.post(url) #post请求核心代码是requests.post(url,data={请求体的字典})
requests.put(url)
requests.delete(url)
requests.head(url)
requests.options(url)

https://cuiqingcai.com/1319.html

Beautiful Soup 将复杂 HTML 文档转换成一个复杂的树形结构，每个节点都是 Python 对象，所有对象可以归纳为 4 种:

Tag：就是 HTML 中的一个个标签，可以利用 soup 加标签名轻松地获取这些标签的内容

NavigableString：获取标签内部的文字用 .string 即可，它的类型是一个 NavigableString

BeautifulSoup：BeautifulSoup 对象表示的是一个文档的全部内容。大部分时候，可以把它当作 Tag 对象，是一个特殊的 Tag，我们可以分别获取它的类型，名称，以及属性

Comment：一个特殊类型的 NavigableString 对象，输出的内容不包括注释符号

#初始化 str 和 list 变量
level_1_name = ""
level_2_name = ""
level_2_code = ""
level_1_names = []
level_2_codes = []
level_2_names = []
level_3_codes = []
level_3_names = []
level_3_notes = []

#进行
for t in tags:if t.name == "h2":level_1_name = t.text    level_2_code = t.textlevel_2_name = t.textelif t.name == "h3":raw = t.textlevel_2_code = re.sub(r"(.*)\((.*)\)",r"\2",raw) #正则表达式：模式字符串：(.*)\((.*)\)；被替换字符串"\2"；被处理字符串：rawlevel_2_name = re.sub(r"(.*)\((.*)\)",r"\1",raw)elif t.name == "h4":raw = t.textlevel_3_code = re.sub(r"(.*) \((.*)\)",r"\1",raw)level_3_name = re.sub(r"(.*) \((.*)\)",r"\2",raw)elif t.name == "p":notes = t.textlevel_1_names.append(level_1_name)level_2_names.append(level_2_name)level_2_codes.append(level_2_code)level_3_names.append(level_3_name)level_3_codes.append(level_3_code)level_3_notes.append(notes)

#根据以上信息生成dataframe格式的数据
df_taxonomy = pd.DataFrame({'group_name' : level_1_names,'archive_name' : level_2_names,'archive_id' : level_2_codes,'category_name' : level_3_names,'categories' : level_3_codes,'category_description': level_3_notes
})

df_taxonomy.head()

	group_name	archive_name	archive_id	category_name	categories	category_description
0	Computer Science	Computer Science	Computer Science	Artificial Intelligence	cs.AI	Covers all areas of AI except Vision, Robotics...
1	Computer Science	Computer Science	Computer Science	Hardware Architecture	cs.AR	Covers systems organization and hardware archi...
2	Computer Science	Computer Science	Computer Science	Computational Complexity	cs.CC	Covers models of computation, complexity class...
3	Computer Science	Computer Science	Computer Science	Computational Engineering, Finance, and Science	cs.CE	Covers applications of computer science to the...
4	Computer Science	Computer Science	Computer Science	Computational Geometry	cs.CG	Roughly includes material in ACM Subject Class...

#按照 "group_name" 进行分组，在组内使用 "archive_name" 进行排序
df_taxonomy.groupby(["group_name","archive_name"])
df_taxonomy

	group_name	archive_name	archive_id	category_name	categories	category_description
0	Computer Science	Computer Science	Computer Science	Artificial Intelligence	cs.AI	Covers all areas of AI except Vision, Robotics...
1	Computer Science	Computer Science	Computer Science	Hardware Architecture	cs.AR	Covers systems organization and hardware archi...
2	Computer Science	Computer Science	Computer Science	Computational Complexity	cs.CC	Covers models of computation, complexity class...
3	Computer Science	Computer Science	Computer Science	Computational Engineering, Finance, and Science	cs.CE	Covers applications of computer science to the...
4	Computer Science	Computer Science	Computer Science	Computational Geometry	cs.CG	Roughly includes material in ACM Subject Class...
...	...	...	...	...	...	...
150	Statistics	Statistics	Statistics	Computation	stat.CO	Algorithms, Simulation, Visualization
151	Statistics	Statistics	Statistics	Methodology	stat.ME	Design, Surveys, Model Selection, Multiple Tes...
152	Statistics	Statistics	Statistics	Machine Learning	stat.ML	Covers machine learning papers (supervised, un...
153	Statistics	Statistics	Statistics	Other Statistics	stat.OT	Work in statistics that does not fit into the ...
154	Statistics	Statistics	Statistics	Statistics Theory	stat.TH	stat.TH is an alias for math.ST. Asymptotics, ...

155 rows × 6 columns

说明一下上面代码中的正则操作，这里我们使用re.sub来用于替换字符串中的匹配项

raw = 'Astrophysics(astro-ph)'
re.sub(r"(.*)\((.*)\)",r"\2",raw)
>>> 'astro-ph'

对应的参数：

正则中的模式字符串 pattern 的格式为 “任意字符” + “(” + “任意字符” + “)”。
替换的字符串 repl 为第2个分组的内容。
要被查找替换的原始字符串 string 为原始的爬取的数据。

这里推荐大家一个在线正则表达式测试的网站：https://tool.oschina.net/regex/

4.3 数据分析及可视化

接下来我们首先看一下所有大类的paper数量分布：

# 使用merge函数，以两个dataframe共同的属性 “categories” 进行合并，并以 “group_name” 作为类别进行统计，统计结果放入 “id” 列中并排序。
_df = data.merge(df_taxonomy, on="categories", how="left").drop_duplicates(["id","group_name"]).groupby("group_name").agg({"id":"count"}).sort_values(by="id",ascending=False).reset_index()_df

	group_name	id
0	Physics	79985
1	Mathematics	51567
2	Computer Science	40067
3	Statistics	4054
4	Electrical Engineering and Systems Science	3297
5	Quantitative Biology	1994
6	Quantitative Finance	826
7	Economics	576

下面我们使用饼图进行上图结果的可视化：

fig = plt.figure(figsize=(15,12))
explode = (0, 0, 0, 0.2, 0.3, 0.3, 0.2, 0.1)
plt.pie(_df["id"],  labels=_df["group_name"], autopct='%1.2f%%', startangle=160, explode=explode)
plt.tight_layout() # tight_layout会自动调整子图参数，使之填充整个图像区域。
plt.show()

https://www.cnblogs.com/biyoulin/p/9565350.html

def pie(x, explode=None, labels=None, colors=None, autopct=None,
pctdistance=0.6, shadow=False, labeldistance=1.1, startangle=None,
radius=None, counterclock=True, wedgeprops=None, textprops=None,
center=(0, 0), frame=False, rotatelabels=False, hold=None, data=None)

下面统计在计算机各个子领域2019年后的paper数量：

# 我们同样使用 merge 函数，对于两个dataframe 共同的特征 categories 进行合并并且进行查询。然后我们再对于数据进行统计和排序：
group_name="Computer Science"
cats = data.merge(df_taxonomy, on="categories").query("group_name == @group_name")
cats.groupby(["year","category_name"]).count().reset_index().pivot(index="category_name", columns="year",values="id")

year	2019	2020
category_name
Artificial Intelligence	558	757
Computation and Language	2153	2906
Computational Complexity	131	188
Computational Engineering, Finance, and Science	108	205
Computational Geometry	199	216
Computer Science and Game Theory	281	323
Computer Vision and Pattern Recognition	5559	6517
Computers and Society	346	564
Cryptography and Security	1067	1238
Data Structures and Algorithms	711	902
Databases	282	342
Digital Libraries	125	157
Discrete Mathematics	84	81
Distributed, Parallel, and Cluster Computing	715	774
Emerging Technologies	101	84
Formal Languages and Automata Theory	152	137
General Literature	5	5
Graphics	116	151
Hardware Architecture	95	159
Human-Computer Interaction	420	580
Information Retrieval	245	331
Logic in Computer Science	470	504
Machine Learning	177	538
Mathematical Software	27	45
Multiagent Systems	85	90
Multimedia	76	66
Networking and Internet Architecture	864	783
Neural and Evolutionary Computing	235	279
Numerical Analysis	40	11
Operating Systems	36	33
Other Computer Science	67	69
Performance	45	51
Programming Languages	268	294
Robotics	917	1298
Social and Information Networks	202	325
Software Engineering	659	804
Sound	7	4
Symbolic Computation	44	36
Systems and Control	415	133

我们可以从结果看出，Computer Vision and Pattern Recognition（计算机视觉与模式识别）类是CS中paper数量最多的子类，遥遥领先于其他的CS子类，并且paper的数量还在逐年增加；另外，Computation and Language（计算与语言）、Cryptography and Security（密码学与安全）以及 Robotics（机器人学）的2019年paper数量均超过1000或接近1000，这与我们的认知是一致的。