案例逐步演示python利用正则表达式提取指定内容并输出到csv

背景和目标

这次我想要处理的是一个txt文件，里面的内容是一台机器定时ping另一台机器的输出结果，想要提取出的内容是时间和rtt值，最后还要把结果输出到csv文件。

1. 明确要提取的内容，编写正则表达式

要提取的文本如下：

第一步是要编写正则表达式，此时可以先不要读取数据文件。先复制一部分数据到str中，方便测试。
编写正则表达式用到了re模块，因为每个人要处理的文本是不一样的，所以需要自己去学习基本的使用方法。re具体使用方法可以参考这篇文章：
https://zhuanlan.zhihu.com/p/139596371

关键就是弄清楚.*?和{}的作用，还有re.S可以匹配到换行符，就可以比较容易地写出正确的表达式。

import re
# 为了方便测试，我把一部分文本先放到str里
str='''
2022-03-11 15:21:48
1
PING 81.71.51.181 (81.71.51.181) 56(84) bytes of data.
64 bytes from 81.71.51.181: icmp_seq=1 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=2 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=3 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=4 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=5 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=6 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=7 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=8 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=9 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=10 ttl=45 time=253 ms--- 81.71.51.181 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9000ms
rtt min/avg/max/mdev = 250.203/250.563/253.202/0.961 ms
2022-03-11 15:22:40
2
PING 81.71.51.181 (81.71.51.181) 56(84) bytes of data.
64 bytes from 81.71.51.181: icmp_seq=1 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=2 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=3 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=4 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=5 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=6 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=7 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=8 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=9 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=10 ttl=45 time=250 ms--- 81.71.51.181 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9009ms
rtt min/avg/max/mdev = 250.181/250.256/250.434/0.636 ms
2022-03-11 15:23:44
3
PING 81.71.51.181 (81.71.51.181) 56(84) bytes of data.
64 bytes from 81.71.51.181: icmp_seq=1 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=2 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=3 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=4 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=5 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=6 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=7 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=8 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=9 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=10 ttl=45 time=250 ms--- 81.71.51.181 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9009ms
rtt min/avg/max/mdev = 250.209/250.320/250.658/0.563 ms
'''# print(re.findall(r'(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2})', str))  # 提取时间
# print(re.findall(r'mdev = (.*?) ms', str))  # 提取rttprint(re.findall(r'(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}).*?mdev = (.*?) ms', data, re.S))  # 提取时间和rtt 包括换行

输出：

D:\python37\python.exe D:/test/data_process.py
['2022-03-11 15:21', '2022-03-11 15:22', '2022-03-11 15:23']
['250.203/250.563/253.202/0.961', '250.181/250.256/250.434/0.636', '250.209/250.320/250.658/0.563']
[('2022-03-11 15:21', '250.203/250.563/253.202/0.961'), ('2022-03-11 15:22', '250.181/250.256/250.434/0.636'), ('2022-03-11 15:23', '250.209/250.320/250.658/0.563')]Process finished with exit code 0

2. 从文件中读入数据

编写出正确的正则表达式后，就可以从文件中读数据了

import re
# 读取文件
with open("ping/ping_flkf_gz.txt","r") as input_file:str = input_file.read()print(re.findall(r'(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}).*?mdev = (.*?) ms', str, re.S))  # 提取时间和延迟 包括换行input_file.close()  # 关闭文件

输出比较多，截取一部分展示：

D:\python37\python.exe D:/test/data_process.py
[('2022-03-11 15:21', '250.203/250.563/253.202/0.961'), ('2022-03-11 15:22', '250.181/250.256/250.434/0.636'), ('2022-03-11 15:23', '250.209/250.320/250.658/0.563'), ('2022-03-11 15:25', '250.183/250.240/250.275/0.225'), ('2022-03-11 15:26', '250.217/250.240/250.300/0.592'), ('2022-03-11 15:27', '250.166/250.362/250.956/0.683'), ('2022-03-11 15:28', '250.186/250.256/250.343/0.319'), ('2022-03-11 15:29', '250.181/250.435/252.077/0.776'), ('2022-03-11 15:30', '250.177/250.249/250.401/0.673'), ('2022-03-11 15:31', '250.210/250.436/251.498/0.376'), ('2022-03-11 15:32', '250.207/250.280/250.588/0.401'), ('2022-03-11 15:33', '250.237/250.336/250.747/0.568'), ('2022-03-11 15:34', '250.217/250.283/250.437/0.675'), ('2022-03-11 15:35', '250.254/250.456/251.092/0.623'), ('2022-03-11 15:36', '250.167/250.236/250.308/0.226'), ('2022-03-11 15:37', '250.162/250.399/251.032/0.667'), ('2022-03-11 15:38', '250.207/250.261/250.406/0.053'), ('2022-03-11 15:39', '250.219/250.657/252.056/0.878')]

这里其实是一个列表，里面的每个元组是我提取出来的时间和rtt。

3. 写入csv

能够正确读取输入文件并提取数据后，下一步就是要把结果写入csv文件，所以用到了csv模块。
for循环遍历列表，使用csv_writer.writerow一行行写入csv文件。

import re
import csv# 读取文件
with open("ping/ping_flkf_gz.txt", "r") as input_file:str = input_file.read()# 用一个列表接收提取出来的数据
list = re.findall(r'(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}).*?mdev = (.*?) ms', str, re.S)  # 1.创建文件对象
output_file = open('res/ping/ping_flkf_gz.csv', 'w', encoding='utf-8', newline='')
# 2.基于文件对象构建csv写入对象
csv_writer = csv.writer(output_file)
# 3. 写入表头
csv_writer.writerow(["time", "latency"])
# 4.遍历列表，写入csv文件
for i in list:csv_writer.writerow([i[0], i[1]])input_file.close()  # 关闭文件
output_file.close()  # 关闭文件

结果就写入到csv文件中了

time,latency
2022-03-11 15:21,250.203/250.563/253.202/0.961
2022-03-11 15:22,250.181/250.256/250.434/0.636
2022-03-11 15:23,250.209/250.320/250.658/0.563
2022-03-11 15:25,250.183/250.240/250.275/0.225
2022-03-11 15:26,250.217/250.240/250.300/0.592
2022-03-11 15:27,250.166/250.362/250.956/0.683
2022-03-11 15:28,250.186/250.256/250.343/0.319
2022-03-11 15:29,250.181/250.435/252.077/0.776
2022-03-11 15:30,250.177/250.249/250.401/0.673
2022-03-11 15:31,250.210/250.436/251.498/0.376

4. 还可以把每个数值分开存放

发现此时latency那一列是这样的250.203/250.563/253.202/0.961
为了后面方便处理，把每个数值单独作为一列，因此要修改正则表达式

import re
import csv# 读取文件
with open("ping/ping_flkf_gz.txt", "r") as input_file:str = input_file.read()# 用一个列表接收提取出来的数据 这里修改了正则表达式，使得每个数值单独作为一列
list = re.findall(r'(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}).*?mdev = (.*?)/(.*?)/(.*?)/(.*?) ms', str, re.S)# 1.创建文件对象
output_file = open('res/ping/ping_flkf_gz.csv', 'w', encoding='utf-8', newline='')
# 2.基于文件对象构建csv写入对象
csv_writer = csv.writer(output_file)
# 3. 写入表头
csv_writer.writerow(["time", "min", "avg", "max", "mdev"])
# 4.遍历列表，写入csv文件
for i in list:csv_writer.writerow([i[0], i[1], i[2], i[3], i[4]])input_file.close()  # 关闭文件
output_file.close()  # 关闭文件

输出到csv文件的效果：

至此就完成了~