python log文件处理_Python 处理大Log文件
最近研究 process monitor,对于取得的Log 需要进行简单的统计得出操作的类型,结果是一个 620000行左右的文件。数据示例如下:
"15:20:33.9935624","cmd.exe","1784","IRP_MJ_CREATE","C:\EDK","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened"
"15:20:33.9935792","cmd.exe","1784","IRP_MJ_DIRECTORY_CONTROL","C:\EDK\build.*","SUCCESS","Type: QueryDirectory, Filter: build.*, 2: Build"
"15:20:33.9936119","cmd.exe","1784","IRP_MJ_DIRECTORY_CONTROL","C:\EDK","NO MORE FILES","Type: QueryDirectory"
"15:20:33.9936272","cmd.exe","1784","IRP_MJ_CLEANUP","C:\EDK","SUCCESS",""
"15:20:33.9936306","cmd.exe","1784","IRP_MJ_CLOSE","C:\EDK","SUCCESS",""
"15:20:33.9937049","cmd.exe","1784","IRP_MJ_CREATE","C:\EDK","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened"
"15:20:33.9937169","cmd.exe","1784","IRP_MJ_DIRECTORY_CONTROL","C:\EDK\build","SUCCESS","Type: QueryDirectory, Filter: build, 2: Build"
"15:20:33.9937342","cmd.exe","1784","IRP_MJ_DIRECTORY_CONTROL","C:\EDK","NO MORE FILES","Type: QueryDirectory"
"15:20:33.9937476","cmd.exe","1784","IRP_MJ_CLEANUP","C:\EDK","SUCCESS",""
"15:20:33.9937541","cmd.exe","1784","IRP_MJ_CLOSE","C:\EDK","SUCCESS",""
"15:20:33.9938281","cmd.exe","1784","IRP_MJ_CREATE","C:\EDK\BaseTools\Bin","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened"
"15:20:33.9938586","cmd.exe","1784","IRP_MJ_DIRECTORY_CONTROL","C:\EDK\BaseTools\Bin\build.*","NO SUCH FILE","Type: QueryDirectory, Filter: build.*"
"15:20:33.9938787","cmd.exe","1784","IRP_MJ_CLEANUP","C:\EDK\BaseTools\Bin","SUCCESS",""
我需要提取出来IRP_MJ_DIRECTORY_CONTROL 和IRP_MJ_CLEANUP这样的操作记录下来。
听说Python擅长于此,于是简单学习一下进行统计。
第一个问题是如何处理大文件(目前的数据不算大,200MB),在【参考1】找到方法;
第二个问题是如何进行统计的问题。我只需要记录一个操作有还是没有,所以使用字典类型最合适不过;
最终代码如下(Python2.7环境下运行)
class Load_Corpus_with_Iteration(object):
def __init__(self,path):
self.path=path
def __iter__(self):
for line in open(self.path):
yield line.split()
corpus = Load_Corpus_with_Iteration('logfile.csv')
operate = {}
index=0
for item in corpus:
list1 = (item[0].split(','))
opStr=list1[3]
operate[opStr]=1
index=index+1
if index % 10000 ==0:
print index,str(operate)
结果如下(为了便于阅读,经过简单排版)
620000 {
'"IRP_MJ_DIRECTORY_CONTROL"': 1,
'"FASTIO_READ"': 1,
'"IRP_MJ_READ"': 1,
'"FASTIO_LOCK"': 1,
'"FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION"': 1,
'"IRP_MJ_CLOSE"': 1,
'"IRP_MJ_QUERY_INFORMATION"': 1,
'"IRP_MJ_SET_INFORMATION"': 1,
'"FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION"': 1,
'"FASTIO_QUERY_INFORMATION"': 1,
'"IRP_MJ_WRITE"': 1,
'"FASTIO_ACQUIRE_FOR_CC_FLUSH"': 1,
'"IRP_MJ_FILE_SYSTEM_CONTROL"': 1,
'"IRP_MJ_QUERY_VOLUME_INFORMATION"': 1,
'"FASTIO_WRITE"': 1,
'"IRP_MJ_CREATE"': 1,
'"FASTIO_NETWORK_QUERY_OPEN"': 1,
'"FASTIO_RELEASE_FOR_CC_FLUSH"': 1,
'"FASTIO_UNLOCK_SINGLE"': 1, '
"IRP_MJ_CLEANUP"': 1,
'"FASTIO_CHECK_IF_POSSIBLE"': 1}
参考:
1. https://blog.csdn.net/chixujohnny/article/details/53069988
python log文件处理_Python 处理大Log文件相关推荐
- python读取大文件内容_python 读取大文件
readlines.read().readline()等一次性读取遇到很大的文件都可能面临内存崩溃,所以我们将读取操作细化一点 1. read(size) + yield 固定每次读取的长度,使用生成 ...
- python 写csv加锁_Python: 对CSV文件读写 和 Md5加密
1. python 有专门的csv包,直接导入即可. import csv: 2. 直接使用普通文件的open方法 csv_reader=open("e:/python/csv_data/l ...
- python创建文件对象_python基础教程:文件读写
在Linux系统中,一切都是文件.但我们通常说的文件是保存在磁盘上的图片.文档.数据.程序等等.而在程序的IO操作中,很多时候就是从磁盘读写文件.本节我们讲解Python中的文件对象如何操作文件. 创 ...
- python装b代码_Python十大装B语法
Python 是一种代表简单思想的语言,其语法相对简单,很容易上手.不过,如果就此小视 Python 语法的精妙和深邃,那就大错特错了.本文精心筛选了最能展现 Python 语法之精妙的十个知识点,并 ...
- dpresult在python中什么意思_Python十大装B语法!你会几种?
本文的文字及图片来源于网络,仅供学习.交流使用,不具有任何商业用途,如有问题请及时联系我们以作处理 Python 是一种代表简单思想的语言,其语法相对简单,很容易上手.不过,如果就此小视 Python ...
- python创建空文本文件_Python干货:「文件处理整合」 创建、打开、写入和删除...
什么是Python文件处理? Python文件处理就是如何在python中创建.打开.读取.写入和删除文件的示例.从文件(EXCELSheet.doc文件.文本文件.cv等)或Add中读取数据.是一个 ...
- python根据文件名获取文件路径_python 查看文件名和文件路径
以下是把sourceDir目录下的以.JPG结尾的文件所有拷贝到targetDir目录下: >>>import os >>> import os.path > ...
- python 写txt 换行_python中写入txt文件需要换行,以及\r 和\n
在Python中,用open()函数打开一个txt文件,写入一行数据之后需要一个换行 如果直接用 f.write('\n') 只会在后面打印一个字符串'\n',而不是换行' 需要用 f.write(' ...
- python递归遍历目录_Python实现递归遍历文件夹并删除文件
思路: 遍历文件夹下面的文件夹 如果文件夹名称等于".svn",则修改文件夹的属性(因为".svn"的文件都是只读的,你不能直接删除) 删除此文件夹 如果文件夹 ...
- python批量读取文件内容_Python之批量读取文件【面试必学】
原博文 2019-12-24 14:00 − python的os模块可以实现普遍的操作系统功能,并且和平台无关.以下为实现根目录下文件的批量读取. os.listdir(dirname)可以列出dir ...
最新文章
- android将拦截短信通过邮箱发送,网易邮箱Android客户端推出 加速布局移动互联网...
- JAVA中方法的类型转换_Java中几种常用数据类型之间转换的方法
- python下载网页里面所有的图片-python批量下载图片的三种方法
- 【Linux】一步一步学Linux——expr命令(265)
- 大型分布式系统需要考虑的问题
- aix java home_java程序员工作日子一(java_home 配置)
- 数据标记系列——图像分割 Curve-GCN
- 有个Linux主机能干什么,什么是Linux主机?
- 多进程爬取Unsplash网站图片
- 嵌入式音视频解决方案 Webrtc vs MetaRTC
- layui表格时间类型显示问题
- ARM CORTEX-M3 内核架构理解归纳
- Python初探(一)
- 10000小时和10000次提交
- sudo与su的区别
- mysql数据库recover_xxx数据库恢复 mariadb数据库误删除恢复 mysql数据库被黑删库恢复
- 在ROIC的框架下精读巴菲特的新作
- BUUCTF-web极客大挑战
- vi/vim简介及使用教程
- MACD金叉不绿选股公式