python基础logging 压缩、根据指定文件夹并进行删除

一、Logging 模块简单介绍

TimedRotatingFileHandler 按日期切分日志，保留多少个backupCount

filename
when='h'
interval=1
backupCount=0
encoding=None
delay=False
utc=False
atTime=None

RotatingFileHandler 按文件大小切分，保留多少文件

filename
mode='a'
maxBytes=0
backupCount=0
encoding=None
delay=False

filename 是输出日志的文件名称前缀，比如说 testServiceLog 这样的就是日志文件名前缀

when 是一个字符串，定义了日志切分的间隔时间单位：参数如下：
"S"：Second 秒
"M"：Minutes 分钟
"H"：Hour 小时
"D"：Days 天
"W"：Week day（0 = Monday）
"midnight"：Roll over at midnight
interval 是间隔时间单位的个数
suffix 避免文件名称重复
backupCount 是保留日志的文件个数
maxBytes 字节个数

二、本人的应用场景

线上日志过去庞大，日志吃掉很多内存
保留14天日志且压缩，删除14天之外的日志
日志文件夹只留300M，大于300M时删日志(神仙需求)
艹，tm我写了两天，网上的都不灵呀，草

三、使用相关模块

zipfile 压缩。经测试579M压缩为1.9M
logging 不解释

base_log.py

import re
import logging
from logging.handlers import TimedRotatingFileHandler
import zipfile
import codecs
import sys
import os
import time
import glob_MIDNIGHT = 24 * 60 * 60  # number of seconds in a dayclass TimedCompressedRotatingFileHandler(TimedRotatingFileHandler):def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False, atTime=None):TimedRotatingFileHandler.__init__(self, filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False, atTime=None)self.when = when.upper()self.backupCount = backupCountself.utc = utcself.atTime = atTimeself.baseFilename = filename# Calculate the real rollover interval, which is just the number of# seconds between rollovers.  Also set the filename suffix used when# a rollover occurs.  Current 'when' events supported:# S - Seconds# M - Minutes# H - Hours# D - Days# midnight - roll over at midnight# W{0-6} - roll over on a certain day; 0 - Monday## Case of the 'when' specifier is not important; lower or upper case# will work.if self.when == 'S':self.interval = 1  # one secondself.suffix = "%Y-%m-%d_%H-%M-%S"self.extMatch = r"^\d{4}-\d{2}-\d{2}_\d{2}-\d{2}-\d{2}(\.\w+)?$"elif self.when == 'M':self.interval = 60  # one minuteself.suffix = "%Y-%m-%d_%H-%M"self.extMatch = r"^\d{4}-\d{2}-\d{2}_\d{2}-\d{2}(\.\w+)?$"elif self.when == 'H':self.interval = 60 * 60  # one hourself.suffix = "%Y-%m-%d_%H"self.extMatch = r"^\d{4}-\d{2}-\d{2}_\d{2}(\.\w+)?$"elif self.when == 'D' or self.when == 'MIDNIGHT':self.interval = 60 * 60 * 24  # one dayself.suffix = "%Y-%m-%d"self.extMatch = r"^\d{4}-\d{2}-\d{2}(\.\w+)?$"elif self.when.startswith('W'):self.interval = 60 * 60 * 24 * 7  # one weekif len(self.when) != 2:raise ValueError("You must specify a day for weekly rollover from 0 to 6 (0 is Monday): %s" % self.when)if self.when[1] < '0' or self.when[1] > '6':raise ValueError("Invalid day specified for weekly rollover: %s" % self.when)self.dayOfWeek = int(self.when[1])self.suffix = "%Y-%m-%d"self.extMatch = r"^\d{4}-\d{2}-\d{2}(\.\w+)?$"else:raise ValueError("Invalid rollover interval specified: %s" % self.when)self.extMatch = re.compile(self.extMatch, re.ASCII)self.interval = self.interval * interval  # multiply by units requested# The following line added because the filename passed in could be a# path object (see Issue #27493), but self.baseFilename will be a stringfilename = self.baseFilenameif os.path.exists(filename):t = os.stat(filename).st_mtimeelse:t = int(time.time())self.rolloverAt = self.computeRollover(t)def computeRollover(self, currentTime):"""Work out the rollover time based on the specified time."""result = currentTime + self.interval# If we are rolling over at midnight or weekly, then the interval is already known.# What we need to figure out is WHEN the next interval is.  In other words,# if you are rolling over at midnight, then your base interval is 1 day,# but you want to start that one day clock at midnight, not now.  So, we# have to fudge the rolloverAt value in order to trigger the first rollover# at the right time.  After that, the regular interval will take care of# the rest.  Note that this code doesn't care about leap seconds. :)# if self.when == 'S' or self.when.startswith('W'):if self.when == 'MIDNIGHT' or self.when.startswith('W'):# This could be done with less code, but I wanted it to be clearif self.utc:t = time.gmtime(currentTime)else:t = time.localtime(currentTime)currentHour = t[3]currentMinute = t[4]currentSecond = t[5]currentDay = t[6]# r is the number of seconds left between now and the next rotationif self.atTime is None:rotate_ts = _MIDNIGHTelse:rotate_ts = ((self.atTime.hour * 60 + self.atTime.minute) * 60 +self.atTime.second)r = rotate_ts - ((currentHour * 60 + currentMinute) * 60 +currentSecond)if r < 0:# Rotate time is before the current time (for example when# self.rotateAt is 13:45 and it now 14:15), rotation is# tomorrow.r += _MIDNIGHTcurrentDay = (currentDay + 1) % 7result = currentTime + r# If we are rolling over on a certain day, add in the number of days until# the next rollover, but offset by 1 since we just calculated the time# until the next day starts.  There are three cases:# Case 1) The day to rollover is today; in this case, do nothing# Case 2) The day to rollover is further in the interval (i.e., today is#         day 2 (Wednesday) and rollover is on day 6 (Sunday).  Days to#         next rollover is simply 6 - 2 - 1, or 3.# Case 3) The day to rollover is behind us in the interval (i.e., today#         is day 5 (Saturday) and rollover is on day 3 (Thursday).#         Days to rollover is 6 - 5 + 3, or 4.  In this case, it's the#         number of days left in the current week (1) plus the number#         of days in the next week until the rollover day (3).# The calculations described in 2) and 3) above need to have a day added.# This is because the above time calculation takes us to midnight on this# day, i.e. the start of the next day.if self.when.startswith('W'):day = currentDay  # 0 is Mondayif day != self.dayOfWeek:if day < self.dayOfWeek:daysToWait = self.dayOfWeek - dayelse:daysToWait = 6 - day + self.dayOfWeek + 1newRolloverAt = result + (daysToWait * (60 * 60 * 24))if not self.utc:dstNow = t[-1]dstAtRollover = time.localtime(newRolloverAt)[-1]if dstNow != dstAtRollover:if not dstNow:  # DST kicks in before next rollover, so we need to deduct an houraddend = -3600else:  # DST bows out before next rollover, so we need to add an houraddend = 3600newRolloverAt += addendresult = newRolloverAtreturn resultdef shouldRollover(self, record):"""Determine if rollover should occur.record is not used, as we are just comparing times, but it is needed sothe method signatures are the same"""t = int(time.time())if t >= self.rolloverAt:return 1return 0def getFilesToDelete(self):"""Determine the files to delete when rolling over.More specific than the earlier method, which just used glob.glob()."""dirName, baseName = os.path.split(self.baseFilename)fileNames = os.listdir(dirName)result = []prefix = baseName + "."plen = len(prefix)file_size = 0for fileName in fileNames:if fileName[:plen] == prefix:suffix = fileName[plen:]filename = os.path.join(dirName, fileName)if not filename.endswith('.zip'):zip_file_name = filename + '.zip'z = zipfile.ZipFile(zip_file_name, 'w', zipfile.ZIP_DEFLATED)z.write(filename, fileName)os.remove(filename)z.close()if self.extMatch.match(suffix):   # 根据文件+. 是否在 suffixresult.append(os.path.join(dirName, fileName))continuetry:file_size += os.stat(filename).st_sizeexcept:continueif file_size >= 1024 * 1024 * 300:result.append(os.path.join(dirName, fileName))if len(result) < self.backupCount:result = []else:result.sort()result = result[:len(result) - self.backupCount]return resultdef doRollover(self):"""do a rollover; in this case, a date/time stamp is appended to the filenamewhen the rollover happens.  However, you want the file to be named for thestart of the interval, not the current time.  If there is a backup count,then we have to get a list of matching filenames, sort them and removethe one with the oldest suffix."""if self.stream:self.stream.close()self.stream = None# get the time that this sequence started at and make it a TimeTuplecurrentTime = int(time.time())dstNow = time.localtime(currentTime)[-1]t = self.rolloverAt - self.intervalif self.utc:timeTuple = time.gmtime(t)else:timeTuple = time.localtime(t)dstThen = timeTuple[-1]if dstNow != dstThen:if dstNow:addend = 3600else:addend = -3600timeTuple = time.localtime(t + addend)dfn = self.rotation_filename(self.baseFilename + "." + time.strftime(self.suffix, timeTuple))if os.path.exists(dfn):os.remove(dfn)self.rotate(self.baseFilename, dfn)if self.backupCount > 0:for s in self.getFilesToDelete():os.remove(s)if not self.delay:self.stream = self._open()newRolloverAt = self.computeRollover(currentTime)while newRolloverAt <= currentTime:newRolloverAt = newRolloverAt + self.interval# If DST changes and midnight or weekly rollover, adjust for this.if (self.when == 'MIDNIGHT' or self.when.startswith('W')) and not self.utc:dstAtRollover = time.localtime(newRolloverAt)[-1]if dstNow != dstAtRollover:if not dstNow:  # DST kicks in before next rollover, so we need to deduct an houraddend = -3600else:  # DST bows out before next rollover, so we need to add an houraddend = 3600newRolloverAt += addendself.rolloverAt = newRolloverAt

注：

切分时请按照 - self.extMatch 字段类型切分
技术控波浪线觉得难看，多继承 BaseRotatingHandler
用法和正常的一样调用即可

`点赞关注，让我看到你`

艹，写了两天，网上的真心不灵。点个赞呗。

python基础logging 压缩、根据指定文件夹并进行删除相关推荐

用c语言批量删除指定文件夹,C语言删除文件夹下所有代码的注释for Mac
#include #include #include char b[1000000]; void delete(char a[]) { int i,j=0; for(i=0;a[i];i++) { i ...
python如何爬取图片到指定文件夹论文_基于Python的图片爬虫程序设计
互联网中包含大量有价值的数据,网络爬虫通过既定规则可以自动地抓取互联网数据并下载至本地存储.研究网络爬虫的工作原理和基于 Python 网络信息爬取技术模块功能,基于 Requests- B ...
python中怎么压缩文件夹_python-对指定文件夹进行压缩
python-对指定文件夹进行压缩目的首先,我试验了一下[1]的效果: import os from zipfile import ZipFile def backupZip(folder): # ...
python解压到指定文件夹_在Python中压缩和解压文件
Python部落(python.freelycode.com)组织翻译,禁止转载,欢迎转发. 如果你已经使用计算机一段时间,你可能遇到了.zip扩展名的文件.它们是可以保存许多其他文件,文件夹和子文件 ...
python批量生成文件夹_python实现批量获取指定文件夹下的所有文件的厂
本文实例讲述了python实现批量获取指定文件夹下的所有文件的厂商信息的方法.分享给大家供大家参考.具体如下: 功能代码如下: import os, string, shutil,re import ...
利用python批量查询企业信息_python实现批量获取指定文件夹下的所有文件的厂商信息...
本文实例讲述了python实现批量获取指定文件夹下的所有文件的厂商信息的方法.分享给大家供大家参考.具体如下: 功能代码如下: import os, string, shutil,re import ...
python找到文件夹下指定文件类型_python 读取指定文件夹中的指定文件类型的文件名...
C# 读取指定文件夹中的全部文件,并按规则生成SQL语句! 本实例的目的在于: 1 了解怎样遍历指定文件夹中的全部文件 2 控制台怎样输入和输出数据代码: using System; using S ...
Excel记录指定文件夹下的所有文件名；批量解压压缩包，处理压缩包套压缩包问题；
10.20 发现一个问题: 当压缩包过大,文件条目超过1048576时,rarfile库中的代码已经不能解决这个压缩包了,需要加装unrar库(from unrar import rarfile)和u ...
chatgpt赋能python：如何在Python中保存文件到指定文件夹
如何在Python中保存文件到指定文件夹如果您正在使用Python编写应用程序或脚本,您可能需要将输出保存到特定的文件夹中. 本文将介绍如何在Python中保存文件到指定文件夹. 使用os模块创建文 ...

python基础logging 压缩、根据指定文件夹并进行删除

一、Logging 模块简单介绍

二、本人的应用场景

三、使用相关模块

注：

`点赞关注，让我看到你`

python基础logging 压缩、根据指定文件夹并进行删除相关推荐

最新文章

热门文章