Redis 6.2的过期删除策略源码分析(定期删除)

都知道redis采用的过期删除策略是定期删除和惰性删除，对于这l两个的解释可以看一下Redis 键的过期删除策略及缓存淘汰策略
下面是根据翻译软件和自己的理解翻译过来的，英文原文也在上面，如果不清楚或者不对可以看一下英文
惰性删除的源码可以看一下我的另一篇文章Redis6.2的过期删除策略源码分析(惰性删除)
刷新频率是多大的可以看另一篇文章Redis6.2 定时删除刷新频率源码

1、定时删除的主要逻辑源码

/* Try to expire a few timed out keys. The algorithm used is adaptive and* will use few CPU cycles if there are few expiring keys, otherwise* it will get more aggressive to avoid that too much memory is used by* keys that can be removed from the keyspace.*  尝试使一些超时的密钥过期。所使用的算法是自适应的，如果过期的密钥很少，它将使用很少的CPU周期，*  否则它将变得更加激进，以避免可以从key空间中删除的key使用过多的内存* Every expire cycle tests multiple databases: the next call will start* again from the next db. No more than CRON_DBS_PER_CALL databases are* tested at every iteration.*每个expire循环测试多个数据库：下一个调用将从下一个db重新开始。在每次迭代中，每个调用测试的数据库不超过CRON_DBS_PER_CALL* The function can perform more or less work, depending on the "type"* argument. It can execute a "fast cycle" or a "slow cycle". The slow* cycle is the main way we collect expired cycles: this happens with* the "server.hz" frequency (usually 10 hertz).*函数可以执行更多或更少的工作，具体取决于“type”参数。它可以执行“快速循环”或“慢速循环”。*慢周期是我们收集过期周期的主要方式：“server.hz”频率（通常为10 hertz）会发生这种情况。* However the slow cycle can exit for timeout, since it used too much time.* For this reason the function is also invoked to perform a fast cycle* at every event loop cycle, in the beforeSleep() function. The fast cycle* will try to perform less work, but will do it much more often.*但是，慢循环会因超时而退出，因为它占用了太多的时间。因此，在beforeSleep（）函数中，*还会调用该函数在每个事件循环中执行快循环。快速循环将尝试执行较少的工作，但会做得更多//这里就看出一般定时删除是慢循环，只有在上次慢循环超时，才会执行快循环，也可以看上面redi刷新源码的博客，里面有源码证据* The following are the details of the two expire cycles and their stop* conditions:*以下是两个过期周期及其停止条件的详细信息:* If type is ACTIVE_EXPIRE_CYCLE_FAST the function will try to run a* "fast" expire cycle that takes no longer than ACTIVE_EXPIRE_CYCLE_FAST_DURATION* microseconds, and is not repeated again before the same amount of time.* The cycle will also refuse to run at all if the latest slow cycle did not* terminate because of a time limit condition.* 如果类型为ACTIVE_EXPIRE_CYCLE_FAST，则函数将尝试运行“快速”过期循环，* 该循环所用时间不超过ACTIVE_EXPIRE_CYCLE_FAST_DURATION微秒，并且在相同时间之前不会再次重复。* 如果最新的慢速循环由于时间限制条件而未终止，则该循环也将拒绝运行* If type is ACTIVE_EXPIRE_CYCLE_SLOW, that normal expire cycle is* executed, where the time limit is a percentage of the REDIS_HZ period* as specified by the ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC define. In the* fast cycle, the check of every database is interrupted once the number* of already expired keys in the database is estimated to be lower than* a given percentage, in order to avoid doing too much work to gain too* little memory.*如果类型为ACTIVE_EXPIRE_CYCLE_SLOW，*则执行正常的过期周期，其中时间限制是由ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC定义指定的REDIS_HZ个周期的百分比。*在快速循环中，一旦数据库中已过期的key数估计低于给定的百分比，就会中断对每个数据库的检查，以避免做太多的工作而获得太少的内存* The configured expire "effort" will modify the baseline parameters in* order to do more work in both the fast and slow expire cycles.*/#define ACTIVE_EXPIRE_CYCLE_KEYS_PER_LOOP 20 /* Keys for each DB loop. 每个循环随机抽取的数量默认为20*/
//微秒，是指在选择删除这一过程中，初始值时长不能超过1ms，不是指activeExpireCycle方法执行的时长
#define ACTIVE_EXPIRE_CYCLE_FAST_DURATION 1000 /* Microseconds. */
#define ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC 25 /* Max % of CPU to use. 要使用的最大CPU百分比，在1s中占比25%用于定时删除*/
#define ACTIVE_EXPIRE_CYCLE_ACCEPTABLE_STALE 10 /* % of stale keys after whichwe do extra efforts.每个周期可接受允许有多少个过期的key，每一个db中随机的key如果过期数比这个大，那么此db重新随机                                                */
//定期刪除策略
void activeExpireCycle(int type) {/* Adjust the running parameters according to the configured expire* effort. The default effort is 1, and the maximum configurable effort* is 10.*根据配置的超时工作调整运行参数。默认工作量为1，最大可配置工作量为10*active-expire-effort 1 取值的范围是1到10，redis清理过期的key是需要消耗内存的，*所以可以设置的过大可以让过期数据清理的更干净,但是同时让redis牺牲更多的CPU取清理更多的过期key，需要权衡*/unsigned longeffort = server.active_expire_effort-1, /* Rescale from 0 to 9. *///如果是1，则就是默认的20，否则多加1相当于随机抽取的数加5，最大抽取65config_keys_per_loop = ACTIVE_EXPIRE_CYCLE_KEYS_PER_LOOP +ACTIVE_EXPIRE_CYCLE_KEYS_PER_LOOP/4*effort,//删除时长用时默认的1ms加上 effort*250μs(微秒) // 可以看一下上面的Redis6.2 定时删除刷新频率源码，里面有服务器每秒多少次的频率计算方法，并且有每次频率调用做了哪些工作的流程图，定时删除只是其中一部分config_cycle_fast_duration = ACTIVE_EXPIRE_CYCLE_FAST_DURATION +ACTIVE_EXPIRE_CYCLE_FAST_DURATION/4*effort,//占比CPU时间，默认是25%，最大43%，如果是100%，那除了定时删除其他的工作都做不了了，所以要做限制config_cycle_slow_time_perc = ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC +2*effort,//允许超时key存在的数量，在一个库里config_cycle_acceptable_stale = ACTIVE_EXPIRE_CYCLE_ACCEPTABLE_STALE-effort;/* This function has some global state in order to continue the work* incrementally across calls.*此函数具有一些全局状态，以便跨调用增量地继续工作*/static unsigned int current_db = 0; /* Next DB to test. 下一个要测试定期删除的DB*/static int timelimit_exit = 0;      /* Time limit hit in previous call? 当这个值不为0时，说明current_db抽取随机key并清理过期key的 没有走完，而是到时间了，就退出了*/static long long last_fast_cycle = 0; /* When last fast cycle ran.最后一个快速循环 */int j, iteration = 0;int dbs_per_call = CRON_DBS_PER_CALL;long long start = ustime(), timelimit, elapsed;/* When clients are paused the dataset should be static not just from the* POV of clients not being able to write, but also from the POV of* expires and evictions of keys not being performed.*当客户端被暂停时，数据集应该是静态的，不仅来自客户端不能写入的POV，还来自过期和不执行的key收回的POV*/if (checkClientPauseTimeoutAndReturnIfPaused()) return;if (type == ACTIVE_EXPIRE_CYCLE_FAST) {/* Don't start a fast cycle if the previous cycle did not exit* for time limit, unless the percentage of estimated stale keys is* too high. Also never repeat a fast cycle for the same period* as the fast cycle total duration itself.*如果前一个周期没有在时间限制内退出，则不要启动快速周期，除非估计的过期key的百分比太高。*也不要在与快速循环总持续时间相同的时间内重复快速循环*/if (!timelimit_exit &&server.stat_expired_stale_perc < config_cycle_acceptable_stale)return;if (start < last_fast_cycle + (long long)config_cycle_fast_duration*2)return;last_fast_cycle = start;}/* We usually should test CRON_DBS_PER_CALL per iteration, with* two exceptions:*我们通常应该在每次迭代的每次调用中测试CRON_DBS_PER_CALL，但有两个例外* 1) Don't test more DBs than we have.不要测试比我们更多的数据库* 2) If last time we hit the time limit, we want to scan all DBs* in this iteration, as there is work to do in some DB and we don't want* expired keys to use memory for too much time.*如果上次我们达到时间限制，我们要扫描所有的数据库*在这个迭代中，由于在一些数据库中有工作要做，我们不希望过期的密钥占用内存太长时间*/if (dbs_per_call > server.dbnum || timelimit_exit)dbs_per_call = server.dbnum;/* We can use at max 'config_cycle_slow_time_perc' percentage of CPU* time per iteration. Since this function gets called with a frequency of* server.hz times per second, the following is the max amount of* microseconds we can spend in this function.*每次迭代最多可以使用config_cycle_slow_time_perc%的CPU时间。*由于该函数以每秒server.hz次的频率被调用，因此下面是我们在该函数中可以花费的最大微秒数**///第一个timelimit是类型ACTIVE_EXPIRE_CYCLE_SLOW的删除某个key最长时间，如果没有删除则会停止//config_cycle_slow_time_perc*1000000/server.hz代表此次定时删除的时长，而且上面config_cycle_fast_duration//也限制了删除此次定时删除key所用的总时间，所以这再 /100 ,限制删除某一个key最大耗时,//这样就不会因为某个键值对过大，删除时间占用时间过长，而导致后面的过期key不能及时删除，所以官方文档上也建议不要设置过大的键值对，如果真过大，还有惰性删除兜底timelimit = config_cycle_slow_time_perc*1000000/server.hz/100;timelimit_exit = 0;if (timelimit <= 0) timelimit = 1;//类型是ACTIVE_EXPIRE_CYCLE_FAST，代表快速循环，也代表上次慢循环超时，删除方法执行的时间等于config_cycle_fast_durationif (type == ACTIVE_EXPIRE_CYCLE_FAST)timelimit = config_cycle_fast_duration; /* in microseconds. *//* Accumulate some global stats as we expire keys, to have some idea* about the number of keys that are already logically expired, but still* existing inside the database.*在密钥过期时积累一些全局统计信息，以便了解逻辑上已经过期但仍存在于数据库中的密钥的数量*/long total_sampled = 0;long total_expired = 0;//这里就循环遍历数据库了for (j = 0; j < dbs_per_call && timelimit_exit == 0; j++) {/* Expired and checked in a single loop. */unsigned long expired, sampled;//指向要处理的数据库redisDb *db = server.db+(current_db % server.dbnum);/* Increment the DB now so we are sure if we run out of time* in the current DB we'll restart from the next. This allows to* distribute the time evenly across DBs.*现在增加DB，这样我们就可以确定时间是否用完了* 在当前数据库中，我们将从下一个数据库重新启动。这允许* 将时间平均分配到DBs*/current_db++;/* Continue to expire if at the end of the cycle there are still* a big percentage of keys to expire, compared to the number of keys* we scanned. The percentage, stored in config_cycle_acceptable_stale* is not fixed, but depends on the Redis configured "expire effort".**如果在循环结束时，与我们扫描的密钥数相比，仍有很大比例的密钥要过期，则继续过期。*存储在config_cycle_acceptable_stale中的百分比不是固定的，而是取决于Redis配置的“expire efforce”*/do {unsigned long num, slots;long long now, ttl_sum;int ttl_samples;iteration++;/* If there is nothing to expire try next DB ASAP.*如果没有要过期的内容，请尽快尝试下一个DB*/if ((num = dictSize(db->expires)) == 0) {db->avg_ttl = 0;break;}slots = dictSlots(db->expires);now = mstime();/* When there are less than 1% filled slots, sampling the key* space is expensive, so stop here waiting for better times...* The dictionary will be resized asap.*当有不到1%的插槽被填满时，对key空间进行采样是非常昂贵的，所以请停止在这里等待更好的时机…字典将尽快调整大小*/if (slots > DICT_HT_INITIAL_SIZE &&(num*100/slots < 1)) break;/* The main collection cycle. Sample random keys among keys* with an expire set, checking for expired ones.*主要收集周期。在具有过期集的key中随机抽取key，检查过期的密钥*/expired = 0;sampled = 0;ttl_sum = 0;ttl_samples = 0;if (num > config_keys_per_loop)num = config_keys_per_loop;/* Here we access the low level representation of the hash table* for speed concerns: this makes this code coupled with dict.c,* but it hardly changed in ten years.*在这里，出于速度考虑，我们访问了哈希表的低级表示：这使得此代码与dict.c结合在一起，但十年来几乎没有变化** Note that certain places of the hash table may be empty,* so we want also a stop condition about the number of* buckets that we scanned. However scanning for free buckets* is very fast: we are in the cache line scanning a sequential* array of NULL pointers, so we can scan a lot more buckets* than keys in the same time.*请注意，哈希表的某些位置可能为空，因此我们还需要一个关于扫描的桶数的停止条件。*然而，扫描空闲的bucket非常快：我们在缓存线中扫描空指针的序列数组，因此我们可以在同一时间扫描比键多得多的bucket*/long max_buckets = num*20;long checked_buckets = 0;while (sampled < num && checked_buckets < max_buckets) {for (int table = 0; table < 2; table++) {if (table == 1 && !dictIsRehashing(db->expires)) break;unsigned long idx = db->expires_cursor;idx &= db->expires->ht[table].sizemask;dictEntry *de = db->expires->ht[table].table[idx];long long ttl;/* Scan the current bucket of the current table.扫描当前表的当前bucket*/checked_buckets++;while(de) {/* Get the next entry now since this entry may get* deleted.*现在获取下一个条目，因为此条目可能会被删除*/dictEntry *e = de;de = de->next;返回键的过期时间ttl = dictGetSignedIntegerVal(e)-now;//判断是否过期if (activeExpireCycleTryExpire(db,e,now)) expired++;if (ttl > 0) {/* We want the average TTL of keys yet* not expired. 我们要的是尚未过期的密钥的平均TTL*/ttl_sum += ttl;ttl_samples++;}sampled++;}}db->expires_cursor++;}total_expired += expired;total_sampled += sampled;/* Update the average TTL stats for this database. 更新此数据库的平均TTL统计信息。*/if (ttl_samples) {long long avg_ttl = ttl_sum/ttl_samples;/* Do a simple running average with a few samples.* We just use the current estimate with a weight of 2%* and the previous estimate with a weight of 98%.*用几个样本做一个简单的运行平均。我们只使用当前权重为2%的估计值和上一个权重为98%的估计值。*/if (db->avg_ttl == 0) db->avg_ttl = avg_ttl;db->avg_ttl = (db->avg_ttl/50)*49 + (avg_ttl/50);}/* We can't block forever here even if there are many keys to* expire. So after a given amount of milliseconds return to the* caller waiting for the other active expire cycle.*即使有很多密钥要过期，我们也不能永远封锁这里。因此，在给定的毫秒数之后，返回到调用方，等待另一个活动的expire循环*/if ((iteration & 0xf) == 0) { /* check once every 16 iterations.每16次迭代检查一次*/elapsed = ustime()-start;if (elapsed > timelimit) {timelimit_exit = 1;server.stat_expired_time_cap_reached_count++;break;}}/* We don't repeat the cycle for the current database if there are* an acceptable amount of stale keys (logically expired but yet* not reclaimed).*如果存在可接受数量的过时密钥（逻辑上已过期但尚未回收），则不会对当前数据库重复此循环*/} while (sampled == 0 ||(expired*100/sampled) > config_cycle_acceptable_stale);}elapsed = ustime()-start;server.stat_expire_cycle_time_used += elapsed;latencyAddSampleIfNeeded("expire-cycle",elapsed/1000);/* Update our estimate of keys existing but yet to be expired.* Running average with this sample accounting for 5%.*更新我们对现有但尚未过期的密钥的估计。该样本的运行平均值占5%。这个很有用，可以及时跳过某次定时删除*/double current_perc;if (total_sampled) {current_perc = (double)total_expired/total_sampled;} elsecurrent_perc = 0;server.stat_expired_stale_perc = (current_perc*0.05)+(server.stat_expired_stale_perc*0.95);
}

2、activeExpireCycleTryExpire(判断过期的方法)

/* Helper function for the activeExpireCycle() function.* This function will try to expire the key that is stored in the hash table* entry 'de' of the 'expires' hash table of a Redis database.** If the key is found to be expired, it is removed from the database and* 1 is returned. Otherwise no operation is performed and 0 is returned.*如果发现key已过期，则会将其从数据库中删除并返回1。否则不执行任何操作并返回0* When a key is expired, server.stat_expiredkeys is incremented.*key过期时，server.stat_expiredkeys将递增。* The parameter 'now' is the current time in milliseconds as is passed* to the function to avoid too many gettimeofday() syscalls.*参数'now'是传递给函数的当前时间（以毫秒为单位），以避免过多的gettimeofday（）系统调用*/
int activeExpireCycleTryExpire(redisDb *db, dictEntry *de, long long now) {long long t = dictGetSignedIntegerVal(de);mstime_t expire_latency;if (now > t) {sds key = dictGetKey(de);robj *keyobj = createStringObject(key,sdslen(key));propagateExpire(db,keyobj,server.lazyfree_lazy_expire);latencyStartMonitor(expire_latency);if (server.lazyfree_lazy_expire)dbAsyncDelete(db,keyobj);elsedbSyncDelete(db,keyobj);latencyEndMonitor(expire_latency);latencyAddSampleIfNeeded("expire-del",expire_latency);notifyKeyspaceEvent(NOTIFY_EXPIRED,"expired",keyobj,db->id);signalModifiedKey(NULL, db, keyobj);decrRefCount(keyobj);server.stat_expiredkeys++;return 1;} else {return 0;}
}

Redis 6.2的过期删除策略源码分析(定期删除)相关推荐

LinkedList中查询(contains)和删除(remove)源码分析
一.contains源码分析本文分析双向链表LinkedList的查询操作源码实现.jdk中源程序中,LinkedList的查询操作,通过contains(Object o)函数实现.具体见下面两部 ...
redis evict.c内存淘汰机制的源码分析
众所周知,redis是一个内存数据库,所有的键值对都是存储在内存中.当数据变多之后,由于内存有限就要淘汰一些键值对,使得内存有足够的空间来保存新的键值对.在redis中,通过设置server.max ...
系统性详解Redis操作Hash类型数据（带源码分析及测试结果）
1 缘起系统讲解Redis的Hash类型CURD, 帮助学习者系统且准确学习Hash数据操作, 逐步养成测试的好习惯, 本文较长,Hash的操作比较多,请耐心看, 既可以集中时间看,亦可以碎片时间学 ...
进程句柄表初始化，扩展，插入删除句柄源码分析
一.为什么要有句柄句柄是一个8字节的结构体,用途是指向内核对象.3环程序无法通过地址直接访问内核对象,所以需要用句柄来间接访问. 本文重点介绍句柄表,句柄本身则留到下一篇博客介绍.但因为接下来介绍句 ...
redis源码分析 ppt_【Redis】redis各类型数据结构和底层实现源码分析
一.简介和应用 Redis是一个由ANSI C语言编写,性能优秀.支持网络.可持久化的K-K内存数据库,并提供多种语言的API.它常用的类型主要是 String.List.Hash.Set.ZSet ...
rm删除命令源码分析
为什么看? 想要在删除文件前,先覆盖文件内容,防止他人恢复文件,从而得到文件原内容:并且需要支持rm命令原本的参数选项: NAME rm - remove files or directories S ...
rocketmq 消息删除_RocketMQ源码分析之文件过期删除机制
1.由于RocketMQ操作CommitLog.ConsumeQueue文件,都是基于内存映射方法并在启动的时候,会加载commitlog.ConsumeQueue目录下的所有文件,为了避免内存与磁盘 ...
redis之十一（Redis 过期策略与源码分析）
在 Redis 中我们可以给一些元素设置过期时间,那当它过期之后 Redis 是如何处理这些过期键呢? 过期键执行流程 Redis 之所以能知道那些键值过期,是因为在 Redis 中维护了一个字典,存 ...
Quartz的Scheduler初始化源码分析
2019独角兽企业重金招聘Python工程师标准>>> Quartz的使用:http://donald-draper.iteye.com/blog/2321886 Quartz的S ...
Redis 过期策略与源码分析
在 Redis 中我们可以给一些元素设置过期时间,那当它过期之后 Redis 是如何处理这些过期键呢? 过期键执行流程 Redis 之所以能知道那些键值过期,是因为在 Redis 中维护了一个字典,存 ...

Redis 6.2的过期删除策略源码分析(定期删除)

1、定时删除的主要逻辑源码

2、activeExpireCycleTryExpire(判断过期的方法)

Redis 6.2的过期删除策略源码分析(定期删除)相关推荐

最新文章

热门文章