1.概述

转载：配置ClickHouse分布式DDL记录自动清理

在ClickHouse集群中，我们可以在DDL语句上附加ON CLUSTER <cluster_name>的语法，使得该DDL语句执行一次即可在集群中所有实例上都执行，简单方便。每执行一条分布式DDL，会在配置文件中<distributed_ddl>指定的ZooKeeper路径上写一条执行记录（路径默认为/clickhouse/task_queue/ddl）。如下图所示。

但是，这个队列默认似乎不会自动清理，造成znode不断增长，官方文档中也没有提供对应的参数来控制。考虑到手动删除znode可能会有风险，遂去ClickHouse源码中寻找蛛丝马迹，最终在dbms/src/interpreters/DDLWorker.h里找到如下定义：

/// Cleaning starts after new node event is received if the last cleaning wasn't made sooner than N seconds ago
Int64 cleanup_delay_period = 60; // minute (in seconds)
/// Delete node if its age is greater than that
Int64 task_max_lifetime = 7 * 24 * 60 * 60; // week (in seconds)
/// How many tasks could be in the queue
size_t max_tasks_in_queue = 1000;

cleanup_delay_period：检查DDL记录清理的间隔，单位为秒，默认60秒。
task_max_lifetime：分布式DDL记录可以保留的最大时长，单位为秒，默认保留7天。
max_tasks_in_queue：分布式DDL队列中可以保留的最大记录数，默认为1000条。

将以上参数加入config.xml的<distributed_ddl>一节即可。

<distributed_ddl><!-- Path in ZooKeeper to queue with DDL queries --><path>/clickhouse/task_queue/ddl</path><cleanup_delay_period>60</cleanup_delay_period><task_max_lifetime>86400</task_max_lifetime><max_tasks_in_queue>200</max_tasks_in_queue>
</distributed_ddl>

ClickHouse内部有专门的线程来清理DDL队列，具体逻辑位于DDLWorker.cpp中，不难，代码录如下。

void DDLWorker::runCleanupThread()
{setThreadName("DDLWorkerClnr");LOG_DEBUG(log, "Started DDLWorker cleanup thread");Int64 last_cleanup_time_seconds = 0;while (!stop_flag){try{cleanup_event->wait();if (stop_flag)break;Int64 current_time_seconds = Poco::Timestamp().epochTime();if (last_cleanup_time_seconds && current_time_seconds < last_cleanup_time_seconds + cleanup_delay_period){LOG_TRACE(log, "Too early to clean queue, will do it later.");continue;}auto zookeeper = tryGetZooKeeper();if (zookeeper->expired())continue;cleanupQueue(current_time_seconds, zookeeper);last_cleanup_time_seconds = current_time_seconds;}catch (...){tryLogCurrentException(log, __PRETTY_FUNCTION__);}}
}void DDLWorker::cleanupQueue(Int64 current_time_seconds, const ZooKeeperPtr & zookeeper)
{LOG_DEBUG(log, "Cleaning queue");Strings queue_nodes = zookeeper->getChildren(queue_dir);filterAndSortQueueNodes(queue_nodes);size_t num_outdated_nodes = (queue_nodes.size() > max_tasks_in_queue) ? queue_nodes.size() - max_tasks_in_queue : 0;auto first_non_outdated_node = queue_nodes.begin() + num_outdated_nodes;for (auto it = queue_nodes.cbegin(); it < queue_nodes.cend(); ++it){if (stop_flag)return;String node_name = *it;String node_path = queue_dir + "/" + node_name;String lock_path = node_path + "/lock";Coordination::Stat stat;String dummy;try{/// Already deletedif (!zookeeper->exists(node_path, &stat))continue;/// Delete node if its lifetime is expired (according to task_max_lifetime parameter)constexpr UInt64 zookeeper_time_resolution = 1000;Int64 zookeeper_time_seconds = stat.ctime / zookeeper_time_resolution;bool node_lifetime_is_expired = zookeeper_time_seconds + task_max_lifetime < current_time_seconds;/// If too many nodes in task queue (> max_tasks_in_queue), delete oldest onebool node_is_outside_max_window = it < first_non_outdated_node;if (!node_lifetime_is_expired && !node_is_outside_max_window)continue;/// Skip if there are active nodes (it is weak guard)if (zookeeper->exists(node_path + "/active", &stat) && stat.numChildren > 0){LOG_INFO(log, "Task " << node_name << " should be deleted, but there are active workers. Skipping it.");continue;}/// Usage of the lock is not necessary now (tryRemoveRecursive correctly removes node in a presence of concurrent cleaners)/// But the lock will be required to implement system.distributed_ddl_queue tableauto lock = createSimpleZooKeeperLock(zookeeper, node_path, "lock", host_fqdn_id);if (!lock->tryLock()){LOG_INFO(log, "Task " << node_name << " should be deleted, but it is locked. Skipping it.");continue;}if (node_lifetime_is_expired)LOG_INFO(log, "Lifetime of task " << node_name << " is expired, deleting it");else if (node_is_outside_max_window)LOG_INFO(log, "Task " << node_name << " is outdated, deleting it");/// Deleting{Strings childs = zookeeper->getChildren(node_path);for (const String & child : childs){if (child != "lock")zookeeper->tryRemoveRecursive(node_path + "/" + child);}/// Remove the lock node and its parent atomicallyCoordination::Requests ops;ops.emplace_back(zkutil::makeRemoveRequest(lock_path, -1));ops.emplace_back(zkutil::makeRemoveRequest(node_path, -1));zookeeper->multi(ops);lock->unlockAssumeLockNodeRemovedManually();}}catch (...){LOG_INFO(log, "An error occured while checking and cleaning task " + node_name + " from queue: " + getCurrentExceptionMessage(false));}}
}

【clickhouse】配置ClickHouse分布式DDL记录自动清理相关推荐

故障分析 | ClickHouse 集群分布式 DDL 被阻塞案例一则
作者:任坤现居珠海,先后担任专职 Oracle 和 MySQL DBA,现在主要负责 MySQL.mongoDB 和 Redis 维护工作. 本文来源:原创投稿 *爱可生开源社区出品,原创内容未经授 ...
Linux/Centos Tomcat 配置日志切分以及脚本自动清理
Tomcat是Apache软件基金会(Apache Software Foundation)的Jakarta项目中的一个核心项目,由Apache,Sun和其他一些公司及个人共同开发而成.由于有了Sun ...
【clickhouse】clickhouse There is no DistributedDDL configuration in server config
1.概述转载:clickhouse on cluster设置在平常使用clickhouse中会有一些建表,删表等操作,但是对于整个集群来说每台服务器都需要建表.在ClickHouse集群中,我们可 ...
【clickhouse】 clickhouse配置查询记录query_log
1.概述转载:clickhouse配置查询记录query_log 最近在使用clickhouse中,看到官方文档上可以配置query_log,但是文档上写的配置比较模糊,特此记录一下具体配置方法,以 ...
【clickhouse】clickhouse配置多块磁盘
1.概述转载:clickhouse配置多块磁盘最近让运维同学新搭了一个clickhouse集群,每台服务器都配置了多块磁盘,但是使用的时候还是按照以前的方式是使用的,导致系统盘空间不够.特此记录一 ...
zookeeper3.4.6配置实现自动清理日志
在使用zookeeper过程中,我们知道,会有dataDir和dataLogDir两个目录,分别用于snapshot和事务日志的输出(默认情况下只有dataDir目录,snapshot和事务日志都保存 ...
PostgreSQL归档配置及自动清理归档日志
PostgreSQL归档配置及自动清理归档日志在生产环境中,数据库都需要开启归档模式,那么PG该如何开始归档呢? PG中归档配置涉及几个参数: # - Archiving - #是否开启归档 #ar ...
zookeeper3.4.6配置实现自动清理日志【转】
在使用zookeeper过程中,我们知道,会有dataDir和dataLogDir两个目录,分别用于snapshot和事务日志的输出(默认情况下只有dataDir目录,snapshot和事务日志都保存 ...
【clickhouse】clickhouse 副本与分片分片详解
1.概述转载:[clickhouse]clickhouse 副本与分片分片详解 clickhouse 中每个服务器节点都可以被称为一个 shard(分片). 假设有 N 台服务器,每个服务器上都有 ...

【clickhouse】配置ClickHouse分布式DDL记录自动清理

文章目录

1.概述

【clickhouse】配置ClickHouse分布式DDL记录自动清理相关推荐

最新文章

热门文章