Ceph集群显示XXX daemons have recently crashed警告

问题：

Ceph集群一直显示XXX daemons have recently crashed，而且数目越来越多;

解决方法：

最近有一个或多个Ceph守护进程崩溃，管理员尚未对该崩溃进行存档(确认)。这可能表示软件错误、硬件问题(例如，故障磁盘)或某些其它问题。

系统中所有的崩溃可以通过以下方式列出：

# ceph crash ls
ID                                                               ENTITY     NEW
2020-05-02_00:53:25.028694Z_b29d405c-2512-4b80-916f-46c45c2cd6a9 osd.94
2020-05-02_00:56:33.807897Z_feea566f-f237-42fd-aadf-45a5e8047896 osd.94
2020-05-02_05:41:03.542296Z_21a06b0b-f2bc-42d1-8d50-5c104e150c9e mon.node01
2020-05-02_09:52:51.146773Z_4e637ead-80df-42df-93f0-42c84ab8feb3 osd.19

新的崩溃可以通过以下方式列出：

# ceph crash ls-new

有关特定崩溃的信息可以通过以下方式检查：

# ceph crash info <crash-id>###例如#### ceph crash info 2020-05-02_05:41:03.542296Z_21a06b0b-f2bc-42d1-8d50-5c104e150c9e
{"os_version_id": "18.04", "utsname_release": "4.15.0-55-generic", "os_name": "Ubuntu", "entity_name": "mon.node01", "timestamp": "2020-05-02 05:41:03.542296Z", "process_name": "ceph-mon", "utsname_machine": "x86_64", "utsname_sysname": "Linux", "os_version": "18.04.3 LTS (Bionic Beaver)", "os_id": "ubuntu", "utsname_version": "#60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019", "backtrace": ["(()+0x12890) [0x7f6c9f2f3890]", "(gsignal()+0xc7) [0x7f6c9e3ebe97]", "(abort()+0x141) [0x7f6c9e3ed801]", "(()+0x8c957) [0x7f6c9ede0957]", "(()+0x92ab6) [0x7f6c9ede6ab6]", "(()+0x92af1) [0x7f6c9ede6af1]", "(()+0x92d24) [0x7f6c9ede6d24]", "(()+0x1424b) [0x7f6c9f51424b]", "(tc_new()+0x283) [0x7f6c9f535943]", "(rocksdb::Arena::AllocateNewBlock(unsigned long)+0x6c) [0x55c1aabe88ac]", "(rocksdb::Arena::AllocateFallback(unsigned long, bool)+0x4b) [0x55c1aabe89db]", "(rocksdb::Arena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0x110) [0x55c1aabe8b80]", "(rocksdb::ConcurrentArena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0xd4) [0x55c1aaaff004]", "(()+0x5a3273) [0x55c1aab6b273]", "(()+0x5a32f0) [0x55c1aab6b2f0]", "(rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, rocksdb::Slice const&, rocksdb::Slice const&, bool, rocksdb::MemTablePostProcessInfo*)+0xfc) [0x55c1aaafa5bc]", "(rocksdb::MemTableInserter::PutCFImpl(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::ValueType)+0x1bd) [0x55c1aab609ed]", "(rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&)+0x26) [0x55c1aab615d6]", "(rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0xa19) [0x55c1aab58de9]", "(rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteThread::WriteGroup&, unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, bool, unsigned long, rocksdb::DB*, bool, bool, bool)+0x14b) [0x55c1aab5cecb]", "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x13f6) [0x55c1aaa80f06]", "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x30) [0x55c1aaa82660]", "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x88) [0x55c1aaa342f8]", "(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x8c) [0x55c1aaa34c3c]", "(MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0x76b) [0x55c1aa80a02b]", "(Paxos::begin(ceph::buffer::v14_2_0::list&)+0x562) [0x55c1aa90bca2]", "(Paxos::propose_pending()+0x127) [0x55c1aa90d5f7]", "(Paxos::finish_round()+0x50a) [0x55c1aa90de1a]", "(Paxos::commit_finish()+0x5fc) [0x55c1aa90fd6c]", "(C_Committed::finish(int)+0x34) [0x55c1aa913d54]", "(Context::complete(int)+0x9) [0x55c1aa84a359]", "(MonitorDBStore::C_DoTransaction::finish(int)+0x94) [0x55c1aa913ac4]", "(Context::complete(int)+0x9) [0x55c1aa84a359]", "(Finisher::finisher_thread_entry()+0x17f) [0x7f6ca05227bf]", "(()+0x76db) [0x7f6c9f2e86db]", "(clone()+0x3f) [0x7f6c9e4ce88f]"], "utsname_hostname": "node01", "crash_id": "2020-05-02_05:41:03.542296Z_21a06b0b-f2bc-42d1-8d50-5c104e150c9e", "archived": "2020-05-06 14:13:12.975173", "ceph_version": "14.2.6"
}

可以通过“存档”崩溃（可能是在管理员检查之后）来消除此警告，从而不会生成此警告：

# ceph crash archive <crash-id>

同样，所有新的崩溃都可以通过以下方式存档：

# ceph crash archive-all

通过ceph crash ls仍然可以看到已存档的崩溃，但不是ceph crash ls-new即可看到。

“recent”所指的时间段由选项mgr/crash/warn_recent_interval控制（默认值：两周）。

可以通过以下方式完全禁用这些警告：

# ceph config set mgr mgr/crash/warn_recent_interval 0

参考：

https://docs.ceph.com/docs/master/rados/operations/health-checks/?highlight=backfillfull%20ratio
https://docs.ceph.com/docs/master/mgr/crash/?highlight=crash

Ceph集群显示XXX daemons have recently crashed警告相关推荐

使用cephadm部署单节点ceph集群，后期可扩容（基于官方文档，靠谱，读起来舒服）
目录 ceph各种部署工具比较(来自官方文档的翻译,靠谱!) 材料准备 cephadm使用条件服务器有外网访问能力服务器没有外网访问能力安装cephadm cephadm的功能两种安装方式基 ...
Ceph (2) - 安装Ceph集群方法 2：使用cephadm配置Nautilus版Ceph集群
<OpenShift 4.x HOL教程汇总> 文章目录安装环境说明 Ceph集群节点说明 Ceph集群主机环境说明用cephadm部署Ceph集群准备节点环境设置环境变量设置h ...
Ceph (1) - 安装Ceph集群方法 1：使用ceph-deploy安装Nautilus版Ceph集群
<OpenShift 4.x HOL教程汇总> 文章目录环境说明 Ceph集群节点说明 Ceph集群主机环境说明用ceph-deploy部署Ceph集群准备节点环境设置环境变量设 ...
CentOS 7 搭建 Ceph 集群（nautilus 版本）
推荐阅读 Helm3(K8S 资源对象管理工具)视频教程:https://edu.csdn.net/course/detail/32506 Helm3(K8S 资源对象管理工具)博客专栏:https: ...
Openstack集群-Ceph集群作为存储的部署
1.安装Ceph集群 1.1 设置ceph的yum源 ceph版本:12.2.5 ceph-deploy版本: 2.0.0 注:此处用控制节点部署mod和mgr ,OSD部署在计算节点上 [root@ ...
Ceph集群报错解决方案笔记
文章目录 0 当前Ceph版本和CentOS版本: 1.节点间配置文件内容不一致错误 2.too few PGs per OSD (21 < min 30)警告解决办法:增加pg数 3.集群状 ...
CEPH集群操作入门--配置
https://www.cnblogs.com/luxiaodai/p/10006036.ht CEPH集群操作入门--配置l CEPH集群操作入门--配置阅读目录(Content) 概述配置存 ...
JavaEE企业级实战项目智牛股第四天 NACOS、ceph集群和Netty
交易平台 - Day 4 学习目标目标1:Nacos背景与基本原理目标2:Nacos的使用目标3:Ceph分布式存储原理目标4:Ceph部署与使用目标5:Netty通讯机制第1章 Naco ...
Ceph集群搭建系列（六）：RBD块设备的使用场景、原理分析及其创建
一.前言下图数据出自2018年ceph社区用户调查报告,Ceph RBD接口的使用场景主要是VM和DB数据库. 而使用Ceph RBD的接口方式的用户数据如下,主要是librbd 和 kernel ...

Ceph集群显示XXX daemons have recently crashed警告

Ceph集群显示XXX daemons have recently crashed警告相关推荐

最新文章

热门文章