【ceph相关】ceph常见问题处理

一、pg相关

1、xx objects unfound

- 问题描述：

dmesg查看磁盘发现读写异常，部分对象损坏（处于objects nofound状态），集群处于ERR状态

root@node1101:~# ceph health detail
HEALTH_ERR noscrub,nodeep-scrub flag(s) set; 13/409798 objects unfound(0.003%)；17 stuck requests are blocked > 4096 sec. Implicated osds 38
OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set
OBJECT_UNFOUND 13/409798 objects unfound (0.003%)pg 5.309 has 1 unfound objectspg 5.2da has 1 unfound objectspg 5.2c9 has 1 unfound objectspg 5.1e2 has 1 unfound objectspg 5.6a has 1 unfound objectspg 5.120 has 1 unfound objectspg 5.148 has 1 unfound objectspg 5.14b has 1 unfound objectspg 5.160 has 1 unfound objectspg 5.35b has 1 unfound objectspg 5.39c has 1 unfound objectspg 5.3ad has 1 unfound objects
REQUEST_STUCK 17 stuck requests are blocked > 4096 sec. Implicated osds 3817 ops are blocked > 67108.9 secosd.38 has stuck requests > 67108.9 sec

- 处理措施：

将unfound pg强制删除，参考命令：ceph pg {pgid} mark_unfound_lost delete
注：如需批量删除unfound pg，则参考命令如下

for i in `ceph health detail | grep pg | awk '{print $2}'`;do ceph pg $i mark_unfound_lost delete;done

2、Reduced data availability: xx pgs inactive

- 问题描述：

磁盘出现读写异常，osd无法启动，强制替换故障盘为新盘加入到集群，出现pgs inactive（unkown）

root@node1106:~# ceph -scluster:id:     7f1aa879-afbb-4b19-9bc3-8f55c8ecbbb4health: HEALTH_WARN4 clients failing to respond to capability release3 MDSs report slow metadata IOs1 MDSs report slow requests3 MDSs behind on trimmingnoscrub,nodeep-scrub flag(s) setReduced data availability: 25 pgs inactive6187 slow requests are blocked > 32 sec. Implicated osds 41services:mon: 3 daemons, quorum node1101,node1102,node1103mgr: node1103(active), standbys: node1102, node1101mds: ceph-3/3/3 up  {0=node1103=up:active,1=node1102=up:active,2=node1104=up:active}, 2 up:standbyosd: 48 osds: 48 up, 48 inflags noscrub,nodeep-scrubdata:pools:   6 pools, 2888 pgsobjects: 474.95k objects, 94.5GiBusage:   267GiB used, 202TiB / 202TiB availpgs:     0.866% pgs unknown2863 active+clean25   unknownroot@node1101:~# ceph pg dump_stuck inactive
ok
PG_STAT STATE   UP UP_PRIMARY ACTING ACTING_PRIMARY
1.166   unknown []         -1     []             -1
1.163   unknown []         -1     []             -1
1.26f   unknown []         -1     []             -1
1.228   unknown []         -1     []             -1
1.213   unknown []         -1     []             -1
1.12f   unknown []         -1     []             -1
1.276   unknown []         -1     []             -1
1.264   unknown []         -1     []             -1
1.32a   unknown []         -1     []             -1
1.151   unknown []         -1     []             -1
1.20d   unknown []         -1     []             -1
1.298   unknown []         -1     []             -1
1.306   unknown []         -1     []             -1
1.2f7   unknown []         -1     []             -1
1.2c8   unknown []         -1     []             -1
1.223   unknown []         -1     []             -1
1.204   unknown []         -1     []             -1
1.374   unknown []         -1     []             -1
1.b5    unknown []         -1     []             -1
1.b6    unknown []         -1     []             -1
1.2b    unknown []         -1     []             -1
1.9f    unknown []         -1     []             -1
1.2ac   unknown []         -1     []             -1
1.78    unknown []         -1     []             -1
1.1c3   unknown []         -1     []             -1
1.1a    unknown []         -1     []             -1
1.d9    unknown []         -1     []             -1

- 处理措施：

强制创建unkown pg，参考命令：ceph osd force-create-pg {pgid}
注：如需批量创建unkown pg，则参考命令如下：

for i in `ceph pg dump_stuck inactive | awk '{if (NR>2){print $1}}'`;do ceph osd force-create-pg $i;done

二、OSD相关

1、osd端口与其他服务固定绑定端口冲突

- 问题描述：

osd先行启动，占用其他服务固定绑定端口，导致其他服务绑定端口失败，无法启动

- 处理措施：

考虑到其他服务涉及组件太多，担心修改不完全导致其他问题发生，尝试修改osd启动端口范围为其他服务之外

修改osd作为服务端的启动端口范围
ceph可通过ms_bind_port_min和ms_bind_port_max参数限制osd和mds守护进程使用端口范围，默认范围为6800：7300
设置端口使用范围为9600:10000，追加参数设置至/etc/ceph/ceph.conf文件中[global]字段内

[root@node111 ~]# cat /etc/ceph/ceph.conf | grep ms_bind_port
ms_bind_port_min = 9600
ms_bind_port_max = 10000
[root@node111 ~]#
[root@node111 ~]# ceph --show-config | grep ms_bind_port
ms_bind_port_max = 10000
ms_bind_port_min = 9600

修改osd作为客户端的启动端口范围
osd作为客户端的启动端口为随机分配的，可通过内核去限制随机端口分配范围
默认端口范围为1024：65000，修改端口范围为7500：65000

--默认端口范围为1024：65000
[root@node111 ~]# cat /proc/sys/net/ipv4/ip_local_port_range
1024    65000
--修改范围为7500：65000
[root@node111 ~]# sed -i 's/net.ipv4.ip_local_port_range=1024 65000/net.ipv4.ip_local_port_range=7500 65000/g' /etc/sysctl.conf
[root@node111 ~]# sysctl -p

2、磁盘热插拔，osd无法上线

- 问题描述：

使用bluestore部署ceph集群，对osd所在磁盘进行热插拔操作，当重新插回之后，osd对应lvm不能自动恢复，导致osd无法上线成功

- 处理措施：

查找故障osd对应uuid：

ceph osd dump | grep {osd-id} | awk '{print $NF}'

参考示例：查找osd.0对应uuid
[root@node127 ~]# ceph osd dump | grep osd.0 | awk '{print $NF}'
57377809-fba4-4389-8703-f9603f16e60d

查找故障osd对应lvm路径：

ls /dev/mapper/ | grep `ceph osd dump | grep {osd-id} | awk '{print $NF}' | sed 's/-/--/g'`

参考示例：通过uuid查找对应lvm路径
注：由于lvm路径对uuid做了处理，需要sed 's/-/--/g'`将-替换为--
[root@node127 ~]# ls /dev/mapper/ | grep `ceph osd dump | grep osd.0 | awk '{print $NF}' | sed 's/-/--/g'`
ceph--3182c42e--f8d8--4c13--ad92--987463d626c8-osd--block--57377809--fba4--4389--8703--f9603f16e60d

删除故障osd对应lvm路径

dmsetup remove /dev/mapper/{lvm-path}

参考示例：删除故障osd对应lvm路径
[root@node127 ~]# dmsetup remove /dev/mapper/ceph--3182c42e--f8d8--4c13--ad92--987463d626c8-osd--block--57377809--fba4--4389--8703--f9603f16e60d

重新激活所有lvm卷组
注：此时可以查看到对应故障osd的lvm信息

vgchange -ay

重新启动osd使得osd上线

systemctl start ceph-volume@lvm-{osd-id}-{osd-uuid}

三、集群相关

1、clock skew detected on mon.node2

- 问题描述：

集群mon节点时间偏差过大，出现clock skew detected on mon.node2 告警信息

- 处理措施：

1、检查集群mon节点时间偏差，使用chronyd时间进行集群时间同步
2、调大集群参数阈值，调整mon_clock_drift_allowed 参数值为2，调整mon_clock_drift_warn_backoff 参数值为30

sed -i "2 amon_clock_drift_allowed = 2" /etc/ceph/ceph.conf
sed -i "3 amon_clock_drift_warn_backoff = 30" /etc/ceph/ceph.conf
ceph tell mon.* injectargs '--mon_clock_drift_allowed 2'
ceph tell mon.* injectargs '--mon_clock_drift_warn_backoff 30'

注：相关参数说明如下：

[root@node147 ~]# ceph --show-config | grep mon_clock_drift
mon_clock_drift_allowed = 0.050000
--当mon节点之间时间偏移超过0.05秒，则不正常
mon_clock_drift_warn_backoff = 5.000000
--当出现5次偏移，则上报告警