ceph osd 磁盘损坏处理

环境：cenot7，ceph luminious，服务器为Proliant DL380 Gen9 安装了 hp ilo4

（一）从 ceph 删除该 osd

1、登陆 ceph mon 节点，查看坏掉的 osd

2、mon 上执行 out osd.x

ceph osd out osd.x

3、从 crush map 中删除 osd.x，防止它再接受数据

ceph osd crush remove osd.x

ceph auth del osd.x

ceph osd rm osd.x

[root@bakmtr01 ~]# ceph -scluster:id:     0e38e7c6-a704-4132-b0e3-76b87f18d8fahealth: HEALTH_OKservices:mon: 3 daemons, quorum bakmtr01,bakmtr02,bakmtr03mgr: bakmtr03(active), standbys: bakmtr01, bakmtr02osd: 99 osds: 99 up, 99 inrgw: 3 daemons active
...

确认已经删除

ceph osd destroy osd.x --yes-i-really-mean-it

这些步骤相当于

ceph osd purge osd.x --yes-i-really-mean-it

4、osd 节点执行 umount /var/lib/ceph/osd/ceph-x

umount /var/lib/ceph/osd/ceph-x

5、查找 osd.x 对应的 device，lv、pv、vg

[root@bakcmp31 ~]# ceph-volume inventory /dev/sdt====== Device report /dev/sdt ======available                 Falserejected reasons          lockedpath                      /dev/sdtscheduler mode            deadlinerotational                1vendor                    HPhuman readable size       1.64 TBsas address               removable                 0model                     LOGICAL VOLUMEro                        0--- Logical Volume ---cluster name              cephname                      osd-data-3f2e912c-f327-4221-b350-a4b3de4376b6osd id                    1cluster fsid              0e38e7c6-a704-4132-b0e3-76b87f18d8fatype                      blockblock uuid                V8RGFc-omqm-B1E2-mKz1-TXfl-2lK3-CF2d0Losd fsid                  2f1aaa8a-f50d-4335-a812-5dd86e8042a3

也可以查看所有磁盘对应的 osd_id

ceph-volume inventory --format json-pretty

还可以通过 ceph-volume lvm list

[root@bakcmp31 ~]# ceph-volume lvm list | grep -A 16 "osd.1 "
====== osd.1 =======[block]    /dev/ceph-757f4a80-60e2-425b-a8fd-629a735a5acd/osd-data-3f2e912c-f327-4221-b350-a4b3de4376b6type                      blockosd id                    1cluster fsid              0e38e7c6-a704-4132-b0e3-76b87f18d8facluster name              cephosd fsid                  2f1aaa8a-f50d-4335-a812-5dd86e8042a3encrypted                 0cephx lockbox secret      block uuid                V8RGFc-omqm-B1E2-mKz1-TXfl-2lK3-CF2d0Lblock device              /dev/ceph-757f4a80-60e2-425b-a8fd-629a735a5acd/osd-data-3f2e912c-f327-4221-b350-a4b3de4376b6vdo                       0crush device class        Nonedevices                   /dev/sdt

6、查看 osd1 对应的 lv、vg

[root@bakcmp31 ~]# ceph-volume lvm list /dev/ceph-757f4a80-60e2-425b-a8fd-629a735a5acd/osd-data-3f2e912c-f327-4221-b350-a4b3de4376b6
====== osd.1 =======[block]    /dev/ceph-757f4a80-60e2-425b-a8fd-629a735a5acd/osd-data-3f2e912c-f327-4221-b350-a4b3de4376b6
...block device              /dev/ceph-757f4a80-60e2-425b-a8fd-629a735a5acd/osd-data-3f2e912c-f327-4221-b350-a4b3de4376b6
...devices                   /dev/sdt

7、删除 lv 、vg

[root@bakcmp31 ~]# lvremove ceph-757f4a80-60e2-425b-a8fd-629a735a5acd/osd-data-3f2e912c-f327-4221-b350-a4b3de4376b6
Do you really want to remove active logical volume ceph-757f4a80-60e2-425b-a8fd-629a735a5acd/osd-data-3f2e912c-f327-4221-b350-a4b3de4376b6? [y/n]: yLogical volume "osd-data-3f2e912c-f327-4221-b350-a4b3de4376b6" successfully removed

[root@bakcmp31 ~]# vgremove ceph-757f4a80-60e2-425b-a8fd-629a735a5acdVolume group "ceph-757f4a80-60e2-425b-a8fd-629a735a5acd" successfully removed

8、删除 pv

找到 osd 对应的 lvs，删除，没有报错的话，删除对应的 vg、pv

[root@cmp17 ~]# lvremove /dev/ceph-a090a75a-bd1c-4c41-9505-55e9919c54c7/osd-data-c9e93977-654c-48ff-9c94-f92ffd1def69WARNING: Device for PV Eeuf0S-XkKi-UwwB-35C8-Eozs-YNFR-0CUSw8 not found or rejected by a filter.Couldn't find device with uuid Eeuf0S-XkKi-UwwB-35C8-Eozs-YNFR-0CUSw8.
Do you really want to remove active logical volume ceph-a090a75a-bd1c-4c41-9505-55e9919c54c7/osd-data-c9e93977-654c-48ff-9c94-f92ffd1def69? [y/n]: yAborting vg_write: No metadata areas to write to!

报错，刷新 pv

pvscan --cache

手工删除pv是不行的，这里需要用到一个pvscan --cache命令去刷新缓存，之后再看pv、vg、lv通通都被清理掉了 (感觉不出来有啥变化)

[root@bakcmp31 ~]# pvscan --cache
[root@bakcmp31 ~]# pvsPV         VG                                        Fmt  Attr PSize  PFree /dev/sdb   ceph-7db7008f-5eea-40b6-b289-6ae7d8a8ed91 lvm2 a--  <1.64t     0
.../dev/sdt                                             lvm2 ---  <1.64t <1.64t/dev/sdu   ceph-02a02c1e-018b-4ea0-8c08-a4fb58547818 lvm2 a--  <1.64t     0

9、依旧可能遇到删除不彻底的问题，如何操作呢？

查看操作

[root@cmp39 ~]# dmsetup ls

删除操作

[root@cmp39 ~]# dmsetup remove ***

（二）更换硬盘后，重建 raid0

根据 ilo 查看对应物理 drive，记录1I:1:20

或者使用 hpssacli 查看 pd 对应的 ld

[root@cmp17 ~]# hpssacli ctrl slot=0 show config detail

安装 hpssacli，从 hp 官网下载 https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_04bffb688a73438598fef81ddd

rpm -ivh hpssacli-2.40-13.0.x86_64.rpm

hpssacli 常用命令

hpssacli ctrl slot=0 pd all show
hpssacli ctrl slot=0 pd all show status
hpssacli ctrl slot=0 ld all show
hpssacli ctrl slot=0 ld all show status

物理 drive 都没问题

[root@cmp17 ~]# hpssacli ctrl slot=0 pd all show status

逻辑 drive 19 error

[root@cmp17 ~]# hpssacli ctrl slot=0 ld all show status
...logicaldrive 19 (1.6 TB, 0): Failedlogicaldrive 20 (1.6 TB, 0): OKlogicaldrive 21 (1.6 TB, 0): O

查看逻辑 drive 19 对应的设备名，没有显示，说明还没有做 raid

[root@cmp17 ~]# hpssacli ctrl slot=0 ld xx show

删除逻辑 drive 19

[root@cmp17 ~]# hpssacli ctrl slot=0 ld xx delete

创建逻辑 drive 19

[root@cmp17 ~]# hpssacli ctrl slot=0 create type=ld drives=1I:1:xx raid=0

（三）节点加入 osd

也可以查看所有磁盘对应的 osd_id

ceph-volume inventory /dev/sdx --format json-pretty

批量创建 osd

batch Creates OSDs from a list of devices using a filestore or bluestore (default) setup

[root@bakcmp31 ~]# ceph-volume lvm batch --bluestore /dev/sdx

ceph-volume lvm activate

[root@bakcmp31 ~]# ceph-volume lvm activate --all

检查

osd 节点

[root@bakcmp31 ~]# systemctl status ceph-osd@x

mon 节点

[root@bakmtr01 ~]# ceph -s