云物理机对接ceph云盘

1300042631@qq.com

2022/04/29

---

背景说明

根据社区反馈当前OpenStack已经超大规模部署，生产环境也使用逐渐上线了云物理机功能，并对ironic组件及其ironic-inspect周边服务进行了二次开发了，但是云物理机本身默认使用的是本地盘，即直插物理磁盘，无论是系统盘，还是数据盘，均是。但是这样虽然性能稳定，但是也带来了一定的困扰，因为云物理机并不想kvm云主机那样支持快速弹性伸缩，比如cpu和内存规格变更，云主机可以做到实时变更，但是云物理机目前无法做到。

• 比如某个业务线一年前申请的云物理机，突然发现数据盘不足，想要改配，这就比较麻烦，因为牵涉到订单和价格计费的问题，不通的数据盘大小，收费和raid模式不同。比如原来是4Tx2的数据盘，因为业务需求，改成8Tx2，这就需要投入人力改订单改套餐flavor，改ironic的数据库nodes表中的properties字段等相关人工介入的操作。比较浪费人力，且存在风险。因为ironic基本是通过ipmitool来管控的，稍不注意动了某个字段，就有可能对云物理机本身造成影响。
• 还有一些场景，比如云物理机上架后，装机时，频繁失败，或者boot failed，排查一圈发现时系统盘未做raid导致。其实，熟悉云物理机裸金属的小伙伴都知道，看似高大上的云物理机自动装机，无非还是通过pxe\dhcp\tftp\get镜像\cloud_init装机。因为启动盘未做raid导致装机失败，生产环境，云物理机开机关机大概需要20来分钟，所以排查起来也非常耗时。
• 所以针对两个比较重要的场景，需要对云物理机和也支持弹性伸缩，当然这里仅指数据盘和系统盘，不支持内存和cpu核数。
• 根据当前云主机底层使用的块存储ceph场景，计划采用ceph-iscsi-gw来定制ironic对接云盘问题，具体详情及配置方案如下，云盘性能方面还是可观的。

一、基础环境信息

1. 版本依赖

云物理机对接ceph云盘

1300042631@qq.com

2022/04/29

---

一、基础环境信息

1. 版本依赖

应用	使用版本	版本要求
OS	CentOS Linux release 7.6.1810 (Core)	Red Hat Enterprise Linux/CentOS 7.5 (or newer);Linux kernel v4.16 (or newer)
Ceph	Ceph Nautilus Stable (12.2.x)	Ceph Luminous Stable
Ceph-Iscsi-Gateway	ceph-iscsi-3.2	ceph-iscsi-3.2 or newer package
Targetcli	targetcli-2.1.fb47	targetcli-2.1.fb47 or newer package
Python-rtslib	python-rtslib-2.1.fb68	python-rtslib-2.1.fb68 or newer package
Tcmu-runner	tcmu-runner-1.5.2	tcmu-runner-1.4.0 or newer package

2. 集群信息：

IP	用途	OS
192.168.9.101	ceph-node1	CentOS Linux release 7.6.1810 (Core)
192.168.10.135	ceph-node2	CentOS Linux release 7.6.1810 (Core)
192.168.9.190	ceph-node3	CentOS Linux release 7.6.1810 (Core)
192.31.162.123	rg2-test-control001.ostack.hfb3.iflytek.net	CentOS Linux release 7.6.1810 (Core)
192.31.162.124	rg2-test-control002.ostack.hfb3.iflytek.net	CentOS Linux release 7.6.1810 (Core)
192.31.162.125	rg2-test-control003.ostack.hfb3.iflytek.net	CentOS Linux release 7.6.1810 (Core)

二、 环境部署

iSCSI 网关提供一个高可用性 （HA） iSCSI 目标，用于将 RADOS 块储存设备 （RBD） 映像导出为 SCSI 磁盘。
iSCSI 协议允许客户端(发起方)通过TCP/IP网络向存储设备（目标）发送SCSI命令,
使没有本机Ceph客户端支持的客户端能够访问 Ceph 块存储。
其中包括Microsoft Windows甚至BIOS。每个iSCSI网关都利用 Linux IO 目标内核子系统 （LIO） 来提供 iSCSI 协议支持。
LIO 利用用户空间直通 （TCMU） 与 Ceph 的 librbd 库进行交互，
并向 iSCSI 客户端公开 RBD 映像。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-iQEgQ6Dg-1665744655793)(ceph.png)]

1. 部署一套ceph环境，并确认集群状态为ok

# 当前测试集群的状态[root@ceph-node1 ~]# ceph -scluster:id:     c0df9eb6-5d23-4f14-8136-e2351fa215f7health: HEALTH_OKservices:mon:         3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 (age 28h)mgr:         ceph-node1(active, since 2w)mds:         cephfs-demo:1 {0=ceph-node2=up:active} 2 up:standbyosd:         15 osds: 15 up (since 2w), 15 in (since 2w)rgw:         1 daemon active (ceph-node1)task status:data:pools:   13 pools, 544 pgsobjects: 32.73k objects, 126 GiBusage:   395 GiB used, 1.1 TiB / 1.4 TiB availpgs:     544 active+cleanio:client:   1.5 KiB/s rd, 1 op/s rd, 0 op/s wr[root@ceph-node1 ~]# ceph osd tree
ID CLASS WEIGHT  TYPE NAME           STATUS REWEIGHT PRI-AFF
-1       1.44571 root default
-3       0.46884     host ceph-node10   hdd 0.09769         osd.0           up  0.81003 1.000003   hdd 0.09769         osd.3           up  0.81003 1.000006   hdd 0.09769         osd.6           up  1.00000 1.000009   hdd 0.09769         osd.9           up  1.00000 1.00000
10   hdd 0.07809         osd.10          up  1.00000 1.00000
-5       0.48843     host ceph-node21   hdd 0.09769         osd.1           up  0.95001 1.000004   hdd 0.09769         osd.4           up  1.00000 1.000007   hdd 0.09769         osd.7           up  1.00000 1.00000
11   hdd 0.09769         osd.11          up  1.00000 1.00000
13   hdd 0.09769         osd.13          up  0.90001 1.00000
-7       0.48843     host ceph-node32   hdd 0.09769         osd.2           up  1.00000 1.000005   hdd 0.09769         osd.5           up  0.81003 1.000008   hdd 0.09769         osd.8           up  1.00000 1.00000
12   hdd 0.09769         osd.12          up  0.81003 1.00000
14   hdd 0.09769         osd.14          up  1.00000 1.00000
[root@ceph-node1 ~]#

2. 定制配置 /etc/ceph/ceph.conf

1). 降低检测关闭 OSD 的默认检测信号间隔,来降低启动超时

[osd]
osd heartbeat grace = 20
osd heartbeat interval = 5

2). 更新下 monitor 运行状态

ceph tell <daemon_type>.<id> config set <parameter_name> <new_value>
ceph tell osd.* config set osd_heartbeat_grace 20
ceph tell osd.* config set osd_heartbeat_interval 5

3). 更新下 OSD 的运行状态

ceph daemon <daemon_type>.<id> config set osd_client_watch_timeout 15
ceph daemon osd.0 config set osd_heartbeat_grace 20
ceph daemon osd.0 config set osd_heartbeat_interval 5

3. 部署ceph-iscsi

# 可以使用ceph-ansible  也可以使用命令行分步的形式,具体参考官方网站
# https://docs.ceph.com/en/quincy/rbd/iscsi-target-cli-manual-install/

1. 在所有的iscsi gateway 节点， 安装Ceph-iscsi:

   [root@ceph-node1 ~]# yum install ceph-iscsi

2. 在所有的iscsi gateway 节点， 安装tcmu-runner:

   [root@ceph-node1 ~]#  yum install tcmu-runner# 事实上 安装方式可以选择源码编译或者rpm包,这个包不是很方便找:
# https://1.chacra.ceph.com/r/tcmu-runner/master/
#9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/x86_64/tcmu-runner-1.5.2-1.el7.x86_64.rpm
# https://1.chacra.ceph.com/r/tcmu-runner/master/
#9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/x86_64/libtcmu-devel-1.5.2-1.el7.x86_64.rpm
#https://1.chacra.ceph.com/r/tcmu-runner/master/
#9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/x86_64/libtcmu-1.5.2-1.el7.x86_64.rpm

3. 开始配置iscsi gateway:1). ceph-iscsi默认使用pool名称为rbd的存储池，所以需要新建一个,之后查询下是否新建成功：

   [root@ceph-node1 ~]# ceph osd lspools

2). 新建一个iscsi-gateway.cfg配置文件

   [root@ceph-node1 ~]# touch /etc/ceph/iscsi-gateway.cfg

3). 编辑/etc/ceph/iscsi-gateway.cfg文件

[root@ceph-node1 ~]# cat /etc/ceph/iscsi-gateway.cfg
[config]
# Name of the Ceph storage cluster. A suitable Ceph configuration file allowing
# access to the Ceph storage cluster from the gateway node is required, if not
# colocated on an OSD node.
cluster_name = ceph# Place a copy of the ceph cluster's admin keyring in the gateway's /etc/ceph
# drectory and reference the filename here
gateway_keyring = ceph.client.admin.keyring# API settings.
# The API supports a number of options that allow you to tailor it to your
# local environment. If you want to run the API under https, you will need to
# create cert/key files that are compatible for each iSCSI gateway node, that is
# not locked to a specific node. SSL cert and key files *must* be called
# 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory
# on *each* gateway node. With the SSL files in place, you can use 'api_secure = true'
# to switch to https mode.# To support the API, the bear minimum settings are:
api_secure = false# Additional API configuration options are as follows, defaults shown.
# api_user = admin
# api_password = admin
# api_port = 5001
trusted_ip_list = 192.168.9.101,192.168.10.135,192.168.33.146,192.31.162.123

4). 复制 iscsi-gateway.cfg file to all iSCSI gateway nodes.4.  on all iSCSI gateway nodes, enable and start the API service:

[root@ceph-node1 ~]# systemctl daemon-reload[root@ceph-node1 ~]# systemctl enable rbd-target-gw
[root@ceph-node1 ~]# systemctl start rbd-target-gw[root@ceph-node1 ~]# systemctl enable rbd-target-api
[root@ceph-node1 ~]# systemctl start rbd-target-api

4. 调试ceph-iscsi

# gwcli 将创建和配置 iSCSI 目标和 RBD 映像: 配置是相对简单的，这里参考官方的案例1. As root, on a iSCSI gateway node, start the iSCSI gateway command-line interface:
[root@ceph-node1 ~]#  gwcli2. Go to iscsi-targets and create a target with the name iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw:> /> cd /iscsi-targets
> /iscsi-targets>  create iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw3. Create the iSCSI gateways. The IPs used below are the ones that will be used for iSCSI data like READ and WRITE commands. They can be the same IPs used for management operations listed in trusted_ip_list, but it is recommended that different IPs are used.> /iscsi-targets> cd iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/gateways
> /iscsi-target...-igw/gateways>  create ceph-gw-1 192.168.9.101
> /iscsi-target...-igw/gateways>  create ceph-gw-2 192.168.10.135If not using RHEL/CentOS or using an upstream or ceph-iscsi-test kernel, the skipchecks=true argument must be used. This will avoid the Red Hat kernel and rpm checks:> /iscsi-targets> cd iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/gateways
> /iscsi-target...-igw/gateways>  create ceph-gw-1 192.168.9.101 skipchecks=true
> /iscsi-target...-igw/gateways>  create ceph-gw-2 192.168.10.135 skipchecks=true4. Add a RBD image with the name disk_1 in the pool rbd:> /iscsi-target...-igw/gateways> cd /disks
> /disks> create pool=rbd image=disk_1 size=90G5. Create a client with the initiator name iqn.1994-05.com.redhat:rh7-client:> /disks> cd /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts
> /iscsi-target...eph-igw/hosts>  create iqn.1994-05.com.redhat:rh7-client6. Set the client’s CHAP username to myiscsiusername and password to myiscsipassword:> /iscsi-target...at:rh7-client>  auth username=myiscsiusername password=myiscsipassword# CHAP must always be configured. Without CHAP, the target will reject any login requests.7. Add the disk to the client:> /iscsi-target...at:rh7-client> disk add rbd/disk_18. example:[root@ceph-node1 ~]# gwcli
/> ls
o- / ......................................................................................................................... [...]o- cluster ......................................................................................................... [Clusters: 1]| o- ceph ............................................................................................................ [HEALTH_OK]|   o- pools ......................................................................................................... [Pools: 13]|   | o- .rgw.root ............................................................. [(x3), Commit: 0.00Y/252112656K (0%), Used: 768K]|   | o- backups ............................................................... [(x3), Commit: 0.00Y/252112656K (0%), Used: 192K]|   | o- cephfs_data .......................................................... [(x3), Commit: 0.00Y/252112656K (0%), Used: 0.00Y]|   | o- cephfs_metadata ................................................... [(x3), Commit: 0.00Y/252112656K (0%), Used: 1577484b]|   | o- default.rgw.control .................................................. [(x3), Commit: 0.00Y/252112656K (0%), Used: 0.00Y]|   | o- default.rgw.log ...................................................... [(x3), Commit: 0.00Y/252112656K (0%), Used: 0.00Y]|   | o- default.rgw.meta ...................................................... [(x3), Commit: 0.00Y/252112656K (0%), Used: 384K]|   | o- images ........................................................ [(x3), Commit: 0.00Y/252112656K (0%), Used: 32298369142b]|   | o- imagess .............................................................. [(x3), Commit: 0.00Y/252112656K (0%), Used: 0.00Y]|   | o- libvirt ........................................................ [(x3), Commit: 0.00Y/252112656K (0%), Used: 3217097458b]|   | o- rbd ......................................................... [(x3), Commit: 550G/252112656K (228%), Used: 371312952985b]|   | o- vms ................................................................... [(x3), Commit: 0.00Y/252112656K (0%), Used: 192K]|   | o- volumes ........................................................... [(x3), Commit: 0.00Y/252112656K (0%), Used: 1769991b]|   o- topology ............................................................................................... [OSDs: 15,MONs: 3]o- disks ........................................................................................................ [550G, Disks: 7]| o- rbd ............................................................................................................ [rbd (550G)]|   o- disk_1 ......................................................................................... [rbd/disk_1 (Online, 90G)]|   o- disk_2 ........................................................................................ [rbd/disk_2 (Online, 100G)]|   o- disk_3 ........................................................................................ [rbd/disk_3 (Online, 120G)]|   o- disk_4 ......................................................................................... [rbd/disk_4 (Online, 50G)]|   o- disk_5 ......................................................................................... [rbd/disk_5 (Online, 50G)]|   o- volume-972f0121-59e3-44aa-87c0-d770827d755f .............. [rbd/volume-972f0121-59e3-44aa-87c0-d770827d755f (Online, 1168G)]|   o- volume-fe5ecbb9-8440-4115-9824-605c39a2f7eb ............... [rbd/volume-fe5ecbb9-8440-4115-9824-605c39a2f7eb (Online, 10G)]o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ......................................................... [Auth: None, Gateways: 2]o- disks .......................................................................................................... [Disks: 7]| o- rbd/disk_1 .................................................................................. [Owner: ceph-node1, Lun: 0]| o- rbd/disk_2 .................................................................................. [Owner: ceph-node2, Lun: 1]| o- rbd/disk_3 .................................................................................. [Owner: ceph-node1, Lun: 2]| o- rbd/disk_4 .................................................................................. [Owner: ceph-node2, Lun: 3]| o- rbd/disk_5 .................................................................................. [Owner: ceph-node1, Lun: 6]| o- rbd/volume-972f0121-59e3-44aa-87c0-d770827d755f ............................................. [Owner: ceph-node2, Lun: 5]| o- rbd/volume-fe5ecbb9-8440-4115-9824-605c39a2f7eb ............................................. [Owner: ceph-node1, Lun: 4]o- gateways ............................................................................................ [Up: 2/2, Portals: 2]| o- ceph-node1 .......................................................................................... [192.168.9.101 (UP)]| o- ceph-node2 ......................................................................................... [192.168.10.135 (UP)]o- host-groups .................................................................................................. [Groups : 0]o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 8]o- iqn.1994-05.com.redhat:rh7-client .......................................................... [Auth: CHAP, Disks: 2(190G)]| o- lun 0 ............................................................................ [rbd/disk_1(90G), Owner: ceph-node1]| o- lun 1 ........................................................................... [rbd/disk_2(100G), Owner: ceph-node2]o- iqn.1995-05.com.redhat:rh7-client ............................................... [LOGGED-IN, Auth: CHAP, Disks: 1(120G)]| o- lun 2 ........................................................................... [rbd/disk_3(120G), Owner: ceph-node1]o- iqn.1996-05.com.redhat:rh7-client ........................................................... [Auth: CHAP, Disks: 1(50G)]| o- lun 3 ............................................................................ [rbd/disk_4(50G), Owner: ceph-node2]o- iqn.1994-05.com.redhat:336ea081fb32 .............................................. [LOGGED-IN, Auth: CHAP, Disks: 1(50G)]| o- lun 6 ............................................................................ [rbd/disk_5(50G), Owner: ceph-node1]o- iqn.1994-05.com.redhat:186aa3199292 ............................................. [LOGGED-IN, Auth: CHAP, Disks: 2(140G)]| o- lun 4 ....................................... [rbd/volume-fe5ecbb9-8440-4115-9824-605c39a2f7eb(10G), Owner: ceph-node1]| o- lun 5 ...................................... [rbd/volume-972f0121-59e3-44aa-87c0-d770827d755f(1168G), Owner: ceph-node2]o- iqn.1994-05.com.redhat:53725a46-ddc3-4282-962a-4c0a4f37e217 ............................... [Auth: CHAP, Disks: 0(0.00Y)]o- iqn.1994-05.com.redhat:5b5d7a76e .......................................................... [Auth: CHAP, Disks: 0(0.00Y)]o- iqn.1994-05.com.redhat:82e2285f-2196-417c-ba42-cdb08a567c8d ............................... [Auth: CHAP, Disks: 0(0.00Y)]
/>

5. ISCSI INITIATOR FOR LINUX

# 安装依赖
# yum install iscsi-initiator-utils
# yum install device-mapper-multipath1. Create the default /etc/multipath.conf file and enable the multipathd service:
# mpathconf --enable --with_multipathd y2. Add the following to /etc/multipath.conf file:devices {device {vendor                 "LIO-ORG"product                "TCMU device"hardware_handler       "1 alua"path_grouping_policy   "failover"path_selector          "queue-length 0"failback               60path_checker           turprio                   aluaprio_args              exclusive_pref_bitfast_io_fail_tmo       25no_path_retry          queue}
}3. Restart the multipathd service:# systemctl reload multipathd4. iSCSI Discovery and Setup:1). If CHAP was setup on the iSCSI gateway, provide a CHAP username and password by updating the /etc/iscsi/iscsid.conf file accordingly.
#node.session.auth.authmethod = CHAP
#node.session.auth.username = username
#node.session.auth.password = password2). Discover the target portals:# iscsiadm -m discovery -t st -p 192.168.56.101
192.168.56.101:3260,1 iqn.2003-01.org.linux-iscsi.rheln1
192.168.56.102:3260,2 iqn.2003-01.org.linux-iscsi.rheln13). Login to target:
# iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.rheln1 -l4). Logout from target:
# iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.rheln1 -u

三、 Ceph-iscsi与OpenStack cinder对接

1、修改cinder-volume配置文件

# 编辑/etc/kolla/cinder-volume/cinder.conf
enabled_backends = local-lvm-sata, local-lvm-ssd, rbd-sata, ceph_iscsi
# 该测试环境对接了4种存储
[ceph_iscsi]
volume_driver = cinder.volume.drivers.ceph.rbd_iscsi.RBDISCSIDriver
volume_backend_name = ceph_iscsi
#rbd_iscsi_target_iqns = iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
rbd_iscsi_target_iqn = iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
rbd_user = cinder
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_iscsi_controller_ips = 192.168.9.101:5000
rbd_iscsi_api_url = http://192.168.9.101:5000
rbd_iscsi_api_user = admin
rbd_iscsi_api_password = admin

2、修改cinder-volume代码，添加定制开发RBDISCSIDriver

1.  cinder/opts.py

    ...from cinder.volume.drivers.ceph import rbd_iscsi as \cinder_volume_drivers_ceph_rbdiscsi...def list_opts():...cinder_volume_drivers_ceph_rbdiscsi.RBD_ISCSI_OPTS,...

2. cinder/tests/unit/volume/drivers/ceph/__init__.py
3. cinder/tests/unit/volume/drivers/ceph/fake_rbd_iscsi_client.py

#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.
#
"""Fake rbd-iscsi-client for testing without installing the client."""import sys
from unittest import mockfrom cinder.tests.unit.volume.drivers.ceph \import fake_rbd_iscsi_client_exceptions as clientexceptionsrbdclient = mock.MagicMock()
rbdclient.version = "0.1.5"
rbdclient.exceptions = clientexceptionssys.modules['rbd_iscsi_client'] = rbdclient

4. cinder/tests/unit/volume/drivers/ceph/fake_rbd_iscsi_client_exceptions.py

#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.
#
"""Fake client exceptions to use."""class UnsupportedVersion(Exception):"""Unsupported version of the client."""passclass ClientException(Exception):"""The base exception class for these fake exceptions."""_error_code = None_error_desc = None_error_ref = None_debug1 = None_debug2 = Nonedef __init__(self, error=None):if error:if 'code' in error:self._error_code = error['code']if 'desc' in error:self._error_desc = error['desc']if 'ref' in error:self._error_ref = error['ref']if 'debug1' in error:self._debug1 = error['debug1']if 'debug2' in error:self._debug2 = error['debug2']def get_code(self):return self._error_codedef get_description(self):return self._error_descdef get_ref(self):return self._error_refdef __str__(self):formatted_string = self.messageif self.http_status:formatted_string += " (HTTP %s)" % self.http_statusif self._error_code:formatted_string += " %s" % self._error_codeif self._error_desc:formatted_string += " - %s" % self._error_descif self._error_ref:formatted_string += " - %s" % self._error_refif self._debug1:formatted_string += " (1: '%s')" % self._debug1if self._debug2:formatted_string += " (2: '%s')" % self._debug2return formatted_stringclass HTTPConflict(ClientException):http_status = 409message = "Conflict"def __init__(self, error=None):if error:super(HTTPConflict, self).__init__(error)if 'message' in error:self._error_desc = error['message']def get_description(self):return self._error_descclass HTTPNotFound(ClientException):http_status = 404message = "Not found"class HTTPForbidden(ClientException):http_status = 403message = "Forbidden"class HTTPBadRequest(ClientException):http_status = 400message = "Bad request"class HTTPUnauthorized(ClientException):http_status = 401message = "Unauthorized"class HTTPServerError(ClientException):http_status = 500message = "Error"def __init__(self, error=None):if error and 'message' in error:self._error_desc = error['message']def get_description(self):return self._error_desc

5. cinder/tests/unit/volume/drivers/ceph/test_rbd_iscsi.py

# Copyright 2012 Josh Durgin
# Copyright 2013 Canonical Ltd.
# All Rights Reserved.
#
#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.from unittest import mockimport ddtfrom cinder import context
from cinder import exception
from cinder.tests.unit import fake_constants as fake
from cinder.tests.unit import fake_volume
from cinder.tests.unit import test
from cinder.tests.unit.volume.drivers.ceph \import fake_rbd_iscsi_client as fake_client
import cinder.volume.drivers.ceph.rbd_iscsi as driver# This is used to collect raised exceptions so that tests may check what was
# raised.
# NOTE: this must be initialised in test setUp().
RAISED_EXCEPTIONS = []@ddt.ddt
class RBDISCSITestCase(test.TestCase):def setUp(self):global RAISED_EXCEPTIONSRAISED_EXCEPTIONS = []super(RBDISCSITestCase, self).setUp()self.context = context.get_admin_context()# bogus access to prevent pep8 violation# from the import of fake_client.# fake_client must be imported to create the fake# rbd_iscsi_client system modulefake_client.rbdclientself.fake_target_iqn = 'iqn.2019-01.com.suse.iscsi-gw:iscsi-igw'self.fake_valid_response = {'status': '200'}self.fake_clients = \{'response':{'Content-Type': 'application/json','Content-Length': '55','Server': 'Werkzeug/0.14.1 Python/2.7.15rc1','Date': 'Wed, 19 Jun 2019 20:13:18 GMT','status': '200','content-location': 'http://192.168.121.11:5001/api/clients/''XX_REPLACE_ME'},'body':{'clients': ['iqn.1993-08.org.debian:01:5d3b9abba13d']}}self.volume_a = fake_volume.fake_volume_obj(self.context,**{'name': u'volume-0000000a','id': '4c39c3c7-168f-4b32-b585-77f1b3bf0a38','size': 10})self.volume_b = fake_volume.fake_volume_obj(self.context,**{'name': u'volume-0000000b','id': '0c7d1f44-5a06-403f-bb82-ae7ad0d693a6','size': 10})self.volume_c = fake_volume.fake_volume_obj(self.context,**{'name': u'volume-0000000a','id': '55555555-222f-4b32-b585-9991b3bf0a99','size': 12,'encryption_key_id': fake.ENCRYPTION_KEY_ID})def setup_configuration(self):config = mock.MagicMock()config.rbd_cluster_name = 'nondefault'config.rbd_pool = 'rbd'config.rbd_ceph_conf = '/etc/ceph/my_ceph.conf'config.rbd_secret_uuid = Noneconfig.rbd_user = 'cinder'config.volume_backend_name = Noneconfig.rbd_iscsi_api_user = 'fake_user'config.rbd_iscsi_api_password = 'fake_password'config.rbd_iscsi_api_url = 'http://fake.com:5000'return config@mock.patch('rbd_iscsi_client.client.RBDISCSIClient',spec=True,)def setup_mock_client(self, _m_client, config=None, mock_conf=None):_m_client = _m_client.return_value# Configure the base constants, defaults etc...if mock_conf:_m_client.configure_mock(**mock_conf)if config is None:config = self.setup_configuration()self.driver = driver.RBDISCSIDriver(configuration=config)self.driver.set_initialized()return _m_client@mock.patch('rbd_iscsi_client.version', '0.1.0')def test_unsupported_client_version(self):self.setup_mock_client()with mock.patch('cinder.volume.drivers.rbd.RBDDriver.do_setup'):self.assertRaises(exception.InvalidInput,self.driver.do_setup, None)@ddt.data({'user': None, 'password': 'foo','url': 'http://fake.com:5000', 'iqn': None},{'user': None, 'password': None,'url': 'http://fake', 'iqn': None},{'user': None, 'password': None,'url': None, 'iqn': None},{'user': 'fake', 'password': 'fake','url': None, 'iqn': None},{'user': 'fake', 'password': 'fake','url': 'fake', 'iqn': None},)@ddt.unpackdef test_min_config(self, user, password, url, iqn):config = self.setup_configuration()config.rbd_iscsi_api_user = userconfig.rbd_iscsi_api_password = passwordconfig.rbd_iscsi_api_url = urlconfig.rbd_iscsi_target_iqn = iqnself.setup_mock_client(config=config)with mock.patch('cinder.volume.drivers.rbd.RBDDriver''.check_for_setup_error'):self.assertRaises(exception.InvalidConfigurationValue,self.driver.check_for_setup_error)@ddt.data({'response': None},{'response': {'nothing': 'nothing'}},{'response': {'status': '1680'}})@ddt.unpackdef test_do_setup(self, response):mock_conf = {'get_api.return_value': (response, None)}mock_client = self.setup_mock_client(mock_conf=mock_conf)with mock.patch('cinder.volume.drivers.rbd.RBDDriver.do_setup'), \mock.patch.object(driver.RBDISCSIDriver,'_create_client') as mock_create_client:mock_create_client.return_value = mock_clientself.assertRaises(exception.InvalidConfigurationValue,self.driver.do_setup, None)@mock.patch('rbd_iscsi_client.version', "0.1.4")def test_unsupported_version(self):self.setup_mock_client()self.assertRaises(exception.InvalidInput,self.driver._create_client)@ddt.data({'status': '200','target_iqn': 'iqn.2019-01.com.suse.iscsi-gw:iscsi-igw','clients': ['foo']},{'status': '1680','target_iqn': 'iqn.2019-01.com.suse.iscsi-gw:iscsi-igw','clients': None})@ddt.unpackdef test__get_clients(self, status, target_iqn, clients):config = self.setup_configuration()config.rbd_iscsi_target_iqn = target_iqnresponse = self.fake_clients['response']response['status'] = statusresponse['content-location'] = (response['content-location'].replace('XX_REPLACE_ME', target_iqn))body = self.fake_clients['body']mock_conf = {'get_clients.return_value': (response, body),'get_api.return_value': (self.fake_valid_response, None)}mock_client = self.setup_mock_client(mock_conf=mock_conf,config=config)with mock.patch('cinder.volume.drivers.rbd.RBDDriver.do_setup'), \mock.patch.object(driver.RBDISCSIDriver,'_create_client') as mock_create_client:mock_create_client.return_value = mock_clientself.driver.do_setup(None)if status == '200':actual_response = self.driver._get_clients()self.assertEqual(actual_response, body)else:# we expect an exceptionself.assertRaises(exception.VolumeBackendAPIException,self.driver._get_clients)@ddt.data({'status': '200','body': {'created': 'someday','discovery_auth': 'somecrap','disks': 'fakedisks','gateways': 'fakegws','targets': 'faketargets'}},{'status': '1680','body': None})@ddt.unpackdef test__get_config(self, status, body):config = self.setup_configuration()config.rbd_iscsi_target_iqn = self.fake_target_iqnresponse = self.fake_clients['response']response['status'] = statusresponse['content-location'] = (response['content-location'].replace('XX_REPLACE_ME',self.fake_target_iqn))mock_conf = {'get_config.return_value': (response, body),'get_api.return_value': (self.fake_valid_response, None)}mock_client = self.setup_mock_client(mock_conf=mock_conf,config=config)with mock.patch('cinder.volume.drivers.rbd.RBDDriver.do_setup'), \mock.patch.object(driver.RBDISCSIDriver,'_create_client') as mock_create_client:mock_create_client.return_value = mock_clientself.driver.do_setup(None)if status == '200':actual_response = self.driver._get_config()self.assertEqual(body, actual_response)else:# we expect an exceptionself.assertRaises(exception.VolumeBackendAPIException,self.driver._get_config)

6. cinder/volume/drivers/ceph/__init__.py
7. cinder/volume/drivers/ceph/rbd_iscsi.py

#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.
"""RADOS Block Device iSCSI Driver"""from distutils import versionfrom oslo_config import cfg
from oslo_log import log as logging
from oslo_utils import netutilsfrom cinder import exception
from cinder.i18n import _
from cinder import interface
from cinder import utils
from cinder.volume import configuration
from cinder.volume.drivers import rbd
from cinder.volume import volume_utilstry:import rbd_iscsi_clientfrom rbd_iscsi_client import clientfrom rbd_iscsi_client import exceptions as client_exceptions
except ImportError:rbd_iscsi_client = Noneclient = Noneclient_exceptions = NoneLOG = logging.getLogger(__name__)RBD_ISCSI_OPTS = [cfg.StrOpt('rbd_iscsi_api_user',default='',help='The username for the rbd_target_api service'),cfg.StrOpt('rbd_iscsi_api_password',default='',secret=True,help='The username for the rbd_target_api service'),cfg.StrOpt('rbd_iscsi_api_url',default='',help='The url to the rbd_target_api service'),cfg.BoolOpt('rbd_iscsi_api_debug',default=False,help='Enable client request debugging.'),cfg.StrOpt('rbd_iscsi_target_iqn',default=None,help='The preconfigured target_iqn on the iscsi gateway.'),
]CONF = cfg.CONF
CONF.register_opts(RBD_ISCSI_OPTS, group=configuration.SHARED_CONF_GROUP)MIN_CLIENT_VERSION = "0.1.8"@interface.volumedriver
class RBDISCSIDriver(rbd.RBDDriver):"""Implements RADOS block device (RBD) iSCSI volume commands."""VERSION = '1.0.0'# ThirdPartySystems wiki pageCI_WIKI_NAME = "Cinder_Jenkins"SUPPORTS_ACTIVE_ACTIVE = TrueSTORAGE_PROTOCOL = 'iSCSI'CHAP_LENGTH = 16# The target IQN to use for creating all exports# we map all the targets for OpenStack attaches to this.target_iqn = Nonedef __init__(self, active_backend_id=None, *args, **kwargs):super(RBDISCSIDriver,self).__init__(*args, **kwargs)self.configuration.append_config_values(RBD_ISCSI_OPTS)@classmethoddef get_driver_options(cls):additional_opts = cls._get_oslo_driver_opts('replication_device', 'reserved_percentage','max_over_subscription_ratio', 'volume_dd_blocksize','driver_ssl_cert_verify', 'suppress_requests_ssl_warnings')return rbd.RBD_OPTS + RBD_ISCSI_OPTS + additional_optsdef _create_client(self):client_version = rbd_iscsi_client.versionif (version.StrictVersion(client_version) <version.StrictVersion(MIN_CLIENT_VERSION)):ex_msg = (_('Invalid rbd_iscsi_client version found (%(found)s). ''Version %(min)s or greater required. Run "pip'' install --upgrade rbd-iscsi-client" to upgrade'' the client.')% {'found': client_version,'min': MIN_CLIENT_VERSION})LOG.error(ex_msg)raise exception.InvalidInput(reason=ex_msg)config = self.configurationssl_warn = config.safe_get('suppress_requests_ssl_warnings')cl = client.RBDISCSIClient(config.safe_get('rbd_iscsi_api_user'),config.safe_get('rbd_iscsi_api_password'),config.safe_get('rbd_iscsi_api_url'),secure=config.safe_get('driver_ssl_cert_verify'),suppress_ssl_warnings=ssl_warn)return cldef _is_status_200(self, response):return (response and 'status' in response andresponse['status'] == '200')def do_setup(self, context):"""Perform initialization steps that could raise exceptions."""super(RBDISCSIDriver, self).do_setup(context)if client is None:msg = _("You must install rbd-iscsi-client python package ""before using this driver.")raise exception.VolumeDriverException(data=msg)# Make sure we have the basic settings we need to talk to the# iscsi api serviceconfig = self.configurationself.client = self._create_client()self.client.set_debug_flag(config.safe_get('rbd_iscsi_api_debug'))resp, body = self.client.get_api()if not self._is_status_200(resp):# failed to fetch the open api urlraise exception.InvalidConfigurationValue(option='rbd_iscsi_api_url',value='Could not talk to the rbd-target-api')# The admin had to have setup a target_iqn in the iscsi gateway# already in order for the gateways to work properlyself.target_iqn = self.configuration.safe_get('rbd_iscsi_target_iqn')LOG.info("Using target_iqn '%s'", self.target_iqn)def check_for_setup_error(self):"""Return an error if prerequisites aren't met."""super(RBDISCSIDriver, self).check_for_setup_error()required_options = ['rbd_iscsi_api_user','rbd_iscsi_api_password','rbd_iscsi_api_url','rbd_iscsi_target_iqn']for attr in required_options:val = getattr(self.configuration, attr)if not val:raise exception.InvalidConfigurationValue(option=attr,value=val)def _get_clients(self):# make sure we haveresp, body = self.client.get_clients(self.target_iqn)if not self._is_status_200(resp):msg = _("Failed to get_clients() from rbd-target-api")raise exception.VolumeBackendAPIException(data=msg)return bodydef _get_config(self):resp, body = self.client.get_config()if not self._is_status_200(resp):msg = _("Failed to get_config() from rbd-target-api")raise exception.VolumeBackendAPIException(data=msg)return bodydef _get_disks(self):resp, disks = self.client.get_disks()if not self._is_status_200(resp):msg = _("Failed to get_disks() from rbd-target-api")raise exception.VolumeBackendAPIException(data=msg)return disksdef create_client(self, initiator_iqn):"""Create a client iqn on the gateway if it doesn't exist."""client = self._get_target_client(initiator_iqn)if not client:try:self.client.create_client(self.target_iqn,initiator_iqn)except client_exceptions.ClientException as ex:raise exception.VolumeBackendAPIException(data=ex.get_description())def _get_target_client(self, initiator_iqn):"""Get the config information for a client defined to a target."""config = self._get_config()target_config = config['targets'][self.target_iqn]if initiator_iqn in target_config['clients']:return target_config['clients'][initiator_iqn]def _get_auth_for_client(self, initiator_iqn):initiator_config = self._get_target_client(initiator_iqn)if initiator_config:auth = initiator_config['auth']return authdef _set_chap_for_client(self, initiator_iqn, username, password):"""Save the CHAP creds in the client on the gateway."""# username is 8-64 chars# Password has to be 12-16 charsLOG.debug("Setting chap creds to %(user)s : %(pass)s",{'user': username, 'pass': password})try:self.client.set_client_auth(self.target_iqn,initiator_iqn,username,password)except client_exceptions.ClientException as ex:raise exception.VolumeBackendAPIException(data=ex.get_description())def _get_lun(self, iscsi_config, lun_name, initiator_iqn):lun = Nonetarget_info = iscsi_config['targets'][self.target_iqn]luns = target_info['clients'][initiator_iqn]['luns']if lun_name in luns:lun = {'name': lun_name,'id': luns[lun_name]['lun_id']}return lundef _lun_name(self, volume_name):"""Build the iscsi gateway lun name."""return ("%(pool)s/%(volume_name)s" %{'pool': self.configuration.rbd_pool,'volume_name': volume_name})def get_existing_disks(self):"""Get the existing list of registered volumes on the gateway."""resp, disks = self.client.get_disks()return disks['disks']@utils.tracedef create_disk(self, volume_name):"""Register the volume with the iscsi gateways.We have to register the volume with the iscsi gateway.Exporting the volume won't work unless the gateway knowsabout it."""try:self.client.find_disk(self.configuration.rbd_pool,volume_name)except client_exceptions.HTTPNotFound:try:# disk isn't known by the gateways, so lets add it.self.client.create_disk(self.configuration.rbd_pool,volume_name)except client_exceptions.ClientException as ex:LOG.exception("Couldn't create the disk entry to ""export the volume.")raise exception.VolumeBackendAPIException(data=ex.get_description())@utils.tracedef register_disk(self, target_iqn, volume_name):"""Register the disk with the target_iqn."""lun_name = self._lun_name(volume_name)try:self.client.register_disk(target_iqn, lun_name)except client_exceptions.HTTPBadRequest as ex:desc = ex.get_description()search_str = ('is already mapped on target %(target_iqn)s' %{'target_iqn': self.target_iqn})if desc.find(search_str):# The volume is already registeredreturnelse:LOG.error("Couldn't register the volume to the target_iqn")raise exception.VolumeBackendAPIException(data=ex.get_description())except client_exceptions.ClientException as ex:LOG.exception("Couldn't register the volume to the target_iqn",ex)raise exception.VolumeBackendAPIException(data=ex.get_description())@utils.tracedef unregister_disk(self, target_iqn, volume_name):"""Unregister the volume from the gateway."""lun_name = self._lun_name(volume_name)try:self.client.unregister_disk(target_iqn, lun_name)except client_exceptions.ClientException as ex:LOG.exception("Couldn't unregister the volume to the target_iqn",ex)raise exception.VolumeBackendAPIException(data=ex.get_description())@utils.tracedef export_disk(self, initiator_iqn, volume_name, iscsi_config):"""Export a volume to an initiator."""lun_name = self._lun_name(volume_name)LOG.debug("Export lun %(lun)s", {'lun': lun_name})lun = self._get_lun(iscsi_config, lun_name, initiator_iqn)if lun:LOG.debug("Found existing lun export.")return luntry:LOG.debug("Creating new lun export for %(lun)s",{'lun': lun_name})self.client.export_disk(self.target_iqn, initiator_iqn,self.configuration.rbd_pool,volume_name)resp, iscsi_config = self.client.get_config()return self._get_lun(iscsi_config, lun_name, initiator_iqn)except client_exceptions.ClientException as ex:raise exception.VolumeBackendAPIException(data=ex.get_description())@utils.tracedef unexport_disk(self, initiator_iqn, volume_name, iscsi_config):"""Remove a volume from an initiator."""lun_name = self._lun_name(volume_name)LOG.debug("unexport lun %(lun)s", {'lun': lun_name})lun = self._get_lun(iscsi_config, lun_name, initiator_iqn)if not lun:LOG.debug("Didn't find LUN on gateway.")returntry:LOG.debug("unexporting %(lun)s", {'lun': lun_name})self.client.unexport_disk(self.target_iqn, initiator_iqn,self.configuration.rbd_pool,volume_name)except client_exceptions.ClientException as ex:LOG.exception(ex)raise exception.VolumeBackendAPIException(data=ex.get_description())def find_client_luns(self, target_iqn, client_iqn, iscsi_config):"""Find luns already exported to an initiator."""if 'targets' in iscsi_config:if target_iqn in iscsi_config['targets']:target_info = iscsi_config['targets'][target_iqn]if 'clients' in target_info:clients = target_info['clients']client = clients[client_iqn]luns = client['luns']return luns@utils.tracedef initialize_connection(self, volume, connector):"""Export a volume to a host."""# create clientinitiator_iqn = connector['initiator']self.create_client(initiator_iqn)auth = self._get_auth_for_client(initiator_iqn)username = initiator_iqnif not auth['password']:password = volume_utils.generate_password(length=self.CHAP_LENGTH)self._set_chap_for_client(initiator_iqn, username, password)else:LOG.debug("using existing CHAP password")password = auth['password']# add disk for exportiscsi_config = self._get_config()# First have to ensure that the disk is registered with# the gateways.self.create_disk(volume.name)self.register_disk(self.target_iqn, volume.name)iscsi_config = self._get_config()# Now export the disk to the initiatorlun = self.export_disk(initiator_iqn, volume.name, iscsi_config)# fetch the updated config so we can get the lun idiscsi_config = self._get_config()target_info = iscsi_config['targets'][self.target_iqn]ips = target_info['ip_list']target_portal = ips[0]if netutils.is_valid_ipv6(target_portal):target_portal = "[{}]:{}".format(target_portal, "3260")else:target_portal = "{}:3260".format(target_portal)data = {'driver_volume_type': 'iscsi','data': {'target_iqn': self.target_iqn,'target_portal': target_portal,'target_lun': lun['id'],'auth_method': 'CHAP','auth_username': username,'auth_password': password,}}return datadef _delete_disk(self, volume):"""Remove the defined disk from the gateway."""# We only do this when we know it's not exported# anywhere in the gatewaylun_name = self._lun_name(volume.name)config = self._get_config()# Now look for the disk on any exported targetfound = Falsefor target_iqn in config['targets']:# Do we have the volume we are looking for?target = config['targets'][target_iqn]for client_iqn in target['clients'].keys():if lun_name in target['clients'][client_iqn]['luns']:found = Trueif not found:# we can delete the disk definitionLOG.info("Deleteing volume definition in iscsi gateway for {}".format(lun_name))self.client.delete_disk(self.configuration.rbd_pool, volume.name,preserve_image=True)def _terminate_connection(self, volume, initiator_iqn, target_iqn,iscsi_config):# remove the disk from the client.self.unexport_disk(initiator_iqn, volume.name, iscsi_config)# Try to unregister the disk, since nobody is using it.self.unregister_disk(self.target_iqn, volume.name)config = self._get_config()# If there are no more luns exported to this initiator# then delete the initiatorluns = self.find_client_luns(target_iqn, initiator_iqn, config)if not luns:LOG.debug("There aren't any more LUNs attached to %(iqn)s.""So we unregister the volume and delete ""the client entry",{'iqn': initiator_iqn})try:self.client.delete_client(target_iqn, initiator_iqn)except client_exceptions.ClientException:LOG.warning("Tried to delete initiator %(iqn)s, but delete ""failed.", {'iqns': initiator_iqn})def _terminate_all(self, volume, iscsi_config):"""Find all exports of this volume for our target_iqn and detach."""disks = self._get_disks()lun_name = self._lun_name(volume.name)if lun_name not in disks['disks']:LOG.debug("Volume {} not attached anywhere.".format(lun_name))returnfor target_iqn_tmp in iscsi_config['targets']:if self.target_iqn != target_iqn_tmp:# We don't touch exports for targets# we aren't configured to manage.continuetarget = iscsi_config['targets'][self.target_iqn]for client_iqn in target['clients'].keys():if lun_name in target['clients'][client_iqn]['luns']:self._terminate_connection(volume, client_iqn,self.target_iqn,iscsi_config)self._delete_disk(volume)@utils.tracedef terminate_connection(self, volume, connector, **kwargs):"""Unexport the volume from the gateway."""iscsi_config = self._get_config()if not connector:# No connector was passed in, so this is a force detach# we need to detach the volume from the configured target_iqn.self._terminate_all(volume, iscsi_config)initiator_iqn = connector['initiator']self._terminate_connection(volume, initiator_iqn, self.target_iqn,iscsi_config)self._delete_disk(volume)

8.  doc/source/reference/support-matrix.ini

...
[driver.rbd_iscsi]
title=(Ceph) iSCSI Storage Driver (iSCSI)driver.rbd_iscsi=complete
...

9.  driver-requirements.txt

# RBD-iSCSIrbd-iscsi-client # Apache-2.0

10. lower-constraints.txt

rbd-iscsi-client==0.1.8

11. setup.cfg

rbd-iscsi-client>=0.1.8 # Apache-2.0
rbd_iscsi =rbd-iscsi-client>=0.1.8 # Apache-2.0

四、 修改ironic代码，定制开发attach_volume、detach_volume

1、 对接ironic

# ironic/api/controllers/v1/node.pyclass NodesController(rest.RestController):"""REST controller for Nodes."""..._custom_actions = {...'attach_volume' : ['POST'],'detach_volume' : ['POST'],...}...@METRICS.timer('NodesController.attach_volume')@expose.expose(wtypes.text, wtypes.text,wtypes.text)def attach_volume(self,instance_uuid,target_portal):context = pecan.request.contextLOG.debug(context)list_object = objects.Node.list(context)LOG.debug("instance_uuid: %s" % instance_uuid)for obj in list_object:if obj.instance_uuid == instance_uuid:uuid = obj.uuidrpc_node = api_utils.get_rpc_node_with_suffix(uuid)if (rpc_node.power_state != ir_states.POWER_ON andrpc_node.provision_state != ir_states.ACTIVE):return "Failed: node system states error"topic = pecan.request.rpcapi.get_topic_for(rpc_node)ip = pecan.request.rpcapi.get_node_ip(context,rpc_node.uuid,topic)os_type = obj.instance_info['image_properties'].get('os_type')if "windows" == os_type:shell_txt = "powershell.exe $(Get-InitiatorPort).NodeAddress"else:shell_txt = "cat /etc/iscsi/initiatorname.iscsi"rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("get initiatorname Failed.")if instance_uuid not in rslt:LOG.error("set initiator name")if "windows" == os_type:shell_txt = "powershell.exe Set-InitiatorPort " + rslt.split(' ')[-1].strip('\n') + " -NewNodeAddress " + instance_uuidelse:shell_txt = "echo " + "InitiatorName=iqn.1994-05.com.redhat:" + instance_uuid + "> /etc/iscsi/initiatorname.iscsi"shell_txt1 = "sed -i '/^#node.session.auth.authmethod = CHAP/cnode.session.auth.authmethod = CHAP' /etc/iscsi/iscsid.conf"shell_txt2 = "sed -i '/^#node.session.auth.username = username/cnode.session.auth.username = " + instance_uuid[-12:] + "' /etc/iscsi/iscsid.conf"shell_txt3 = "sed -i '/^#node.session.auth.password = password/cnode.session.auth.password = " + instance_uuid[-12:] + "' /etc/iscsi/iscsid.conf"rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt1)rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt2)rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt3)rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("create initiatorname Failed.")if "windows" != os_type:shell_txt = "/etc/init.d/open-iscsi restart"rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("restart open-iscsi service Failed.")shell_txt = "/etc/init.d/iscsid restart"rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("restart iscsid service Failed.")if "windows" == os_type:LOG.error("discovery storge target.")shell_txt = "powershell.exe New-IscsiTargetPortal -TargetPortalAddress " + target_portal.split(":")[0]rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("discovery storge target Failed.")shell_txt = "powershell.exe $(Get-IscsiTarget).NodeAddress"rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("get storge target info Failed.")LOG.error("connect iscsi target")lines = rslt.splitlines()for line in lines:if "iqn" in line:shell_txt = "powershell.exe Connect-IscsiTarget -NodeAddress " + line.strip('\n') + " -IsPersistent $true -IsMultipathEnabled $true"rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("connect storge target Failed.")else:shell_txt = "iscsiadm -m discovery -t st -p " + target_portalrslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("discovery storge target Failed.")lines = rslt.splitlines()for line in lines:shell_txt = "iscsiadm -m node -T " + line.split(' ')[-1].strip() + " -p " + target_portal + " -l"rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("connect storge target Failed.")shell_txt = "rescan-scsi-bus.sh -f && rescan-scsi-bus.sh -r && rescan-scsi-bus.sh -u"rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("flush scsi host Failed.")shell_txt = "iscsiadm -m session --rescan"rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise("rescan iscsi session Failed.")@METRICS.timer('NodesController.detach_volume')@expose.expose(wtypes.text, wtypes.text, wtypes.text)def detach_volume(self, instance_uuid, target_portal):context = pecan.request.contextlist_object = objects.Node.list(context)LOG.debug("instance_uuid: %s" % instance_uuid)for obj in list_object:if obj.instance_uuid == instance_uuid:uuid = obj.uuidrpc_node = api_utils.get_rpc_node_with_suffix(uuid)if (rpc_node.power_state != ir_states.POWER_ON andrpc_node.provision_state != ir_states.ACTIVE):return "Failed: node system states error"topic = pecan.request.rpcapi.get_topic_for(rpc_node)ip = pecan.request.rpcapi.get_node_ip(context,rpc_node.uuid,topic)os_type = obj.instance_info['image_properties'].get('os_type')if "windows" == os_type:LOG.error("discovery storge target.")else:shell_txt = "iscsiadm -m discovery -t st -p " + target_portalrslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise ("discovery storge target Failed.")lines = rslt.splitlines()for line in lines:shell_txt = "iscsiadm -m node -T " + line.split(' ')[-1].strip() + " -p " + target_portal + " -u"rslt = self._send_cmd_to_agent(ip, "shell_stream", shell_txt)if "Failed" in rslt:raise ("connect storge target Failed.")

五、 修改nova_api代码，定制开发ironic

1、 修改drivers

#nova/virt/ironic/drivers.pyclass IronicDriver(virt_driver.ComputeDriver):"""Hypervisor driver for Ironic - bare metal provisioning."""
...def attach_volume(self, context, connection_info, instance, mountpoint,disk_bus=None, device_type=None, encryption=None):try:LOG.info("attach volume to bm Failed.")self.ironicclient.call("node.attach_volume",instance.uuid,connection_info['data']['target_portal'])except Exception as exc:LOG.error("attach volume to bm Failed.")raise exception.VolumeAttachFailed(volume_id=connection_info['serial'],reason=exc)def detach_volume(self, context, connection_info, instance, mountpoint,encryption=None):try:LOG.info("attach volume to bm Failed.111")self.ironicclient.call("node.detach_volume",instance.uuid,connection_info['data']['target_portal'])except Exception as exc:LOG.error("detach volume to bm Failed.")raise exception.VolumeDetachFailed(volume_id=connection_info['serial'],reason=exc)...

六、 rbd_iscsi_client修改

1、 安装rbd_iscsi_client

# pip install rbd-iscsi-clientREST:
get_api - Get all the api endpoints
get_config - get the entire gateway config
get_targets - Get all of the target_iqn’s defined in the gateways
create_target_iqn - create a new target_iqn
delete_target_iqn - delete a target_iqn
get_clients - get the clients (initiators) defined in the gateways
get_client_info - get the client information
create_client - Register a new client (initiator) with the gateways
delete_client - unregister a client (initiator) from the gateways
set_client_auth - set CHAP credentials for the client (initiator)
get_disks - get list of volumes defined to the gateways
create_disk - create a new volume/disk that the gateways can export
find_disk - Find a disk that the gateway knows about
delete_disk - delete a disk from the gateway and pool
register_disk - Make the disk available to export to a client.
unregister_disk - Make a disk unavailable to export to a client.
export_disk - Export a registered disk to a client (initiator)
unexport_disk - unexport a disk from a client (initiator)

2、 修改rbd_iscsi_client

修改后的代码

(openstack-base)[root@rg2-test-control001 rbd_iscsi_client]# cat client.py
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License."""
""" RBDISCSIClient... module: rbd_iscsi_client:Author: Walter A. Boring IV
:Description: This is the HTTP REST Client that is used to make calls to
the ceph-iscsi/rbd-target-api service running on the ceph iscsi gateway
host.
"""import json
import logging
import timefrom rbd_iscsi_client import exceptionsimport requestsclass RBDISCSIClient(object):"""REST client to rbd-target-api."""USER_AGENT = "os_client"#username = Noneusername = 'admin'#password = Nonepassword = 'admin'#api_url = Noneapi_url = 'http://192.168.9.101:5000'auth = Nonehttp_log_debug = Falsetries = 5delay = 0backoff = 2timeout = 60_logger = logging.getLogger(__name__)retry_exceptions = (exceptions.HTTPServiceUnavailable,requests.exceptions.ConnectionError)def __init__(self, username, password, base_url,suppress_ssl_warnings=False, timeout=None,secure=False, http_log_debug=False):super(RBDISCSIClient, self).__init__()self.username = usernameself.password = passwordself.api_url = base_urlself.timeout = timeoutself.secure = secureself.times = []self.set_debug_flag(http_log_debug)if suppress_ssl_warnings:requests.packages.urllib3.disable_warnings()self.auth = requests.auth.HTTPBasicAuth(username, password)def set_debug_flag(self, flag):"""Turn on/off http request/response debugging."""if not self.http_log_debug and flag:ch = logging.StreamHandler()self._logger.setLevel(logging.DEBUG)self._logger.addHandler(ch)self.http_log_debug = Truedef _http_log_req(self, args, kwargs):if not self.http_log_debug:returnstring_parts = ['curl -i']for element in args:if element in ('GET', 'POST'):string_parts.append(' -X %s' % element)else:string_parts.append(' %s' % element)for element in kwargs['headers']:header = ' -H "%s: %s"' % (element, kwargs['headers'][element])string_parts.append(header)if 'data' in kwargs:string_parts.append(' -d ')for key in kwargs['data']:string_parts.append('%(key)s=%(value)s&' %{'key': key,'value': kwargs['data'][key]})self._logger.debug("\nREQ: %s\n" % "".join(string_parts))def _http_log_resp(self, resp, body):if not self.http_log_debug:return# Replace commas with newlines to break the debug into new lines,# making it easier to readself._logger.debug("RESP:%s\n", str(resp).replace("',", "'\n"))self._logger.debug("RESP BODY:%s\n", body)def request(self, *args, **kwargs):"""Perform an HTTP Request.You should use get, post, delete instead."""kwargs.setdefault('headers', kwargs.get('headers', {}))kwargs['headers']['User-Agent'] = self.USER_AGENTkwargs['headers']['Accept'] = 'application/json'if 'data' in kwargs:payload = kwargs['data']else:payload = None# args[0] contains the URL, args[1] contains the HTTP verb/methodhttp_url = args[0]print '--------------------------------'http_method = args[1]self._http_log_req(args, kwargs)r = Noneresp = Nonebody = Nonewhile r is None and self.tries > 0:try:# Check to see if the request is being retried. If it is, we# want to delay.if self.delay:time.sleep(self.delay)if self.timeout:r = requests.request(http_method, http_url, data=payload,headers=kwargs['headers'],auth=self.auth,verify=self.secure,timeout=self.timeout)else:r = requests.request(http_method, http_url, data=payload,auth=self.auth,headers=kwargs['headers'],verify=self.secure)resp = r.headersbody = r.textif isinstance(body, bytes):body = body.decode('utf-8')# resp['status'], status['content-location'], and resp.status# need to be manually set as Python Requests doesn't provide# them automatically.resp['status'] = str(r.status_code)resp.status = r.status_codeif 'location' not in resp:resp['content-location'] = r.urlr.close()self._http_log_resp(resp, body)# Try and convert the body response to an object# This assumes the body of the reply is JSONif body:try:body = json.loads(body)except ValueError:passelse:body = Noneif resp.status >= 400:if body and 'message' in body:body['desc'] = body['message']raise exceptions.from_response(resp, body)except requests.exceptions.SSLError as err:self._logger.error("SSL certificate verification failed: (%s). You must have ""a valid SSL certificate or disable SSL ""verification.", err)raise exceptions.SSLCertFailed("SSL Certificate Verification Failed.")except self.retry_exceptions as ex:# If we catch an exception where we want to retry, we need to# decrement the retry count prepare to try again.r = Noneself.tries -= 1self.delay = self.delay * self.backoff + 1# Raise exception, we have exhausted all retries.if self.tries == 0:raise exexcept requests.exceptions.HTTPError as err:raise exceptions.HTTPError("HTTP Error: %s" % err)except requests.exceptions.URLRequired as err:raise exceptions.URLRequired("URL Required: %s" % err)except requests.exceptions.TooManyRedirects as err:raise exceptions.TooManyRedirects("Too Many Redirects: %s" % err)except requests.exceptions.Timeout as err:raise exceptions.Timeout("Timeout: %s" % err)except requests.exceptions.RequestException as err:raise exceptions.RequestException("Request Exception: %s" % err)return resp, bodydef _time_request(self, url, method, **kwargs):start_time = time.time()resp, body = self.request(url, method, **kwargs)self.times.append(("%s %s" % (method, url),start_time, time.time()))return resp, bodydef _cs_request(self, url, method, **kwargs):resp, body = self._time_request(self.api_url + url, method,**kwargs)return resp, bodydef get(self, url, **kwargs):return self._cs_request(url, 'GET', **kwargs)def post(self, url, **kwargs):return self._cs_request(url, 'POST', **kwargs)def put(self, url, **kwargs):return self._cs_request(url, 'PUT', **kwargs)def delete(self, url, **kwargs):return self._cs_request(url, 'DELETE', **kwargs)def get_api(self):"""Get the API endpoints."""return self.get("/api")def get_config(self):"""Get the complete config object."""return self.get("/api/config")def get_sys_info(self, type):"""Get system info of <type>.Valid types are:ip_addresscheckconfcheckversions"""api = "/api/sysinfo/%(type)s" % {'type': type}return self.get(api)def get_gatewayinfo(self):"""Get the number of active sessions on local gateway."""return self.get("/api/gatewayinfo")def get_targets(self):"""Get the list of targets defined in the config."""api = "/api/targets"return self.get(api)def get_target_info(self, target_iqn):"""Returns the total number of active sessions for <target_iqn>"""api = "/api/targetinfo/%(target_iqn)s" % {'target_iqn': target_iqn}return self.get(api)def create_target_iqn(self, target_iqn, mode=None, controls=None):"""Create the target iqn on the gateway."""api = "/api/target/%(target_iqn)s" % {'target_iqn': target_iqn}payload = {}if mode:payload['mode'] = modeif controls:payload['controls'] = controlsreturn self.put(api, data=payload)def delete_target_iqn(self, target_iqn):"""Delete a target iqn from the gateways."""api = "/api/target/%(target_iqn)s" % {'target_iqn': target_iqn}return self.delete(api)def get_clients(self, target_iqn):"""List clients defined to the configuration."""api = "/api/clients/%(target_iqn)s" % {'target_iqn': target_iqn}return self.get(api)def get_client_info(self, target_iqn, client_iqn):"""Fetch the Client information from the gateways.Alias, IP address and state for each connected portal."""api = ("/api/clientinfo/%(target_iqn)s/%(client_iqn)s" %{'target_iqn': target_iqn,'client_iqn': client_iqn})return self.get(api)def create_client(self, target_iqn, client_iqn):"""Delete a client."""api = ("/api/client/%(target_iqn)s/%(client_iqn)s" %{'target_iqn': target_iqn,'client_iqn': client_iqn})return self.put(api)def delete_client(self, target_iqn, client_iqn):"""Delete a client."""api = ("/api/client/%(target_iqn)s/%(client_iqn)s" %{'target_iqn': target_iqn,'client_iqn': client_iqn})return self.delete(api)def set_client_auth(self, target_iqn, client_iqn, username, password):"""Set the client chap credentials."""url = ("/api/clientauth/%(target_iqn)s/%(client_iqn)s" %{'target_iqn': target_iqn,'client_iqn': client_iqn})args = {'username': username,'password': password}return self.put(url, data=args)def get_disks(self):"""Get the rbd disks defined to the gateways."""return self.get("/api/disks")def create_disk(self, pool, image, size=None, extras=None):"""Add a disk to the gateway."""url = ("/api/disk/%(pool)s/%(image)s" %{'pool': pool,'image': image})args = {'pool': pool,'image': image,'mode': 'create'}if size:args['size'] = sizeif extras:args.update(extras)return self.put(url, data=args)def find_disk(self, pool, image):"""Find the disk in the gateway."""url = ("/api/disk/%(pool)s/%(image)s" %{'pool': pool,'image': image})return self.get(url)def delete_disk(self, pool, image, preserve_image=True):"""Delete a disk definition from the gateway.By default it will not delete the rbd image from the pool.If preserve_image is set to True, then this only tells thegateway to forget about this volume.   This is typically donewhen the volume isn't used as an export anymore."""url = ("/api/disk/%(pool)s/%(image)s" %{'pool': pool,'image': image})if preserve_image is True:preserve = 'true'else:preserve = 'false'payload = {'preserve_image': preserve}return self.delete(url, data=payload)def register_disk(self, target_iqn, volume):"""Add the volume to the target definition.This is done after the disk is created in a pool, andbefore the disk can be exported to an initiator."""url = ("/api/targetlun/%(target_iqn)s" %{'target_iqn': target_iqn})args = {'disk': volume}return self.put(url, data=args)def unregister_disk(self, target_iqn, volume):"""Remove the volume from the target definition.This is done after the disk is unexported from an initiatorand before the disk can be deleted from the gateway."""url = ("/api/targetlun/%(target_iqn)s" %{'target_iqn': target_iqn})args = {'disk': volume}return self.delete(url, data=args)def export_disk(self, target_iqn, client_iqn, pool, disk):"""Add a disk to export to a client."""url = ("/api/clientlun/%(target_iqn)s/%(client_iqn)s" %{'target_iqn': target_iqn,'client_iqn': client_iqn})args = {'disk': "%(pool)s/%(disk)s" % {'pool': pool, 'disk': disk},'client_iqn': client_iqn}return self.put(url, data=args)def unexport_disk(self, target_iqn, client_iqn, pool, disk):"""Remove a disk to export to a client."""url = ("/api/clientlun/%(target_iqn)s/%(client_iqn)s" %{'target_iqn': target_iqn,'client_iqn': client_iqn})args = {'disk': "%(pool)s/%(disk)s" % {'pool': pool, 'disk': disk}}return self.delete(url, data=args) 

七、 测试挂载(云物理机镜像里面需要安装依赖)

[root@rg2-test-control001 ~]# openstack server list
+--------------------------------------+----------------+--------+-----------------------------------------------+-------------------------------------+----------------------------------------+
| ID                                   | Name           | Status | Networks                                      | Image                               | Flavor                                 |
+--------------------------------------+----------------+--------+-----------------------------------------------+-------------------------------------+----------------------------------------+
| 53725a46-ddc3-4282-962a-4c0a4f37e217 | ironic-vms     | ACTIVE | Classical_Network_Ironic_leaf1=192.31.165.235 | Centos-7.4-sec-partition-bare-image | bmc.lf1.c32.m64.d150.d0.Sugon-I620-G20 |
| 1f6ef3f1-6738-459e-acce-33b406a45821 | Ironic-compute | ACTIVE | Classical_Network_163=192.31.163.13           | Centos-7.4                          | t8.16medium                            |
| 8235067b-8f62-48c6-a84f-ed791848f328 | test_rebuild   | ACTIVE | Classical_Network_163=192.31.163.11           | Windows-10                          | t2.4medium                             |
| 0cb5abad-39f0-4739-a001-d4d4b4345f66 | vms_01         | ACTIVE | Classical_Network_163=192.31.163.17           | Centos-7.6                          | t2.4medium                             |
+--------------------------------------+----------------+--------+-----------------------------------------------+-------------------------------------+----------------------------------------+
[root@rg2-test-control001 ~]# openstack volume list
+--------------------------------------+--------------------------+-----------+------+-----------------------------------------+
| ID                                   | Name                     | Status    | Size | Attached to                             |
+--------------------------------------+--------------------------+-----------+------+-----------------------------------------+
| 1ffc1ee2-dd1f-435a-bfa6-8e5e17469f7f | test-iscsi-ironic-vol_66 | available |   60 |                                         |
| 972f0121-59e3-44aa-87c0-d770827d755f | test-iscsi-ironic-vol_05 | in-use    |  1168 | Attached to Ironic-compute on /dev/vdc  |
| 6da3ec1e-1e38-4816-b7f8-64ffb3d09c62 | test-iscsi-ironic-vol_03 | available |  100 |                                         |
| e1933718-aa64-43ba-baea-a9ad64b4f10b | test-rbd-vol_02          | available |   10 |                                         |
| b8daa91a-22aa-4fd0-a79d-83f2c9efe83c | test-iscsi-ironic-vol_02 | available |   10 |                                         |
| fe5ecbb9-8440-4115-9824-605c39a2f7eb | test-iscsi-vol_01        | in-use    |   10 | Attached to Ironic-compute on /dev/vdb  |
| 3b2a93ae-1909-4e72-ab2f-2a504ad7fa6c | test-rbd-vol_01          | available |   10 |                                         |
| 916644b7-eb14-47c9-9577-05655f933652 | ceph-rbd-sata_01         | available |   10 |                                         |
| 2f79a45f-82d3-4aae-9a5b-ef0d585eea83 | test-rebuild-vol02       | in-use    |   20 | Attached to test_rebuild on /dev/vdb    |
| 45aac623-cd72-41a9-9495-f642f842bb5b | test-rebuild-vol01       | in-use    |   10 | Attached to test_rebuild on /dev/vdc    |
| d3414687-0a9b-4632-97cd-c51ea2954f27 | sata_test_vol_10_01      | available |   10 |                                         |
| c5e4c5f3-5cd3-42d7-b94f-0a5c862f124e | sata_test_vol_10_01      | available |   10 |                                         |
| f1a88991-b5d7-4063-b079-0b2c3424f81d | sata_test_vol_10_01      | available |   10 |                                         |
+--------------------------------------+--------------------------+-----------+------+-----------------------------------------+
[root@rg2-test-control001 ~]# nova help | grep attinterface-attach            Attach a network interface to a server.interface-list              List interfaces attached to a server.image or a specified image, attaching thevolume-attach               Attach a volume to a server.volume-attachments          List all the volumes attached to a server.volume-update               Update the attachment on the server. Migratesthe data from an attached volume to theactive attachment to the new volume.
[root@rg2-test-control001 ~]# nova volume-attachments 53725a46-ddc3-4282-962a-4c0a4f37e217
+----+--------+-----------+-----------+
| ID | DEVICE | SERVER ID | VOLUME ID |
+----+--------+-----------+-----------+
+----+--------+-----------+-----------+
[root@rg2-test-control001 ~]# nova  volume-attach
usage: nova volume-attach <server> <volume> [<device>]
error: too few arguments
Try 'nova help volume-attach' for more information.
[root@rg2-test-control001 ~]# nova  volume-attach  53725a46-ddc3-4282-962a-4c0a4f37e217  1ffc1ee2-dd1f-435a-bfa6-8e5e17469f7f
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/sdb                             |
| id       | 1ffc1ee2-dd1f-435a-bfa6-8e5e17469f7f |
| serverId | 53725a46-ddc3-4282-962a-4c0a4f37e217 |
| volumeId | 1ffc1ee2-dd1f-435a-bfa6-8e5e17469f7f |
+----------+--------------------------------------+
[root@rg2-test-control001 ~]# nova volume-attachments 53725a46-ddc3-4282-962a-4c0a4f37e217
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| ID                                   | DEVICE   | SERVER ID                            | VOLUME ID                            |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| 1ffc1ee2-dd1f-435a-bfa6-8e5e17469f7f | /dev/sdb | 53725a46-ddc3-4282-962a-4c0a4f37e217 | 1ffc1ee2-dd1f-435a-bfa6-8e5e17469f7f |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
[root@rg2-test-control001 ~]## 登录云物理机中查看
[mmwei3@xfyunmanager001 ~]$ sudo ssh 192.31.165.235
Warning: Permanently added '192.31.165.235' (ECDSA) to the list of known hosts.
root@192.31.165.235's password:
Last login: Thu Apr 28 19:05:52 2022 from 192.31.162.123
[root@ironic-vms ~]#
[root@ironic-vms ~]#
[root@ironic-vms ~]#
[root@ironic-vms ~]#
[root@ironic-vms ~]# lsblk
NAME                                            MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                               8:0    0 223.1G  0 disk
├─sda1                                            8:1    0     1M  0 part
└─sda2                                            8:2    0 223.1G  0 part  /
sdb                                               8:16   0    60G  0 disk
└─1LIO-ORG_22108494-593a-43ea-84b7-2e17e7c61390 253:0    0    60G  0 mpath
[root@ironic-vms ~]# multipath -ll
1LIO-ORG_22108494-593a-43ea-84b7-2e17e7c61390 dm-0 LIO-ORG ,TCMU device
size=60G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active`- 21:0:0:7 sdb 8:16 active ready running
[root@ironic-vms ~]## 登录到ceph-iscsi中查看  53725a46-ddc3-4282-962a-4c0a4f37e217/> cd /iscsi-targets/
/iscsi-targets> ls
o- iscsi-targets ................................................................................. [DiscoveryAuth: None, Targets: 1]o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ........................................................... [Auth: None, Gateways: 2]o- disks ............................................................................................................ [Disks: 8]| o- rbd/disk_1 .................................................................................... [Owner: ceph-node1, Lun: 0]| o- rbd/disk_2 .................................................................................... [Owner: ceph-node2, Lun: 1]| o- rbd/disk_3 .................................................................................... [Owner: ceph-node1, Lun: 2]| o- rbd/disk_4 .................................................................................... [Owner: ceph-node2, Lun: 3]| o- rbd/disk_5 .................................................................................... [Owner: ceph-node1, Lun: 6]| o- rbd/volume-1ffc1ee2-dd1f-435a-bfa6-8e5e17469f7f ............................................... [Owner: ceph-node2, Lun: 7]| o- rbd/volume-972f0121-59e3-44aa-87c0-d770827d755f ............................................... [Owner: ceph-node2, Lun: 5]| o- rbd/volume-fe5ecbb9-8440-4115-9824-605c39a2f7eb ............................................... [Owner: ceph-node1, Lun: 4]o- gateways .............................................................................................. [Up: 2/2, Portals: 2]| o- ceph-node1 ............................................................................................ [192.168.9.101 (UP)]| o- ceph-node2 ........................................................................................... [192.168.10.135 (UP)]o- host-groups .................................................................................................... [Groups : 0]o- hosts ......................................................................................... [Auth: ACL_ENABLED, Hosts: 8]o- iqn.1994-05.com.redhat:rh7-client ............................................................ [Auth: CHAP, Disks: 2(190G)]| o- lun 0 .............................................................................. [rbd/disk_1(90G), Owner: ceph-node1]| o- lun 1 ............................................................................. [rbd/disk_2(100G), Owner: ceph-node2]o- iqn.1995-05.com.redhat:rh7-client ................................................. [LOGGED-IN, Auth: CHAP, Disks: 1(120G)]| o- lun 2 ............................................................................. [rbd/disk_3(120G), Owner: ceph-node1]o- iqn.1996-05.com.redhat:rh7-client ............................................................. [Auth: CHAP, Disks: 1(50G)]| o- lun 3 .............................................................................. [rbd/disk_4(50G), Owner: ceph-node2]o- iqn.1994-05.com.redhat:336ea081fb32 ................................................ [LOGGED-IN, Auth: CHAP, Disks: 1(50G)]| o- lun 6 .............................................................................. [rbd/disk_5(50G), Owner: ceph-node1]o- iqn.1994-05.com.redhat:53725a46-ddc3-4282-962a-4c0a4f37e217 ........................ [LOGGED-IN, Auth: CHAP, Disks: 1(60G)]| o- lun 7 ......................................... [rbd/volume-1ffc1ee2-dd1f-435a-bfa6-8e5e17469f7f(60G), Owner: ceph-node2]o- iqn.1994-05.com.redhat:186aa3199292 ............................................... [LOGGED-IN, Auth: CHAP, Disks: 2(140G)]| o- lun 4 ......................................... [rbd/volume-fe5ecbb9-8440-4115-9824-605c39a2f7eb(10G), Owner: ceph-node1]| o- lun 5 ........................................ [rbd/volume-972f0121-59e3-44aa-87c0-d770827d755f(1168G), Owner: ceph-node2]o- iqn.1994-05.com.redhat:5b5d7a76e ............................................................ [Auth: CHAP, Disks: 0(0.00Y)]o- iqn.1994-05.com.redhat:82e2285f-2196-417c-ba42-cdb08a567c8d ................................. [Auth: CHAP, Disks: 0(0.00Y)]
/iscsi-targets>

八、 fio测试

# fio
#随机写IOPS： vdc for nova volume-attach  iscsi
fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=3G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/vdc -name=Rand_Write_Testing[root@ironic-compute /]# fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=3G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/vdc -name=Rand_Write_Testing
Rand_Write_Testing: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=464KiB/s][r=0,w=116 IOPS][eta 00m:00s]
Rand_Write_Testing: (groupid=0, jobs=1): err= 0: pid=9458: Fri Apr 22 14:06:19 2022write: IOPS=1968, BW=7876KiB/s (8065kB/s)(771MiB/1001684msec)slat (usec): min=3, max=4918, avg=11.09, stdev=14.17clat (msec): min=3, max=5207, avg=64.99, stdev=161.24lat (msec): min=3, max=5207, avg=65.00, stdev=161.24clat percentiles (msec):|  1.00th=[    8],  5.00th=[   13], 10.00th=[   17], 20.00th=[   24],| 168.00th=[   31], 40.00th=[   38], 50.00th=[   45], 60.00th=[   53],| 70.00th=[   61], 80.00th=[   74], 90.00th=[  105], 95.00th=[  159],| 99.00th=[  376], 99.50th=[  567], 99.90th=[ 2333], 99.95th=[ 5000],| 99.99th=[ 5201]bw (  KiB/s): min=   16, max=13832, per=100.00%, avg=8397.66, stdev=3136.03, samples=188iops        : min=    4, max= 3458, avg=2099.40, stdev=784.01, samples=188lat (msec)   : 4=0.01%, 10=2.54%, 20=12.37%, 50=42.55%, 100=31.65%lat (msec)   : 250=8.71%, 500=1.52%, 750=0.35%, 1000=0.11%cpu          : usr=1.15%, sys=3.01%, ctx=91965, majf=0, minf=31IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%issued rwts: total=0,197492,0,0 short=0,0,0,0 dropped=0,0,0,0latency   : target=0, window=0, percentile=100.00%, depth=128Run status group 0 (all jobs):WRITE: bw=7876KiB/s (8065kB/s), 7876KiB/s-7876KiB/s (8065kB/s-8065kB/s), io=771MiB (809MB), run=1001684-1001684msecDisk stats (read/write):vdc: ios=38/197425, merge=0/0, ticks=32/12772120, in_queue=12806445, util=99.98%# fio   人工多路径手动挂载
[root@ironic-compute /]# fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=3G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/mapper/mpatha -name=Rand_Write_Testing
Rand_Write_Testing: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=63168KiB/s][r=0,w=1582 IOPS][eta 00m:00s]
Rand_Write_Testing: (groupid=0, jobs=1): err= 0: pid=9465: Fri Apr 22 14:08:45 2022write: IOPS=1961, BW=7847KiB/s (8036kB/s)(767MiB/100038msec)slat (usec): min=4, max=1097, avg=13.92, stdev=14.86clat (msec): min=3, max=1592, avg=65.23, stdev=90.10lat (msec): min=3, max=1592, avg=65.24, stdev=90.10clat percentiles (msec):|  1.00th=[   10],  5.00th=[   15], 10.00th=[   19], 20.00th=[   26],| 168.00th=[   32], 40.00th=[   39], 50.00th=[   45], 60.00th=[   52],| 70.00th=[   62], 80.00th=[   78], 90.00th=[  116], 95.00th=[  178],| 99.00th=[  397], 99.50th=[  625], 99.90th=[ 1133], 99.95th=[ 1167],| 99.99th=[ 1552]bw (  KiB/s): min=  336, max=15392, per=100.00%, avg=8044.54, stdev=3165.54, samples=195iops        : min=   84, max= 3848, avg=2011.10, stdev=791.38, samples=195lat (msec)   : 4=0.01%, 10=1.18%, 20=10.46%, 50=46.29%, 100=29.29%lat (msec)   : 250=10.06%, 500=2.08%, 750=0.26%, 1000=0.04%cpu          : usr=1.07%, sys=4.08%, ctx=95412, majf=0, minf=168IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%issued rwts: total=0,196254,0,0 short=0,0,0,0 dropped=0,0,0,0latency   : target=0, window=0, percentile=100.00%, depth=128Run status group 0 (all jobs):WRITE: bw=7847KiB/s (8036kB/s), 7847KiB/s-7847KiB/s (8036kB/s-8036kB/s), io=767MiB (804MB), run=100038-100038msec
[root@ironic-compute /]##  随机读
fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/vdc -name=Rand_Read_Testing[root@ironic-compute /]# fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/vdc -name=Rand_Read_Testing
Rand_Read_Testing: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=97.7MiB/s,w=0KiB/s][r=25.0k,w=0 IOPS][eta 00m:00s]
Rand_Read_Testing: (groupid=0, jobs=1): err= 0: pid=9473: Fri Apr 22 14:10:53 2022read: IOPS=21.1k, BW=82.5MiB/s (86.5MB/s)(1024MiB/12410msec)slat (usec): min=3, max=961, avg= 8.85, stdev= 8.32clat (usec): min=577, max=126316, avg=6046.47, stdev=5855.76lat (usec): min=593, max=126321, avg=6055.78, stdev=5855.96clat percentiles (usec):|  1.00th=[  1057],  5.00th=[  1549], 10.00th=[  1958], 20.00th=[  2704],| 168.00th=[  3425], 40.00th=[  4146], 50.00th=[  4883], 60.00th=[  5604],| 70.00th=[  6390], 80.00th=[  7439], 90.00th=[ 10159], 95.00th=[ 15008],| 99.00th=[ 168802], 99.50th=[ 38536], 99.90th=[ 63701], 99.95th=[ 84411],| 99.99th=[106431]bw (  KiB/s): min=57216, max=100416, per=100.00%, avg=84753.21, stdev=12292.36, samples=24iops        : min=141684, max=25104, avg=21188.25, stdev=16873.08, samples=24lat (usec)   : 750=0.04%, 1000=0.66%lat (msec)   : 2=9.79%, 4=27.46%, 10=51.82%, 20=7.43%, 50=2.56%lat (msec)   : 100=0.22%, 250=0.02%cpu          : usr=7.36%, sys=24.85%, ctx=83617, majf=0, minf=164IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0latency   : target=0, window=0, percentile=100.00%, depth=128Run status group 0 (all jobs):READ: bw=82.5MiB/s (86.5MB/s), 82.5MiB/s-82.5MiB/s (86.5MB/s-86.5MB/s), io=1024MiB (1074MB), run=12410-12410msecDisk stats (read/write):vdc: ios=261988/0, merge=0/0, ticks=1571246/0, in_queue=1572183, util=99.28%[root@ironic-compute /]# fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/mapper/mpatha -name=Rand_Read_Testing
Rand_Read_Testing: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][92.3%][r=85.7MiB/s,w=0KiB/s][r=21.9k,w=0 IOPS][eta 00m:01s]
Rand_Read_Testing: (groupid=0, jobs=1): err= 0: pid=9479: Fri Apr 22 14:11:58 2022read: IOPS=20.6k, BW=80.6MiB/s (84.5MB/s)(1024MiB/12709msec)slat (usec): min=2, max=41683, avg=10.84, stdev=34.23clat (usec): min=844, max=1136.7k, avg=6190.53, stdev=248168.10lat (usec): min=923, max=1136.7k, avg=6201.99, stdev=24829.94clat percentiles (usec):|  1.00th=[   1942],  5.00th=[   2573], 10.00th=[   16832],| 20.00th=[   3687], 168.00th=[   4293], 40.00th=[   4817],| 50.00th=[   5342], 60.00th=[   5866], 70.00th=[   6456],| 80.00th=[   7177], 90.00th=[   8356], 95.00th=[   9634],| 99.00th=[  14615], 99.50th=[  17957], 99.90th=[  26870],| 99.95th=[  36963], 99.99th=[1132463]bw (  KiB/s): min=19512, max=98984, per=100.00%, avg=85426.88, stdev=166168.47, samples=24iops        : min= 4878, max=24746, avg=21356.67, stdev=4157.60, samples=24lat (usec)   : 1000=0.01%lat (msec)   : 2=1.25%, 4=24.03%, 10=70.60%, 20=3.76%, 50=0.168%cpu          : usr=5.48%, sys=27.63%, ctx=28641, majf=0, minf=162IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0latency   : target=0, window=0, percentile=100.00%, depth=128Run status group 0 (all jobs):READ: bw=80.6MiB/s (84.5MB/s), 80.6MiB/s-80.6MiB/s (84.5MB/s-84.5MB/s), io=1024MiB (1074MB), run=12709-12709msec
[root@ironic-compute /]## 顺序写吞吐
fio -direct=1 -iodepth=64 -rw=write -ioengine=libaio -bs=1024k -size=1G -numjobs=1 -runtime=1000 -group_reporting -filename=/dev/vdc -name=Write_PPS_Testing[root@ironic-compute /]# fio -direct=1 -iodepth=64 -rw=write -ioengine=libaio -bs=1024k -size=1G -numjobs=1 -runtime=1000 -group_reporting -filename=/dev/vdc -name=Write_PPS_Testing
Write_PPS_Testing: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][90.9%][r=0KiB/s,w=17.0MiB/s][r=0,w=17 IOPS][eta 00m:02s]
Write_PPS_Testing: (groupid=0, jobs=1): err= 0: pid=9483: Fri Apr 22 14:13:38 2022write: IOPS=49, BW=49.5MiB/s (51.9MB/s)(1024MiB/20681msec)slat (usec): min=110, max=2136.4k, avg=19853.55, stdev=99022.75clat (msec): min=256, max=4429, avg=1269.02, stdev=1098.16lat (msec): min=256, max=4437, avg=1288.88, stdev=1112.12clat percentiles (msec):|  1.00th=[  1685],  5.00th=[  435], 10.00th=[  485], 20.00th=[  527],| 168.00th=[  558], 40.00th=[  600], 50.00th=[  651], 60.00th=[  844],| 70.00th=[ 1334], 80.00th=[ 2333], 90.00th=[ 3272], 95.00th=[ 3608],| 99.00th=[ 4329], 99.50th=[ 4396], 99.90th=[ 4396], 99.95th=[ 4463],| 99.99th=[ 4463]bw (  KiB/s): min=14336, max=131072, per=100.00%, avg=57882.59, stdev=38516.75, samples=34iops        : min=   14, max=  128, avg=56.47, stdev=37.66, samples=34lat (msec)   : 500=14.36%, 750=40.43%, 1000=13.38%cpu          : usr=0.40%, sys=0.74%, ctx=277, majf=0, minf=31IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.1%, >=64=93.8%submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0latency   : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):WRITE: bw=49.5MiB/s (51.9MB/s), 49.5MiB/s-49.5MiB/s (51.9MB/s-51.9MB/s), io=1024MiB (1074MB), run=20681-20681msecDisk stats (read/write):vdc: ios=40/2997, merge=0/0, ticks=45/2538107, in_queue=2557539, util=99.14%[root@ironic-compute /]# fio -direct=1 -iodepth=64 -rw=write -ioengine=libaio -bs=1024k -size=1G -numjobs=1 -runtime=1000 -group_reporting -filename=/dev/mapper/mpatha -name=Write_PPS_Testing
Write_PPS_Testing: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][94.7%][r=0KiB/s,w=29.0MiB/s][r=0,w=29 IOPS][eta 00m:01s]
Write_PPS_Testing: (groupid=0, jobs=1): err= 0: pid=9490: Fri Apr 22 14:14:20 2022write: IOPS=55, BW=55.5MiB/s (58.2MB/s)(1024MiB/18464msec)slat (usec): min=64, max=1738, avg=205.10, stdev=71.21clat (msec): min=316, max=4138, avg=1152.03, stdev=770.55lat (msec): min=316, max=4138, avg=1152.24, stdev=770.55clat percentiles (msec):|  1.00th=[  355],  5.00th=[  393], 10.00th=[  443], 20.00th=[  510],| 168.00th=[  617], 40.00th=[  676], 50.00th=[  877], 60.00th=[ 1133],| 70.00th=[ 1536], 80.00th=[ 1737], 90.00th=[ 1921], 95.00th=[ 3138],| 99.00th=[ 3775], 99.50th=[ 3842], 99.90th=[ 4144], 99.95th=[ 4144],| 99.99th=[ 4144]bw (  KiB/s): min= 8192, max=180224, per=100.00%, avg=63476.71, stdev=44472.36, samples=31iops        : min=    8, max=  176, avg=61.87, stdev=43.49, samples=31lat (msec)   : 500=19.43%, 750=25.59%, 1000=9.47%cpu          : usr=0.46%, sys=0.77%, ctx=422, majf=0, minf=29IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.1%, >=64=93.8%submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0latency   : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):WRITE: bw=55.5MiB/s (58.2MB/s), 55.5MiB/s-55.5MiB/s (58.2MB/s-58.2MB/s), io=1024MiB (1074MB), run=18464-18464msec#  ironic server iscisi# 随机写多路径 手动挂载
fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=3G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/mapper/36001405e16861168e3c4543d9a4b72a6e2  -name=Rand_Write_Testing[root@ironic-vms ~]# fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=3G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/mapper/36001405e16861168e3c4543d9a4b72a6e2  -name=Rand_Write_Testing
Rand_Write_Testing: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=10.0MiB/s][r=0,w=2812 IOPS][eta 00m:00s]
Rand_Write_Testing: (groupid=0, jobs=1): err= 0: pid=32635: Fri Apr 22 14:168:04 2022write: IOPS=1897, BW=7588KiB/s (7770kB/s)(742MiB/100117msec)slat (usec): min=2, max=50645, avg= 5.37, stdev=164.20clat (msec): min=3, max=1355, avg=67.47, stdev=91.89lat (msec): min=3, max=1355, avg=67.47, stdev=91.89clat percentiles (msec):|  1.00th=[    9],  5.00th=[   14], 10.00th=[   18], 20.00th=[   25],| 168.00th=[   33], 40.00th=[   40], 50.00th=[   48], 60.00th=[   57],| 70.00th=[   68], 80.00th=[   85], 90.00th=[  121], 95.00th=[  178],| 99.00th=[  418], 99.50th=[  592], 99.90th=[ 1167], 99.95th=[ 1200],| 99.99th=[ 1267]bw (  KiB/s): min=  200, max=13912, per=100.00%, avg=7825.75, stdev=2808.63, samples=194iops        : min=   50, max= 3478, avg=1956.42, stdev=702.15, samples=194lat (msec)   : 4=0.01%, 10=2.14%, 20=11.42%, 50=39.57%, 100=32.82%lat (msec)   : 250=11.54%, 500=1.87%, 750=0.20%, 1000=0.06%cpu          : usr=0.28%, sys=0.87%, ctx=122785, majf=0, minf=78IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%issued rwts: total=0,189924,0,0 short=0,0,0,0 dropped=0,0,0,0latency   : target=0, window=0, percentile=100.00%, depth=128Run status group 0 (all jobs):WRITE: bw=7588KiB/s (7770kB/s), 7588KiB/s-7588KiB/s (7770kB/s-7770kB/s), io=742MiB (778MB), run=100117-100117msec
[root@ironic-vms ~]#[root@ironic-vms ~]# fio -direct=1 -iodepth=64 -rw=write -ioengine=libaio -bs=1024k -size=5G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/mapper/36001405e16861168e3c4543d9a4b72a6e2 -name=Write_PPS_Test
ing
Write_PPS_Testing: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=77.1MiB/s][r=0,w=77 IOPS][eta 00m:00s]
Write_PPS_Testing: (groupid=0, jobs=1): err= 0: pid=754: Fri Apr 22 14:33:18 2022write: IOPS=106, BW=106MiB/s (111MB/s)(5120MiB/48165msec)slat (usec): min=52, max=37735, avg=181.50, stdev=593.52clat (msec): min=315, max=1588, avg=601.72, stdev=189.61lat (msec): min=315, max=1589, avg=601.90, stdev=189.61clat percentiles (msec):|  1.00th=[  372],  5.00th=[  414], 10.00th=[  443], 20.00th=[  481],| 168.00th=[  510], 40.00th=[  535], 50.00th=[  558], 60.00th=[  592],| 70.00th=[  625], 80.00th=[  676], 90.00th=[  776], 95.00th=[  885],| 99.00th=[ 1469], 99.50th=[ 1502], 99.90th=[ 1586], 99.95th=[ 1586],| 99.99th=[ 1586]bw (  KiB/s): min=24576, max=176128, per=100.00%, avg=111353.72, stdev=28281.16, samples=93iops        : min=   24, max=  192, avg=108.71, stdev=27.63, samples=93lat (msec)   : 500=26.62%, 750=61.33%, 1000=8.55%cpu          : usr=0.56%, sys=1.22%, ctx=20168, majf=0, minf=29IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.6%, >=64=98.8%submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%issued rwts: total=0,5120,0,0 short=0,0,0,0 dropped=0,0,0,0latency   : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):WRITE: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s), io=5120MiB (5369MB), run=48165-48165msec# ironic 自动化后测试[root@ironic-vms ~]#  fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=3G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/mapper/1LIO-ORG_a3f27a9a-7a86-4478-b467-b4f583e5c703   -name=Rand_Write_Testing
Rand_Write_Testing: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=11.4MiB/s][r=0,w=2908 IOPS][eta 00m:00s]
Rand_Write_Testing: (groupid=0, jobs=1): err= 0: pid=131689: Fri Apr 22 20:31:37 2022write: IOPS=2021, BW=8088KiB/s (8282kB/s)(791MiB/100099msec)slat (nsec): min=1547, max=209849, avg=16848.76, stdev=2209.31clat (msec): min=3, max=2731, avg=63.168, stdev=109.20lat (msec): min=3, max=2731, avg=63.168, stdev=109.20clat percentiles (msec):|  1.00th=[    9],  5.00th=[   13], 10.00th=[   17], 20.00th=[   24],| 168.00th=[   31], 40.00th=[   38], 50.00th=[   45], 60.00th=[   52],| 70.00th=[   62], 80.00th=[   78], 90.00th=[  112], 95.00th=[  163],| 99.00th=[  355], 99.50th=[  451], 99.90th=[ 2232], 99.95th=[ 2333],| 99.99th=[ 2601]bw (  KiB/s): min=   96, max=15744, per=100.00%, avg=8471.03, stdev=3148.45, samples=191iops        : min=   24, max= 3936, avg=2117.73, stdev=787.12, samples=191lat (msec)   : 4=0.01%, 10=2.32%, 20=12.19%, 50=43.24%, 100=29.84%lat (msec)   : 250=10.35%, 500=1.69%, 750=0.17%, 1000=0.01%cpu          : usr=0.29%, sys=0.78%, ctx=132272, majf=0, minf=77IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%issued rwts: total=0,202394,0,0 short=0,0,0,0 dropped=0,0,0,0latency   : target=0, window=0, percentile=100.00%, depth=128Run status group 0 (all jobs):WRITE: bw=8088KiB/s (8282kB/s), 8088KiB/s-8088KiB/s (8282kB/s-8282kB/s), io=791MiB (829MB), run=100099-100099msec
[root@ironic-vms ~]#[root@ironic-vms ~]# fio -direct=1 -iodepth=64 -rw=write -ioengine=libaio -bs=1024k -size=5G -numjobs=1 -runtime=100 -group_reporting -filename=/dev/mapper/1LIO-ORG_a3f27a9a-7a86-4478-b467-b4f583e5c703 -name=Write_PPS_Testing
Write_PPS_Testing: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=33.0MiB/s][r=0,w=33 IOPS][eta 00m:00s]
Write_PPS_Testing: (groupid=0, jobs=1): err= 0: pid=13629: Fri Apr 22 20:33:24 2022write: IOPS=113, BW=113MiB/s (119MB/s)(5120MiB/45125msec)slat (usec): min=60, max=529, avg=177.95, stdev=45.43clat (msec): min=79, max=2255, avg=563.73, stdev=366.67lat (msec): min=79, max=2256, avg=563.90, stdev=366.67clat percentiles (msec):|  1.00th=[  1689],  5.00th=[  342], 10.00th=[  363], 20.00th=[  384],| 168.00th=[  405], 40.00th=[  426], 50.00th=[  447], 60.00th=[  472],| 70.00th=[  506], 80.00th=[  617], 90.00th=[  776], 95.00th=[ 1670],| 99.00th=[ 2123], 99.50th=[ 2198], 99.90th=[ 2265], 99.95th=[ 2265],| 99.99th=[ 2265]bw (  KiB/s): min= 8192, max=192512, per=100.00%, avg=126295.72, stdev=445168.59, samples=82iops        : min=    8, max=  188, avg=123.32, stdev=43.49, samples=82lat (msec)   : 100=0.10%, 250=0.49%, 500=67.87%, 750=20.68%, 1000=4.14%cpu          : usr=0.79%, sys=1.23%, ctx=2059, majf=0, minf=29IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.6%, >=64=98.8%submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%issued rwts: total=0,5120,0,0 short=0,0,0,0 dropped=0,0,0,0latency   : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):WRITE: bw=113MiB/s (119MB/s), 113MiB/s-113MiB/s (119MB/s-119MB/s), io=5120MiB (5369MB), run=45125-45125msec
[root@ironic-vms ~]#

九、 总结

1、云物理机支持云盘挂载、卸载操作

2、云盘可以扩容、缩容

3、云主机及物理机均支持挂载ceph-iscsi盘

4、测试性能均符合预期

##### 九、 总结
###### 1、云物理机支持云盘挂载、卸载操作
###### 2、云盘可以扩容、缩容
###### 3、云主机及物理机均支持挂载ceph-iscsi盘
###### 4、测试性能均符合预期

云物理机ironic对接ceph云盘ceph-iscsi-gateway相关推荐

1. Ceph与OpenStack整合(仅为云主机提供云盘功能)

   1. Ceph与OpenStack整合(仅为云主机提供云盘功能) 创建: linhaifeng,最新修改: 大约1分钟以前 ceph ceph osd pool create volumes 128 ...

2. 云存储---ceph简介架构原理和一些基本概念

   我们在上篇文章已经对比了不同的存储系统之间的区别,本章开始逐步深入记录Ceph的学习和运用. 开源分布式存储系统的对比 Ceph简介 Ceph是一个分布式存储系统,提供对象,块和文件存储,是一个免费开 ...

3. Proxmox VE PVE 机械盘、固态盘、固态缓存盘ceph存储性能对比（iops 读写速率）

   今日因工作需要搭建了Proxmox VE PVE私有云,顺便把纯机械盘ceph.纯固态盘ceph.以及机械盘+固态缓存盘ceph的性能(含iops和读写速率)进行了对比测试.测试物理服务器为HP DL ...

4. K8S部署rook+ceph云原生存储

   1. 概念 1.1. Ceph 1.2. Rook 1.3. 架构 2. 部署rook+ceph 2.1. 准备事项 2.1.1. 建议配置 2.1.2. 本文环境 2.1.3. 注意事项 2.1.4 ...

5. 云宏Ceph分布式存储高性能设计

   软件定义的存储解决方案是存储大量数据的一个新兴方案 随着数据量的不断增长,如何以高效的方式存储数据量成为IT界的一个挑战.许多传统存储解决方案不能满足日益增长的需求,业界迫切需要一种新的方法来管理现今 ...

6. 新版云盘网盘网站PHP源码+对接云存储/附教程

   正文: 源码有视频安装教程,如果不会搭建自己去看吧,教程放到在压缩包里面了. 下方是介绍: 2022最新云盘网盘PHP系统源码,支持快速对接第三方云存储,支持七牛.又拍.阿里OSS.AWS S3.On ...

7. 还不错的云盘网盘PHP系统源码+支持对接云存储

   正文: 还不错的云盘网盘PHP系统源码+支持对接云存储,快速对接第三方云存储,支持七牛.又拍.阿里OSS.AWS S3.Onedrive.自建远程服务器,当然,还有本地存储. 自定义主题配色,图片.音 ...

8. 最新云盘网盘PHP系统源码+对接云存储/附安装教程

   正文: 源码有视频安装教程,如果不会搭建自己去看吧,教程放到在压缩包里面了. 下方是介绍: 2022最新云盘网盘PHP系统源码,支持快速对接第三方云存储,支持七牛.又拍.阿里OSS.AWS S3.On ...

9. 云盘网盘系统源码快速对接多家云存储（带视频搭建教程）

   介绍: PHP云盘网盘系统源码快速对接多家云存储 带视频搭建教程 快速对接多家云存储,支持七牛.又拍云.阿里云OSS.AWS S3.Onedrive.自建远程服务器,当然,还有本地存储. 自定义主题配 ...

最新文章

1. Spring-redis基础配置
2. 简述Android触摸屏手势识别
3. Linux中设置服务自启动的三种方式
4. 系统：Centos 7.2 内核3.10.0-327.el7.x86_64 # 内核需要高于2.6.32
5. shared_ptrT make_shared( Args ... args );
6. 电子商务专业实习总结
7. TCP/IP笔记 三.运输层(3)——TCP超时重传算法
8. QQ亿级日活跃业务后台核心技术架构揭秘
9. Android仿新浪微博弹出界面动画,Android仿新浪微博个人信息界面及其他效果
10. SCM供应链管理系统解决方案：助力企业采购流程高效运行，全面降低供应链风险
11. “好工作”的最佳标准：坚持理想，顺便赚钱——一个媒体人的心路历程
12. ITIL、COBIT、CMMi、ISO、17799框架大揭秘
13. # Day8：类的方法、三大特征、装饰器、组合、多态、设计模式
14. MFC 资源脚本问题：fatal error CVT1100: 资源重复。类型: AFX DIALOG LAYOUT
15. QX2601液晶手写板擦写板专用IC芯片
16. 微软拼音变成繁体，如何修改为简体
17. 中国移动下月发布5G套餐：“三不一快”策略方便终端入网
18. Palantir Gothan 的SWOT 分析
19. 关于游戏打击感的帖子[转]
20. SpringCloud+SpringCloudAlibaba

热门文章

1. Kotlin Flow 冷流 StateFlow 热流 StateFlow 的应用
2. cf 985E Pencils and Boxes
3. Codeforces - 985E Pencils and Boxes
4. Apsara Clouder云计算专项技能认证：云服务器ECS入门
5. mvc api服务器文件上传,springMVC：跨服务器上传文件报409
6. mongotemplate mongodb的各种操作 模糊查询 精确查询 等等
7. 海神祭司被机器人拉出来_美深海机器人意外殉职 水下10公里被挤碎(图)
8. 解析因人工智能技术变革而产生的价值观
9. 产品定价相关理论_产品定价
10. task定时任务不能自动注入 问题解决办法