Oracle RAC集群增加新共享硬盘并使用AFD的式增加新ASM磁盘组

此实验是在ASM共享磁盘的RAC环境下增加新共享硬盘并新建ASM磁盘组创建相应的表空间和表数据进行测试过程。参考资料包括了https://www.oracle.com/cn/technical-resources/article/database/asmfd.html Oracle ASM Filter Driver相关资料。

具体实验记录如下：

实验环境背景

数据库集群版本

[grid@rac2 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [19.0.0.0.0]

ASM磁盘开启的文件

ASMCMD> lsof
DB_Name  Instance_Name  Path
+ASM     +ASM2          +ORS.255.1038877513
racdb    racdb2         +DATA/RACDB/86B637B62FE07A65E053F706E80A27CA/DATAFILE/sysaux.266.1038939581
racdb    racdb2         +DATA/RACDB/86B637B62FE07A65E053F706E80A27CA/DATAFILE/system.265.1038939581
racdb    racdb2         +DATA/RACDB/86B637B62FE07A65E053F706E80A27CA/DATAFILE/undotbs1.267.1038939581
racdb    racdb2         +DATA/RACDB/A4582AEF3E8210F9E0533C23100A3339/TEMPFILE/temp.268.1038939611
racdb    racdb2         +DATA/RACDB/CONTROLFILE/current.261.1038938883
racdb    racdb2         +DATA/RACDB/DATAFILE/sysaux.258.1038938777
racdb    racdb2         +DATA/RACDB/DATAFILE/system.257.1038938723
racdb    racdb2         +DATA/RACDB/DATAFILE/undotbs1.259.1038938803
racdb    racdb2         +DATA/RACDB/DATAFILE/undotbs2.269.1038940101
racdb    racdb2         +DATA/RACDB/DATAFILE/users.260.1038938803
racdb    racdb2         +DATA/RACDB/ONLINELOG/group_1.262.1038938887
racdb    racdb2         +DATA/RACDB/ONLINELOG/group_2.263.1038938887
racdb    racdb2         +DATA/RACDB/ONLINELOG/group_3.270.1038940635
racdb    racdb2         +DATA/RACDB/ONLINELOG/group_4.271.1038940637
racdb    racdb2         +DATA/RACDB/TEMPFILE/temp.264.1038938901

ASM客户端信息

ASMCMD> lsct
DB_Name  Status     Software_Version  Compatible_version  Instance_Name  Disk_Group
+ASM     CONNECTED        19.0.0.0.0          19.0.0.0.0  +ASM2          ORS
+ASM     CONNECTED        19.0.0.0.0          19.0.0.0.0  +ASM2          DATA
_OCR     CONNECTED                 -                   -  rac2           ORS
racdb    CONNECTED        19.0.0.0.0          19.0.0.0.0  racdb2         DATA

实验步骤

新增共享硬盘

本次新增硬盘到添加ASM新的磁盘的整个过程和创建搭建数据库时的ASM磁盘组类似，最后会有尝试使用ASMFD的新功能(这是一个可以取代 ASMLIB 和 udev 设置的新功能)直接绑定新硬盘然后加入数据库来使用。

节点1:新增硬盘9，容量为50G

节点2:使用节点1的硬盘9，容量为50G

使用udev配置Asm磁盘(两节点都做)

虚拟机在要之前配置共享存储时在各自的虚拟机安装路径中的vmx文件中添加disk.enableUUID = "TRUE"才会显示UUID。udev是一种工具，它能够根据系统中的硬件设备的状态动态更新设备文件，包括设备文件的创建，删除等。设备文件通常放在/dev目录下。使用udev后，在/dev目录下就只包含系统中真正存在的设备。

 [root@rac1 ~]# fdisk -l    //查看新增硬盘Disk /dev/sdi: 53.7 GB, 53687091200 bytes, 104857600 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytes//按步骤增加磁盘的分区1root@rac1 ~]# fdisk /dev/sdiWelcome to fdisk (util-linux 2.23.2).Changes will remain in memory only, until you decide to write them.Be careful before using the write command.Device does not contain a recognized partition tableBuilding a new DOS disklabel with disk identifier 0x9a414875.Command (m for help): nPartition type:p   primary (0 primary, 0 extended, 4 free)e   extendedSelect (default p): Using default response pPartition number (1-4, default 1): First sector (2048-104857599, default 2048): Using default value 2048Last sector, +sectors or +size{K,M,G} (2048-104857599, default 104857599): Using default value 104857599Partition 1 of type Linux and of size 50 GiB is setCommand (m for help): wThe partition table has been altered!Calling ioctl() to re-read partition table.Syncing disks.

两节点分别fdisk -l ，显示都一致

 Disk /dev/sdi: 53.7 GB, 53687091200 bytes, 104857600 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0x9a414875Device Boot      Start         End      Blocks   Id  System/dev/sdi1            2048   104857599    52427776   83  Linux

但是节点2没有sdi1 且没有UUID显示

 [root@rac2 ~]# partprobe /dev/sdsda   sda1  sda2  sdb   sdb1  sdc   sdc1  sdd   sdd1  sde   sde1  sdf   sdf1  sdg   sdg1  sdh   sdh1  sdi   [root@rac2 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdi1[root@rac2 ~]# [root@rac2 ~]# partprobe /dev/sdi  //通知内核sdi盘变化 [root@rac2 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdi1         //可以出现UUID代表正常36000c295d9c2913ec150e329c3465337

新增物理盘为ASM磁盘组成员，节点1：

 [root@rac1 ~]# vi sdi1.sh#!/bin/bashfor i in i ;doecho "KERNEL==\"sd?1\", SUBSYSTEM==\"block\", PROGRAM==\"/usr/lib/udev/scsi_id -g -u -d /dev/\$parent\", RESULT==\"`/usr/lib/udev/scsi_id -g -u -d /dev/sd\$i`\", SYMLINK+=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\""done[root@rac1 ~]# ./sdi1.sh KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c295d9c2913ec150e329c3465337", SYMLINK+="asm-diski", OWNER="grid", GROUP="asmadmin", MODE="0660"[root@rac1 ~]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules  //新增上述结果[root@rac1 ~]# udevadm control --reload[root@rac1 ~]# udevadm trigger[root@rac1 ~]# ls -l /dev/sd?1  brw-rw---- 1 root disk     8,   1 May 13 11:37 /dev/sda1brw-rw---- 1 grid asmadmin 8,  17 May 13 11:37 /dev/sdb1brw-rw---- 1 grid asmadmin 8,  33 May 13 11:37 /dev/sdc1brw-rw---- 1 grid asmadmin 8,  49 May 13 11:37 /dev/sdd1brw-rw---- 1 grid asmadmin 8,  65 May 13 11:37 /dev/sde1brw-rw---- 1 grid asmadmin 8,  81 May 13 11:37 /dev/sdf1brw-rw---- 1 grid asmadmin 8,  97 May 13 11:37 /dev/sdg1brw-rw---- 1 root disk     8, 113 May 13 11:37 /dev/sdh1brw-rw---- 1 grid asmadmin 8, 129 May 13 11:37 /dev/sdi1

节点2：

 [root@rac2 ~]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules  //增加上述结果[root@rac2 ~]# udevadm control --reload[root@rac2 ~]# udevadm trigger[root@rac2 ~]# ls -l /dev/sd?1brw-rw---- 1 root disk     8,   1 May 13 11:51 /dev/sda1brw-rw---- 1 grid asmadmin 8,  17 May 13 11:51 /dev/sdb1brw-rw---- 1 grid asmadmin 8,  33 May 13 11:51 /dev/sdc1brw-rw---- 1 grid asmadmin 8,  49 May 13 11:51 /dev/sdd1brw-rw---- 1 grid asmadmin 8,  65 May 13 11:51 /dev/sde1brw-rw---- 1 grid asmadmin 8,  81 May 13 11:51 /dev/sdf1brw-rw---- 1 grid asmadmin 8,  97 May 13 11:51 /dev/sdg1brw-rw---- 1 root disk     8, 113 May 13 11:51 /dev/sdh1brw-rw---- 1 grid asmadmin 8, 129 May 13 11:51 /dev/sdi1

查看现有ASM磁盘组的情况

 [grid@rac1 ~]$ sqlplus  / as sysasmSQL*Plus: Release 19.0.0.0.0 - Production on Wed May 13 13:00:57 2020Version 19.3.0.0.0Copyright (c) 1982, 2019, Oracle.  All rights reserved.Connected to:Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - ProductionVersion 19.3.0.0.0SQL> show parameter asm_diskstring;   //查看磁盘组支持的模式NAME                                 TYPE        VALUE------------------------------------ ----------- ------------------------------asm_diskstring                       string      /dev/asm*, AFD:*SQL> set lin 200SQL> col path for a40SQL> select group_number,disk_number,total_mb,name,failgroup,path from v$asm_disk order by 1;GROUP_NUMBER DISK_NUMBER   TOTAL_MB NAME                           FAILGROUP                      PATH------------ ----------- ---------- ------------------------------ ------------------------------ ----------------------------------------0           0          0                                                               /dev/asm-diskf0           1          0                                                               /dev/asm-diskg0           2          0                                                               /dev/asm-diskc0           3          0                                                               /dev/asm-diske0           4          0                                                               /dev/asm-diskd0           5          0                                                               /dev/asm-diskb0           6          0                                                               /dev/asm-diski1           0      20476 DATA1                          DATA1                          AFD:DATA11           1      20476 DATA2                          DATA2                          AFD:DATA22           0      20476 FRA1                           FRA1                           AFD:FRA13           0       1020 ORS1                           ORS1                           AFD:ORS1GROUP_NUMBER DISK_NUMBER   TOTAL_MB NAME                           FAILGROUP                      PATH------------ ----------- ---------- ------------------------------ ------------------------------ ----------------------------------------3           1       1020 ORS2                           ORS2                           AFD:ORS23           2       1020 ORS3                           ORS3                           AFD:ORS313 rows selected.

可以从以上内容看到支持的有AFD模式的磁盘组并且所有的资源和数据都是写在此类磁盘组的上面，上述的磁盘组都是通过grid集群图形化界面或ASMCA磁盘管理软件界面创建的，创建后默认的模式是就是AFD。之前手动设置的udev rules文件通过查资料其实都可以不用做，以下方式使用shell命令行的形式新建AFD模式的磁盘组。

直接路径的方式创建ASM磁盘组

先直接使用/dev/asm-diski直接路径的方式创建

 SQL> create diskgroup bak_test external redundancy    disk  '/dev/asm-diski' name baktest  attribute 'au_size'='4M','compatible.asm' = '19.0.0.0', 'compatible.rdbms' = '19.0.0.0';Diskgroup created.//查看新创建的磁盘组SQL> select group_number,disk_number,total_mb,name,failgroup,path from v$asm_disk order by 1;GROUP_NUMBER DISK_NUMBER   TOTAL_MB NAME                           FAILGROUP                      PATH------------ ----------- ---------- ------------------------------ ------------------------------ ----------------------------------------0           0          0                                                               /dev/asm-diskf0           1          0                                                               /dev/asm-diskg0           2          0                                                               /dev/asm-diskc0           3          0                                                               /dev/asm-diske0           4          0                                                               /dev/asm-diskd0           5          0                                                               /dev/asm-diskb1           0      20476 DATA1                          DATA1                          AFD:DATA11           1      20476 DATA2                          DATA2                          AFD:DATA22           0      20476 FRA1                           FRA1                           AFD:FRA13           0       1020 ORS1                           ORS1                           AFD:ORS13           1       1020 ORS2                           ORS2                           AFD:ORS2GROUP_NUMBER DISK_NUMBER   TOTAL_MB NAME                           FAILGROUP                      PATH------------ ----------- ---------- ------------------------------ ------------------------------ ----------------------------------------3           2       1020 ORS3                           ORS3                           AFD:ORS34           0      51196 BAKTEST                        BAKTEST                        /dev/asm-diski    //这里可以看到模式是直接路径的方式13 rows selected.

上述最后一行的信息可以看到新增磁盘组的路径是直接路径的方式，此方式和默认的方式都不同，应修改为AFD的模式。修改模式如下：

 SQL> select name,state from v$asm_diskgroup;   //查看现有磁盘组STATE信息NAME                           STATE------------------------------ -----------DATA                           MOUNTEDFRA                            MOUNTEDORS                            MOUNTEDBAK_TEST                       MOUNTEDSQL> select name,path from v$asm_disk_stat;  //查案磁盘组路径信息NAME                           PATH------------------------------ ----------------------------------------BAKTEST                        /dev/asm-diskiORS1                           AFD:ORS1ORS2                           AFD:ORS2ORS3                           AFD:ORS3DATA1                          AFD:DATA1DATA2                          AFD:DATA2FRA1                           AFD:FRA17 rows selected.SQL> alter diskgroup BAK_TEST dismount;  //dismount磁盘组Diskgroup altered.SQL> select name,state from v$asm_diskgroup;NAME                           STATE------------------------------ -----------BAK_TEST                       DISMOUNTEDDATA                           MOUNTEDFRA                            MOUNTEDORS                            MOUNTEDSQL> drop diskgroup BAKTEST;   //删除磁盘组失败报错drop diskgroup BAKTEST*ERROR at line 1:ORA-15039: diskgroup not droppedORA-15073: diskgroup BAKTEST is mounted by another ASM instance切换至节点2：[grid@rac2 ~]$ asmcmdASMCMD> umount BAKTESTASMCMD> lsdgSQL> drop diskgroup BAKTEST;  //删除成功Diskgroup dropped.

AFD方式创建新磁盘组

 ASMCMD> afd_state   //查看AFD状态ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'rac1'ASMCMD> afd_label  baktest01 /dev/asm-diskiASMCMD> afd_lsdsk--------------------------------------------------------------------------------Label                     Filtering   Path================================================================================BAKTEST01                   ENABLED   /dev/asm-diskiDATA1                       ENABLED   /dev/asm-diskeDATA2                       ENABLED   /dev/asm-diskfFRA1                        ENABLED   /dev/asm-diskgORS1                        ENABLED   /dev/asm-diskbORS2                        ENABLED   /dev/asm-diskcORS3                        ENABLED   /dev/asm-diskdSQL> create diskgroup bak_afdtest external redundancy    disk  'AFD:BAKTEST01'  //在AFD模式上创建磁盘组SQL> alter diskgroup BAK_AFDTEST  mount;  //两节点分别mountSQL> CREATE tablespace AFD_test datafile '+BAK_AFDTEST' SIZE 100M;SQL> CREATE TABLE t_afd (n NUMBER) tablespace AFD_test;SQL> INSERT INTO t_afd VALUES(1);SQL> commit;Commit complete.SQL> select * from t_afd;  //查询信息正常N----------1

后续

后续会继续整理相关AFD相关的知识，包括直接创建AFD硬盘不生成ASM磁盘组。