环境介绍：

故事发生在Oracle 11.2.0.4 RAC/ Linux 6.8 的数据库集群服务器上。

DATA/ ARCH/ 磁盘组空间不足，进行扩容。

首先，如果要在线使用udev方式扩容asm，建议先备份两个节点所有在用网卡信息，避免bug误删除网卡信息。备份至/tmp目录下。(包括the ifcfg-eth0 (public), ifcfg-eth1 (private) and ifcfg-eth2 (backup))，注意11.2.0.3及以后在linux上的bug，见如下官网解释：

Network interface going down when dynamically adding disks to storage using udev in RHEL 6 (文档 ID 1569028.1)

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later

Oracle Net Services - Version 11.2.0.4 to 11.2.0.4 [Release 11.2]

Linux x86-64

Symptoms

Dynamically added new disks to storage using udev rules(as per the docs note 414897.1, 1528148.1, 371814.1 etc).

During start_udev, udev has deleted the public network interface and this caused the listener to crash, and clusterware moved all services, scan listeners and the VIP on node 1 to node 2.

Changes

Trying to add disk to storage using udev

Cause

udev has deleted the public network interface.

Solution

As per RHEL you are advised to set HOTPLUG="no" for the network configuration scripts.

Add HOTPLUG="no" to the ifcfg-eth0 (public), ifcfg-eth1 (private) and ifcfg-eth2 (backup) network config files in /etc/sysconfig/network-scripts directory.

建议先做数据库备份

1.停止数据库及集群服务

su - grid

srvctl stop listener

srvctl stop database -d jyrac

rac1/rac2

su - root

/u01/11.2.0/grid/bin/crsctl stop crs

####################### rac1/rac2 begin ##############################

2.网卡信息修改

注意：如果是使用网卡绑定，比如绑定后的网卡为bond0，则要在bond0的配置文件里添加hotplug配置信息，在eth0里添加不起作用。(实战中所有在用网卡均添加)

备份所有网卡信息至/tmp 目录

rac1/rac2

cp ifcfg-eth0 /tmp/ifcfg-eth0bak

cp ifcfg-bond0 /tmp/ifcfg-bond0bak

cp ifcfg-eth1 /tmp/ifcfg-eth1bak

...

实际：

[root@jyrac1 network-scripts]# cp * /tmp/wkbak

[root@jyrac1 network-scripts]#

[root@jyrac2 network-scripts]# cp * /tmp/wkbak/

[root@jyrac2 network-scripts]#

修改网卡信息

HOTPLUG="no"

重启网络服务

service network restart

[root@jyrac1 network-scripts]# service network restart

Shutting down interface bond0: [ OK ]

Shutting down interface eth0: [ OK ]

Shutting down loopback interface: [ OK ]

Bringing up loopback interface: [ OK ]

Bringing up interface bond0: Determining if ip address 192.168.20.11 is already in use for device bond0...

[ OK ]

Bringing up interface eth0: Determining if ip address 187.187.2.61 is already in use for device eth0...

[ OK ]

[root@jyrac1 network-scripts]#

[root@jyrac2 network-scripts]# service network restart

Shutting down interface bond0: [ OK ]

Shutting down interface eth0: [ OK ]

Shutting down loopback interface: [ OK ]

Bringing up loopback interface: [ OK ]

Bringing up interface bond0: Determining if ip address 192.168.20.12 is already in use for device bond0...

[ OK ]

Bringing up interface eth0: Determining if ip address 187.187.2.62 is already in use for device eth0...

[ OK ]

[root@jyrac2 network-scripts]#

3.修改udev权限文件

备份udev文件至/tmp 目录

[root@jyrac1 network-scripts]# cp /etc/udev/rules.d/12-dm-permissions.rules /tmp/wkbak/

vi /etc/udev/rules.d/12-dm-permissions.rules

增加：

ENV{DM_NAME}=="arch4", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

ENV{DM_NAME}=="data3", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

4.加载配置文件

[root@rac-db1 /]# /sbin/udevadm control reload-rules

5.添加udev磁盘

(生产库千万别重启udev，否则vip会漂移，且网卡信息可能会被删除，甚至宕库，重启监听可解决)

udevadm trigger --subsystem-match=block --action=add

[root@jyrac1 network-scripts]# udevadm control reload-rules

unrecognized command

udevadm[27284]: unrecognized command

[root@jyrac1 network-scripts]#

[root@jyrac1 network-scripts]#

[root@jyrac1 network-scripts]#

[root@jyrac1 network-scripts]# /sbin/udevadm control reload-rules

unrecognized command

udevadm[27682]: unrecognized command

[root@jyrac1 network-scripts]# find / -name udevadm

/sbin/udevadm

^C

[root@jyrac1 network-scripts]#

[root@jyrac1 network-scripts]#

[root@jyrac1 network-scripts]#

[root@jyrac1 network-scripts]# cd

[root@jyrac1 ~]# /sbin/udevadm control reload-rules

unrecognized command

udevadm[27987]: unrecognized command

[root@jyrac1 ~]# ls -la /sbin/udevadm

-rwxr-xr-x. 1 root root 137256 Oct 17 2013 /sbin/udevadm

[root@jyrac1 ~]# /sbin/udevadm control reload-rules

unrecognized command

udevadm[28151]: unrecognized command

[root@jyrac1 ~]# /sbin/start_udev

Starting udev: [ OK ]

[root@jyrac1 ~]#

6.检查磁盘属性

ll -l /dev/dm*

如果有问题，可以考虑重启multipath 或者udev试试。

/sbin/start_udev

or

service multipathd restart

再次检查属性，确认没问题之后考虑启动集群服务。

[root@jyrac1 ~]# ll -l /dev/dm*

brw-rw----. 1 root disk 253, 0 Apr 11 18:23 /dev/dm-0

brw-rw----. 1 root disk 253, 1 Apr 11 18:23 /dev/dm-1

brw-rw----. 1 grid asmadmin 253, 10 Apr 11 18:23 /dev/dm-10

brw-rw----. 1 root disk 253, 11 Apr 11 18:23 /dev/dm-11

brw-rw----. 1 root disk 253, 12 Apr 11 18:23 /dev/dm-12

brw-rw----. 1 grid asmadmin 253, 13 Apr 11 18:23 /dev/dm-13

brw-rw----. 1 grid asmadmin 253, 14 Apr 11 18:23 /dev/dm-14

brw-rw----. 1 root disk 253, 2 Apr 11 18:23 /dev/dm-2

brw-rw----. 1 grid asmadmin 253, 3 Apr 11 18:23 /dev/dm-3

brw-rw----. 1 grid asmadmin 253, 4 Apr 11 18:23 /dev/dm-4

brw-rw----. 1 grid asmadmin 253, 5 Apr 11 18:23 /dev/dm-5

brw-rw----. 1 grid asmadmin 253, 6 Apr 11 18:23 /dev/dm-6

brw-rw----. 1 grid asmadmin 253, 7 Apr 11 18:23 /dev/dm-7

brw-rw----. 1 grid asmadmin 253, 8 Apr 11 18:23 /dev/dm-8

brw-rw----. 1 grid asmadmin 253, 9 Apr 11 18:23 /dev/dm-9

[root@jyrac1 ~]#

[root@jyrac2 ~]# ll -l /dev/dm*

brw-rw---- 1 root disk 253, 0 Apr 11 18:25 /dev/dm-0

brw-rw---- 1 root disk 253, 1 Apr 11 18:25 /dev/dm-1

brw-rw---- 1 grid asmadmin 253, 10 Apr 11 18:25 /dev/dm-10

brw-rw---- 1 root disk 253, 11 Apr 11 18:25 /dev/dm-11

brw-rw---- 1 root disk 253, 12 Apr 11 18:25 /dev/dm-12

brw-rw---- 1 grid asmadmin 253, 13 Apr 11 18:25 /dev/dm-13

brw-rw---- 1 grid asmadmin 253, 14 Apr 11 18:25 /dev/dm-14

brw-rw---- 1 root disk 253, 2 Apr 11 18:25 /dev/dm-2

brw-rw---- 1 grid asmadmin 253, 3 Apr 11 18:25 /dev/dm-3

brw-rw---- 1 grid asmadmin 253, 4 Apr 11 18:25 /dev/dm-4

brw-rw---- 1 grid asmadmin 253, 5 Apr 11 18:25 /dev/dm-5

brw-rw---- 1 grid asmadmin 253, 6 Apr 11 18:25 /dev/dm-6

brw-rw---- 1 grid asmadmin 253, 7 Apr 11 18:25 /dev/dm-7

brw-rw---- 1 grid asmadmin 253, 8 Apr 11 18:25 /dev/dm-8

brw-rw---- 1 grid asmadmin 253, 9 Apr 11 18:25 /dev/dm-9

[root@jyrac2 ~]#

7.启动crs集群服务

rac1/rac2

su - root

/u01/11.2.0/grid/bin/crsctl start crs

######################## rac1/rac2 end############################

8.检查crs服务

su - grid

crs_stat -t

crsctl stat res -t

9.启动数据库

srvctl start database -d jyrac

10.检查数据库状态

crsctl stat res -t

11.检查asm是否识别到未添加的磁盘

set linesize 300

col name for a20

select group_number,name,state,type,total_mb,free_mb,usable_file_mb,allocation_unit_size/1024/1024 unit_mb from v$asm_diskgroup order by 1;

GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB USABLE_FILE_MB UNIT_MB

------------ -------------------- ---------------------- ------------ ---------- ---------- -------------- ----------

1 ARCH MOUNTED EXTERN 614400 120548 120548 1

2 DATA MOUNTED EXTERN 409600 83773 83773 1

3 OCR MOUNTED NORMAL 15360 14434 4657 1

set linesize 300

col path for a50

select group_number,name,path,mount_status,state,redundancy,total_mb,free_mb,failgroup,create_date from v$asm_disk order by 1,2;

GROUP_NUMBER NAME PATH MOUNT_STATUS STATE REDUNDANCY TOTAL_MB FREE_MB FAILGROUP CREATE_DATE

------------ -------------------- -------------------------------------------------- -------------- ---------------- -------------- ---------- ---------- ------------------------------------------------------------ -----------------------

0 /dev/mapper/arch4 CLOSED NORMAL UNKNOWN 0 0

0 /dev/mapper/data3 CLOSED NORMAL UNKNOWN 0 0

1 ARCH_0000 /dev/mapper/arch1 CACHED NORMAL UNKNOWN 204800 40180 ARCH_0000 21-MAY-2015 11:00:32

1 ARCH_0001 /dev/mapper/arch2 CACHED NORMAL UNKNOWN 204800 40183 ARCH_0001 21-MAY-2015 11:00:32

1 ARCH_0002 /dev/mapper/arch3 CACHED NORMAL UNKNOWN 204800 40185 ARCH_0002 21-MAY-2015 11:00:32

2 DATA_0000 /dev/mapper/data1 CACHED NORMAL UNKNOWN 204800 41879 DATA_0000 21-MAY-2015 10:59:33

2 DATA_0001 /dev/mapper/data2 CACHED NORMAL UNKNOWN 204800 41894 DATA_0001 21-MAY-2015 10:59:33

3 OCR_0000 /dev/mapper/crs1 CACHED NORMAL UNKNOWN 5120 4812 OCR_0000 21-MAY-2015 10:16:24

3 OCR_0001 /dev/mapper/crs2 CACHED NORMAL UNKNOWN 5120 4810 OCR_0001 21-MAY-2015 10:16:24

3 OCR_0002 /dev/mapper/crs3 CACHED NORMAL UNKNOWN 5120 4812 OCR_0002 21-MAY-2015 10:16:24

10 rows selected.

12.ASM扩容

alter diskgroup DATA add disk '/dev/mapper/data3' rebalance power 8;

alter diskgroup ARCH add disk '/dev/mapper/arch4' rebalance power 8;

13.检查进度

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE

------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----------------------------------------------------------------------------------------

1 REBAL WAIT 8

2 REBAL RUN 8 8 33914 108615 5606 13

SQL>

set linesize 300

col path for a50

select group_number,name,path,mount_status,state,redundancy,total_mb,free_mb,failgroup,create_date from v$asm_disk order by 1,2;

转自

邮箱：286397146@qq.com

昵称：A这货不是DBA