As with all systems, a disk failure can happen and so we had a failing disk on one of our ODA’s. As mentioned in the ODA documentation, replacing a disk is easy and OAK does everything for you. Actually it is doing everything for you, from all OS related actions (like multipath configuration and partitioning) to dropping and adding the replaced disk in ASM.
It all looked fine until we got a close look after OAK was done adding the disk. Replacing the disk introduced the following 2 problems:
1. The ASM alertlog keeps logging:
WARNING: Read Failed. group:0 disk:1 AU:0 offset:0 size:4096 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_25561.trc: ORA-27061: waiting for async I/Os failed Linux-x86_64 Error: 5: Input/output error Additional information: -1 Additional information: 4096
and the /var/log/messages keeps logging messages like:
May 23 22:14:58 emn-odadb-ts1-03 kernel: end_request: I/O error, dev dm-22, sector May 23 22:14:58 emn-odadb-ts1-03 kernel: Buffer I/O error on device dm-46, logical block 0 ....
2. The size of the second partition on the new disk (used by diskgroup RECO) is not the same as the size of all other (orginal) disks. Partition 2 on original disks are 80800 MB but the partition 2 on the new disk is just 75080 MB.
Cause problem 1:
At first it looked like problem 1 (I/O errors in ASM alertlog and OS messages file) was because of the new disk also failing, but looking further the logfile /opt/oracle/oak/log/<ODA node name>/oak/oakd.log showed the new disk got a new device name, was partitioned and correctly added to the RECO diskgroup. The I/O errors the alertlog and messages file are showing are about the OLD (physically remove) disk and it is pretty hard to read/write to a physically removed disk :-).
The reason why the old disk device still exists, is because there are still processes (Grid Infra/ASM/Database) having open file descriptors to the old device so Linux is not able to remove the device.
Use the following commands to get a list of process id’s for the processes that are still having open file descriptors to the device file of the removed disk:
/sbin/fuser /dev/mapper/HDD_E1_S05_992884312p2 /dev/mapper/HDD_E1_S05_992884312p2: 3254 3298 5196 ps -ef|grep 3298 grid 3298 1 0 2012 ? 00:00:40 asm_vbg0_+ASM1
Cause problem 2:The /sbin/parted command that is executed by OAK when the new disk is inserted, is called to create 2 partitions on the new disk with using an partition size for partition 2, starting at the next cylinder after where partition 1 ends and ending on the cylinder based on 99% of the disk. When deploying the ODA these partitions are defined exactly the same way, but it seems that due to a change in the parted utility or because the new disk is different (Vendor: HITACHI, Model: HUS1560SCSUN600G) than the original disk (Vendor: SEAGATE, Model: ST360057SSUN600G) the 99% results in different end cylinder number, resulting in a smaller partition (around 6 GB smaller).
Solution problem 1:
There is no real solution for getting rid of the old disk device and there is a generic ASM problem (MOS note 1485163.1: Disks of Dismounted Diskgroup Are Still Hold / Lock By Oracle Process) created, and still open, for this problem. It is not an ODA specific problem. The only way to get rid of the old device is to restart CRS stack per node, thus including all database instances running on that node.
Solution problem 2:This problem is said to be fixed in ODA 126.96.36.199.0 where you have the oppertunity to reinitialize the disk using oakcli which will recreate the disk partitions with the correct size (and of course all OS and ASM related actions too).