ASM on Red Hat 7 and OEL 7 with UDEV

Preamble

I know UDEV, at least the word, since a quite long time but I have never tried it in a concrete example. While discussing in the OCM workshop I came to know that ASMLib was not mandatory to install Automatic Storage Management (ASM). Also Oracle has not been super fast to provide the ASMLib library for Red Hat 7, their initial communication was that they are not going even to provide it on Red Hat 7. Apparently now you can download the one of OEL 7 and install it on Red Hat 7. I’m not even mentioning all blog posts and forum discussions about the obsolescence of ASMLib versus UDEV. Some even mentioned that ASMLib brings nothing versus the native kernel UDEV feature. At least from what I have tested it eases the management of disks.

While trying to implement UDEV I have found tons of blog posts on Red Hat 5 and 6 but nothing really on Red Hat 7, or at least with the latest capabilities of UDEV rules.

Very high level UDEV is a dynamic device manager managing your entities called nodes in /dev directory. What is worth to have in mind is that after a reboot your disk devices may change their names which might be a small problem for your database. This tricky situation can be solved by writing UDEV rules to have a consistent naming convention for your devices.

For licensing reason I have tested using a virtual machine using Oracle Linux Server release 7.1 (equivalent of Red Hat Enterprise Linux Server release 7.1 (Maipo)) and Oracle Grid Infrastructure 12.1.0.2.0.

UDEV Linux configuration

What I’m preparing in the background is a RAC installation using two virtual machine running under VirtualBox. For this I have attached to my first virtual machine four fixed size of 1GB disks like this:

udev01
udev01

To attach those disks in another virtual machine go to Virtual Media Manager and change their property to shareable like this:

udev02
udev02

Once you start your guest OS you should see something:

[root@server3 ~]# fdisk -l
.
.
.
Disk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
 
 
Disk /dev/sdc: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
 
 
Disk /dev/sdd: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
 
 
Disk /dev/sde: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
.
.
.

In many blog post I have seen creation of a unique primary partition on those disks with:

[root@server3 ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2).
 
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
 
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xa4cc3f96.
 
Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):
Using default value 2097151
Partition 1 of type Linux and of size 1023 MiB is set
 
Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
Syncing disks.

Or use parted if you want to automate the creation with a shell script:

[root@server3 ~]# parted /dev/sdd mklabel msdos
Information: You may need to update /etc/fstab.
 
[root@server3 ~]# parted /dev/sdd mkpart primary 1m 100%
Information: You may need to update /etc/fstab.
 
[root@server3 ~]# fdisk -l /dev/sdd
 
Disk /dev/sdd: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000df279
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048     2097151     1047552   83  Linux
[root@server3 ~]# parted /dev/sdb print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sdb: 1074MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
 
Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1074MB  1073MB  primary

But honestly I’m wondering the need of this unique primary partition filling entirely the disk. Apparently it is needed only with ASMLib so as not using it decided to give a try without creating them…

So deleted the one I created (fdisk can also be used):

[root@server3 rules.d]# parted /dev/sde rm 1
Information: You may need to update /etc/fstab.

The idea of UDEV is to associate the fixed device id to a more understandable name. On many blog posts you will see the usage of iscsi_id binary. That moved from /sbin in Red Hat 5 and 6 to /usr/lib/udev in Red Hat 7.

[root@server3 ~]# for disk in `ls /dev/sd*`
                  do
                    echo $disk
                    /usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=$disk
                  done
/dev/sda
1ATA_VBOX_HARDDISK_VB35905f56-9625bd7c
/dev/sda1
1ATA_VBOX_HARDDISK_VB35905f56-9625bd7c
/dev/sda2
1ATA_VBOX_HARDDISK_VB35905f56-9625bd7c
/dev/sdb
1ATA_VBOX_HARDDISK_VB420a8d94-2a375f5b
/dev/sdc
1ATA_VBOX_HARDDISK_VB63ab3b43-33327cc0
/dev/sdd
1ATA_VBOX_HARDDISK_VBce02798e-7d13dc9a
/dev/sde
1ATA_VBOX_HARDDISK_VB9c27c116-7154ebea

Which can now be replaced (since Red Hat 6) with udevadm (udevadm info -h for more information):

[root@server3 ~]# udevadm info --query=property /dev/sdb
DEVLINKS=/dev/disk/by-id/ata-VBOX_HARDDISK_VB420a8d94-2a375f5b
DEVNAME=/dev/sdb
DEVPATH=/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
DEVTYPE=disk
ID_ATA=1
ID_ATA_FEATURE_SET_PM=1
ID_ATA_FEATURE_SET_PM_ENABLED=1
ID_ATA_SATA=1
ID_ATA_SATA_SIGNAL_RATE_GEN2=1
ID_ATA_WRITE_CACHE=1
ID_ATA_WRITE_CACHE_ENABLED=1
ID_BUS=ata
ID_MODEL=VBOX_HARDDISK
ID_MODEL_ENC=VBOX\x20HARDDISK\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
ID_PART_TABLE_TYPE=dos
ID_REVISION=1.0
ID_SERIAL=VBOX_HARDDISK_VB420a8d94-2a375f5b
ID_SERIAL_SHORT=VB420a8d94-2a375f5b
ID_TYPE=disk
MAJOR=8
MINOR=16
MPATH_SBIN_PATH=/sbin
SUBSYSTEM=block
TAGS=:systemd:
USEC_INITIALIZED=85769

The target is to write UDEV rules to associate our /dev/sdb to /dev/sde disk devices to symbolic names with a taste of ASMLib i.e. /dev/oracleasm/disks/DISKxx. To have a first insight of how to write UDEV rules you can start with man udev command. When using legacy method it gives below rules that you see on many articles, I have decided to put the rules in Oracle suggested file i.e. 99-oracle-asmdevices.rules located in /etc/udev/rules.d directory:

[root@server3 ~]# head -n 1 /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?", PROGRAM=="/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB420a8d94-2a375f5b", SYMLINK+="oracleasm/disks/DISK01", OWNER="oracle", GROUP="dba", MODE="0660"

I rate it a bit cumbersome and hard to read, with newest UDEV rules it can gives something brighter like below:

[root@server3 ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?", ENV{ID_SERIAL}=="VBOX_HARDDISK_VB420a8d94-2a375f5b", SYMLINK+="oracleasm/disks/DISK01", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="VBOX_HARDDISK_VB63ab3b43-33327cc0", SYMLINK+="oracleasm/disks/DISK02", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="VBOX_HARDDISK_VBce02798e-7d13dc9a", SYMLINK+="oracleasm/disks/DISK03", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="VBOX_HARDDISK_VB9c27c116-7154ebea", SYMLINK+="oracleasm/disks/DISK04", OWNER="oracle", GROUP="dba", MODE="0660"

Easier isn’t it ? Notice the sd? to take all SATA disk, even the system one but as we have not specified its serial id no rule will apply to it. The ENV{ID_SERIAL} match the serial ID of the disk to a generic name, so even if after reboot your devices names have changed the serial ID will not change. The SYMLINK will create a more understandable link to our devices and OWNER, GROUP and MODE gives ownership and permission.

Reload and apply the UDEV rules with:

[root@server3 ~]# udevadm control --reload-rules
[root@server3 ~]# udevadm trigger

Then you can see the link nodes that have been created:

[root@server3 ~]# ll /dev/oracleasm/disks/
total 0
lrwxrwxrwx 1 root root 9 Jul  2 12:33 DISK01 -> ../../sdb
lrwxrwxrwx 1 root root 9 Jul  2 12:33 DISK02 -> ../../sdc
lrwxrwxrwx 1 root root 9 Jul  2 12:33 DISK03 -> ../../sdd
lrwxrwxrwx 1 root root 9 Jul  2 12:33 DISK04 -> ../../sde

You can test your UDEV rules with:

[root@server3 ~]# udevadm test /sys/block/sdb

The rights are set only devices but not on symlinks:

[root@server3 hugepages]# ll /dev/sd*
brw-rw---- 1 root   disk 8,  0 Jul  6 13:04 /dev/sda
brw-rw---- 1 root   disk 8,  1 Jul  6 13:04 /dev/sda1
brw-rw---- 1 root   disk 8,  2 Jul  6 13:04 /dev/sda2
brw-rw---- 1 oracle dba  8, 16 Jul  7 15:45 /dev/sdb
brw-rw---- 1 oracle dba  8, 32 Jul  7 15:46 /dev/sdc
brw-rw---- 1 oracle dba  8, 48 Jul  6 13:27 /dev/sdd
brw-rw---- 1 oracle dba  8, 64 Jul  6 13:27 /dev/sde

ASM configuration with UDEV

Once the UDEV rules have been created you can install Grid Infrastructure and when done use asmca to create your ASM instance and create your first ASM device group on your newly created disks (I have first changed the discovery path to /dev/oracleasm/disks):

udev03
udev03

I got the “Oracle Grid Infrastructure is not configured properly. ASMCA needs Oracle Grid Infrastructure to configure ASM” error:

udev04
udev04

What I did not notice when executing Grid Infrastructure root.sh is the below message:

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/crs/install /u01/app/12.1.0/grid/crs/install/roothas.pl
 
 
To configure Grid Infrastructure for a Cluster execute the following command as oracle user:
/u01/app/12.1.0/grid/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.

Once script executed the configuration went better. I had few others error message but they were all related to memory allocation of my virtual machine. Oracle Grid Infrastructure in 12cR1 is consuming almost 1 GB of memory so you need a quite big /dev/shm (MEMORY_TARGET parameter).

Once the ASM instance has been created even without any partition created on disks everything looks fine:

SQL> col PATH FOR a30
SQL> SET lines 200
SQL> SELECT name,total_mb FROM v$asm_diskgroup;
 
NAME                             TOTAL_MB
------------------------------ ----------
DATA                                 3072
 
SQL> SELECT name,PATH,mode_status,os_mb FROM v$asm_disk
 
NAME                           PATH                           MODE_ST      OS_MB
------------------------------ ------------------------------ ------- ----------
                               /dev/oracleasm/disks/DISK01    ONLINE        1024
DATA_0000                      /dev/oracleasm/disks/DISK02    ONLINE        1024
DATA_0002                      /dev/oracleasm/disks/DISK04    ONLINE        1024
DATA_0001                      /dev/oracleasm/disks/DISK03    ONLINE        1024

References

One thought on “ASM on Red Hat 7 and OEL 7 with UDEV

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>