Table of contents
Preamble
I know UDEV, at least the word, since a quite long time but I have never tried it in a concrete example. While discussing in the OCM workshop I came to know that ASMLib was not mandatory to install Automatic Storage Management (ASM). Also Oracle has not been super fast to provide the ASMLib library for Red Hat 7, their initial communication was that they are not going even to provide it on Red Hat 7. Apparently now you can download the one of OEL 7 and install it on Red Hat 7. I’m not even mentioning all blog posts and forum discussions about the obsolescence of ASMLib versus UDEV. Some even mentioned that ASMLib brings nothing versus the native kernel UDEV feature. At least from what I have tested it eases the management of disks.
While trying to implement UDEV I have found tons of blog posts on Red Hat 5 and 6 but nothing really on Red Hat 7, or at least with the latest capabilities of UDEV rules.
Very high level UDEV is a dynamic device manager managing your entities called nodes in /dev directory. What is worth to have in mind is that after a reboot your disk devices may change their names which might be a small problem for your database. This tricky situation can be solved by writing UDEV rules to have a consistent naming convention for your devices.
For licensing reason I have tested using a virtual machine using Oracle Linux Server release 7.1 (equivalent of Red Hat Enterprise Linux Server release 7.1 (Maipo)) and Oracle Grid Infrastructure 12.1.0.2.0.
UDEV Linux configuration
What I’m preparing in the background is a RAC installation using two virtual machine running under VirtualBox. For this I have attached to my first virtual machine four fixed size of 1GB disks like this:
To attach those disks in another virtual machine go to Virtual Media Manager and change their property to shareable like this:
Once you start your guest OS you should see something:
[root@server3 ~]# fdisk -l . . . Disk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdc: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdd: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sde: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes . . . |
In many blog post I have seen creation of a unique primary partition on those disks with:
[root@server3 ~]# fdisk /dev/sdc Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0xa4cc3f96. Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): 1 First sector (2048-2097151, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): Using default value 2097151 Partition 1 of type Linux and of size 1023 MiB is set Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. |
Or use parted if you want to automate the creation with a shell script:
[root@server3 ~]# parted /dev/sdd mklabel msdos Information: You may need to update /etc/fstab. [root@server3 ~]# parted /dev/sdd mkpart primary 1m 100% Information: You may need to update /etc/fstab. [root@server3 ~]# fdisk -l /dev/sdd Disk /dev/sdd: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x000df279 Device Boot Start End Blocks Id System /dev/sdd1 2048 2097151 1047552 83 Linux [root@server3 ~]# parted /dev/sdb print Model: ATA VBOX HARDDISK (scsi) Disk /dev/sdb: 1074MB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 1049kB 1074MB 1073MB primary |
But honestly I’m wondering the need of this unique primary partition filling entirely the disk. Apparently it is needed only with ASMLib so as not using it decided to give a try without creating them…
So deleted the one I created (fdisk can also be used):
[root@server3 rules.d]# parted /dev/sde rm 1 Information: You may need to update /etc/fstab. |
The idea of UDEV is to associate the fixed device id to a more understandable name. On many blog posts you will see the usage of iscsi_id binary. That moved from /sbin in Red Hat 5 and 6 to /usr/lib/udev in Red Hat 7.
[root@server3 ~]# for disk in `ls /dev/sd*` do echo $disk /usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=$disk done /dev/sda 1ATA_VBOX_HARDDISK_VB35905f56-9625bd7c /dev/sda1 1ATA_VBOX_HARDDISK_VB35905f56-9625bd7c /dev/sda2 1ATA_VBOX_HARDDISK_VB35905f56-9625bd7c /dev/sdb 1ATA_VBOX_HARDDISK_VB420a8d94-2a375f5b /dev/sdc 1ATA_VBOX_HARDDISK_VB63ab3b43-33327cc0 /dev/sdd 1ATA_VBOX_HARDDISK_VBce02798e-7d13dc9a /dev/sde 1ATA_VBOX_HARDDISK_VB9c27c116-7154ebea |
Which can now be replaced (since Red Hat 6) with udevadm (udevadm info -h for more information):
[root@server3 ~]# udevadm info --query=property /dev/sdb DEVLINKS=/dev/disk/by-id/ata-VBOX_HARDDISK_VB420a8d94-2a375f5b DEVNAME=/dev/sdb DEVPATH=/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb DEVTYPE=disk ID_ATA=1 ID_ATA_FEATURE_SET_PM=1 ID_ATA_FEATURE_SET_PM_ENABLED=1 ID_ATA_SATA=1 ID_ATA_SATA_SIGNAL_RATE_GEN2=1 ID_ATA_WRITE_CACHE=1 ID_ATA_WRITE_CACHE_ENABLED=1 ID_BUS=ata ID_MODEL=VBOX_HARDDISK ID_MODEL_ENC=VBOX\x20HARDDISK\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20 ID_PART_TABLE_TYPE=dos ID_REVISION=1.0 ID_SERIAL=VBOX_HARDDISK_VB420a8d94-2a375f5b ID_SERIAL_SHORT=VB420a8d94-2a375f5b ID_TYPE=disk MAJOR=8 MINOR=16 MPATH_SBIN_PATH=/sbin SUBSYSTEM=block TAGS=:systemd: USEC_INITIALIZED=85769 |
The target is to write UDEV rules to associate our /dev/sdb to /dev/sde disk devices to symbolic names with a taste of ASMLib i.e. /dev/oracleasm/disks/DISKxx. To have a first insight of how to write UDEV rules you can start with man udev command. When using legacy method it gives below rules that you see on many articles, I have decided to put the rules in Oracle suggested file i.e. 99-oracle-asmdevices.rules located in /etc/udev/rules.d directory:
[root@server3 ~]# head -n 1 /etc/udev/rules.d/99-oracle-asmdevices.rules KERNEL=="sd?", PROGRAM=="/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB420a8d94-2a375f5b", SYMLINK+="oracleasm/disks/DISK01", OWNER="oracle", GROUP="dba", MODE="0660" |
I rate it a bit cumbersome and hard to read, with newest UDEV rules it can gives something brighter like below:
[root@server3 ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules KERNEL=="sd?", ENV{ID_SERIAL}=="VBOX_HARDDISK_VB420a8d94-2a375f5b", SYMLINK+="oracleasm/disks/DISK01", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?", ENV{ID_SERIAL}=="VBOX_HARDDISK_VB63ab3b43-33327cc0", SYMLINK+="oracleasm/disks/DISK02", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?", ENV{ID_SERIAL}=="VBOX_HARDDISK_VBce02798e-7d13dc9a", SYMLINK+="oracleasm/disks/DISK03", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?", ENV{ID_SERIAL}=="VBOX_HARDDISK_VB9c27c116-7154ebea", SYMLINK+="oracleasm/disks/DISK04", OWNER="oracle", GROUP="dba", MODE="0660" |
Easier isn’t it ? Notice the sd? to take all SATA disk, even the system one but as we have not specified its serial id no rule will apply to it. The ENV{ID_SERIAL} match the serial ID of the disk to a generic name, so even if after reboot your devices names have changed the serial ID will not change. The SYMLINK will create a more understandable link to our devices and OWNER, GROUP and MODE gives ownership and permission.
Reload and apply the UDEV rules with:
[root@server3 ~]# udevadm control --reload-rules [root@server3 ~]# udevadm trigger |
Then you can see the link nodes that have been created:
[root@server3 ~]# ll /dev/oracleasm/disks/ total 0 lrwxrwxrwx 1 root root 9 Jul 2 12:33 DISK01 -> ../../sdb lrwxrwxrwx 1 root root 9 Jul 2 12:33 DISK02 -> ../../sdc lrwxrwxrwx 1 root root 9 Jul 2 12:33 DISK03 -> ../../sdd lrwxrwxrwx 1 root root 9 Jul 2 12:33 DISK04 -> ../../sde |
You can test your UDEV rules with:
[root@server3 ~]# udevadm test /sys/block/sdb |
The rights are set only devices but not on symlinks:
[root@server3 hugepages]# ll /dev/sd* brw-rw---- 1 root disk 8, 0 Jul 6 13:04 /dev/sda brw-rw---- 1 root disk 8, 1 Jul 6 13:04 /dev/sda1 brw-rw---- 1 root disk 8, 2 Jul 6 13:04 /dev/sda2 brw-rw---- 1 oracle dba 8, 16 Jul 7 15:45 /dev/sdb brw-rw---- 1 oracle dba 8, 32 Jul 7 15:46 /dev/sdc brw-rw---- 1 oracle dba 8, 48 Jul 6 13:27 /dev/sdd brw-rw---- 1 oracle dba 8, 64 Jul 6 13:27 /dev/sde |
ASM configuration with UDEV
Once the UDEV rules have been created you can install Grid Infrastructure and when done use asmca to create your ASM instance and create your first ASM device group on your newly created disks (I have first changed the discovery path to /dev/oracleasm/disks):
I got the “Oracle Grid Infrastructure is not configured properly. ASMCA needs Oracle Grid Infrastructure to configure ASM” error:
What I did not notice when executing Grid Infrastructure root.sh is the below message:
To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user: /u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/crs/install /u01/app/12.1.0/grid/crs/install/roothas.pl To configure Grid Infrastructure for a Cluster execute the following command as oracle user: /u01/app/12.1.0/grid/crs/config/config.sh This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media. |
Once script executed the configuration went better. I had few others error message but they were all related to memory allocation of my virtual machine. Oracle Grid Infrastructure in 12cR1 is consuming almost 1 GB of memory so you need a quite big /dev/shm (MEMORY_TARGET parameter).
Once the ASM instance has been created even without any partition created on disks everything looks fine:
SQL> col PATH FOR a30 SQL> SET lines 200 SQL> SELECT name,total_mb FROM v$asm_diskgroup; NAME TOTAL_MB ------------------------------ ---------- DATA 3072 SQL> SELECT name,PATH,mode_status,os_mb FROM v$asm_disk NAME PATH MODE_ST OS_MB ------------------------------ ------------------------------ ------- ---------- /dev/oracleasm/disks/DISK01 ONLINE 1024 DATA_0000 /dev/oracleasm/disks/DISK02 ONLINE 1024 DATA_0002 /dev/oracleasm/disks/DISK04 ONLINE 1024 DATA_0001 /dev/oracleasm/disks/DISK03 ONLINE 1024 |
References
- Configuring Disk Devices Manually for Oracle ASM
- How To Replace ASMLib With UDEV (Doc ID 1461321.1)
- How to Configure LUNs for ASM Disks using WWID, DM-Multipathing, and ASMLIB on RHEL 5/OL 5 and RHEL 6/OL 6 (Doc ID 1365511.1)
- How To Setup ASM on Linux Using ASMLIB Disks, Raw Devices, Block Devices or UDEV Devices? (Doc ID 580153.1)
- How To Setup Partitioned Linux Block Devices Using UDEV (Non-ASMLIB) And Assign Them To ASM? (Doc ID 1528148.1)
- UDEV SCSI Rules Configuration for ASM in Oracle Linux 5 and 6
- Udev Rules
- Fun with Linux UDEV and ASM: Using UDEV to create ASM disk volumes
- Oracle ASMLib Software Update and Support Policy for Red Hat Enterprise Linux (Doc ID 1089399.1)
- ASMCA returns “Oracle Grid Infrastructure is not configured properly. ASMCA needs Oracle Grid Infrastructure to configure ASM” (Doc ID 2008316.1)
