Written by Table Of Contents
Preamble
Before entering in Veritas Volume Manager (VxVM) details we must go in fine detail of our hardware configuration. Let’s first confirm at Linux level that I see my 2 fiber channel (FC) cards with each 2 FC ports:
[root@server1 ~]# lspci | grep -i fibre 09:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03) 09:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03) 0c:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03) 0c:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03) |
[root@server1 ~]# ll /sys/class/fc_host/host*/device lrwxrwxrwx 1 root root 0 Jun 5 17:49 /sys/class/fc_host/host0/device -> ../../../devices/pci0000:00/0000:00:03.0/0000:09:00.0/host0 lrwxrwxrwx 1 root root 0 Jun 5 17:51 /sys/class/fc_host/host1/device -> ../../../devices/pci0000:00/0000:00:03.0/0000:09:00.1/host1 lrwxrwxrwx 1 root root 0 Jun 5 17:51 /sys/class/fc_host/host2/device -> ../../../devices/pci0000:00/0000:00:05.0/0000:0c:00.0/host2 lrwxrwxrwx 1 root root 0 Jun 5 17:51 /sys/class/fc_host/host3/device -> ../../../devices/pci0000:00/0000:00:05.0/0000:0c:00.1/host3 |
We will see this later but we need correspondence between SCSI chain number and its World Wide Name (WWN), here it is:
[root@server1 ~]# for host in `ls /sys/class/fc_host/`; do echo "$host WWN: `cat /sys/class/fc_host/$host/port_name`"; done host0 WWN: 0x50060b0000c35e0c host1 WWN: 0x50060b0000c35e0e host2 WWN: 0x50060b0000c35e08 host3 WWN: 0x50060b0000c35e0a |
So if everything is fiber wired (FC cards/ports to FC switches) I should see any disk array (enclosure) device/LUN as 4 equivalent name. Please note that this is absolutely not mandatory, what’s really important in my configuration is to have each disk visible through my 2 FC cards (so by one port minimum so each disk/LUN will have 2 names).
Please note that this post is more for DBAs to see and understand what have been done by you system teammates i.e. I’m not entering in the creation/modification of VxVM disk/disk group/controller and so on…
All commands have been executed on a Red Hat Enterprise Linux Server release 5.5 (Tikanga) server configured with Veritas Volume Manager release VRTSvxvm-5.1.132.000-SP1RP2_RHEL5.
To avoid putting real server name I’m using the below tips for my prompt customization:
export PS1="[\u@server1 \W]# " |
Dynamic Multipathing (DMP)
Veritas Dynamic Multi-Pathing (DMP or VxDMP) is the VxVM functionality implemented to support disk availability from multiple paths. Even if it looks complex to start with this functionality it is unfortunately mandatory before doing the more simple things as it is based configuration on which all other components will be created (subdisks, plexes, volumes).
Configured disk arrays or enclosures accessible from this server:
[root@server1 ~]# vxdmpadm listenclosure ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE LUN_COUNT ======================================================================================= emc0 EMC 000290102696 CONNECTED A/A 54 emc1 EMC 000290105264 CONNECTED A/A 1 other_disks OTHER_DISKS OTHER_DISKS CONNECTED OTHER_DISKS 1 |
I/O policy default configuration of this enclosure:
[root@server1 ~]# vxdmpadm getattr enclosure emc0 ENCLR_NAME ATTR_NAME DEFAULT CURRENT ============================================================================ emc0 iopolicy MinimumQ MinimumQ emc0 partitionsize 512 512 emc0 use_all_paths - - emc0 failover_policy Global Global emc0 recoveryoption[throttle] Nothrottle[0] Nothrottle[0] emc0 recoveryoption[errorretry] Timebound[300] Timebound[300] emc0 redundancy 0 0 emc0 dmp_lun_retry_timeout 0 0 emc0 failovermode - - |
Disk default naming convention of VxVM:
[root@server1 ~]# vxddladm get namingscheme NAMING_SCHEME PERSISTENCE LOWERCASE USE_AVID ============================================================ Enclosure Based Yes Yes Yes |
List of configured disk on your system:
[root@server1 ~]# vxdisk -e list DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR cciss/c0d0 auto:none - - online invalid cciss/c0d0 - emc0_00f3 auto:cdsdisk - - online sdbg std emc0_3d6b auto:cdsdisk vgp1148_01 vgp1148 online clone_disk sdbn srdf-r1 emc0_3d7b auto:cdsdisk vgp1148_03 vgp1148 online clone_disk sdbp srdf-r1 emc0_3d73 auto:cdsdisk vgp1148_02 vgp1148 online clone_disk sdbo srdf-r1 emc0_3ebb auto:cdsdisk vgp1118_02 vgp1118 online clone_disk sdca srdf-r1 emc0_3eb3 auto:cdsdisk vgp1118_01 vgp1118 online clone_disk sdbx srdf-r1 emc0_3ecb auto:cdsdisk vgp1318_01 vgp1318 online clone_disk sdcc srdf-r1 emc0_3ec3 auto:cdsdisk vgp1118_03 vgp1118 online clone_disk sdcb srdf-r1 . . |
Disks of a disk group (please note it makes relationship between VxVM disks and the ones you see at system level):
[root@server1 ~]# vxdisk -g vgp1417 -e list DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR emc0_084b auto:cdsdisk vgp1417_04 vgp1417 online clone_disk sdax srdf-r1 emc0_084f auto:cdsdisk vgp1417_05 vgp1417 online clone_disk sdaz srdf-r1 emc0_085b auto:cdsdisk vgp1417_08 vgp1417 online clone_disk sdbd srdf-r1 emc0_526f auto:cdsdisk vgp1417_09 vgp1417 online clone_disk sdbh srdf-r1 emc0_527b auto:cdsdisk vgp1417_10 vgp1417 online clone_disk sdbi srdf-r1 emc0_527f auto:cdsdisk vgp1417_11 vgp1417 online clone_disk sdbj srdf-r1 emc0_0853 auto:cdsdisk vgp1417_06 vgp1417 online clone_disk sdbb srdf-r1 emc0_0857 auto:cdsdisk vgp1417_07 vgp1417 online clone_disk sdbc srdf-r1 |
Main problem with above command is that you don’t see multipathing details of VxVM disk, to go deeper in detail use:
[root@server1 ~]# vxdisk list emc0_084b Device: emc0_084b devicetag: emc0_084b type: auto hostid: server1 disk: name=vgp1417_04 id=1325653859.61.server1 group: name=vgp1417 id=1318435685.29.server1 info: format=cdsdisk,privoffset=256,pubslice=3,privslice=3 flags: online ready private autoconfig noautoimport imported clone_disk pubpaths: block=/dev/vx/dmp/emc0_084bs3 char=/dev/vx/rdmp/emc0_084bs3 guid: - udid: EMC%5FSYMMETRIX%5F000290102696%5F960084B000 site: - version: 3.1 iosize: min=512 (bytes) max=1024 (blocks) public: slice=3 offset=65792 len=70539872 disk_offset=0 private: slice=3 offset=256 len=65536 disk_offset=0 update: time=1335608893 seqno=0.17 ssb: actual_seqno=0.0 headers: 0 240 configs: count=1 len=51360 logs: count=1 len=4096 Defined regions: config priv 000048-000239[000192]: copy=01 offset=000000 enabled config priv 000256-051423[051168]: copy=01 offset=000192 enabled log priv 051424-055519[004096]: copy=01 offset=000000 enabled lockrgn priv 055520-055663[000144]: part=00 offset=000000 Multipathing information: numpaths: 2 sdax state=enabled sdh state=enabled |
Remark:
You may also use the opposite command:
[root@server1 ~]# vxdisk list sdax Device: emc0_084b devicetag: emc0_084b type: auto hostid: server1 disk: name=vgp1417_04 id=1325653859.61.server1 group: name=vgp1417 id=1318435685.29.server1 info: format=cdsdisk,privoffset=256,pubslice=3,privslice=3 flags: online ready private autoconfig noautoimport imported clone_disk pubpaths: block=/dev/vx/dmp/emc0_084bs3 char=/dev/vx/rdmp/emc0_084bs3 guid: - udid: EMC%5FSYMMETRIX%5F000290102696%5F960084B000 site: - version: 3.1 iosize: min=512 (bytes) max=1024 (blocks) public: slice=3 offset=65792 len=70539872 disk_offset=0 private: slice=3 offset=256 len=65536 disk_offset=0 update: time=1335608893 seqno=0.17 ssb: actual_seqno=0.0 headers: 0 240 configs: count=1 len=51360 logs: count=1 len=4096 Defined regions: config priv 000048-000239[000192]: copy=01 offset=000000 enabled config priv 000256-051423[051168]: copy=01 offset=000192 enabled log priv 051424-055519[004096]: copy=01 offset=000000 enabled lockrgn priv 055520-055663[000144]: part=00 offset=000000 Multipathing information: numpaths: 2 sdax state=enabled sdh state=enabled |
Using vxdmpadm command you can also display complete multipathing information and obviously also confirm that each VxVM disk is accessible through 2 paths:
[root@server1 ~]# vxdmpadm getsubpaths dmpnodename=emc0_084b NAME STATE[A] PATH-TYPE[M] CTLR-NAME ENCLR-TYPE ENCLR-NAME ATTRS ================================================================================ sdax ENABLED(A) - c2 EMC emc0 - sdh ENABLED(A) - c0 EMC emc0 - |
[root@server1 ~]# vxdmpadm -v getdmpnode nodename=sdax NAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME SERIAL-NO ARRAY_VOL_ID ======================================================================================================== emc0_084b ENABLED EMC 2 2 0 emc0 960084B000 084B |
[root@server1 ~]# vxdmpadm getdmpnode enclosure=emc0 NAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME ============================================================================== emc0_00f3 ENABLED EMC 2 2 0 emc0 emc0_3d6b ENABLED EMC 2 2 0 emc0 emc0_3d7b ENABLED EMC 2 2 0 emc0 emc0_3d73 ENABLED EMC 2 2 0 emc0 emc0_3ebb ENABLED EMC 2 2 0 emc0 emc0_3eb3 ENABLED EMC 2 2 0 emc0 emc0_3ecb ENABLED EMC 2 2 0 emc0 emc0_3ec3 ENABLED EMC 2 2 0 emc0 . . |
[root@server1 ~]# vxdmpadm list dmpnode dmpnodename=emc0_084b dmpdev = emc0_084b state = enabled enclosure = emc0 cab-sno = 000290102696 asl = libvxemc.so vid = EMC pid = SYMMETRIX array-name = EMC array-type = A/A iopolicy = MinimumQ avid = 084B lun-sno = 960084B000 udid = EMC%5FSYMMETRIX%5F000290102696%5F960084B000 dev-attr = srdf-r1 ###path = name state type transport ctlr hwpath aportID aportWWN attr path = sdax enabled(a) - FC c2 c2 7b-b 50:06:04:84:52:a7:6a:36 - path = sdh enabled(a) - FC c0 c0 10b-b 50:06:04:84:52:a7:6a:39 - |
[root@server1 ~]# vxdmpadm getlungroup dmpnodename=sdax NAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME ========================================================================= emc0_084b ENABLED EMC 2 2 0 emc0 |
[root@server1 ~]# vxdmpadm getsubpaths dmpnodename=sdax NAME STATE[A] PATH-TYPE[M] CTLR-NAME ENCLR-TYPE ENCLR-NAME ATTRS ================================================================================ sdax ENABLED(A) - c2 EMC emc0 - sdh ENABLED(A) - c0 EMC emc0 - |
Remark:
Please note the two controllers that are linked to the FC card we have identified in previous chapter:
[root@server1 ~]# vxdmpadm getsubpaths enclosure=emc0 ctlr=c0 NAME STATE[A] PATH-TYPE[M] DMPNODENAME ENCLR-NAME CTLR ATTRS ================================================================================ sdo ENABLED(A) - emc0_00f3 emc0 c0 - sdz ENABLED(A) - emc0_0837 emc0 c0 - sdh ENABLED(A) - emc0_084b emc0 c0 - sdi ENABLED(A) - emc0_084f emc0 c0 - sdj ENABLED(A) - emc0_0853 emc0 c0 - sdk ENABLED(A) - emc0_0857 emc0 c0 - sdl ENABLED(A) - emc0_085b emc0 c0 - . . |
The vxdisk command can also display the multipathing information:
[root@server1 ~]# vxdisk path -g vgp1417 SUBPATH DANAME DMNAME GROUP STATE sdax emc0_084b vgp1417_04 vgp1417 ENABLED sdh emc0_084b vgp1417_04 vgp1417 ENABLED sdaz emc0_084f vgp1417_05 vgp1417 ENABLED sdi emc0_084f vgp1417_05 vgp1417 ENABLED sdbd emc0_085b vgp1417_08 vgp1417 ENABLED sdl emc0_085b vgp1417_08 vgp1417 ENABLED sdbh emc0_526f vgp1417_09 vgp1417 ENABLED sdp emc0_526f vgp1417_09 vgp1417 ENABLED sdbi emc0_527b vgp1417_10 vgp1417 ENABLED sdq emc0_527b vgp1417_10 vgp1417 ENABLED sdbj emc0_527f vgp1417_11 vgp1417 ENABLED sdr emc0_527f vgp1417_11 vgp1417 ENABLED sdbb emc0_0853 vgp1417_06 vgp1417 ENABLED sdj emc0_0853 vgp1417_06 vgp1417 ENABLED sdbc emc0_0857 vgp1417_07 vgp1417 ENABLED sdk emc0_0857 vgp1417_07 vgp1417 ENABLED |
Let’s come back to the controller and display the ones configured on my system (each enclosure is configured with two controllers):
[root@server1 ~]# vxdmpadm listctlr all CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME ===================================================== c0 EMC ENABLED emc0 c2 EMC ENABLED emc0 c0 EMC ENABLED emc1 c2 EMC ENABLED emc1 c360 OTHER_DISKS ENABLED other_disks |
We can re-conciliate WWN identified in previous chapter with VxDMP commands:
[root@server1 ~]# vxdmpadm getctlr c0 LNAME PNAME VENDOR CTLR-ID ============================================================================================= c0 c0 Qlogic-Corp. 50:06:0b:00:00:c3:5e:0c [root@server1 ~]# vxdmpadm getctlr c2 LNAME PNAME VENDOR CTLR-ID ============================================================================================= c2 c2 Qlogic-Corp. 50:06:0b:00:00:c3:5e:08 |
And confirm, as anticipated, that each of my FC card is wired with only one FC cable. Each of my FC or Host Bus Adapter (HBA) cards has two ports and only one is used:
[root@server1 ~]# vxddladm list hbas HBA-ID DRIVER FIRMWARE DISCOVERY STATE ADDRESS ====================================================================================== c1 8.03.07.03.05.07-k 5.03.16 (496) fabric Offline 50:06:0b:00:00:c3:5e:0e c3 8.03.07.03.05.07-k 5.03.16 (496) fabric Offline 50:06:0b:00:00:c3:5e:0a c360 - - - Online - c0 8.03.07.03.05.07-k 5.03.16 (496) fabric Online 50:06:0b:00:00:c3:5e:0c c2 8.03.07.03.05.07-k 5.03.16 (496) fabric Online 50:06:0b:00:00:c3:5e:08 |
To obtain the configured ports:
[root@server1 ~]# vxddladm list ports PORT-ID HBA-ID STATE ADDRESS ========================================== c0_p3 c0 Online 50:06:04:84:52:a7:6a:39 c0_p2 c0 Online 50:06:04:84:52:a7:6a:08 c0_p1 c0 Online 50:06:04:84:52:a9:ec:09 c2_p3 c2 Online 50:06:04:84:52:a7:6a:36 c2_p2 c2 Online 50:06:04:84:52:a7:6a:07 c2_p1 c2 Online 50:06:04:84:52:a9:ec:06 [root@server1 ~]# vxddladm list targets TARGET-ID ALIAS HBA-ID STATE ADDRESS ===================================================== c0_p3_t0 - c0 Online 50:06:04:84:52:a7:6a:39 c0_p2_t0 - c0 Online 50:06:04:84:52:a7:6a:08 c0_p1_t0 - c0 Online 50:06:04:84:52:a9:ec:09 c2_p3_t0 - c2 Online 50:06:04:84:52:a7:6a:36 c2_p2_t0 - c2 Online 50:06:04:84:52:a7:6a:07 c2_p1_t0 - c2 Online 50:06:04:84:52:a9:ec:06 [root@server1 ~]# vxddladm list targets hba=c0 TARGET-ID ALIAS HBA-ID STATE ADDRESS ===================================================== c0_p3_t0 - c0 Online 50:06:04:84:52:a7:6a:39 c0_p2_t0 - c0 Online 50:06:04:84:52:a7:6a:08 c0_p1_t0 - c0 Online 50:06:04:84:52:a9:ec:09 |
To understand which devices are using which ports and the name of the VxVM driver:
[root@server1 ~]# vxddladm list devices DEVICE TARGET-ID STATE DDL-STATUS (ASL) =============================================================== cciss/c0d0 - Online - sdg c0_p3_t0 Online SKIPPED sdi c0_p3_t0 Online CLAIMED (libvxemc.so) sdf c0_p3_t0 Online SKIPPED sdh c0_p3_t0 Online CLAIMED (libvxemc.so) sde c0_p3_t0 Online SKIPPED sdd c0_p3_t0 Online SKIPPED sdr c0_p3_t0 Online CLAIMED (libvxemc.so) . . [root@server1 ~]# vxddladm list devices target=c0_p1_t0 DEVICE TARGET-ID STATE DDL-STATUS (ASL) =============================================================== sdbl c0_p1_t0 Online CLAIMED (libvxemc.so) sdbk c0_p1_t0 Online SKIPPED |
To get the list of supported enclosures:
[root@server1 ~]# vxddladm listsupport all LIBNAME VID PID ================================================================================================= libvxpurple.so SUN T300 libvxveritas.so VERITAS All libvxhillsannet2.so DotHill SANnet II FC, SANnet II SCSI, SANnet II SATA, SANnet II U320 libvxhds.so HITACHI All libvxpp.so EMC, DGC All libvxFJTSYe8k.so FUJITSU All libvxfje3k4ka.so FUJITSU E3000, E400A libvxibmds6k.so IBM 1750 libvxsun7x10.so SUN Sun Storage 7410, Sun Storage 7310, Sun Storage 7210, Sun Storage 7110, ZFS Storage 7120, ZFS Storage 7320, ZFS Storage 7420, ZFS Storage 7720 libvxfsc.so FSC FibreCAT_SX1 libvxxp1281024.so HP All libvxlsiall.so IBM, SUN, STK, SGI All libvxnipnyis.so NEC iStorage 1000, iStorage 2000, iStorage 4000 libvxtsbaf.so TOSHIBA AF_AF3500, AF_AF1500, AF2_AF7000, AF2_AF2000, AF3_AF7500, AF3_AF2500 libvxfusionio.so FIO ioDrive libvxhpalua.so HP, COMPAQ HSV101, HSV111 (C)COMPAQ, HSV111, HSV200, HSV210, HSV300, HSV400, HSV450 libvxxp12k.so HP All libvxhitachi.so HITACHI DF350, DF400, DF400F, DF500, DF500F libvxpillaraxiom.so Pillar Axiom 300, Axiom 500, Axiom 600 libvxFJTSYe6k.so FUJITSU E6000 libvxxiv.so XIV, IBM NEXTRA, 2810XIV libvxxp256.so HP All libvxramsan.so TMS RamSan 400 libvxhdsalua.so HITACHI DF600, DF600-V, DF600F, DF600F-V libvxhpmsa.so HP MSA VOLUME libvxhds9980.so HITACHI All libvxhdsusp.so HITACHI All libvxCLARiiON.so DGC All libvxnetapp.so NETAPP All libvxeqlogic.so EQLOGIC 100E-00 libvxemc.so EMC SYMMETRIX libvxddns2a.so DDN S2A 9550, S2A 9900, S2A 9700 libvxfjtsye2k.so FUJITSU E2000, ETERNUS_DXL libvxmsa2kfc_sa.so HP MSA2312fc, MSA2324fc, MSA2012sa, MSA2312sa, MSA2324sa, MSA2312i, MSA2324i, P2000 G3 FC, P2000G3 FC/iSCSI, P2000 G3 SAS, P2000 G3 iSCSI libvxdothill.so DotHill R/Evo 2730-2R, R/Evo 2530-2R, R/Evo 2330-2R, R/Evo 2130-2RX, R/Evo 2130-2J, R/Evo 5730-2R libvxcompellent.so COMPELNT Compellent Vol libvxibmds8k.so IBM 2107 libvxmsa2k.so HP MSA2012fc, MSA2212fc, MSA2012i libvxxiotechE5k.so XIOTECH ISE1400 libvxcopan.so COPANSYS 8814, 8818 libvxibmsvc.so IBM 2145, 2062 libvxhuawei.so HUAWEI, HS S5300, S5500, S5600, S2300, S2100, V1800, V1500, VIS6000, S2300E, S2600, S6800E, S8000-I, S8000 libvx3par.so 3PARdata VV libvxhpsvsp.so HP HSVX740 libvxshark.so IBM 2105 |
And more information (Active/Active mode) on the one I use:
[root@server1 ~]# vxddladm listsupport libname=libvxemc.so ATTR_NAME ATTR_VALUE ================================================================================================= LIBNAME libvxemc.so VID EMC PID SYMMETRIX ARRAY_TYPE A/A ARRAY_NAME EMC |
Basic commands
Disk groups
How to determine if a disk is associated to a disk group:
[root@server1 ~]# vxdisk -s list sdax Disk: emc0_084b type: auto flags: online ready private autoconfig noautoimport imported clone_disk guid: - udid: EMC%5FSYMMETRIX%5F000290102696%5F960084B000 site: - diskid: 1325653859.61.server1 dgname: vgp1417 dgid: 1318435685.29.server1 hostid: server1 info: format=cdsdisk,privoffset=256,pubslice=3,privslice=3 |
Disk groups list:
[root@server1 ~]# vxdg list NAME STATE ID vgp1417a enabled,cds 1318432411.19.server1 vgoem3 enabled,cds 1320851456.27.server1 vgptws02a enabled,cds 1318847636.93.server1 . . |
More information of a disk group:
[root@server1 ~]# vxdg list vgp1417 Group: vgp1417 dgid: 1318435685.29.server1 import-id: 1024.62 flags: cds version: 160 alignment: 8192 (bytes) ssb: on autotagging: on detach-policy: global dg-fail-policy: dgdisable copies: nconfig=default nlog=default config: seqno=0.1556 permlen=51360 free=51325 templen=17 loglen=4096 config disk emc0_084b copy 1 len=51360 state=clean online config disk emc0_084f copy 1 len=51360 state=clean online config disk emc0_085b copy 1 len=51360 state=clean online config disk emc0_526f copy 1 len=51360 disabled config disk emc0_527b copy 1 len=51360 disabled config disk emc0_527f copy 1 len=51360 disabled config disk emc0_0853 copy 1 len=51360 state=clean online config disk emc0_0857 copy 1 len=51360 state=clean online log disk emc0_084b copy 1 len=4096 log disk emc0_084f copy 1 len=4096 log disk emc0_085b copy 1 len=4096 log disk emc0_526f copy 1 len=4096 disabled log disk emc0_527b copy 1 len=4096 disabled log disk emc0_527f copy 1 len=4096 disabled log disk emc0_0853 copy 1 len=4096 log disk emc0_0857 copy 1 len=4096 |
Free space of a disk group:
[root@server1 ~]# vxdg -g vgp1417 free DISK DEVICE TAG OFFSET LENGTH FLAGS vgp1417_04 emc0_084b emc0_084b 56623104 13916768 - vgp1417_05 emc0_084f emc0_084f 64389600 6150272 - vgp1417_09 emc0_526f emc0_526f 17592064 52947808 - vgp1417_10 emc0_527b emc0_527b 0 70539872 - vgp1417_11 emc0_527f emc0_527f 0 70539872 - |
Remark:
So how much at the end ? For vgp1417_04 (emc0_084b) it is 13916768*512 (sector size)/(1024 * 1024) MB i.e. 6795MB free (I have already seen more user friendly command output)…
Even if not written in official VxVM documentation there is an alternative command with a much better output:
[root@server1 ~]# vxassist -g vgp1417 maxsize Maximum volume size: 109616431104 (104538Mb) |
Complete information of a disk group (subdisks, plexes, …):
[root@server1 ~]# vxprint -g vgp1417 TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0 dg vgp1417 vgp1417 - - - - - - dm vgp1417_04 emc0_084b - 70539872 - - - - dm vgp1417_05 emc0_084f - 70539872 - - - - dm vgp1417_06 emc0_0853 - 70539872 - - - - dm vgp1417_07 emc0_0857 - 70539872 - - - - dm vgp1417_08 emc0_085b - 70539872 - - - - dm vgp1417_09 emc0_526f - 70539872 - - - - dm vgp1417_10 emc0_527b - 70539872 - - - - dm vgp1417_11 emc0_527f - 70539872 - - - - v lvol1 fsgen ENABLED 6291456 - ACTIVE - - pl lvol1-01 lvol1 ENABLED 6291456 - ACTIVE - - sd vgp1417_04-01 lvol1-01 ENABLED 6291456 0 - - - . . |
Volumes
Volumes of a disk group:
[root@server1 ~]# vxprint -hvt -g vgp1417 V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE DC NAME PARENTVOL LOGVOL SP NAME SNAPVOL DCO EX NAME ASSOC VC PERMS MODE STATE v lvol1 - ENABLED ACTIVE 6291456 SELECT - fsgen pl lvol1-01 lvol1 ENABLED ACTIVE 6291456 CONCAT - RW sd vgp1417_04-01 lvol1-01 vgp1417_04 0 6291456 0 emc0_084b ENA v lvol2 - ENABLED ACTIVE 20971520 SELECT - fsgen . . |
More information of a particular volume:
[root@server1 ~]# vxprint -g vgp1417 -t lvol1 V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE v lvol1 - ENABLED ACTIVE 6291456 SELECT - fsgen |
Plexes
Plexes of a disk group:
[root@server1 ~]# vxprint -pt -g vgp1417 PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE pl lvol1-01 lvol1 ENABLED ACTIVE 6291456 CONCAT - RW pl lvol2-01 lvol2 ENABLED ACTIVE 20971520 CONCAT - RW pl lvol3-01 lvol3 ENABLED ACTIVE 2097152 CONCAT - RW pl lvol4-01 lvol4 ENABLED ACTIVE 6291456 CONCAT - RW pl lvol5-01 lvol5 ENABLED ACTIVE 4194304 CONCAT - RW pl lvol6-01 lvol6 ENABLED ACTIVE 14680064 CONCAT - RW pl lvol7-01 lvol7 ENABLED ACTIVE 44040192 CONCAT - RW pl lvol8-01 lvol8 ENABLED ACTIVE 169869312 CONCAT - RW pl lvol9-01 lvol9 ENABLED ACTIVE 12582912 CONCAT - RW pl lvol10-01 lvol10 ENABLED ACTIVE 27262976 CONCAT - RW pl lvol11-01 lvol11 ENABLED ACTIVE 41943040 CONCAT - RW |
More information of a plex:
[root@server1 ~]# vxprint -g vgp1417 -l lvol1-01 Plex: lvol1-01 info: len=6291456 type: layout=CONCAT state: state=ACTIVE kernel=ENABLED io=read-write assoc: vol=lvol1 sd=vgp1417_04-01 flags: busy complete mediatype: hdd |
Subdisks
Subdisks of a particular disk group:
[root@server1 ~]# vxprint -st -g vgp1417 SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE sd vgp1417_04-01 lvol1-01 vgp1417_04 0 6291456 0 emc0_084b ENA sd vgp1417_04-02 lvol9-01 vgp1417_04 6291456 12582912 0 emc0_084b ENA sd vgp1417_04-03 lvol10-01 vgp1417_04 18874368 27262976 0 emc0_084b ENA sd vgp1417_04-04 lvol6-01 vgp1417_04 46137344 10485760 4194304 emc0_084b ENA sd vgp1417_05-01 lvol5-01 vgp1417_05 0 4194304 0 emc0_084f ENA sd vgp1417_05-02 lvol7-01 vgp1417_05 4194304 31457280 0 emc0_084f ENA sd vgp1417_05-03 lvol11-01 vgp1417_05 35651584 16155104 0 emc0_084f ENA sd vgp1417_05-04 lvol7-01 vgp1417_05 51806688 12582912 31457280 emc0_084f ENA sd vgp1417_06-01 lvol2-01 vgp1417_06 0 20971520 0 emc0_0853 ENA sd vgp1417_06-02 lvol11-01 vgp1417_06 20971520 25787936 16155104 emc0_0853 ENA sd vgp1417_06-03 lvol8-01 vgp1417_06 46759456 23780416 114269008 emc0_0853 ENA sd vgp1417_07-01 lvol8-01 vgp1417_07 0 56312048 0 emc0_0857 ENA sd vgp1417_07-02 lvol8-01 vgp1417_07 56312048 14227824 138049424 emc0_0857 ENA sd vgp1417_08-01 lvol3-01 vgp1417_08 0 2097152 0 emc0_085b ENA sd vgp1417_08-02 lvol4-01 vgp1417_08 2097152 6291456 0 emc0_085b ENA sd vgp1417_08-03 lvol6-01 vgp1417_08 8388608 4194304 0 emc0_085b ENA sd vgp1417_08-04 lvol8-01 vgp1417_08 12582912 57956960 56312048 emc0_085b ENA sd vgp1417_09-01 lvol8-01 vgp1417_09 0 17592064 152277248 emc0_526f ENA |
More information of a subdisk:
[root@server1 ~]# vxprint -g vgp1417 -l vgp1417_04 Disk: vgp1417_04 info: diskid=1325653859.61.server1 assoc: device=emc0_084b type=auto flags: autoconfig device: path=/dev/vx/dmp/emc0_084bs3 devinfo: publen=70539872 privlen=65536 mediatype: hdd |
Performance monitoring
When having I/O issues the standard command you would use could be iostat:
[root@server1 ~]# iostat -mdx Linux 2.6.18-274.3.1.el5 (server1) 06/07/2012 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util cciss/c0d0 6.45 40.26 3.54 25.84 0.05 0.17 15.73 0.82 28.01 5.13 15.07 cciss/c0d0p1 0.00 0.00 0.00 0.00 0.00 0.00 31.97 0.00 21.14 14.42 0.00 cciss/c0d0p2 6.45 40.26 3.54 25.84 0.05 0.17 15.73 0.82 28.01 5.13 15.07 sda 0.00 0.00 0.00 0.00 0.00 0.00 3.98 0.00 1.87 1.87 0.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 4.51 4.51 0.00 sdc 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 1.83 1.83 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 1.66 1.66 0.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 2.36 2.36 0.00 sdf 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 2.51 2.51 0.00 sdg 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 2.34 2.34 0.00 . . |
Remark:
As stated in man pages if you do not specify interval and count parameter displayed statistics are since server boot time so not really relevant…
[root@server1 ~]# iostat -md -p sda 1 2 Linux 2.6.18-274.3.1.el5 (server1) 06/07/2012 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 0.00 0.00 0.00 7 0 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 0.00 0.00 0.00 0 0 |
You can also use nmon.
With the command we have seen in previous chapter you are now able to identify to which disk group/volume/VXVM disks it is associated…
But even more interesting you can use VxVM command to get detailed information of disk using Operation System disk names:
[root@server1 ~]# vxdmpadm iostat show pathname=sdax cpu usage = 374381us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms) PATHNAME READS WRITES READS WRITES READS WRITES sdax 1182902 4756775 328093320 294705430 5.10 20.82 |
[root@server1 ~]# vxdmpadm iostat show dmpnodename=sdax cpu usage = 375891us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms) PATHNAME READS WRITES READS WRITES READS WRITES sdax 1183412 4762187 328098810 295062539 5.10 20.82 sdh 1133886 4773058 324904565 294251856 5.35 20.75 |
[root@server1 ~]# vxdmpadm iostat show pathname=sdax interval=2 count=2 cpu usage = 374396us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms) PATHNAME READS WRITES READS WRITES READS WRITES sdax 1182902 4756777 328093320 294705543 5.10 20.82 sdax 0 0 0 0 0.00 0.00 |
[root@server1 ~]# vxdmpadm -e iostat show pathname=sdax cpu usage = 374509us per cpu memory = 1114112b ERROR I/Os PATHNAME READS WRITES sdax 0 0 |
As well as using VxVM disk names:
[root@server1 ~]# vxdmpadm -q iostat show dmpnodename=emc0_084b cpu usage = 374484us per cpu memory = 1114112b QUEUED I/Os PENDING I/Os DMPNODENAME READS WRITES emc0_084b 0 0 0 |
[root@server1 ~]# vxdmpadm -z iostat show dmpnodename=emc0_084b cpu usage = 374558us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms) PATHNAME READS WRITES READS WRITES READS WRITES sdax 1182910 4756814 328093384 294707962 5.10 20.82 sdh 1133430 4767643 324899821 293884532 5.35 20.76 |
[root@server1 ~]# vxdmpadm -u bytes -z iostat show dmpnodename=emc0_084b cpu usage = 374888us per cpu memory = 1114112b OPERATIONS BYTES AVG TIME(ms) PATHNAME READS WRITES READS WRITES READS WRITES sdax 1183012 4760612 480699392 707400192 5.10 20.82 sdh 1133494 4771329 3140286976 287247360 5.35 20.76 |
You can display information of controllers to see if your multipathing policy is working well:
[root@server1 ~]# vxdmpadm iostat show groupby=ctlr ctlr=c0 cpu usage = 374524us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms) CTLRNAME READS WRITES READS WRITES READS WRITES c0 135753949 218251362 3572140375 2841606614 7.62 7.10 [root@server1 ~]# vxdmpadm iostat show groupby=ctlr ctlr=c2 cpu usage = 374541us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms) CTLRNAME READS WRITES READS WRITES READS WRITES c2 135349009 215663711 3357968941 2737547261 7.96 6.92 |
Those interesting information can also be display at disk group level:
[root@server1 ~]# vxstat -u m -g vgp1417 OPERATIONS BLOCKS AVG TIME(ms) TYP NAME READ WRITE READ WRITE READ WRITE vol lvol1 827072 413478 17059m 29032m 1.51 7.34 vol lvol2 761459 357294 3465m 2781m 4.46 10.80 vol lvol3 9518191 7008657 175801m 56452m 0.26 0.02 vol lvol4 610864 15518552 259528m 176116m 1.86 0.19 vol lvol5 170534 303104 886m 998m 4.50 8.68 vol lvol6 2646674 2379866 69607m 16601m 1.19 1.66 vol lvol7 16854588 20012885 206030m 119118m 0.10 0.16 vol lvol8 79042007 128500851 899927m 689835m 0.04 0.02 vol lvol9 301556 4788315 2090m 74546m 3.00 0.83 vol lvol10 980251 4487076 287228m 193539m 3.20 0.62 vol lvol11 82761 10693 338m 82m 2.00 11.79 |
Remark
-d, -p, -s and -u m options are interesting to display information at different granularity level. -r option can also be used to reset statistics and focus on current period only (to remove boot time statistics or reorganization operations)…
There is also vxtrace command to trace operations on volumes but I have, most probably, not really understood its added value as the output is too much verbose and does not contains any synthetic information:
[root@server1 ~]# vxtrace -g vgp1417 -o disk lvol8 57 START write disk emc0_0857 op 0 block 50782432 len 16 57 END write disk emc0_0857 op 0 block 50782432 len 16 time 3 58 START write disk emc0_0857 op 0 block 195779 len 1 58 END write disk emc0_0857 op 0 block 195779 len 1 time 1 59 START write disk emc0_0857 op 0 block 1246640 len 16 59 END write disk emc0_0857 op 0 block 1246640 len 16 time 2 . . |
Remark:
I/O statistics are gathered by default but when you system is stable or when you have high I/O demand it may be interesting to deactivate it to maximize throughput. To check if statistics are gathered:
[root@server1 ~]# vxtune vol_stats_enable 1 |
Then to de-activate them, as well as for DMP part (not possible to check if there are activated or not):
[root@server1 ~]# vxtune vol_stats_enable 0 [root@server1 ~]# vxdmpadm iostat stop |
If you have slow response from a device group it may be also interesting to check if you have running operation in background:
[root@server1 ~]# vxtask -h list TASKID PTID TYPE/STATE PCT PROGRESS |
Even if default VxVM parameters suit most workloads you can change them using vxdmpadm command. To display tunable VxDMP parameters use:
[root@server1 ~]# vxdmpadm gettune all Tunable Current Value Default Value ------------------------------ ------------- ------------- dmp_cache_open on on dmp_daemon_count 10 10 dmp_delayq_interval 15 15 dmp_enable_restore on on dmp_fast_recovery on on dmp_health_time 60 60 dmp_log_level 1 1 dmp_low_impact_probe on on dmp_lun_retry_timeout 0 0 dmp_path_age 300 300 dmp_pathswitch_blks_shift 9 9 dmp_probe_idle_lun on on dmp_probe_threshold 5 5 dmp_queue_depth 32 32 dmp_restore_cycles 10 10 dmp_restore_interval 300 300 dmp_restore_policy check_disabled check_disabled dmp_retry_count 5 5 dmp_scsi_timeout 20 20 dmp_sfg_threshold 1 1 dmp_stat_interval 1 1 dmp_monitor_fabric on on dmp_monitor_osevent on on dmp_native_support off off |
Remark:
You may also want to use vxbench available in VRTSspt package…
Very good information , Thanks Mr.Yannick
Thanks to you for stopping by !
Yannick.
Great information. Very helpful when troubleshooting performance issues.
Very good information , Thanks Mr.Yannick