Veritas Volume Manager (VxVM) command line examples

Preamble

Before entering in Veritas Volume Manager (VxVM) details we must go in fine detail of our hardware configuration. Let’s first confirm at Linux level that I see my 2 fiber channel (FC) cards with each 2 FC ports:

[root@server1 ~]# lspci | grep -i fibre
09:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
09:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
0c:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
0c:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
[root@server1 ~]# ll /sys/class/fc_host/host*/device
lrwxrwxrwx 1 root root 0 Jun  5 17:49 /sys/class/fc_host/host0/device -> ../../../devices/pci0000:00/0000:00:03.0/0000:09:00.0/host0
lrwxrwxrwx 1 root root 0 Jun  5 17:51 /sys/class/fc_host/host1/device -> ../../../devices/pci0000:00/0000:00:03.0/0000:09:00.1/host1
lrwxrwxrwx 1 root root 0 Jun  5 17:51 /sys/class/fc_host/host2/device -> ../../../devices/pci0000:00/0000:00:05.0/0000:0c:00.0/host2
lrwxrwxrwx 1 root root 0 Jun  5 17:51 /sys/class/fc_host/host3/device -> ../../../devices/pci0000:00/0000:00:05.0/0000:0c:00.1/host3

We will see this later but we need correspondence between SCSI chain number and its World Wide Name (WWN), here it is:

[root@server1 ~]# for host in `ls /sys/class/fc_host/`; do echo "$host WWN: `cat /sys/class/fc_host/$host/port_name`"; done
host0 WWN: 0x50060b0000c35e0c
host1 WWN: 0x50060b0000c35e0e
host2 WWN: 0x50060b0000c35e08
host3 WWN: 0x50060b0000c35e0a

So if everything is fiber wired (FC cards/ports to FC switches) I should see any disk array (enclosure) device/LUN as 4 equivalent name. Please note that this is absolutely not mandatory, what’s really important in my configuration is to have each disk visible through my 2 FC cards (so by one port minimum so each disk/LUN will have 2 names).

Please note that this post is more for DBAs to see and understand what have been done by you system teammates i.e. I’m not entering in the creation/modification of VxVM disk/disk group/controller and so on…

All commands have been executed on a Red Hat Enterprise Linux Server release 5.5 (Tikanga) server configured with Veritas Volume Manager release VRTSvxvm-5.1.132.000-SP1RP2_RHEL5.

To avoid putting real server name I’m using the below tips for my prompt customization:

export PS1="[\u@server1 \W]# "

Dynamic Multipathing (DMP)

Veritas Dynamic Multi-Pathing (DMP or VxDMP) is the VxVM functionality implemented to support disk availability from multiple paths. Even if it looks complex to start with this functionality it is unfortunately mandatory before doing the more simple things as it is based configuration on which all other components will be created (subdisks, plexes, volumes).

Configured disk arrays or enclosures accessible from this server:

[root@server1 ~]# vxdmpadm listenclosure
ENCLR_NAME        ENCLR_TYPE     ENCLR_SNO      STATUS       ARRAY_TYPE     LUN_COUNT
=======================================================================================
emc0              EMC            000290102696         CONNECTED    A/A        54
emc1              EMC            000290105264         CONNECTED    A/A         1
other_disks       OTHER_DISKS    OTHER_DISKS          CONNECTED    OTHER_DISKS   1

I/O policy default configuration of this enclosure:

[root@server1 ~]# vxdmpadm getattr enclosure emc0
ENCLR_NAME      ATTR_NAME                     DEFAULT        CURRENT
============================================================================
emc0           iopolicy                      MinimumQ       MinimumQ
emc0           partitionsize                 512            512
emc0           use_all_paths                 -              -
emc0           failover_policy               Global         Global
emc0           recoveryoption[throttle]      Nothrottle[0]  Nothrottle[0]
emc0           recoveryoption[errorretry]    Timebound[300] Timebound[300]
emc0           redundancy                    0              0
emc0           dmp_lun_retry_timeout         0              0
emc0           failovermode                  -              -

Disk default naming convention of VxVM:

[root@server1 ~]# vxddladm get namingscheme
NAMING_SCHEME       PERSISTENCE    LOWERCASE      USE_AVID
============================================================
Enclosure Based     Yes            Yes            Yes

List of configured disk on your system:

[root@server1 ~]# vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
cciss/c0d0   auto:none      -            -           online invalid       cciss/c0d0       -
emc0_00f3    auto:cdsdisk   -            -           online               sdbg             std
emc0_3d6b    auto:cdsdisk   vgp1148_01   vgp1148     online clone_disk    sdbn             srdf-r1
emc0_3d7b    auto:cdsdisk   vgp1148_03   vgp1148     online clone_disk    sdbp             srdf-r1
emc0_3d73    auto:cdsdisk   vgp1148_02   vgp1148     online clone_disk    sdbo             srdf-r1
emc0_3ebb    auto:cdsdisk   vgp1118_02   vgp1118     online clone_disk    sdca             srdf-r1
emc0_3eb3    auto:cdsdisk   vgp1118_01   vgp1118     online clone_disk    sdbx             srdf-r1
emc0_3ecb    auto:cdsdisk   vgp1318_01   vgp1318     online clone_disk    sdcc             srdf-r1
emc0_3ec3    auto:cdsdisk   vgp1118_03   vgp1118     online clone_disk    sdcb             srdf-r1
.
.

Disks of a disk group (please note it makes relationship between VxVM disks and the ones you see at system level):

[root@server1 ~]# vxdisk -g vgp1417 -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
emc0_084b    auto:cdsdisk   vgp1417_04   vgp1417     online clone_disk    sdax             srdf-r1
emc0_084f    auto:cdsdisk   vgp1417_05   vgp1417     online clone_disk    sdaz             srdf-r1
emc0_085b    auto:cdsdisk   vgp1417_08   vgp1417     online clone_disk    sdbd             srdf-r1
emc0_526f    auto:cdsdisk   vgp1417_09   vgp1417     online clone_disk    sdbh             srdf-r1
emc0_527b    auto:cdsdisk   vgp1417_10   vgp1417     online clone_disk    sdbi             srdf-r1
emc0_527f    auto:cdsdisk   vgp1417_11   vgp1417     online clone_disk    sdbj             srdf-r1
emc0_0853    auto:cdsdisk   vgp1417_06   vgp1417     online clone_disk    sdbb             srdf-r1
emc0_0857    auto:cdsdisk   vgp1417_07   vgp1417     online clone_disk    sdbc             srdf-r1

Main problem with above command is that you don’t see multipathing details of VxVM disk, to go deeper in detail use:

[root@server1 ~]# vxdisk list emc0_084b
Device:    emc0_084b
devicetag: emc0_084b
type:      auto
hostid:    server1
disk:      name=vgp1417_04 id=1325653859.61.server1
group:     name=vgp1417 id=1318435685.29.server1
info:      format=cdsdisk,privoffset=256,pubslice=3,privslice=3
flags:     online ready private autoconfig noautoimport imported clone_disk
pubpaths:  block=/dev/vx/dmp/emc0_084bs3 char=/dev/vx/rdmp/emc0_084bs3
guid:      -
udid:      EMC%5FSYMMETRIX%5F000290102696%5F960084B000
site:      -
version:   3.1
iosize:    min=512 (bytes) max=1024 (blocks)
public:    slice=3 offset=65792 len=70539872 disk_offset=0
private:   slice=3 offset=256 len=65536 disk_offset=0
update:    time=1335608893 seqno=0.17
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=51360
logs:      count=1 len=4096
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-051423[051168]: copy=01 offset=000192 enabled
 log      priv 051424-055519[004096]: copy=01 offset=000000 enabled
 lockrgn  priv 055520-055663[000144]: part=00 offset=000000
Multipathing information:
numpaths:   2
sdax            state=enabled
sdh             state=enabled

Remark:
You may also use the opposite command:

[root@server1 ~]# vxdisk list sdax
Device:    emc0_084b
devicetag: emc0_084b
type:      auto
hostid:    server1
disk:      name=vgp1417_04 id=1325653859.61.server1
group:     name=vgp1417 id=1318435685.29.server1
info:      format=cdsdisk,privoffset=256,pubslice=3,privslice=3
flags:     online ready private autoconfig noautoimport imported clone_disk
pubpaths:  block=/dev/vx/dmp/emc0_084bs3 char=/dev/vx/rdmp/emc0_084bs3
guid:      -
udid:      EMC%5FSYMMETRIX%5F000290102696%5F960084B000
site:      -
version:   3.1
iosize:    min=512 (bytes) max=1024 (blocks)
public:    slice=3 offset=65792 len=70539872 disk_offset=0
private:   slice=3 offset=256 len=65536 disk_offset=0
update:    time=1335608893 seqno=0.17
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=51360
logs:      count=1 len=4096
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-051423[051168]: copy=01 offset=000192 enabled
 log      priv 051424-055519[004096]: copy=01 offset=000000 enabled
 lockrgn  priv 055520-055663[000144]: part=00 offset=000000
Multipathing information:
numpaths:   2
sdax            state=enabled
sdh             state=enabled

Using vxdmpadm command you can also display complete multipathing information and obviously also confirm that each VxVM disk is accessible through 2 paths:

[root@server1 ~]# vxdmpadm getsubpaths dmpnodename=emc0_084b
NAME         STATE[A]   PATH-TYPE[M] CTLR-NAME  ENCLR-TYPE   ENCLR-NAME    ATTRS
================================================================================
sdax         ENABLED(A)    -          c2         EMC          emc0             -
sdh          ENABLED(A)    -          c0         EMC          emc0             -
[root@server1 ~]# vxdmpadm -v getdmpnode nodename=sdax
NAME                 STATE        ENCLR-TYPE   PATHS  ENBL  DSBL  ENCLR-NAME   SERIAL-NO            ARRAY_VOL_ID
========================================================================================================
emc0_084b            ENABLED      EMC          2      2     0     emc0         960084B000           084B
[root@server1 ~]# vxdmpadm getdmpnode enclosure=emc0
NAME                 STATE        ENCLR-TYPE   PATHS  ENBL  DSBL  ENCLR-NAME
==============================================================================
emc0_00f3            ENABLED      EMC          2      2     0     emc0
emc0_3d6b            ENABLED      EMC          2      2     0     emc0
emc0_3d7b            ENABLED      EMC          2      2     0     emc0
emc0_3d73            ENABLED      EMC          2      2     0     emc0
emc0_3ebb            ENABLED      EMC          2      2     0     emc0
emc0_3eb3            ENABLED      EMC          2      2     0     emc0
emc0_3ecb            ENABLED      EMC          2      2     0     emc0
emc0_3ec3            ENABLED      EMC          2      2     0     emc0
.
.
[root@server1 ~]# vxdmpadm list dmpnode dmpnodename=emc0_084b
dmpdev          = emc0_084b
state           = enabled
enclosure       = emc0
cab-sno         = 000290102696
asl             = libvxemc.so
vid             = EMC
pid             = SYMMETRIX
array-name      = EMC
array-type      = A/A
iopolicy        = MinimumQ
avid            = 084B
lun-sno         = 960084B000
udid            = EMC%5FSYMMETRIX%5F000290102696%5F960084B000
dev-attr        = srdf-r1
###path         = name state type transport ctlr hwpath aportID aportWWN attr
path            = sdax enabled(a) - FC c2 c2 7b-b 50:06:04:84:52:a7:6a:36 -
path            = sdh enabled(a) - FC c0 c0 10b-b 50:06:04:84:52:a7:6a:39 -
[root@server1 ~]# vxdmpadm getlungroup dmpnodename=sdax
NAME                 STATE     ENCLR-TYPE   PATHS  ENBL  DSBL  ENCLR-NAME
=========================================================================
emc0_084b            ENABLED      EMC          2      2     0     emc0
[root@server1 ~]# vxdmpadm getsubpaths dmpnodename=sdax
NAME         STATE[A]   PATH-TYPE[M] CTLR-NAME  ENCLR-TYPE   ENCLR-NAME    ATTRS
================================================================================
sdax         ENABLED(A)    -          c2         EMC          emc0             -
sdh          ENABLED(A)    -          c0         EMC          emc0             -

Remark:
Please note the two controllers that are linked to the FC card we have identified in previous chapter:

[root@server1 ~]# vxdmpadm getsubpaths enclosure=emc0 ctlr=c0
NAME         STATE[A]   PATH-TYPE[M] DMPNODENAME  ENCLR-NAME   CTLR    ATTRS
================================================================================
sdo          ENABLED(A)   -          emc0_00f3    emc0         c0        -
sdz          ENABLED(A)   -          emc0_0837    emc0         c0        -
sdh          ENABLED(A)   -          emc0_084b    emc0         c0        -
sdi          ENABLED(A)   -          emc0_084f    emc0         c0        -
sdj          ENABLED(A)   -          emc0_0853    emc0         c0        -
sdk          ENABLED(A)   -          emc0_0857    emc0         c0        -
sdl          ENABLED(A)   -          emc0_085b    emc0         c0        -
.
.

The vxdisk command can also display the multipathing information:

[root@server1 ~]# vxdisk path -g vgp1417
SUBPATH                     DANAME               DMNAME       GROUP        STATE
sdax                        emc0_084b            vgp1417_04   vgp1417      ENABLED
sdh                         emc0_084b            vgp1417_04   vgp1417      ENABLED
sdaz                        emc0_084f            vgp1417_05   vgp1417      ENABLED
sdi                         emc0_084f            vgp1417_05   vgp1417      ENABLED
sdbd                        emc0_085b            vgp1417_08   vgp1417      ENABLED
sdl                         emc0_085b            vgp1417_08   vgp1417      ENABLED
sdbh                        emc0_526f            vgp1417_09   vgp1417      ENABLED
sdp                         emc0_526f            vgp1417_09   vgp1417      ENABLED
sdbi                        emc0_527b            vgp1417_10   vgp1417      ENABLED
sdq                         emc0_527b            vgp1417_10   vgp1417      ENABLED
sdbj                        emc0_527f            vgp1417_11   vgp1417      ENABLED
sdr                         emc0_527f            vgp1417_11   vgp1417      ENABLED
sdbb                        emc0_0853            vgp1417_06   vgp1417      ENABLED
sdj                         emc0_0853            vgp1417_06   vgp1417      ENABLED
sdbc                        emc0_0857            vgp1417_07   vgp1417      ENABLED
sdk                         emc0_0857            vgp1417_07   vgp1417      ENABLED

Let’s come back to the controller and display the ones configured on my system (each enclosure is configured with two controllers):

[root@server1 ~]# vxdmpadm listctlr all
CTLR-NAME       ENCLR-TYPE      STATE      ENCLR-NAME
=====================================================
c0              EMC             ENABLED      emc0
c2              EMC             ENABLED      emc0
c0              EMC             ENABLED      emc1
c2              EMC             ENABLED      emc1
c360            OTHER_DISKS     ENABLED      other_disks

We can re-conciliate WWN identified in previous chapter with VxDMP commands:

[root@server1 ~]# vxdmpadm getctlr c0
LNAME     PNAME                                    VENDOR               CTLR-ID
=============================================================================================
c0        c0                                       Qlogic-Corp.         50:06:0b:00:00:c3:5e:0c
[root@server1 ~]# vxdmpadm getctlr c2
LNAME     PNAME                                    VENDOR               CTLR-ID
=============================================================================================
c2        c2                                       Qlogic-Corp.         50:06:0b:00:00:c3:5e:08

And confirm, as anticipated, that each of my FC card is wired with only one FC cable. Each of my FC or Host Bus Adapter (HBA) cards has two ports and only one is used:

[root@server1 ~]# vxddladm list hbas
HBA-ID  DRIVER          FIRMWARE        DISCOVERY                 STATE    ADDRESS
======================================================================================
c1      8.03.07.03.05.07-k 5.03.16 (496)   fabric                    Offline 50:06:0b:00:00:c3:5e:0e
c3      8.03.07.03.05.07-k 5.03.16 (496)   fabric                    Offline 50:06:0b:00:00:c3:5e:0a
c360    -               -               -                         Online  -
c0      8.03.07.03.05.07-k 5.03.16 (496)   fabric                    Online  50:06:0b:00:00:c3:5e:0c
c2      8.03.07.03.05.07-k 5.03.16 (496)   fabric                    Online  50:06:0b:00:00:c3:5e:08

To obtain the configured ports:

[root@server1 ~]# vxddladm list ports
PORT-ID      HBA-ID   STATE    ADDRESS
==========================================
c0_p3        c0       Online   50:06:04:84:52:a7:6a:39
c0_p2        c0       Online   50:06:04:84:52:a7:6a:08
c0_p1        c0       Online   50:06:04:84:52:a9:ec:09
c2_p3        c2       Online   50:06:04:84:52:a7:6a:36
c2_p2        c2       Online   50:06:04:84:52:a7:6a:07
c2_p1        c2       Online   50:06:04:84:52:a9:ec:06
[root@server1 ~]# vxddladm list targets
TARGET-ID    ALIAS      HBA-ID   STATE    ADDRESS
=====================================================
c0_p3_t0     -          c0       Online   50:06:04:84:52:a7:6a:39
c0_p2_t0     -          c0       Online   50:06:04:84:52:a7:6a:08
c0_p1_t0     -          c0       Online   50:06:04:84:52:a9:ec:09
c2_p3_t0     -          c2       Online   50:06:04:84:52:a7:6a:36
c2_p2_t0     -          c2       Online   50:06:04:84:52:a7:6a:07
c2_p1_t0     -          c2       Online   50:06:04:84:52:a9:ec:06
[root@server1 ~]# vxddladm list targets hba=c0
TARGET-ID    ALIAS      HBA-ID   STATE    ADDRESS
=====================================================
c0_p3_t0     -          c0       Online   50:06:04:84:52:a7:6a:39
c0_p2_t0     -          c0       Online   50:06:04:84:52:a7:6a:08
c0_p1_t0     -          c0       Online   50:06:04:84:52:a9:ec:09

To understand which devices are using which ports and the name of the VxVM driver:

[root@server1 ~]# vxddladm list devices
DEVICE               TARGET-ID    STATE   DDL-STATUS (ASL)
===============================================================
cciss/c0d0           -            Online  -
sdg                  c0_p3_t0     Online  SKIPPED
sdi                  c0_p3_t0     Online  CLAIMED (libvxemc.so)
sdf                  c0_p3_t0     Online  SKIPPED
sdh                  c0_p3_t0     Online  CLAIMED (libvxemc.so)
sde                  c0_p3_t0     Online  SKIPPED
sdd                  c0_p3_t0     Online  SKIPPED
sdr                  c0_p3_t0     Online  CLAIMED (libvxemc.so)
.
.
[root@server1 ~]# vxddladm list devices target=c0_p1_t0
DEVICE               TARGET-ID    STATE   DDL-STATUS (ASL)
===============================================================
sdbl                 c0_p1_t0     Online  CLAIMED (libvxemc.so)
sdbk                 c0_p1_t0     Online  SKIPPED

To get the list of supported enclosures:

[root@server1 ~]# vxddladm listsupport all
LIBNAME             VID                 PID
=================================================================================================
libvxpurple.so      SUN                 T300
libvxveritas.so     VERITAS             All
libvxhillsannet2.so DotHill             SANnet II FC, SANnet II SCSI, SANnet II SATA, SANnet II U320
libvxhds.so         HITACHI             All
libvxpp.so          EMC, DGC            All
libvxFJTSYe8k.so    FUJITSU             All
libvxfje3k4ka.so    FUJITSU             E3000, E400A
libvxibmds6k.so     IBM                 1750
libvxsun7x10.so     SUN                 Sun Storage 7410, Sun Storage 7310, Sun Storage 7210, Sun Storage 7110, ZFS Storage 7120, ZFS Storage 7320, ZFS Storage 7420, ZFS Storage 7720
libvxfsc.so         FSC                 FibreCAT_SX1
libvxxp1281024.so   HP                  All
libvxlsiall.so      IBM, SUN, STK, SGI  All
libvxnipnyis.so     NEC                 iStorage 1000, iStorage 2000, iStorage 4000
libvxtsbaf.so       TOSHIBA             AF_AF3500, AF_AF1500, AF2_AF7000, AF2_AF2000, AF3_AF7500, AF3_AF2500
libvxfusionio.so    FIO                 ioDrive
libvxhpalua.so      HP, COMPAQ          HSV101, HSV111 (C)COMPAQ, HSV111, HSV200, HSV210, HSV300, HSV400, HSV450
libvxxp12k.so       HP                  All
libvxhitachi.so     HITACHI             DF350, DF400, DF400F, DF500, DF500F
libvxpillaraxiom.so Pillar              Axiom 300, Axiom 500, Axiom 600
libvxFJTSYe6k.so    FUJITSU             E6000
libvxxiv.so         XIV, IBM            NEXTRA, 2810XIV
libvxxp256.so       HP                  All
libvxramsan.so      TMS                 RamSan 400
libvxhdsalua.so     HITACHI             DF600, DF600-V, DF600F, DF600F-V
libvxhpmsa.so       HP                  MSA VOLUME
libvxhds9980.so     HITACHI             All
libvxhdsusp.so      HITACHI             All
libvxCLARiiON.so    DGC                 All
libvxnetapp.so      NETAPP              All
libvxeqlogic.so     EQLOGIC             100E-00
libvxemc.so         EMC                 SYMMETRIX
libvxddns2a.so      DDN                 S2A 9550, S2A 9900, S2A 9700
libvxfjtsye2k.so    FUJITSU             E2000, ETERNUS_DXL
libvxmsa2kfc_sa.so  HP                  MSA2312fc, MSA2324fc, MSA2012sa, MSA2312sa, MSA2324sa, MSA2312i, MSA2324i, P2000 G3 FC, P2000G3 FC/iSCSI, P2000 G3 SAS, P2000 G3 iSCSI
libvxdothill.so     DotHill             R/Evo 2730-2R, R/Evo 2530-2R, R/Evo 2330-2R, R/Evo 2130-2RX, R/Evo 2130-2J, R/Evo 5730-2R
libvxcompellent.so  COMPELNT            Compellent Vol
libvxibmds8k.so     IBM                 2107
libvxmsa2k.so       HP                  MSA2012fc, MSA2212fc, MSA2012i
libvxxiotechE5k.so  XIOTECH             ISE1400
libvxcopan.so       COPANSYS            8814, 8818
libvxibmsvc.so      IBM                 2145, 2062
libvxhuawei.so      HUAWEI, HS          S5300, S5500, S5600, S2300, S2100, V1800, V1500, VIS6000, S2300E, S2600, S6800E, S8000-I, S8000
libvx3par.so        3PARdata            VV
libvxhpsvsp.so      HP                  HSVX740
libvxshark.so       IBM                 2105

And more information (Active/Active mode) on the one I use:

[root@server1 ~]# vxddladm listsupport libname=libvxemc.so
ATTR_NAME           ATTR_VALUE
=================================================================================================
LIBNAME             libvxemc.so
VID                 EMC
PID                 SYMMETRIX
ARRAY_TYPE          A/A
ARRAY_NAME          EMC

VxVM Basic commands

Disk groups

How to determine if a disk is associated to a disk group:

[root@server1 ~]# vxdisk -s list sdax
Disk:   emc0_084b
type:   auto
flags:  online ready private autoconfig noautoimport imported clone_disk
guid:   -
udid:   EMC%5FSYMMETRIX%5F000290102696%5F960084B000
site:    -
diskid: 1325653859.61.server1
dgname: vgp1417
dgid:   1318435685.29.server1
hostid: server1
info:   format=cdsdisk,privoffset=256,pubslice=3,privslice=3

Disk groups list:

[root@server1 ~]# vxdg list
NAME         STATE           ID
vgp1417a     enabled,cds          1318432411.19.server1
vgoem3       enabled,cds          1320851456.27.server1
vgptws02a    enabled,cds          1318847636.93.server1
.
.

More information of a disk group:

[root@server1 ~]# vxdg list vgp1417
Group:     vgp1417
dgid:      1318435685.29.server1
import-id: 1024.62
flags:     cds
version:   160
alignment: 8192 (bytes)
ssb:            on
autotagging:    on
detach-policy: global
dg-fail-policy: dgdisable
copies:    nconfig=default nlog=default
config:    seqno=0.1556 permlen=51360 free=51325 templen=17 loglen=4096
config disk emc0_084b copy 1 len=51360 state=clean online
config disk emc0_084f copy 1 len=51360 state=clean online
config disk emc0_085b copy 1 len=51360 state=clean online
config disk emc0_526f copy 1 len=51360 disabled
config disk emc0_527b copy 1 len=51360 disabled
config disk emc0_527f copy 1 len=51360 disabled
config disk emc0_0853 copy 1 len=51360 state=clean online
config disk emc0_0857 copy 1 len=51360 state=clean online
log disk emc0_084b copy 1 len=4096
log disk emc0_084f copy 1 len=4096
log disk emc0_085b copy 1 len=4096
log disk emc0_526f copy 1 len=4096 disabled
log disk emc0_527b copy 1 len=4096 disabled
log disk emc0_527f copy 1 len=4096 disabled
log disk emc0_0853 copy 1 len=4096
log disk emc0_0857 copy 1 len=4096

Free space of a disk group:

[root@server1 ~]# vxdg -g vgp1417 free
DISK         DEVICE       TAG          OFFSET    LENGTH    FLAGS
vgp1417_04   emc0_084b    emc0_084b    56623104  13916768  -
vgp1417_05   emc0_084f    emc0_084f    64389600  6150272   -
vgp1417_09   emc0_526f    emc0_526f    17592064  52947808  -
vgp1417_10   emc0_527b    emc0_527b    0         70539872  -
vgp1417_11   emc0_527f    emc0_527f    0         70539872  -

Remark:
So how much at the end ? For vgp1417_04 (emc0_084b) it is 13916768*512 (sector size)/(1024 * 1024) MB i.e. 6795MB free (I have already seen more user friendly command output)…

Even if not written in official VxVM documentation there is an alternative command with a much better output:

[root@server1 ~]# vxassist -g vgp1417 maxsize
Maximum volume size: 109616431104 (104538Mb)

Complete information of a disk group (subdisks, plexes, …):

[root@server1 ~]# vxprint -g vgp1417
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg vgp1417      vgp1417      -        -        -        -        -       -
 
dm vgp1417_04   emc0_084b    -        70539872 -        -        -       -
dm vgp1417_05   emc0_084f    -        70539872 -        -        -       -
dm vgp1417_06   emc0_0853    -        70539872 -        -        -       -
dm vgp1417_07   emc0_0857    -        70539872 -        -        -       -
dm vgp1417_08   emc0_085b    -        70539872 -        -        -       -
dm vgp1417_09   emc0_526f    -        70539872 -        -        -       -
dm vgp1417_10   emc0_527b    -        70539872 -        -        -       -
dm vgp1417_11   emc0_527f    -        70539872 -        -        -       -
 
v  lvol1        fsgen        ENABLED  6291456  -        ACTIVE   -       -
pl lvol1-01     lvol1        ENABLED  6291456  -        ACTIVE   -       -
sd vgp1417_04-01 lvol1-01    ENABLED  6291456  0        -        -       -
.
.

Volumes

Volumes of a disk group:

[root@server1 ~]# vxprint -hvt -g vgp1417
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
 
v  lvol1        -            ENABLED  ACTIVE   6291456  SELECT    -        fsgen
pl lvol1-01     lvol1        ENABLED  ACTIVE   6291456  CONCAT    -        RW
sd vgp1417_04-01 lvol1-01    vgp1417_04 0      6291456  0         emc0_084b ENA
 
v  lvol2        -            ENABLED  ACTIVE   20971520 SELECT    -        fsgen
.
.

More information of a particular volume:

[root@server1 ~]# vxprint -g vgp1417 -t lvol1
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
 
v  lvol1        -            ENABLED  ACTIVE   6291456  SELECT    -        fsgen

Plexes

Plexes of a disk group:

[root@server1 ~]# vxprint -pt -g vgp1417
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
 
pl lvol1-01     lvol1        ENABLED  ACTIVE   6291456  CONCAT    -        RW
pl lvol2-01     lvol2        ENABLED  ACTIVE   20971520 CONCAT    -        RW
pl lvol3-01     lvol3        ENABLED  ACTIVE   2097152  CONCAT    -        RW
pl lvol4-01     lvol4        ENABLED  ACTIVE   6291456  CONCAT    -        RW
pl lvol5-01     lvol5        ENABLED  ACTIVE   4194304  CONCAT    -        RW
pl lvol6-01     lvol6        ENABLED  ACTIVE   14680064 CONCAT    -        RW
pl lvol7-01     lvol7        ENABLED  ACTIVE   44040192 CONCAT    -        RW
pl lvol8-01     lvol8        ENABLED  ACTIVE   169869312 CONCAT   -        RW
pl lvol9-01     lvol9        ENABLED  ACTIVE   12582912 CONCAT    -        RW
pl lvol10-01    lvol10       ENABLED  ACTIVE   27262976 CONCAT    -        RW
pl lvol11-01    lvol11       ENABLED  ACTIVE   41943040 CONCAT    -        RW

More information of a plex:

[root@server1 ~]# vxprint -g vgp1417 -l lvol1-01
Plex:     lvol1-01
info:     len=6291456
type:     layout=CONCAT
state:    state=ACTIVE kernel=ENABLED io=read-write
assoc:    vol=lvol1 sd=vgp1417_04-01
flags:    busy complete
mediatype: hdd

Subdisks

Subdisks of a particular disk group:

[root@server1 ~]# vxprint -st -g vgp1417
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
 
sd vgp1417_04-01 lvol1-01    vgp1417_04 0      6291456  0         emc0_084b ENA
sd vgp1417_04-02 lvol9-01    vgp1417_04 6291456 12582912 0        emc0_084b ENA
sd vgp1417_04-03 lvol10-01   vgp1417_04 18874368 27262976 0       emc0_084b ENA
sd vgp1417_04-04 lvol6-01    vgp1417_04 46137344 10485760 4194304 emc0_084b ENA
sd vgp1417_05-01 lvol5-01    vgp1417_05 0      4194304  0         emc0_084f ENA
sd vgp1417_05-02 lvol7-01    vgp1417_05 4194304 31457280 0        emc0_084f ENA
sd vgp1417_05-03 lvol11-01   vgp1417_05 35651584 16155104 0       emc0_084f ENA
sd vgp1417_05-04 lvol7-01    vgp1417_05 51806688 12582912 31457280 emc0_084f ENA
sd vgp1417_06-01 lvol2-01    vgp1417_06 0      20971520 0         emc0_0853 ENA
sd vgp1417_06-02 lvol11-01   vgp1417_06 20971520 25787936 16155104 emc0_0853 ENA
sd vgp1417_06-03 lvol8-01    vgp1417_06 46759456 23780416 114269008 emc0_0853 ENA
sd vgp1417_07-01 lvol8-01    vgp1417_07 0      56312048 0         emc0_0857 ENA
sd vgp1417_07-02 lvol8-01    vgp1417_07 56312048 14227824 138049424 emc0_0857 ENA
sd vgp1417_08-01 lvol3-01    vgp1417_08 0      2097152  0         emc0_085b ENA
sd vgp1417_08-02 lvol4-01    vgp1417_08 2097152 6291456 0         emc0_085b ENA
sd vgp1417_08-03 lvol6-01    vgp1417_08 8388608 4194304 0         emc0_085b ENA
sd vgp1417_08-04 lvol8-01    vgp1417_08 12582912 57956960 56312048 emc0_085b ENA
sd vgp1417_09-01 lvol8-01    vgp1417_09 0      17592064 152277248 emc0_526f ENA

More information of a subdisk:

[root@server1 ~]# vxprint -g vgp1417 -l vgp1417_04
Disk:     vgp1417_04
info:     diskid=1325653859.61.server1
assoc:    device=emc0_084b type=auto
flags:    autoconfig
device:   path=/dev/vx/dmp/emc0_084bs3
devinfo:  publen=70539872 privlen=65536
mediatype: hdd

Performance monitoring

When having I/O issues the standard command you would use could be iostat:

[root@server1 ~]# iostat -mdx
Linux 2.6.18-274.3.1.el5 (server1)      06/07/2012
 
Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0        6.45    40.26  3.54 25.84     0.05     0.17    15.73     0.82   28.01   5.13  15.07
cciss/c0d0p1      0.00     0.00  0.00  0.00     0.00     0.00    31.97     0.00   21.14  14.42   0.00
cciss/c0d0p2      6.45    40.26  3.54 25.84     0.05     0.17    15.73     0.82   28.01   5.13  15.07
sda               0.00     0.00  0.00  0.00     0.00     0.00     3.98     0.00    1.87   1.87   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     9.71     0.00    4.51   4.51   0.00
sdc               0.00     0.00  0.00  0.00     0.00     0.00     9.71     0.00    1.83   1.83   0.00
sdd               0.00     0.00  0.00  0.00     0.00     0.00     9.71     0.00    1.66   1.66   0.00
sde               0.00     0.00  0.00  0.00     0.00     0.00     9.71     0.00    2.36   2.36   0.00
sdf               0.00     0.00  0.00  0.00     0.00     0.00     9.71     0.00    2.51   2.51   0.00
sdg               0.00     0.00  0.00  0.00     0.00     0.00     9.71     0.00    2.34   2.34   0.00
.
.

Remark:
As stated in man pages if you do not specify interval and count parameter displayed statistics are since server boot time so not really relevant…

[root@server1 ~]# iostat -md -p sda 1 2
Linux 2.6.18-274.3.1.el5 (server1)      06/07/2012
 
Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               0.00         0.00         0.00          7          0
 
Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               0.00         0.00         0.00          0          0

You can also use nmon.

With the command we have seen in previous chapter you are now able to identify to which disk group/volume/VXVM disks it is associated…

But even more interesting you can use VxVM command to get detailed information of disk using Operation System disk names:

[root@server1 ~]# vxdmpadm iostat show pathname=sdax
                       cpu usage = 374381us    per cpu memory = 1114112b
                        OPERATIONS            BLOCKS          AVG TIME(ms)
PATHNAME             READS    WRITES     READS    WRITES     READS    WRITES
sdax               1182902   4756775 328093320 294705430    5.10    20.82
[root@server1 ~]# vxdmpadm iostat show dmpnodename=sdax
                       cpu usage = 375891us    per cpu memory = 1114112b
                        OPERATIONS            BLOCKS          AVG TIME(ms)
PATHNAME             READS    WRITES     READS    WRITES     READS    WRITES
sdax               1183412   4762187 328098810 295062539    5.10    20.82
sdh                1133886   4773058 324904565 294251856    5.35    20.75
[root@server1 ~]# vxdmpadm iostat show pathname=sdax interval=2 count=2
                       cpu usage = 374396us    per cpu memory = 1114112b
                        OPERATIONS            BLOCKS          AVG TIME(ms)
PATHNAME             READS    WRITES     READS    WRITES     READS    WRITES
sdax               1182902   4756777 328093320 294705543    5.10    20.82
 
sdax                     0         0         0         0    0.00     0.00
[root@server1 ~]# vxdmpadm -e iostat show pathname=sdax
                       cpu usage = 374509us    per cpu memory = 1114112b
                        ERROR I/Os
PATHNAME             READS    WRITES
sdax                     0         0

As well as using VxVM disk names:

[root@server1 ~]# vxdmpadm -q iostat show dmpnodename=emc0_084b
                       cpu usage = 374484us    per cpu memory = 1114112b
                        QUEUED I/Os   PENDING I/Os
DMPNODENAME          READS    WRITES
emc0_084b                0         0         0
[root@server1 ~]# vxdmpadm -z iostat show dmpnodename=emc0_084b
                       cpu usage = 374558us    per cpu memory = 1114112b
                        OPERATIONS            BLOCKS          AVG TIME(ms)
PATHNAME             READS    WRITES     READS    WRITES     READS    WRITES
sdax               1182910   4756814 328093384 294707962    5.10    20.82
sdh                1133430   4767643 324899821 293884532    5.35    20.76
[root@server1 ~]# vxdmpadm -u bytes -z iostat show dmpnodename=emc0_084b
                       cpu usage = 374888us    per cpu memory = 1114112b
                        OPERATIONS            BYTES           AVG TIME(ms)
PATHNAME             READS    WRITES     READS    WRITES     READS    WRITES
sdax               1183012   4760612 480699392 707400192    5.10    20.82
sdh                1133494   4771329 3140286976 287247360    5.35    20.76

You can display information of controllers to see if your multipathing policy is working well:

[root@server1 ~]# vxdmpadm iostat show groupby=ctlr ctlr=c0
                       cpu usage = 374524us    per cpu memory = 1114112b
                        OPERATIONS            BLOCKS          AVG TIME(ms)
CTLRNAME             READS    WRITES     READS    WRITES     READS    WRITES
c0               135753949 218251362 3572140375 2841606614    7.62     7.10
[root@server1 ~]# vxdmpadm iostat show groupby=ctlr ctlr=c2
                       cpu usage = 374541us    per cpu memory = 1114112b
                        OPERATIONS            BLOCKS          AVG TIME(ms)
CTLRNAME             READS    WRITES     READS    WRITES     READS    WRITES
c2               135349009 215663711 3357968941 2737547261    7.96     6.92

Those interesting information can also be display at disk group level:

[root@server1 ~]# vxstat -u m -g vgp1417
                      OPERATIONS          BLOCKS           AVG TIME(ms)
TYP NAME              READ     WRITE      READ     WRITE   READ  WRITE
vol lvol1           827072    413478    17059m    29032m   1.51   7.34
vol lvol2           761459    357294     3465m     2781m   4.46  10.80
vol lvol3          9518191   7008657   175801m    56452m   0.26   0.02
vol lvol4           610864  15518552   259528m   176116m   1.86   0.19
vol lvol5           170534    303104      886m      998m   4.50   8.68
vol lvol6          2646674   2379866    69607m    16601m   1.19   1.66
vol lvol7         16854588  20012885   206030m   119118m   0.10   0.16
vol lvol8         79042007 128500851   899927m   689835m   0.04   0.02
vol lvol9           301556   4788315     2090m    74546m   3.00   0.83
vol lvol10          980251   4487076   287228m   193539m   3.20   0.62
vol lvol11           82761     10693      338m       82m   2.00  11.79

Remark
-d, -p, -s and -u m options are interesting to display information at different granularity level. -r option can also be used to reset statistics and focus on current period only (to remove boot time statistics or reorganization operations)…

There is also vxtrace command to trace operations on volumes but I have, most probably, not really understood its added value as the output is too much verbose and does not contains any synthetic information:

[root@server1 ~]# vxtrace -g vgp1417 -o disk lvol8
57 START write disk emc0_0857 op 0 block 50782432 len 16
57 END write disk emc0_0857 op 0 block 50782432 len 16 time 3
58 START write disk emc0_0857 op 0 block 195779 len 1
58 END write disk emc0_0857 op 0 block 195779 len 1 time 1
59 START write disk emc0_0857 op 0 block 1246640 len 16
59 END write disk emc0_0857 op 0 block 1246640 len 16 time 2
.
.

Remark:
I/O statistics are gathered by default but when you system is stable or when you have high I/O demand it may be interesting to deactivate it to maximize throughput. To check if statistics are gathered:

[root@server1 ~]# vxtune vol_stats_enable
1

Then to de-activate them, as well as for DMP part (not possible to check if there are activated or not):

[root@server1 ~]# vxtune vol_stats_enable 0
[root@server1 ~]# vxdmpadm iostat stop

If you have slow response from a device group it may be also interesting to check if you have running operation in background:

[root@server1 ~]# vxtask -h list
TASKID  PTID TYPE/STATE    PCT   PROGRESS

Even if default VxVM parameters suit most workloads you can change them using vxdmpadm command. To display tunable VxDMP parameters use:

[root@server1 ~]# vxdmpadm gettune all
            Tunable               Current Value  Default Value
------------------------------    -------------  -------------
dmp_cache_open                           on               on
dmp_daemon_count                         10               10
dmp_delayq_interval                      15               15
dmp_enable_restore                       on               on
dmp_fast_recovery                        on               on
dmp_health_time                          60               60
dmp_log_level                             1                1
dmp_low_impact_probe                     on               on
dmp_lun_retry_timeout                     0                0
dmp_path_age                            300              300
dmp_pathswitch_blks_shift                 9                9
dmp_probe_idle_lun                       on               on
dmp_probe_threshold                       5                5
dmp_queue_depth                          32               32
dmp_restore_cycles                       10               10
dmp_restore_interval                    300              300
dmp_restore_policy             check_disabled   check_disabled
dmp_retry_count                           5                5
dmp_scsi_timeout                         20               20
dmp_sfg_threshold                         1                1
dmp_stat_interval                         1                1
dmp_monitor_fabric                       on               on
dmp_monitor_osevent                      on               on
dmp_native_support                      off              off

Remark:
You may also want to use vxbench available in VRTSspt package…

References

About Post Author

This entry was posted in Linux and tagged , . Bookmark the permalink.

15 thoughts on “Veritas Volume Manager (VxVM) command line examples

  1. Hi Yannick,

    Occassionally, found this a bit old article.
    I have HP MSA 2050 storage and connected my servers to them via iSCSI ports. OS is RHEL 7.2.

    I have VCS installed and the problem is that VCS services loaded before ISCSI finished scanning paths. So VxDMP not aware about LUNs.

    I can refresh it manually with “vxdctl enable” or “vxdisk scandisks”. But I would like to have more standard solution.

    May be you have some tricks for that.

    Thank you in advance.

    BR,
    Fred

  2. Arun Gohil says:

    Very good information . However I have a queston .
    when you have two sites ( Site A & Site B ) with SRDF Replication how can you tell when the disks are imported to and writed enabled at which site ?

    Help appreciated – Thank You .

    Arun Gohil

    • Thanks for stopping by. Honestly I have ever dealt with that configuration as it is exactly the part I’m expecting to be handle properly by cluster product in use. Where I work we massively use Veritas Cluster Server (VCS) with SRDF replication as you mentionned. On your slave node you should see no active volume group (they will be imported at cluster switchover/failover) so nothing should be writable by default…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>