Fusion I/O installation and benchmarking

 

Preamble

For one of our Business Intelligence (BI) project we have decided to purchase a HP 365GB Multi Level Cell G2 PCIe ioDrive2 for ProLiant Servers in one of our blade server. The initial aim was to use Oracle Smart Flash Cache (db_flash_cache_file and db_flash_cache_size initialization parameters) but as the feature is available only on Oracle Enterprise Linux and Solaris (pure marketing limitation) we turn down to Veritas SmartIO.

Before we configured the Fusion I/O card I have been able to perform few tests using an I/O bench marking product.

Testing has been done using Red Hat Enterprise Linux Server release 6.4 (Santiago) and the card we have in our server is a HP ioDrive2 Blade Mezzanine Card 365 GB, HP SKU number = QK761A. The server itself has 120GB of RAM and 2 quad core sockets Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz.

What Fusion I/O claim as brute performance:

ioDrive2 365GB
NAND Type MLC (Multi Level Cell)
Read Bandwidth (1MB) 910 MB/s
Write Bandwidth (1MB) 590 MB/s
Ran. Read IOPS (512B) 137,000
Ran.Write IOPS (512B) 535,000
Ran. Read IOPS (4K) 110,000
Ran. Write IOPS (4K) 140,000
Read Access Latency 68 µs
Write Access Latency 15 µs

That is a bit higher than what HP claim (HP ioDrive2 Blade Mezzanine Card 365 GB, HP SKU number = QK761A):

ioDrive2 365GB
NAND Type MLC (Multi Level Cell)
Read max sequential throughput 860 MB/s
Write max sequential throughput 560 MB/s
Read IOPS (4 Kb, Q16) 71,000
Write IOPS (4 Kb, Q16) 23,500
Read average access latency (4 Kb, Q1) 185 µs
Write average access latency (4 Kb, Q1) 70 µs

Installation

The latest Kernel I had was kernel-2.6.32-431.20.3.el6.x86_64 so I have obliged to modify Grub configuration and reboot with previous kernel as the latest available driver in zip files was:

iomemory-vsl-2.6.32-358.el6.x86_64-3.2.6.1212-1.0.el6.x86_64.rpm

Then I have also installed:

fio-util-3.2.6.1212-1.0.el6.x86_64.rpm
fio-firmware-ioaccelerator-3.2.6.20131003-1.noarch.rpm
libvsl-3.2.6.1212-1.0.el6.x86_64.rpm
fio-common-3.2.6.1212-1.0.el6.x86_64.rpm
fio-sysvinit-3.2.6.1212-1.0.el6.x86_64.rpm

Finally loaded the driver is memory with:

modprobe iomemory-vsl
[root@server1 ~]# fio-status
 
Found 1 ioMemory device in this system
Driver version: 3.2.6 build 1212
 
Adapter: HP Adrenaline 3 Adapter
        HP 365GB MLC IO Accelerator for BladeSystem c-Class, Product Number:QK761A, SN:3UN340B04A
        External Power: NOT connected
        PCIe Power limit threshold: 24.75W
        Connected ioMemory modules:
          fct0: Product Number:QK761A, SN:1338D0B5C
 
fct0    Attached
        HP IO Accelerator 2, SN:1338D0B5C
        Located in slot 0 Center of Adrenaline III Adapter SN:1340A010F
        PCI:23:00.0, Slot Number:2
        Firmware v7.1.15, rev 110356 Public
        365.00 GBytes device size
        Internal temperature: 43.31 degC, max 49.22 degC
        Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00%
        Contained VSUs:
          fioa: ID:0, UUID:3d4d60ea-a338-45a9-add3-0a9d2f3618d0
 
fioa    State: Online, Type: block device
        ID:0, UUID:3d4d60ea-a338-45a9-add3-0a9d2f3618d0
        365.00 GBytes device size

I wanted to detach the card to upgrade the firmware and format it (even if what format does is unclear so far) I was not able to do it (the force option is not available):

[root@server1 ~]# fio-detach /dev/fct0
Detaching: [====================] (100%)
Error: failed to detach /dev/fct0: device busy.

With fio-status -a I have been able to see that firmware was already the latest one available so gave up on this part…

Then I created a filesystem on it with:

mkfs -t ext4 /de/fioa

And mounted it using below option, /etc/fstab:

# Fusion I/O card
#/dev/fioa /mnt/fioa vxfs defaults 1 2

To have the ioSphere graphical interface I also installed (client and server part as I have only one test server):

fio-agent-3.7.0.181-1.0.x86_64.rpm
hp-io-accel-msrv-3.7.0.181-1.0.x86_64.r

To start server and agent part use:

fio-msrv start
fio-agent start

I initially did a simple:

mkfs -t ext4 /dev/fioa

And mounted the device in a file system then one of my colleague challenged me stating that I should have formatted the card using VxFS to get more performances, so wanted to test this option. I’m not gonna display any ext4 performance testing result but from what I have seen VxFS is giving around 2-5% more performance. I thought it would be straightforward but encountered many issues:

[root@server1 ~]# umount /mnt/fioa
[root@server1 ~]# mkfs -t vxfs -o bsize=8192,largefiles /dev/fioa
UX:vxfs mkfs.vxfs: ERROR: V-3-25584: /dev/fioa is not a supported device

Ok, I think I have to create a VxVM device group and a logical volume on it to make this working, nice refresh from old memories:

[root@server1 ~]# vxdisk -s list fioa
Disk:   fusionio0_0
type:   auto
flags:  online ready private autoconfig invalid
guid:   -
udid:   FIO%5FioDrive%5FFusionIO%5F1338D0B5C
site:    -
diskid:
dgname:
dgid:
hostid:
info:   format=none
[root@server1 ~]# vxdisksetup -i fusionio0_0
VxVM vxdisksetup ERROR V-5-2-0 Disk fusionio0_0 has  Linux ext4 FS on it
        Use the '-f' flag to force the operation
[root@server1 ~]# vxdisksetup -f -i fusionio0_0
VxVM vxparms ERROR V-5-1-6536 error reading partitions
VxVM vxdisksetup ERROR V-5-2-43 fusionio0_0: Invalid disk device for vxdisksetup
[root@server1 ~]# vxdisk init fusionio0_0
VxVM vxdisk ERROR V-5-1-5433 Device fusionio0_0: init failed:
        Disk Ioctl failed

Then using fdisk /dev/fioa I have tried to correct below warning using, obviously, c, u and w:

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
 
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

And retried:

[root@server1 ~]# vxdisksetup -i -f /dev/vx/dmp/fusionio0_0
VxVM vxdisksetup ERROR V-5-2-1118 /dev/vx/dmp/fusionio0_0: Can not get disk information.

Strangely I discovered that full path cannot be supplied in this command, so using:

[root@server1 ~]# vxdisksetup -i -f fusionio0_0
[root@server1 ~]# vxdisk -o alldgs list|grep f
fusionio0_0  auto:cdsdisk    -            -            online ssdtrim
[root@server1 ~]# fdisk -l /dev/fioa
 
Disk /dev/fioa (Sun disk label): 16 heads, 680 sectors, 65521 cylinders
Units = cylinders of 10880 * 512 bytes
 
    Device Flag    Start       End    Blocks   Id  System
/dev/fioa3  u          0     65521 356434240    5  Whole disk
/dev/fioa8  u          0     65521 356434240    f  Unknown

The after VxVM disk initialization, VxVM device group creation:

[root@server1 ~]# vxdg init fusionio fusionio0_0
[root@server1 ~]# vxdg list fusionio
Group:     fusionio
dgid:      1405514597.10441.server1
import-id: 1024.10440
flags:     cds
version:   180
alignment: 8192 (bytes)
ssb:            on
autotagging:    on
detach-policy: global
dg-fail-policy: obsolete
ioship: off
copies:    nconfig=default nlog=default
config:    seqno=0.1027 permlen=51360 free=51357 templen=2 loglen=4096
config disk fusionio0_0 copy 1 len=51360 state=clean online
log disk fusionio0_0 copy 1 len=4096
[root@server1 ~]# vxdg -g fusionio free
DISK         DEVICE       TAG          OFFSET    LENGTH    FLAGS
fusionio0_0  fusionio0_0  fusionio0_0  0         712802688 -
[root@server1 ~]# vxassist -g fusionio maxsize
Maximum volume size: 712802304 (348048Mb)

Remark
Ultra small difference in real available storage

Creation of a logical volume and mount it under a file system:

[root@server1 ~]# vxassist -g fusionio make lvol01 712802688
[root@server1 ~]# vxdg -g fusionio free
VxVM vxdg INFO V-5-1-15790 No free space remaining for diskgroup fusionio
[root@server1 ~]# vxassist -g fusionio maxsize
VxVM vxassist ERROR V-5-1-15809 No free space remaining in diskgroup fusionio with given constraints
[root@server1 ~]# mkfs -t vxfs -o bsize=8192,largefiles /dev/vx/rdsk/fusionio/lvol01
    version 9 layout
    712802688 sectors, 44550168 blocks of size 8192, log size 8192 blocks
    rcq size 1024 blocks
    largefiles supported
[root@server1 ~]# mount -t vxfs -o delaylog,nodatainlog,convosync=direct,mincache=direct,largefiles /dev/vx/dsk/fusionio/lvol01 /mnt/fioa
[root@server1 ~]# df -P /mnt/fioa
Filesystem                  1024-blocks  Used Available Capacity Mounted on
/dev/vx/dsk/fusionio/lvol01   356401344 85904 353531736       1% /mnt/fioa

Remark:
I initially did not choose any particular mount options nor VxFS options as benchmark tool I used has an option to bypass buffered I/O (direct=1 use non-buffered I/O, usually O_DIRECT). I have performed few I/O benchmarks to compare and I have not seen any differences so the option is working well…

Finally as suggested in Fusion I/O quick installation guide I installed fio 2.1.10 benchmark to taking the RPM from http://www.rpmfind.net/linux/rpm2html/search.php?query=fio(x86-64).

Reading a bit the man page of fio I have decided to use non-buffered I/O to bypass Linux File system caching and libaio (Linux native asynchronous I/O) as ioengine.

On top of comparing with brute performance Fusion I/O and HP claim I have added the random mixed reads and writes that is closer to an Oracle database behavior in my opinion…:-)

The presented result is the best average run of multiple ones. Even if I cannot explain why some run are strangely going wild and provide results that is far from the expected average. I have also seen that the generated files if used with a different blocksize than the one for which they have been created can provide as well strange results…

Fusion I/O with VxFS

Sequential read of 1MB:

[root@server1 ~]# fio --name=files --directory=/mnt/fioa --runtime=300 --readwrite=read --size=10g --blocksize=1m --numjobs=30 --group_reporting --ioengine=libaio --direct=1
files: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=1
...
fio-2.1.10
Starting 30 processes
Jobs: 14 (f=14): [RR_RR___R_____RRR_____RRRRR_R_] [63.2% done] [863.0MB/0KB/0KB /s] [863/0/0 iops] [eta 02m:55s]
files: (groupid=0, jobs=30): err= 0: pid=33727: Mon Jul 21 15:02:03 2014
  read : io=263109MB, bw=898043KB/s, iops=876, runt=300012msec
    slat (usec): min=2, max=512, avg= 6.36, stdev= 5.89
    clat (msec): min=2, max=86408, avg=33.40, stdev=567.24
     lat (msec): min=2, max=86408, avg=33.41, stdev=567.24
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    9], 10.00th=[   11], 20.00th=[   15],
     | 30.00th=[   19], 40.00th=[   23], 50.00th=[   27], 60.00th=[   31],
     | 70.00th=[   37], 80.00th=[   43], 90.00th=[   52], 95.00th=[   61],
     | 99.00th=[   79], 99.50th=[   86], 99.90th=[  103], 99.95th=[  110],
     | 99.99th=[  147]
    bw (KB  /s): min=   13, max=136942, per=3.91%, avg=35107.94, stdev=11757.79
    lat (msec) : 4=0.12%, 10=8.75%, 20=25.86%, 50=53.74%, 100=11.39%
    lat (msec) : 250=0.13%, 2000=0.01%, >=2000=0.01%
  cpu          : usr=0.01%, sys=0.03%, ctx=263123, majf=0, minf=8550
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=263109/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
   READ: io=263109MB, aggrb=898042KB/s, minb=898042KB/s, maxb=898042KB/s, mint=300012msec, maxt=300012msec
 
Disk stats (read/write):
  VxVM11000: ios=526029/23, merge=0/0, ticks=12700305/178152, in_queue=12878178, util=100.00%

Sequential write of 1MB:

[root@server1 ~]# fio --name=files --directory=/mnt/fioa --runtime=300 --readwrite=write --size=10g --blocksize=1m --numjobs=30 --group_reporting --ioengine=libaio --direct=1
files: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=1
...
fio-2.1.10
Starting 30 processes
Jobs: 30 (f=30): [WWWWWWWWWWWWWWWWWWWWWWWWWWWWWW] [54.1% done] [0KB/481.0MB/0KB /s] [0/481/0 iops] [eta 04m:15s]
files: (groupid=0, jobs=30): err= 0: pid=34058: Mon Jul 21 15:14:47 2014
  write: io=166221MB, bw=567245KB/s, iops=553, runt=300065msec
    slat (usec): min=47, max=745, avg=291.29, stdev=89.11
    clat (msec): min=2, max=204, avg=53.86, stdev= 3.27
     lat (msec): min=2, max=204, avg=54.15, stdev= 3.28
    clat percentiles (msec):
     |  1.00th=[   51],  5.00th=[   51], 10.00th=[   52], 20.00th=[   52],
     | 30.00th=[   52], 40.00th=[   52], 50.00th=[   53], 60.00th=[   53],
     | 70.00th=[   55], 80.00th=[   57], 90.00th=[   59], 95.00th=[   61],
     | 99.00th=[   64], 99.50th=[   66], 99.90th=[   70], 99.95th=[   72],
     | 99.99th=[  101]
    bw (KB  /s): min=13864, max=22000, per=3.34%, avg=18930.40, stdev=704.91
    lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.35%, 100=99.63%
    lat (msec) : 250=0.01%
  cpu          : usr=0.54%, sys=0.02%, ctx=166266, majf=0, minf=813
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=166221/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
  WRITE: io=166221MB, aggrb=567244KB/s, minb=567244KB/s, maxb=567244KB/s, mint=300065msec, maxt=300065msec
 
Disk stats (read/write):
  VxVM11000: ios=0/332412, merge=0/0, ticks=0/17730170, in_queue=17731799, util=100.00%

What can be also graphically seen with HP IO Accelerator Management Tool:

fusionio01
fusionio01

Remark:
As there are small differences I think HP IO Accelerator Management Tool is displaying in “real” MB (megabyte i.e. 1000^2) and not in the usual MB (megabyte or MiB i.e. 1024^2)

At the end with block of 1MB I almost achieve the maximum throughput i.e. 835MB for read, 554 MB for write. No need to say that this is far from a real life workload…

Sequential read of 4K:

[root@server1 ~]# fio --name=files --directory=/mnt/fioa --runtime=300 --readwrite=read --size=10g --blocksize=4k --numjobs=30 --group_reporting --ioengine=libaio --direct=1
files: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
...
fio-2.1.10
Starting 30 processes
Jobs: 30 (f=30): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [100.0% done] [370.2MB/0KB/0KB /s] [94.8K/0/0 iops] [eta 00m:00s]
files: (groupid=0, jobs=30): err= 0: pid=62709: Tue Jul 22 10:37:16 2014
  read : io=108292MB, bw=369636KB/s, iops=92409, runt=300001msec
    slat (usec): min=2, max=352, avg= 6.94, stdev= 5.41
    clat (usec): min=38, max=18497, avg=315.84, stdev=159.05
     lat (usec): min=90, max=18503, avg=322.94, stdev=158.91
    clat percentiles (usec):
     |  1.00th=[  139],  5.00th=[  163], 10.00th=[  175], 20.00th=[  191],
     | 30.00th=[  213], 40.00th=[  237], 50.00th=[  262], 60.00th=[  306],
     | 70.00th=[  358], 80.00th=[  430], 90.00th=[  540], 95.00th=[  636],
     | 99.00th=[  812], 99.50th=[  876], 99.90th=[  988], 99.95th=[ 1032],
     | 99.99th=[ 1144]
    bw (KB  /s): min= 7849, max=15624, per=3.33%, avg=12325.96, stdev=691.03
    lat (usec) : 50=0.01%, 100=0.01%, 250=45.33%, 500=41.85%, 750=10.91%
    lat (usec) : 1000=1.82%
    lat (msec) : 2=0.07%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=1.31%, sys=2.91%, ctx=27722931, majf=0, minf=883
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=27722807/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
   READ: io=108292MB, aggrb=369636KB/s, minb=369636KB/s, maxb=369636KB/s, mint=300001msec, maxt=300001msec
 
Disk stats (read/write):
  VxVM11000: ios=27714318/11, merge=0/0, ticks=8230869/0, in_queue=8207494, util=100.00%

The Q16 and Q1 in Read IOPS (4 Kb, Q16) and Read average access latency (4 Kb, Q1) performance figures are unexplained to me but I have slightly different results that HP ones, more inline with the one of Fusion I/O this time. With 4KB sequential read I got 92409 IOPS and a latency of 316 µs.

Sequential write of 4K:

[root@server1 ~]# fio --name=files --directory=/mnt/fioa --runtime=300 --readwrite=write --size=10g --blocksize=4k --numjobs=30 --group_reporting --ioengine=libaio --direct=1
files: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
...
fio-2.1.10
Starting 30 processes
Jobs: 30 (f=30): [WWWWWWWWWWWWWWWWWWWWWWWWWWWWWW] [100.0% done] [0KB/435.5MB/0KB /s] [0/111K/0 iops] [eta 00m:00s]
files: (groupid=0, jobs=30): err= 0: pid=57592: Thu Jul 17 09:42:58 2014
  write: io=136424MB, bw=465659KB/s, iops=116414, runt=300001msec
    slat (usec): min=1, max=321, avg=10.35, stdev= 7.20
    clat (usec): min=1, max=35258, avg=245.38, stdev=220.97
     lat (usec): min=52, max=35304, avg=255.90, stdev=220.93
    clat percentiles (usec):
     |  1.00th=[  106],  5.00th=[  139], 10.00th=[  157], 20.00th=[  181],
     | 30.00th=[  197], 40.00th=[  213], 50.00th=[  229], 60.00th=[  243],
     | 70.00th=[  262], 80.00th=[  282], 90.00th=[  330], 95.00th=[  390],
     | 99.00th=[  580], 99.50th=[  636], 99.90th=[ 1144], 99.95th=[ 5728],
     | 99.99th=[10816]
    bw (KB  /s): min=10672, max=17680, per=3.33%, avg=15528.46, stdev=640.17
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.67%, 250=63.71%, 500=33.39%, 750=2.04%, 1000=0.08%
    lat (msec) : 2=0.02%, 4=0.02%, 10=0.06%, 20=0.01%, 50=0.01%
  cpu          : usr=1.41%, sys=4.74%, ctx=35637317, majf=0, minf=827
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=34924535/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
  WRITE: io=136424MB, aggrb=465658KB/s, minb=465658KB/s, maxb=465658KB/s, mint=300001msec, maxt=300001msec
 
Disk stats (read/write):
  VxVM11000: ios=21/34917946, merge=0/0, ticks=5/6391626, in_queue=6328451, util=100.00%

Same comment as before. With 4KB sequential write I got 116414 IOPS and a latency of 245 µs.

Random read of 4K:

[root@server1 ~]# fio --name=files --directory=/mnt/fioa --runtime=300 --readwrite=randread --size=10g --blocksize=4k --numjobs=30 --group_reporting --ioengine=libaio --direct=1
files: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
...
fio-2.1.10
Starting 30 processes
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [100.0% done] [374.8MB/0KB/0KB /s] [95.1K/0/0 iops] [eta 00m:00s]
files: (groupid=0, jobs=30): err= 0: pid=59221: Thu Jul 17 10:35:22 2014
  read : io=111971MB, bw=382194KB/s, iops=95548, runt=300001msec
    slat (usec): min=2, max=346, avg= 7.29, stdev= 5.69
    clat (usec): min=51, max=29179, avg=304.26, stdev=138.28
     lat (usec): min=87, max=29184, avg=311.71, stdev=138.18
    clat percentiles (usec):
     |  1.00th=[  135],  5.00th=[  159], 10.00th=[  175], 20.00th=[  199],
     | 30.00th=[  221], 40.00th=[  245], 50.00th=[  270], 60.00th=[  302],
     | 70.00th=[  342], 80.00th=[  398], 90.00th=[  482], 95.00th=[  564],
     | 99.00th=[  716], 99.50th=[  772], 99.90th=[  884], 99.95th=[  932],
     | 99.99th=[ 1128]
    bw (KB  /s): min=10816, max=13352, per=3.33%, avg=12744.23, stdev=154.52
    lat (usec) : 100=0.01%, 250=42.09%, 500=49.24%, 750=8.02%, 1000=0.62%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.48%, sys=3.15%, ctx=28665013, majf=0, minf=942
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=28664658/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
   READ: io=111971MB, aggrb=382194KB/s, minb=382194KB/s, maxb=382194KB/s, mint=300001msec, maxt=300001msec
 
Disk stats (read/write):
  VxVM11000: ios=28657912/7, merge=0/0, ticks=8168935/0, in_queue=8143204, util=100.00%

Random write of 4K:

[root@server1 ~]# fio --name=files --directory=/mnt/fioa --runtime=300 --readwrite=randwrite --size=10g --blocksize=4k --numjobs=30 --group_reporting --ioengine=libaio --direct=1
files: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
...
fio-2.1.10
Starting 30 processes
Jobs: 30 (f=30): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done] [0KB/482.5MB/0KB /s] [0/124K/0 iops] [eta 00m:00s]
files: (groupid=0, jobs=30): err= 0: pid=5461: Tue Jul 22 15:27:26 2014
  write: io=141570MB, bw=483225KB/s, iops=120806, runt=300001msec
    slat (usec): min=2, max=361, avg=10.27, stdev= 7.80
    clat (usec): min=1, max=18886, avg=235.65, stdev=195.69
     lat (usec): min=45, max=18904, avg=246.09, stdev=195.60
    clat percentiles (usec):
     |  1.00th=[   98],  5.00th=[  129], 10.00th=[  147], 20.00th=[  171],
     | 30.00th=[  187], 40.00th=[  203], 50.00th=[  217], 60.00th=[  229],
     | 70.00th=[  245], 80.00th=[  266], 90.00th=[  310], 95.00th=[  378],
     | 99.00th=[  684], 99.50th=[  732], 99.90th=[ 1032], 99.95th=[ 5728],
     | 99.99th=[ 6816]
    bw (KB  /s): min=11088, max=18000, per=3.33%, avg=16111.16, stdev=725.93
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=1.12%, 250=71.60%, 500=24.18%, 750=2.68%, 1000=0.31%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.08%, 20=0.01%
  cpu          : usr=1.61%, sys=4.70%, ctx=38028601, majf=0, minf=852
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=36242014/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
  WRITE: io=141570MB, aggrb=483225KB/s, minb=483225KB/s, maxb=483225KB/s, mint=300001msec, maxt=300001msec
 
Disk stats (read/write):
  VxVM11000: ios=1/36226598, merge=0/0, ticks=0/6531861, in_queue=6474813, util=100.00%

Mixed random read and write of 1M:

[root@server1 ~]# fio --name=files --directory=/mnt/fioa --runtime=300 --readwrite=randrw --size=10g --blocksize=1m --numjobs=30 --group_reporting --ioengine=libaio --direct=1
files: (g=0): rw=randrw, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=1
...
fio-2.1.10
Starting 30 processes
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [34.5% done] [126.0MB/167.0MB/0KB /s] [126/167/0 iops] [eta 09m:31s]
files: (groupid=0, jobs=30): err= 0: pid=33339: Mon Jul 21 14:50:38 2014
  read : io=54960MB, bw=187433KB/s, iops=183, runt=300262msec
    slat (usec): min=2, max=379269, avg=15981.97, stdev=63095.57
    clat (usec): min=0, max=354420, avg=20844.16, stdev=13271.35
     lat (msec): min=2, max=379, avg=36.83, stdev=59.09
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    2], 10.00th=[ 7008], 20.00th=[11968],
     | 30.00th=[14784], 40.00th=[17024], 50.00th=[19328], 60.00th=[21888],
     | 70.00th=[24704], 80.00th=[28544], 90.00th=[35072], 95.00th=[42240],
     | 99.00th=[65280], 99.50th=[77312], 99.90th=[108032], 99.95th=[123392],
     | 99.99th=[288768]
    bw (KB  /s): min= 1140, max=34816, per=3.54%, avg=6632.32, stdev=4081.97
  write: io=52502MB, bw=179050KB/s, iops=174, runt=300262msec
    slat (usec): min=38, max=370268, avg=19282.13, stdev=67946.45
    clat (msec): min=1, max=506, avg=113.71, stdev=58.55
     lat (msec): min=2, max=507, avg=133.00, stdev=73.42
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   45], 10.00th=[   65], 20.00th=[   87],
     | 30.00th=[   97], 40.00th=[  103], 50.00th=[  108], 60.00th=[  112],
     | 70.00th=[  117], 80.00th=[  124], 90.00th=[  137], 95.00th=[  269],
     | 99.00th=[  355], 99.50th=[  367], 99.90th=[  396], 99.95th=[  408],
     | 99.99th=[  441]
    bw (KB  /s): min= 1345, max=19768, per=3.39%, avg=6062.94, stdev=1788.28
    lat (usec) : 2=1.18%, 4=1.94%, 10=0.10%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%
    lat (msec) : 2=0.02%, 4=0.17%, 10=5.04%, 20=19.36%, 50=24.96%
    lat (msec) : 100=15.43%, 250=29.04%, 500=2.75%, 750=0.01%
  cpu          : usr=0.18%, sys=0.01%, ctx=111544, majf=0, minf=895
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=54960/w=52502/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
   READ: io=54960MB, aggrb=187433KB/s, minb=187433KB/s, maxb=187433KB/s, mint=300262msec, maxt=300262msec
  WRITE: io=52502MB, aggrb=179050KB/s, minb=179050KB/s, maxb=179050KB/s, mint=300262msec, maxt=300262msec
 
Disk stats (read/write):
  VxVM11000: ios=109920/104966, merge=0/0, ticks=2225189/13669617, in_queue=15899414, util=100.00%

As a summary:

HP ioDrive2 Blade Mezzanine Card 365 GB, HP SKU number = QK761A
Sequential Read Bandwidth (1MB) 877 MB/s
Sequential Write Bandwidth (1MB) 554 MB/s
Sequential Read IOPS (4K) 92,409
Sequential Write IOPS (4K) 116,414
Random Read IOPS (4K) 95,548
Random Write IOPS (4K) 120,806
Mixed random reads and writes IOPS & bandwidth (4K) 18762 + 18718, 75 MB/s + 75 MB/s
Mixed random reads and writes IOPS & bandwidth (8K) 9527 + 9522, 76 MB/s + 76 MB/s
Mixed random reads and writes IOPS & bandwidth (16K) 4976 + 4966, 79 MB/s + 79 MB/s
Sequential Read Access Latency (4K) 316 µs
Sequential Write Access Latency (4K) 245 µs

With ext4 (defaults mount options) I got:

Sequential Read Bandwidth (1MB) 830 MB/s
Sequential Write Bandwidth (1MB) 549 MB/s
Mixed random reads and writes IOPS & bandwidth (4K) 16402 + 16394, 64 MB/s + 64 MB/s

VxFS testing

No need to say that I’m a bit amazed by the throughput with a block size of 1MB, I used the below mount option to bypass the operating system file cache but the direct=1 option of fio does already the job from what I have tested:

delaylog,nodatainlog,convosync=direct,mincache=direct,largefiles
VxFS file system (Raid5, HP3PAR, HBA)
Sequential Read Bandwidth (1MB) 1153 MB/s
Sequential Write Bandwidth (1MB) 1543 MB/s
Sequential Read IOPS (4K) 65574
Sequential Write IOPS (4K) 34045
Random Read IOPS (4K) 5149
Random Write IOPS (4K) 13194
Mixed random reads and writes IOPS & bandwidth (4K) 3141 + 3144, 12 MB/s + 12 MB/s
Mixed random reads and writes IOPS & bandwidth (8K) 3108 + 3110, 24 MB/s + 24 MB/s
Mixed random reads and writes IOPS & bandwidth (16K) 3092 + 3093, 48 MB/s + 48 MB/s
Sequential Read Access Latency (4K) 20 msec
Sequential Write Access Latency (4K) 20 msec

Remark:
As the array behind (HP 3PAR) is shared by multiple environments the results can be quite heretic…

References

This entry was posted in Linux and tagged . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>