ZFS iSCSI Benchmark Tests on ESX
Let me just start out by saying that my setup is not perfect, I would actually call it bad. All I have is a 1 1TB SATA disk and 4GB of RAM. There is no redundancy in the setup or a dedicated SSD for ZIL. It’s just a regular machine that I had laying around and I wanted to see how OpenSolaris with Comstar would turn out.
The Hardware
As I mentioned, this is just a regular machine without any PCI controller cards with regular SATA disks. The machine has a dual core CPU:
root@zfs:~# psrinfo -pv
The physical processor has 2 virtual processors (0-1)
x86 (GenuineIntel 6F2 family 6 model 15 step 2 clock 2133 MHz)
Intel(r) Core(tm)2 CPU 6400 @ 2.13GHz
It is capable of handling 64 bit:
root@zfs:~# isainfo -bv
64-bit amd64 applications
vmx ssse3 cx16 sse3 sse2 sse fxsr mmx cmov amd_sysc cx8 tsc fpu
It has 4GB of RAM:
root@zfs:~# prtconf | grep Memory
Memory size: 4094 Megabytes
And here is the information about the SATA disk:
root@zfs:~# cfgadm -al -s "select=type(disk),cols=ap_id:info" sata1/1::dsk/c2t1d0
Ap_Id Information
sata1/1::dsk/c2t1d0 Mod: ST1000DM003-1CH162 FRev: CC47 SN: S1DEJVQ5
And lastly here is the S.M.A.R.T information regarding the disk:
root@zfs:~# smartctl -i -d sat /dev/rdsk/c2t1d0
smartctl 6.2 2013-07-26 r3841 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST1000DM003-1CH162
Serial Number: S1DEJVQ5
LU WWN Device Id: 5 000c50 06c885059
Firmware Version: CC47
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sun Dec 22 10:39:36 2013 MST
Nothing crazy or expensive.
Local Write Benchmarks
Before we present an iSCSI LUN to our ESXi host let’s see what speeds we get locally. I actually installed napp-it on my omnios install (if you need help with that, check out Jarret’s post here). In the napp-it web-gui there are a bunch of available tests:
Doing a quick DD test yielded the following:
I was actually impressed. Checking out the data sheet of the hardrive, I saw the following:
The Avg Data rate is 156MB/s, the Max Data rate is 210 MB/s and I got 177MB/s. This was a regular DD test, I am sure if I did a bonnie++ I would get closer to the 210MB/s mark. But I wanted a realistic test, with FileSystem Cache and OS limitations involved. Doing the same test from the command line, I saw the following:
root@zfs:~# dd if=/dev/zero of=/data/dd.tst bs=1M count=5K
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB) copied, 25.7934 s, 208 MB/s
And checking out the IO stats as the above test was going, I saw the following:
root@zfs:~# iostat -xM sd1 1
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 9.9 1474.2 0.1 172.2 8.3 0.8 6.2 88 83
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 0.0 1699.9 0.0 210.9 9.0 1.0 5.8 100 98
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 2.0 1501.4 0.1 173.9 7.8 0.9 5.8 90 88
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 0.0 1686.9 0.0 209.3 9.0 1.0 5.9 100 98
The iostat columns are described in the man page quite well:
wait
average number of transactions waiting for service
(queue length)
This is the number of I/O operations held in the
device driver queue waiting for acceptance by the
device.
actv
average number of transactions actively being ser-
viced (removed from the queue but not yet com-
pleted)
This is the number of I/O operations accepted, but
not yet serviced, by the device.
svc_t
average response time of transactions, in mil-
liseconds
The svc_t output reports the overall response
time, rather than the service time, of a device.
The overall time includes the time that transac-
tions are in queue and the time that transactions
are being serviced. The time spent in queue is
shown with the -x option in the wsvc_t output
column. The time spent servicing transactions is
the true service time. Service time is also shown
with the -x option and appears in the asvc_t out-
put column of the same report.
%w
percent of time there are transactions waiting for
service (queue non-empty)
%b
percent of time the disk is busy (transactions in
progress)
We can see that the device was definitely pushed to the max: the percent busy (%b) was almost 100% and same for the percent wait (%w). But the service time (svc_t) was still really good, below 6.5 ms and below the avg seek time as mentioned in the data sheet above. Just for fun I ran the iozone 1gb test and here is what I saw:
The sequential write was about the same speed 175MB/s. A bonnie++ test, yielded slightly higher speeds:
Network Benchmark
At home I had a 1GB un-managed switch and I wanted to see if a Linux machine on the same switch could get close to that speed when bench-marking against the onmnios server. Since omnios was just a regular machine, I only had one NIC:
root@zfs:~# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
e1000g0 Ethernet up 1000 full e1000g0
The setup is getting less perfect as we keep going :). The iperf binary comes with the napp-it install, so let’s test it out. On the zfs machine I started the server and then connected from the local Linux machine as the iperf client. Here were the results:
root@zfs:~# /var/web-gui/data/tools/iperf/iperf -s -w 2m
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 125 KByte (WARNING: requested 2.00 MByte)
------------------------------------------------------------
[ 4] local 192.168.1.101 port 5001 connected with 192.168.1.107 port 40146
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.09 GBytes 933 Mbits/sec
Not too shabby.
Next I wanted to try a similar test between the omnios machine and the ESXi machine. Back in the day I compiled a static netperf binary on a CentOS 4.8 Linux distribution and it actually worked on an ESXi host (another person did a similar compile with ipmitool as described here). So first I followed the instructions laid out in this blog on how to compile netperf for solaris. After I compiled netperf-2.4.5, I started the server on the omnios machine:
root@zfs:~# netserver
Starting netserver at port 12865
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC
Then I copied the old binary to the ESXi machine and it still worked out :) Here are my results:
~ # /tmp/netperf -H zfs
TCP STREAM TEST to zfs
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
128000 32768 32768 10.00 935.09
Similar results, I was pushing close the full 1GB through the switch.
ESXi Network iSCSI setup
I had two NICs on the ESXi server, so I dedicated 1 Nic to the VM and Mgmt traffic, and the other NIC to the iSCSI connection. I probably didn’t have to do this since the iSCSI network was already separate from the other networks, but I went ahead and bound the vmk that I used for the iSCSI connection to the physical NIC that was assigned to the vSwitch:
~ # esxcli iscsi networkportal list
vmhba32
Adapter: vmhba32
Vmknic: vmk1
iSCSI Benchmark
Using napp-it I went ahead and presented a LUN using Comstar, the process was pretty easy:
- create volume (under the Disks section)
- create a volume LU (under the Comstar section)
- create a Target
- create a Target Group with Target as member
- create a view ( To allow LU to be accessed by the TG we created above, and by host group if you have one)
After a rescan on the ESXi host, the LUN was present:
~ # esxcli storage core adapter rescan -A vmhba32
~ # esxcfg-scsidevs -c | grep SUN
naa.600144f0876a9c47000052b744210009 Direct-Access /vmfs/devices/disks/naa.600144f0876a9c47000052b744210009 10240MB NMP SUN iSCSI Disk (naa.600144f0876a9c47000052b744210009)
Here is a little more information about the device:
~ # esxcli storage core device list -d naa.600144f0876a9c47000052b744210009
naa.600144f0876a9c47000052b744210009
Display Name: SUN iSCSI Disk (naa.600144f0876a9c47000052b744210009)
Has Settable Display Name: true
Size: 10240
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.600144f0876a9c47000052b744210009
Vendor: SUN
Model: COMSTAR
Revision: 1.0
SCSI Level: 5
Is Pseudo: false
Status: degraded
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
Is Offline: false
Is Perennially Reserved: false
Thin Provisioning Status: yes
Attached Filters:
VAAI Status: unknown
Other UIDs: vml.0200000000600144f0876a9c47000052b744210009434f4d535441
I went ahead and put VMFS on it and then I did a quick dd test from the ESXi test. I know it’s not a valid test, but I wanted to see even an estimate:
~ # time dd if=/dev/zero of=/vmfs/volumes/test/dd.tst bs=1M count=2K
2048+0 records in
2048+0 records out
real 10m 12.82s
user 0m 4.26s
sys 0m 0.00s
As the above test was running I ran esxtop and here is what I saw:
That didn’t look good at all. I also checked out iostat on the omnios machine and I saw the following:
root@zfs:~# iostat -xM sd1 1
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 2.0 266.0 0.1 12.6 0.4 0.1 1.9 34 10
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 0.0 242.8 0.0 17.8 1.0 0.1 4.6 95 12
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 0.0 176.2 0.0 6.9 0.7 0.1 4.1 62 6
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 0.0 281.0 0.0 10.8 0.1 0.1 0.6 5 8
That is really slow, but it’s not like the omnios machine was over loaded or anything.
iSCSI ZFS with WriteBack Cache
So then I checked out the LUN and I saw that write-back cache was not enabled. Per this VMware KB, it’s recommended to enable that. So inside napp-it, I went ahead and enabled that:
After I enabled that I went ahead and did another dd test:
~ # time dd if=/dev/zero of=/vmfs/volumes/test/dd.tst bs=1M count=3K
3072+0 records in
3072+0 records out
real 0m 40.40s
user 0m 5.92s
sys 0m 0.00s
Also checked out esxtop and saw the following:
That is way better, now I am writing at ~80MB/s.
IOmeter Tests with iSCSI
I then installed a windows VM and installed IOmeter on it. I then added a VMDK to the VM that resided on my test LUN. After that I followed the instructions laid out in this VMware KB to do some performance testing. The KB provides a predefined set of IOmeter tests; they are mostly 4K-32K, Read and Write tests. The output of IOmeter is in CSV format and you can use Excel to check out the results. There is also a pretty cool perl script which grabs a couple of columns and generates an HTML file of those columns. I modified the script to grab latency stats as well. I also modified the IOmeter results to only include sequential tests (excluded the random tests). Here is the result of my initial test:
That is actually pretty good, I am getting about 105-111 MB/s for small IO sizes between 4K and 32K. We can see that the latency looks good as well 1-9ms. And of course as the block size increases so does the latency :) I decided to do a test with larger IO. I modified the test to include 128K-1M sequential read and writes. Here were the results:
The results are really impressive but so is the latency. Our bandwidth is 107-178MB/s but our latency varies from 24ms to 408ms
Performance from a Linux VM
At this point I wanted to see how the iSCSI LUN would fair against a Linux VM. After I added the VMDK to the VM, I saw the disk inside the OS:
$dmesg -T | tail -4
[Sun Dec 22 14:34:03 2013] sdb: unknown partition table
[Sun Dec 22 14:34:03 2013] sd 3:0:0:0: [sdb] Cache data unavailable
[Sun Dec 22 14:34:03 2013] sd 3:0:0:0: [sdb] Assuming drive cache: write through
[Sun Dec 22 14:34:03 2013] sd 3:0:0:0: [sdb] Attached SCSI disk
Then doing a dd test just to the disk, I saw the following:
~# dd if=/dev/zero of=/dev/sdb bs=1M count=3K
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 54.911 s, 58.7 MB/s
At the same time checking out esxtop here is what I saw:
That wasn’t so good. As I ran the dd test, I also ran an iostat within the VM and here is what I saw:
$iostat -xm sdb 1
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 10403.19 0.00 87.23 0.00 43.62 1024.00 151.91 1320.93 0.00 1320.93 12.20 106.38
avg-cpu: %user %nice %system %iowait %steal %idle
1.03 0.00 6.19 92.78 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 16758.76 0.00 174.23 0.00 87.11 1024.00 144.03 1378.67 0.00 1378.67 5.92 103.09
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 3.23 96.77 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 18025.81 0.00 92.47 0.00 46.24 1024.00 150.16 849.49 0.00 849.49 11.63 107.53
avg-cpu: %user %nice %system %iowait %steal %idle
1.05 0.00 2.11 96.84 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 12833.68 0.00 105.26 0.00 52.63 1024.00 155.04 1462.88 0.00 1462.88 10.00 105.26
We can also see the latency is pretty high with the OS. We can also see that the avgrq-sz and avgqu-sz which are defined from the man page as such:
avgrq-sz
The average size (in sectors) of the requests that were issued to the device.
avgqu-sz
The average queue length of the requests that were issued to the device.
Are pretty high, the avg request size is 1024 sectors (512KB) and the avg queue length is ~150. Actually all the iostat columns are described here:
- r/s and w/s— respectively, the number of read and write requests issued by processes to the OS for a device.
- rsec/s and wsec/s — sectors read/written (each sector 512 bytes).
- rkB/s and wkB/s — kilobytes read/written.
- avgrq-sz — average sectors per request (for both reads and writes). Do the math — (rsec + wsec) / (r + w) = (134.13+178.84)/(10.18+9.78)=15.6798597
- If you want it in kilobytes, divide by 2.
- If you want it separate for reads and writes — do you own math using rkB/s and wkB/s.
- avgqu-sz — average queue length for this device.
- await — average response time (ms) of IO requests to a device. The name is a bit confusing as this is the total response time including wait time in the requests queue (let call it qutim), and service time that device was working servicing the requests (see next column — svctim).So the formula is await = qutim + svctim.
- svctim — average time (ms) a device was servicing requests. This is a component of total response time of IO requests.
- %util — this is a pretty confusing value. The man page defines it as, Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%. A bit difficult to digest. Perhaps it’s better to think of it as percentage of time the device was servicing requests as opposed to being idle. To understand it better here is the formula:
- utilization = ( (read requests + write requests) * service time in ms / 1000 ms ) * 100%
- %util = ( r + w ) * svctim /10 = ( 10.18 + 9.78 ) * 8.88 = 17.72448
From the above output we can see that avg wait time is pretty high (~1000ms). So we are sending a lot of large IO to the device (512kb) and a lot of simultaneous requests (~128), the device is taking a while to process this IO and therefore the avg wait time is high.
Lower IO Size and Request Numbers in Linux VM
The IO size and the queue length are controlled by these two sysfs variables:
$cat /sys/block/sdb/queue/max_sectors_kb
512
$cat /sys/block/sdb/queue/nr_requests
128
From the sysfs documentation:
max_sectors_kb (RW)
-------------------
This is the maximum number of kilobytes that the block layer will allow
for a filesystem request. Must be smaller than or equal to the maximum
size allowed by the hardware.
...
...
nr_requests (RW)
----------------
This controls how many requests may be allocated in the block layer for
read or write requests. Note that the total allocated number may be twice
this amount, since it applies only to reads or writes (not the accumulated
sum).
So let’s go ahead and change the IO size to 32KB and see if that helps out at all:
# echo 32 > /sys/block/sdb/queue/max_sectors_kb
Now running the dd command again:
# dd if=/dev/zero of=/dev/sdb bs=1M count=3K
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 56.1516 s, 57.4 MB/s
The speed was the same, but when I checked out esxtop, I saw DAVG go down:
Checking out iostat from within the VM I saw the following:
$ iostat -xm sdb 1
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 12957.45 0.00 1855.32 0.00 57.98 64.00 156.00 81.42 0.00 81.42 0.57 106.38
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 2.20 97.80 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 12369.23 0.00 1780.22 0.00 55.63 64.00 157.79 95.12 0.00 95.12 0.62 109.89
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 3.26 96.74 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 14091.30 0.00 2006.52 0.00 62.70 64.00 159.10 76.70 0.00 76.70 0.54 108.70
We can see that the avrtq-sz changed to 64 (32kb), which is good and we now see that the avg wait time went down to ~80ms (from ~1000ms). Lowering the number of requests to 4:
# echo 4 > /sys/block/sdb/queue/nr_requests
lowered the DAVG to practically nothing, but the speed wasn’t that great. Here is the dd test:
~# dd if=/dev/zero of=/dev/sdb bs=1M count=3K
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 64.5064 s, 49.9 MB/s
and here is iostat:
$iostat -xm sdb 1
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 10666.67 0.00 1519.79 0.00 47.46 63.96 9.49 3.53 0.00 3.53 0.69 104.17
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 2.08 97.92 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 13234.38 0.00 1893.75 0.00 59.13 63.94 12.18 5.73 0.00 5.73 0.55 104.17
avg-cpu: %user %nice %system %iowait %steal %idle
2.06 0.00 2.06 95.88 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 10716.49 0.00 1531.96 0.00 47.87 64.00 12.12 3.88 0.00 3.88 0.67 103.09
We can see that the wait time is now ~4ms.
Change Zpool Block Size
When I created the zfs volume, I set the block size to 128KB:
root@zfs:~# zfs get volblocksize data/test
NAME PROPERTY VALUE SOURCE
data/test volblocksize 128K -
After reading a couple of sites:
FreeNAS 9.10 on VMware ESXi 6.0 Guide
- I use 64K block-size based on some benchmarks I’ve done comparing 16K (the default), 64K, and 128K. 64K blocks didn’t really hurt random I/O but helped some on sequential performance, and also gives a better compression ratio. 128K blocks had the best better compression ratio but random I/O started to suffer so I think 64K is a good middle-ground. Various workloads will probably benefit from different block sizes.
- You must also consider I/O block size before creating a ZFS store this is not something that can be changed later so now is the time. It’s done by adding the –b 64K to the ZFS create command. I chose to use 64k for the block size which aligns with VMWare default allocation size thus optimizing performance.
It seems that the default value of 64K is better. I also ran into this forum: Block alignment and sizing in ZFS volumes for iSCSI to vSphere 5. From that forum:
Latest best practice documents suggest 8k block size with iscsi. At least the ones I know with latest VMware file systems in vsphere5.
This is ok for ZFS backends, but anything up to 32k for iSCSI should work fine. In a ZFS system the balance is between metadata and data: small data block size means more metadata is needed. In general 4K is too small, 8-16-32 are generally ok.
I tried setting it on the fly, but it failed:
root@zfs:~# zfs set volblocksize=32K data/testing
cannot set property for 'data/testing': 'volblocksize' is readonly
So I deleted the volume and re-created it. I tested with both 64KB and 32KB, for me 32KB worked out a little better. Here are a couple of dd tests (these were without any changes to the Linux OS):
# dd if=/dev/zero of=/dev/sdb bs=1M count=3K
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 34.476 s, 93.4 MB/s
this was for 64KB and here is the 32KB one:
# dd if=/dev/zero of=/dev/sdb bs=1M count=3K
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 31.1633 s, 103 MB/s
The DAVG was pretty high for both (~70ms). Modifying the max_sectors_kb and nr_requests to 32 and 4 respectively , yielded very low latency but the bandwidth was lower as well:
# dd if=/dev/zero of=/dev/sdb bs=1M count=3K
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 47.1101 s, 68.4 MB/s
Still better than 128KB ZVol Block size. If you don’t mind a little latency then leave the default values for max_sectors_kb and nr_requests sysfs variables.
Linux VM File System performance tweaks
After I put a file system on the disk and tried the same dd test, I saw the following:
# dd if=/dev/zero of=/mnt/test/dd.tst bs=1M count=3K
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 31.1195 s, 104 MB/s
Looking over this site, I decided to try out some mount options:
# mount /dev/sdb1 /mnt/test -o noatime,data=writeback
That yielded very little increase:
# dd if=/dev/zero of=/mnt/test/dd.tst bs=1M count=3K
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 29.8348 s, 108 MB/s
The above options helped out my Fedora box more than my Debian VM.
Other Tweaks
Throughout my testing I did a couple of things. I would say the below had minimal if no impact on the performance:
- Install VMware tools
- From VMware KB 1017652 use the pvscsi virtual adapter
- From Linux 2.6 kernel-based virtual machines experience slow disk I/O performance set the I/O Scheduler to noop (
echo noop > /sys/block/sdb/queue/scheduler). - Per this HP KB, change the MaxIOSize option in VMware (
esxcli system settings advanced set -o /Disk/DiskMaxIOSize -i 128) - Modify the following ZFS settings on the ZFS Volume, taken from this blog.
- Turn off the checksum (
zfs set checksum=off data/test) - Change Primary Cache (
zfs set primarycache=metadata data/test)
- Turn off the checksum (
- Disable sync (there is a discussion here), it seems that it mostly applies to NFS, but I decided to give it a try just to see if it helps out(
zfs set sync=disabled data/test)
Summary
In summary, I think the following settings made the biggest impact:
- Enable Write Back Cache
- Use 64KB or 32KB block size on the ZFS Volume
- If latency is important to you, change the IO size from within the Linux OS (max_sectors_kb and nr_requests)
In the end here was my final dd result from my Linux VM:
# dd if=/dev/zero of=/mnt/test/dd.tst bs=1M count=3K
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 27.5574 s, 117 MB/s
And checking out my NIC Utilization during the above test:
Almost the full pipe :) Lastly here is how the omnios iostat looked like:
root@zfs:~# iostat -xM sd1 1
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 41.0 881.1 0.2 103.4 4.4 0.9 5.8 98 93
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 24.0 811.1 0.1 100.1 5.6 0.8 7.6 98 75
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd1 25.0 868.7 0.1 102.2 3.0 1.0 4.5 99 98
Good enough for me.