I have an ESXi 5.1 environment set up for experiment purpose. It has two Datastores: One on a 4TB local disk, another on iSCSI pointing to a FreeNAS target which has a pair of mirrored 2TB drives. All 10 powered on VMs are placed on the local disk Datastore, but they were doing nothing so the Virtual Disk latency was not bad until I did this:
I added a second Virtual Disk to one of the VMs. This disk was placed on the separate datastore on iSCSI. And then I ran IOMeter to test performance on this second disk. It produced data transfer satuating the GB Network. However, I realized that the latency of the datastore on local disk increased to over 500 ms and all VMs responsed very slowly, while the latency on the iSCSI datastore remain low. I'm wondering why the latency was not on the iSCSI datastore being stress test but it impacted the other one which is not related?