After that prompting I checked the vmkwarning log and found the following:
/var/log # tail vmkwarning.log
2013-07-09T01:10:13.901Z cpu14:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan
2013-07-09T01:10:13.901Z cpu14:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa
2013-07-09T01:10:13.920Z cpu14:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan
2013-07-09T01:10:13.920Z cpu14:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa
2013-07-09T01:10:14.010Z cpu2:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expand
2013-07-09T01:10:14.010Z cpu2:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fai
2013-07-09T01:11:03.238Z cpu0:326782)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expand
2013-07-09T01:11:03.238Z cpu0:326782)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fai
2013-07-09T01:19:39.291Z cpu13:327237)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan
2013-07-09T01:19:39.291Z cpu13:327237)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa
Which lead me to the following VMware KB: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004424
Using PowerCLI, I checked the total size of all VMDK files on one member of the cluster with the following command:
PowerCLI > Get-HardDisk -Datastore <datastore> | select -ExpandProperty CapacityGB | measure -sum -ave -max -min
Count : 78
Average : 197.730180789263
Sum : 15422.9541015625
Maximum : 544.49609375
Minimum : 50
Property :
According to that article, the maximum supported size of all opened VMDK files was only 8TB, so I was well above that, therefore running into issues.
I've adjusted the VMFS3.MaxHeapSizeMB to the maxmum 256MB to compensate, and that has resolved this issue.