Here's the way I see it. There are two things holding vSGA back:
1) Lack of DirectX 10/11 (I think that last game to use DirectX 9.0c was like Star Wars: Empire at War) and OpenGL 3+ -> We also need HDCP support but that's a different gripe.
2) limited to only 256Mb of GPU RAM and 256Mb of vRAM (the GeForce 5900 Ultra had 256Mb of physical RAM).
I think we're closer to breaking through #2.
I created a 5.2 pool and set the 3D settings to use vSphere's settings. Here's the gpuvm output looking at vShpere:
Xserver unix:0, GPU maximum memory 2076672KB
pid 10779, VM "MediaPC3", reserved 262144KB of GPU memory.
GPU memory left 1814528KB.
I edited the [svga.vramSize = "536870912"] line in the VMX file and changed the 5 to a 9. I booted the VM back up and here's the output from gpuvm:
Xserver unix:0, GPU maximum memory 2076672KB
pid 12296, VM "MediaPC3", reserved 457457KB of GPU memory.
GPU memory left 1619215KB.
Looks like the VM has taken a pretty healthy chunk out of the physical GPU RAM with the forced setting. Does this fix vSGA performce? Not necessarily, plus it's a bit of a hack that I'd frown on if put into a production environment but sometimes people just need it to work. It also looks like the Half GPU RAM/Half vRAM still applies, thus we're only pulling 450Mb for 900MB total. Need to look into it more to see if the Viewserver overrides that setting when the pool mandates the RAM size.
How we lookin on number 1?