I'd recommend you take a look at the Windows Task Manager to see which processes use a large amount of memory. I could bet it's the SQL server, which usually grabs as much memory as possible. From my personal experience (which may not necessarily be a supported configuration) it should be sufficient to provision the vCenter Server with 4GB for the mentioned size, if you are low on physical memory.
André