VirtualBox limits the maximum amount of ram a VM can use on a 32bit host to 3584MB. There is no technical reason for this specific limit, whilst its true most 32bit hosts are limited to ~3.5GB of ram, servers with PAE enabled can far surpass this limit. My desktop runs win2k3 server, and can utilize all 16GB of installed ram. Wanting to run a 64bit guest with a decent amount of ram I set about to fix this pointless limitation.
After downloading the source for VirtualBox I was able to locate the problem code, when built for 32bit the constant MM_RAM_MAX_IN_MB is set to 3584. It was believed that to fix this it would be a simple matter of increasing MM_RAM_MAX_IN_MB and recompiling.
#if HC_ARCH_BITS == 64
# define MM_RAM_MAX_IN_MB UINT32_C(2097152)
# define MM_RAM_MAX_IN_MB UINT32_C(3584)
A simple matter in theory; there was however no way I was going to actually try to build this from source, as with most large projects, it would take forever just to setup the build environment and dependencies, it would then likely fail to build in a numerous inexplicable ways.
Since only a single constant was required to be changed it was decided to modify the binary instead. A simple find and replace, 3584 -> 16384 was applied to VBoxSVC.exe, this was facilitated by a simple patch tool written for this purpose, the tool takes confirmation for each address to prevent altering unrelated code (in my case every instance of 3584 was replaced, this might not be the case for every VBoxSVC.exe version so checking the disassemble is a must). Testing found it to work fully, a 64bit guest was run and was able to utilize 12GB of memory on the 32bit host. http://mjsstuf.x10host.com/files/vbox-mod.rar