Hmm general opinion along my colleagues is to avoid Vista at all cost, windows 7 seems ok.
As for xp 64bit, it lack drivers etc but everything works if you use Vista drivers.
Know a couple of guys who have been using it for couple of years and never had any problems with it
I would recommend either xp64 or Win7 but everyone has the right to choose for himself.
By the way why do you need so much RAM?
You have a powerful PC and a crappy video card so don't see the point in adding more RAM (unless you want to play all newest games)

Card is crap for newest video games.
For older ones it should suffice.
If you have for instance 4GB RAM and if the game stutters it is 99% due to video card not being able to cope with the graphics and not RAM or main CPU.
Issues like this can be reduced by installing the most recent drivers, reducing resolution and details in game, but it usually means that you need better video card.
If it works better on his Windows7 and you have the same PC check for most recent drivers for your card.