I never listen to my thoughts Lawry, they only lead me astray.
Without a multimeter I'd probably leave the switch out of the equation for the moment, and instead just connect *one* pickup hot (white or yellow) directly to the outer lug of the volume pot. Check for buzz and then repeat with the other pickup hot. You said the buzz was there without the pots in the circuit, right? So we now need to work out if it's a specific pickup or the switch causing the issue.