Yeah, Im not sure it that is going to work with the Brad Point though, the centering for it comes from the spike in the centre of the bit, if you've a smaller hole then that spike is hitting nothing and the edges of the flute are the first thing that will make contact. (At least thats how it appears on the ones I have).
If you're using a standard drill bit then what you say is probably true, but I'd imagine that the possibility of tearing out increases if all your torque at start up is on the flutes.
I guess the best solution is to run a couple of test holes on stock and see what happens. I've not had great success with drilling out like that, but i suppose you are only talking 1/32" a side.