Even paint needs something to grab on to. If you sand that fine and then paint, it will just peel off eventually.

Like FW said, the reason for starting on a low grit and working your way up is to remove scratches and marks you might not even see until you hit it with stain and then it's a much bigger job to rectify.