I suggest that its also important to wash the dreck off the sandpaper regularly. That both keeps the paper wet enough for the cut to work and keeps it cutting cleanly.
But most of all, try and make sure there are no bare areas. Brush round holes with a bit of varnish, top coat, whatever, it hardly matters, and you can just ease them back out to full size with a drill or file when you're done if they end up a tad too small with the extra coating.