I wouldn't worry too much until you have done all the finishing you are going to do. A few layers of [your choice of finish] in the holes may be all that's needed to turn loose fit into satisfactory.