End grain is always the hardest to get smooth. Can you feel them if you lightly run your finger tips over it? If yes, keep sanding, if you are painting a solid colour your primer and top coat will do a lot of filling of minor scratches. If staining your clear will fill a lot too but get it as smooth as say, the feeling of newspaper on your finger tips and you can't go wrong.