If they aren't moving, I wouldn't bother unless the small gap around the edges bothered you. If you keep them in, they'll be more secure than pulling them out and pushing them back in again as the wood won't grip them quite so firmly next time.