I would advise against gluing them. However, if you do, use a very small amount. Like say a couple of small drops of woodworkers glue (Titebond or similar) applied to each post with a toothpick. Do not use CA.
If you ever need to remove them in future and they are completely glued in, there is an increased risk of damaging the wood of finish.

As sonic said, these are meant to be friction-fit. I suggest taking up the slack by wrapping the posts with something like a thin tape so that will make them fit snug. Snug is all you need. Copper shielding tape can work well for these kinds of fixes.

There are other methods to fix this issue, but "shimming" is the quickest and easiest.