You are the only one who can answer that question. If it bothers you now, most likely it will in the future and therefore best to deal with it now and do what you feel you want.

That said, need to take the least amount of damage path back out of all that burst colour and hopefully some thinners (if oil based) will help to remove a lot of the colour with the remainder residual being faint enough that any sanding back is significantly reduced.