Unfortunately, your analysis seems correct. Starting over is probably the way to go.

Now, I'm gonna say, most likely, the spray and the brushed on poly are incompatible. Next time stick with one or the other, personally I would choose spray. Also, sanding through and removing colour is a very common problem with glitter finishes. That clear coat needs to be thick enough to bury the uneven surface of the glitter and survive a leveling. Err on the side of caution and build it up slowly with lots of coats.

Persistence is the key. You will get there in the end.