But that makes me wonder about how to deal with the transition and potential incompatibilities.
General rule is lacquer over poly = ok; poly over lacquer = not ok. However, there's always a risk and test pieces are always a good idea.

As for transition, I just did a gloss poly front of headstock on a satin sides & back and I masked the edge (side) all the way around about 1.5mm below the front. This allows the gloss to just roll over the edge contiguously. I'll just hit it lightly with a synthetic sanding pad to get a smooth transition from one to the other.

Granted both coatings were poly, but the method would also work with what you're looking to do.