Sometimes you just have to do what you have to do.

If the fretboard is flat (have you checked the edge of your notched straight edge against another straight edge to confirm that it hasn't got a bow in it?), then you'll just have to carry on sanding the frets down. With a flat neck, the frets should end up no lower than the 1st fret, which shouldn't be an issue.

Have you checked that you haven't simply got a single high fret at the 3rd fret by using a fret rocker?

And are you checking with a full straight edge along the top of the frets? You don't want to create a mini reverse 'ski-jump' by using uneven pressure of sanding amounts whereby he upper frets are lower than most of the the lower frets (apart from the first two frets). You obviously want all the frets the same height.