Do you have a notched straight edge to check if the fretboard is straight? If you put on a ruler on the frets, you can't really check if the fretwire is uneven or if it is the neck/fretboard.
I usually first check if the neck is straight with a notched straight edge before glueing and stringing it up. If it isn't (they usually are), I adjust them slightly on the truss rod. Then, I use a credit card (or a fret rocker) to check for uneven frets. I mark them and then do filing/sanding/polishing.
Then, AFTER it's strung up and finished, the neck will slightly bend under the tension (VERY slightly, it's not like it's gonna look like a bow) and sometimes the frets need a slight touch up then, if a correction on the truss rod won't do.