I must admit the method of sanding all of the frets at once is not the way I go about it.
Seems like 'chemo-therapy' killing off good cells as well as the bad ones you are supposed to be targeting.

However, when you are crowning with that file, I would suggest approaching it from each side of the fret and try not to let it ride over the top.

Cheers, Mark.