Sometimes you have to take a step back to then move forwards. I'm doing it all the time. Yes, TO will sand off. You've either got to just add loads of TO coats, so that it's thick enough to sand back to a flat surface over rough grain, or else you get the wood really smooth first before applying any TO, so that you have to apply fewer coats of TO. Most people here end up putting around 30 coats of TO on, with sandings every 4-5 coats or so.

Towards the end, you can thin the TO with turps about 50/50, so it's a bit more liquid and doesn't leave so many ridges in the finish when wiping it on. It obviously goes on a bit thinner as well, so wait until you've done the main building-up with the full strength TO.