The more accurate you can get, the better. You'll notice any variation a lot more on the high frets as 0.1mm becomes a much greater proportion of the string length.

0.001" is about 0.026mm, so I'd use inches for the scale length if I was doing it.

There is the accuracy/tolerance of the cnc router to consider as well. If that's say +/- 0.05mm, then the slot could end up further out than you think, so the closer the cnc can get to the ideal position, the better.

However, the software may still be using the mm figure you typed in, but is just displaying it to one decimal place.

If you have a chance, you could do a couple of test cuts on a piece of pine, and measure with callipers how accurate the cuts are.

Don't forget that the ultimate position will be decided by how well you centre the fret crown when you profile it!