Hi Maxence
Are you sanding in the same direction of the grain or across the grain?
You should sand in the same direction of the grain.
Starting with 120 sandpaper is good to take down excess filler then try switching to a finer grade say 240 and see how that works.
I had fill 2 times and even a 3rd time in some spots on the body on mine i found that the coarse sandpaper was tearing out the filler a bit as well if i sanded to hard.
Follow the filling steps that Simon mentioned in an earlier post and it should work out well.