I'm answering myself, probably it can help someone with the same question.
So it's possible to sand "by hand".
I started again with some 180 grain sanding paper, and i bought a little sanding block with "clips" to maintain the paper.
It took me 2 hours to have a good result.
So if you have some time and you want to do some sport, it will be ok without any tool !
The work is not finish, I think I need to work an hour more with what i have left in 180, and I want to "finish" with a 240 grain, before clearcoating.
The 120 grain was very scary, I tested it on a plank and it look like the wood was "destroy" by the process. It's why I started with 180.
IMG_2002 (resized).JPG
IMG_2003 (resized).JPG