Not saying to do this but if was me and I was going to sand it I'd make sure that only the speck itself gets hit with the sandpaper, at first anyway, and maybe tape a really thin mask with a tiny hole over the speck just to be sure what doesn't need sanding doesn't get it.