I had similar issue with my Mata Hari backbox. I used small pieces of wood (tedious) and lots of wood putty to seal and make everything relatively smooth.
Let's not forget that this is against the wall, and that you will never see it again..... So choose your battles accordingly...
Yves