Ugh, that's not good if the connector won't come off. It could be melted on there. It sounds like the header pins may need to be replaced. Absolute worst case, you snip all the wires right at the connector, get the board out of there, unsolder the header pins, remove the header, solder in a new header, and crimp on a new connector with the wires in the correct order according to the manual.
If the fuse holder is loose on the board, that's not great, but it could be that they just didn't get it tight up against the board. You're going to have to be the judge of that or send the board off to someone to have them replace the header and fuse holder.
Definitely desolder that wire. If you're going to keep that fuse holder, clean as much of the solder off as possible just for cosmetic reasons. Don't heat it hot enough to melt the plastic.