If you don't want to go to the trouble of removing the connector pins (not sure how comfortable you are with board work), you could try just re-flowing the pins on J4 for the meantime. If it fixes the problem again, then you *know* you'll have to replace the male headers if it should happen yet again.
I also find it easier to remove the headers by re-flowing the pins first anyway, the fresh bit of fresh solder helps make removal easier when you go at it with a desoldering tool.