Quoted from Tuukka:Usually the problem with lamp sockets is the rivet getting loose. That can be fixed by soldering parts together.
Rust removing chemicals do not hurt either, but clean thoroughly after using those.
Also spray some clear coat on the metal parts after Evaporust and cleaning them. It will prevent/slow down rust from returning. You can buy cans of clear at your local hardware store and probably Amazon.