You can use a deep-well socket, and tighten the retaining nut.
If you don't have any deeps, then take the 2 screws out and tighten with a normal wrench.
There should be a locking (toothed) washer between the nut and plate to keep it from happening again.