The squares are the optos on the motors. They should both toggle as the hand moves.
Try a factory reset; I was having the same issue homing despite the mech working fine until I did that.
Does the hand move at all when you enter the test? (with interlock closed)
Do both your home switches register in switch test?