When the machine resets, does the reset process immediately stop after the score reels reset to zero? Of does the score motor continue to run after the score reels reset? From your description, it sounds like the machine isn't recognizing there is a ball in the outhole. Do you have a schematic for the machine? You would want to look at the circuit that contains the outhole solenoid (ball return solenoid). You'll need to determine what switches send power to the outhole solenoid. O relay switch is part of the outhole solenoid circuit. There is also a score motor switch in this circuit. Also, check relays Q, AX and A that send power to the O relay coil. We'll need to determine which score motor switches send power within these circuits. Similar Gottlieb machines use score motor switch 1C for the O relay coil and score motor switch 4C for the outhole solenoid...

I believe the outhole solenoid and playfield hole kicker solenoid share the same circuit. Before the circuit branches off, there is a score motor switch that sends power to both of these solenoids. Look at the outhole solenoid (ball return solenoid) on the schematic and work back through the circuit to see what score motor switch is in the circuit. When you have determined the score motor switch in the circuit, find the switch on the score motor and thoroughly clean the switch. Check any other switches down the line from the score motor switch, and make sure the switches are clean and adjusted properly... If this doesn't solve the issue, then check the circuit that contains the O relay coil...

You can identify the score motor switches by their wire colors listed on the schematic...

It's great you found the problem!