Since you say the switch clicks every time you press it, the two things I would do is loosen the mounting screws on the switch and see if that fixes it (takes pressure off the internal parts), if not then replace the switch.

One other thing, test with a ball rather than your finger because the ball may not be pressing it down as much as your finger. You can bend the arm on the switch to make it more sensitive to the pressure of the ball.

Also, make sure the wires are soldered on well.

Check the trough opto switches.

As you said this part shows that the switch at the door worked unless the game is programmed to kick out a new ball into the shooter lane regardless of whether the castle door switch is triggered or not:
*Hit Monkey targets to enable ball capture.
*Make right orbit to freeze ball on magnet.
*Monkey captures ball and drops it behind door.
*New ball ejected into plunger lane.

This part appears to show that the trough opto did not recognize that the ball drained unless the game is programmed as stated in the first part above:
*If new ball drains without achieving RESCUE multiball then
---Game continues with no balls
---Ball search opens castle door, releasing captured ball
---Game continues until captured ball drains
---No other ball can be captured (monkey targets are solid, not flashing)
*If RESCUE multiball is achieved with new ball then
---Game continues with captured ball and RESCUE ball (2-ball multiball)
---Once both balls drain, ball over
---Monkey targets are re-enabled (flashing) for another shot at monkey ball capture.

I still suspect it is a flaky trough opto or the castle switch is bad, but it also could be a switch matrix issue. Bring up the switch test screen and see if the status of any of the switches is in the wrong state. I would check the trough with and without balls as well as putting a ball on the castle door switch during the trough testing.

I just hand tested mine. I believe you have an intermittently working castle switch. I can reproduce your symptoms by removing the ball behind the door. I would first try making the switch more sensitive. If that does not fix the problem then change out the switch.