Thanks for the reply Pin-Guy, that’s a good idea, but all of the optos were working normally.
I’ve solved the problem and I decided to reply to myself and describe the situation, in case any other STTNG owners have the same problem in future, because its a really odd one.
Whenever a mode starts and balls drain, without ending the mode, my natural first thought is to check for a faulty opto. In this case, negative, all optos working perfectly and normally after thorough testing - both bottom and top opto troughs.
So my mind wandered to the 16 opto board. Luckily I own a Dirty Harry so I have an identical board I could swap in for a blind test. Fault continued exactly the same.... no change. Not the 16 opto board either. Thats actually good news, but it doesnt solve the problem.
So I filled the upper trough with 4 balls and emptied the bottom trough. Then I checked that all 4 balls were visible on the switch matrix, which they were. Then went into solenoid test, left hand popper, and ejected the first ball only. I did this by using the door switch as the trigger. After the 1st ball was ejected I returned to switch mode and made my first important discovery on the pathway to a solution.
Switch 3 AND switch 4 were now closed, meaning 2 balls were missing, rather than one! This explains why the ball arriving at the lower trough wasnt ending the mode, because its expecting 2 balls.
So why was it missing and why does this only happen in this specific set of circumstances? Why is it only when 4 balls are loaded in the upper trough and 1 is ejected? Because of physics.
To pop the first ball upward, the design of the VUK & trough requires the ball to move past the 2nd queued ball, which is immediately behind it. Because the 1st ball is seated in its popper hole, it is slightly lower than ball 2 and being a sphere, this means it will collide with ball 2 on the way past and upward. The kinetic energy is then distributed like a newtons cradle back through all of the queued balls making them all jiggle out of place and then re-seat..... not a problem when there’s only 2,or 3 balls in the trough, but when its fully loaded with 4 balls, there is enough kinetic energy to knock ball 4 back into the subway towards the ball diverter, permenety out of sight of the opto and thus the trough is suddenly seeing 2 missing balls at once.
The fix was to cut down and modify a nice new trough shim to fit the upper ball trough so that the balls want to sit forward in the V and are less inclined to roll backward. Also this helps ball 1 sit at roughly the same height as ball 2, limiting the energy transferred through the staged balls.
I hope this discovery helps someone in future.