I'm pretty sure the machine thinks it's captured and would just kick a ball out of trough instead of castle area when Dorothy multi ball starts. This is pretty common.

It is the way the game is programmed to operate.

I assume so. The switch is triggered to say the shot was hit, so ball captured. It's just that once in a while the ball doesn't fully make it within magnets range. It's rare, but does happen. Game still awards the locked ball, it's just not a physical lock at that point. That's just how I look at it when playing mine. Doesn't affect gameplay, so it's ok in my book.

I don't know if this applies to your game or not. If you understand how to access diagnostics, see if the monkey sensor is operating correctly.

On my game if its a failed monkey capture, the monkey will activate but not go all the way to the top.
It will go up a bit then go back down.

After one of the software updates, I started getting a diagnostic code that referred to the monkey sensor. I'll try to explain the sensor so how it may apply to your situation. My understanding of the sensor is that is that it is part of the magnetic coil. There is no individual switch or sensor. It is simple software recognizing a magnetic "event". After the software update, for some reason, my game had a problem with the (I hope I'm using the proper term) polarity of that sensor. I was instructed to remove and flip the magnetic coil. This cleared that code and the monkey sensor returned to a proper operation.

I'm not suggesting that you do the same but my experience may be similar to what you are seeing. IOW, I was seeing a lot of missed "captures" and I believe that sensor code was part of it.

The Monkey Magnet sensor is a separate sensor mounted on the backboard near the base of monkey mech, it's purpose is to detect when a ball is actually captured by the Playfield magnet. It does this by sensing a larger magnetic field when the ball is on the magnet. It is very common that this sensor needs to be adjusted or moved to make it more accurate with regards to sending the Monkey down to get the ball. In other words, when all is working as designed a weak shot that triggers the magnet (via rollover switches in the loop) but does not get captured by the magnet on the PF will not trigger the Monkey to travel down and it will NOT award a lock. The code obviously compensates for when it's not working properly by awarding the lock even though no ball gets deposited in the castle. There is a specific test in the menu (Page B-18 of the manual) for this sensor and if you run those tests with a ball you can move the sensor around until you find a spot that is more reliably accurate. A few things to remember; you may have to move it pretty far and there is a lot of stuff (monkey mech) in the way back there, don't let the magnet stay energized too long as it can get quite hot and finally patience is a virtue - mine was not easy to find a good spot and the first couple tries worked for a while and then I had to move it again. Since the last time though it works very well and I very rarely have any false captures.

Here's a pretty good video someone did for this and is shows the sensor, etc.:

The video is helpful but I found it to also be confusing for a couple reasons - the biggest is that you can't see the ball and the screen at the same time to know when sensor should be active. However the overall problem is that the PF magnet itself triggers the ball sensor so it always thinks a ball has been captured when it turns on - this is exactly what my game did when I got it. The first thing you need to do is go into the test and see how much the sensor needs to be moved before the magnet no longer triggers it, then start fine tuning it back toward the magnet until you find the sweet spot / orientation. Remember to not let the magnet get too hot, to let it cool down completely and retest once you think you have the sweet spot established, and then to test again when you put the PF back down in the game. I used some thick double sided tape to temporarily position the sensor when testing.