OP

Quoted from zacaj:

At this point, I've got enough stuff mounted that I should be able to do some test plays.... once I wire stuff up. I'm not really interested in doing 'temporary' wiring, so getting enough to do some flipping, hit some targets means I'm gonna wire up everything I possibly can in its most final form possible
This time, unlike on my last build, I'm gonna put connectors on every mech, since I know I'll be doing at least one playfield transfer. Before I only did this on mechs like drop targets which I knew I'd need to service, reasoning that unsoldering two wires wasn't a big deal and could eliminate 3 potential points of failure. [quoted image]
On the last build, I also had the advantage of having big playfield support rails running the whole length of the playfield, which made wire management easy. Each wire just ran to the mech, and then back to the rail, and the wires ran along the rail, safely above all the mechs. This time there's no room for rails, so I'm going to have to route all the wires around the mechs. The switch wiring will be done directly on the playfield wood, and the coil wiring will be at the top of these ladders from PBL to keep the low voltage isolated for noise reduction: [quoted image]
Using connectors means that I can't just easily daisy chain the power wiring from one coil to the next, soldering 2-3 wires to the lug of the coil, and I don't like putting two wires, especially large ones, into a connector. Plus, doing daisy chaining like that makes it really messy to adjust stuff if you need to add/remove mechs later, so I decided I wanted a new system. The wago connectors I used for wiring in the cabinet seemed like an obvious route, so I designed some 3d printed wago mounts that snap into the ladders and have a built in slot for a zip tie to help with routing. My thinking is that I'll put one of these near each group fo coils, and then run four coils to each connector, along with one main power line.
[quoted image]
I then planned out a simple route for the harness to work its way through the playfield to all the mechs. This got a bit weird because there are some places where there's no 'clear' path for the wire to get through, thanks to the subway, and I don't know exactly where the subway is going to go yet, so I just left some slack in the wires and crossed my fingers it'll work out somehow. [quoted image]
The wiring... didn't come out as cleanly as I'd hoped. Having connectors on everything naturally leads to a bit of messiness, but the wago mounts also look bad. Since I don't want to just run wires directly into them I had to loop everything through the connector first, and then into the connectors, so it ends up with 5 ugly loops of wire on each one. [quoted image]
The size of the connectors+mounts didn't help either. Despite them seeming pretty small in your hand, once you've got them stuffed into a playfield with lots of wires going into them they're actually pretty bulky, and the fact that all the wires have to enter from the same direction isn't very good either. Wire nuts have the same problem. I feel like when I'm joining two wires together, it's pretty much always because they're coming from different places! Why not design a wago connector with holes in both sides? Sigh
While wiring, I also started to have problems with my homemade wire striper (a 2x4 with some holes in it to hold a wire and a marker). The stripes were coming out really ugly, and one of my markers got ruined after its tip got covered with left over paint from another marker. So I took a break to design something better:
[quoted image]
(Cutaway view to show how it works)
This is a little 3d printed box with a hole for the wire to go through and a spot on top to fit a marker in. After a few revisions I got it tuned pretty well so that the markers just snap right in and you can run wires though, and get much cleaner results than my old system. Plus, if they ever get 'contaminated' with paint I can just print another one

OP

Quoted from zacaj:

<why people don't design their own boards, part 3>
After some more investigation, I discover that this random voltage drop when I fire a coil... isn't related to firing a coil! It even happens when I have the coil voltage turned off. So at this point I have no idea why it'd be happening. But at this point I'm also not (to my knowledge) doing anything out of the ordinary so... maybe this is ordinary? it *is* a pretty small dip, so it could be that this happens all the time, but most machines aren't sensitive to pick it up? (I know many only poll their inputs every few ms)
My goal with this system was to avoid any debouncing, at least for the flipper/bumper switches, to make sure they reacted as fast as possible, but if there's no way to avoid these tiny voltage drops, I'll need to at least do something. And hey, if my input routine is really 10-40x as fast as comparable systems... even if I do the simplest debounce in the world (just wait until a signal has been seen twice), then, while that may be a 50% slowdown, I'm still... 5-20x faster than other systems, and that's still pretty good!
So I make a small tweak to my input code. It reads all the inputs. If anything has changed, it immediately reads them again. If the change went away, that must have been one of these tiny voltage drops. Otherwise, lets assume it's a real thing.
I put this new code on one of my node boards, and do some stress tests, flipping the flippers as much as possible to try and generate any issue, but everything checks out! I'm not too happy with this solution, especially since it's covering up for a hardware issue, but at least since it's software I can easily tweak it further later on if necessary, and I still have 5x more performance window to work work (and that's before doing any optimizations...), so I guess it's good enough for now
With that out of the way, I flash the new firmware to my second node board too, and hook everything up again for some more serious testing. But as soon as I flip the switch, the pop bumper connected to the new node board locks on. Uh oh. Apparently when I was replacing bad components last time I forgot this one. I replace the transistor but now none of the coils work. Did I blow the coil fuse? No. I try the other board too: nothing. Then I find the 12V fuse, which is used to power the transistors, was blown. I guess when the pop bumper locked on, it pulled enough through the gate driver chip to blow it? I replace that fuse, turn the game back on, and one of the flippers on the first node board fires once and dies. 12V fuse is blown again. After some more checking I find out that all 7 of my 12V gate driver chips are blown now. As far as I can tell, trying to fire a coil without the 12V, instead of just not working like I'd expect since the chip doesn't have power, instead somehow fries the chip completely. And a fried chip, instead of just doing nothing like I'd expect, both blows the 12V fuse and locks on its transistor. Great....
When every component is good, this all works. But if anything dies, the issues seem to cascade, to the point where I'm replacing as many components after a coil locks on as the new coil would cost. And the coil didn't even go bad! On Poker, I had issues occasionally with my 5V level shifter that controlled the transistor dying, especially when the 5V fuse blew, but that was localized, and at least didn't lock on the transistor. I figured that these 12V gate driver chips, which are specifically designed for driving transistors at high power, would handle this much better, but it seems they're even more fragile. This definitely isn't sustainable. I don't plan on frying more coils, but since i want to do a bunch of experimentation with coil drive patterns, etc it's definitely a possibility. Unless I can find a simple way to avoid this issue, I may need to redesign these boards to use a different method of driving the transistors.

OP

Quoted from zacaj:

<why people don't design their own boards, part 3>
After some more investigation, I discover that this random voltage drop when I fire a coil... isn't related to firing a coil! It even happens when I have the coil voltage turned off. So at this point I have no idea why it'd be happening. But at this point I'm also not (to my knowledge) doing anything out of the ordinary so... maybe this is ordinary? it *is* a pretty small dip, so it could be that this happens all the time, but most machines aren't sensitive to pick it up? (I know many only poll their inputs every few ms)
My goal with this system was to avoid any debouncing, at least for the flipper/bumper switches, to make sure they reacted as fast as possible, but if there's no way to avoid these tiny voltage drops, I'll need to at least do something. And hey, if my input routine is really 10-40x as fast as comparable systems... even if I do the simplest debounce in the world (just wait until a signal has been seen twice), then, while that may be a 50% slowdown, I'm still... 5-20x faster than other systems, and that's still pretty good!
So I make a small tweak to my input code. It reads all the inputs. If anything has changed, it immediately reads them again. If the change went away, that must have been one of these tiny voltage drops. Otherwise, lets assume it's a real thing.
I put this new code on one of my node boards, and do some stress tests, flipping the flippers as much as possible to try and generate any issue, but everything checks out! I'm not too happy with this solution, especially since it's covering up for a hardware issue, but at least since it's software I can easily tweak it further later on if necessary, and I still have 5x more performance window to work work (and that's before doing any optimizations...), so I guess it's good enough for now
With that out of the way, I flash the new firmware to my second node board too, and hook everything up again for some more serious testing. But as soon as I flip the switch, the pop bumper connected to the new node board locks on. Uh oh. Apparently when I was replacing bad components last time I forgot this one. I replace the transistor but now none of the coils work. Did I blow the coil fuse? No. I try the other board too: nothing. Then I find the 12V fuse, which is used to power the transistors, was blown. I guess when the pop bumper locked on, it pulled enough through the gate driver chip to blow it? I replace that fuse, turn the game back on, and one of the flippers on the first node board fires once and dies. 12V fuse is blown again. After some more checking I find out that all 7 of my 12V gate driver chips are blown now. As far as I can tell, trying to fire a coil without the 12V, instead of just not working like I'd expect since the chip doesn't have power, instead somehow fries the chip completely. And a fried chip, instead of just doing nothing like I'd expect, both blows the 12V fuse and locks on its transistor. Great....
When every component is good, this all works. But if anything dies, the issues seem to cascade, to the point where I'm replacing as many components after a coil locks on as the new coil would cost. And the coil didn't even go bad! On Poker, I had issues occasionally with my 5V level shifter that controlled the transistor dying, especially when the 5V fuse blew, but that was localized, and at least didn't lock on the transistor. I figured that these 12V gate driver chips, which are specifically designed for driving transistors at high power, would handle this much better, but it seems they're even more fragile. This definitely isn't sustainable. I don't plan on frying more coils, but since i want to do a bunch of experimentation with coil drive patterns, etc it's definitely a possibility. Unless I can find a simple way to avoid this issue, I may need to redesign these boards to use a different method of driving the transistors.

OP

Quoted from zacaj:

Diodes are good if you've got too much voltage, or current going backwards, but I don't think either of those are the issue here.
Voltage regulators in the system is a good idea, and in fact this is how stern's node boards work. They only send their 48V coil power in, and then each node board has voltage regulators onboard to change that into the 3.3V, 6V, etc they need. On one hand, you eliminate a lot of wiring, and the potential connector issues that come with it. On the other hand, they don't have any fuses on their boards, so if something were to go wrong that blows my 12V fuse in my system, they'd instead blow their regulator on the board... and then they'd have lost voltage... and the same issue would occur. Now, they also have software current monitoring to detect shorts and try to prevent the need for fuses, which is cool and something I'd love to have, but regulators and power supplies on the boards, fuses on the boards, current monitors on the boards: that's all more circuitry for me to design, test, troubleshoot, buy, and install. And I don't have a good history with designing voltage regulator circuits, which is the simplest of those! Back two or three builds ago I was trying to do that but I ended up just frying lots of power supplies AND lots of boards I have zero electronics experience so troubleshooting these designs is basically how I learn, so I don't want to make anything more complicated than necessary, although obviously keeping things simple can provide its own complications; I just try to keep the complications ones that I understand.
Speaking of current sensing, my next step forward on this is to repurpose one of the solenoid drive lines on my cpu as a voltage sensor to detect when the 12V is lost and go into a safety shut down mode. Similar purpose to Stern's current sensing, but voltage sensing is a built in CPU feature and much easier for me to confirm and measure, while I don't even know what current my coils actually pull, so I'm hoping that this voltage check will at least stop the vicious cycle of part failures when it happens and allow me to continue development until I (hopefully) understand the actual issue here and can prevent it

OP

Quoted from zacaj:

<why I do design my own boards, part 1>
With one node board hooked up and solid, I can test my flippers! They are very strong, which should be expected since I threw FL-11629s, the strongest option williams had, in a game with no ramps. That was on purpose though, like with the slings, I want to turn them down and tweak them to where I like.
Unlike on Poker, this time I'm using solid state flippers. Each coil has a power and a hold winding, and they have individual transistors powering them, just like WPC did in the 90s, so that's new territory I need to figure out. And judging by the struggles even most commercial manufacturers are having these days, it's not a simple thing. There are lots of concerns to balance:
- flipper strength
- flipper 'snappiness'
- flipper responsiveness
- control: can you do tiny taps, tip passes, and other finesse moves?
- they shouldn't drop when a fast moving ball hits them
- no 'JPP feel' where the outer-most shots on the playfield are weak
- no 'flipper fade' from the coils overheating
A lot of this has to do with the EOS switch. Some games don't have one. Most games don't seem to need one. I'd rather not have one on this game either, since I'm running low on switch inputs. But a lot of these problems would also just go away if I simply added a high voltage EOS and controlled the entire flipper via a single transistor, letting the EOS take care of managing the power winding. And I can always fall back to that if needed. But I think that an EOS limits your power somewhat. Since the switch has to open before the flipper has reached the end of its stroke, there'll necessarily be at least some portion of the stroke that's at low power when it could be at high power. Optimally the flipper would always be at high power until the moment *after* the ball leaves the flipper.
The way that most systems seem (I can't see their source code so this is all guesses and measurements) to use a fully solid state flipper is pretty simple. When the flipper button closes, fire the power winding for 40ms. (sometimes less, I think JJP is closer to 30ms, sometimes this is configurable, but both WPC and DE seem to use 40). If the EOS switch opens (eg, the bat got pushed down a bit)... fire the power winding for 40ms. That's it. Simple, easy to implement. Data East games don't even use a CPU for this, it's just some logic chips and timers. This same logic is also laid out in the tutorials+docs for many homebrew systems.
So I started with that on my game too. The game played good, the flippers felt fine. But then I checked the logs on my node board and noticed some issues. Often, the act of pressing in the flipper button would result in 2-3 'triggers' of the switch... which would result in 2-3 overlapping 40ms pulses. In other words, the flipper was actually firing for more than 40ms, sometimes as much as 65. Flipper feels fine to the player, seems to be working properly, and might even be more powerful than before, if the 40ms wasn't actually enough. But the flipper is spending a lot of time energized when it doesn't need to be, which means more heat!
Next issue: while just holding the flipper button in, the coils are triggering again! Turns out just moving your fingers around on the button while holding it in is enough to get signal drops from the contacts, which surprises me. These are new contacts too. So randomly, even at a stand still, the coils are being pulsed for 40ms again and heating up.
Issue three: when you release the button, sometimes you get another quick on/off pulse from the switch contacts. Sometimes you'll actually see this, the flipper will drop a bit then fire again, then drop for real. Other times it's not easily perceptible, but the flipper drop is effectively lagging 40ms behind when it should.
All three of these could be addressed with some aggressive debouncing (and in fact, the game code itself does, so the lane change isn't going crazy all the time), but adding some debounce logic from scratch on the node board's tiny processor will be a challenge. And besides, any debouncing of the flipper buttons will lead to lag, and probably prevent some of the finess+control I'm aiming for, so I want to avoid it if possible.
I tried adding some more rules to cut out specific cases like this, and things seemed good at first, but sooner or later I'd manage to create some weird edge case that'd result in a flipper being down while the button was pressed, and the code was becoming increasingly complicated, so it was clear I was approaching this from the wrong angle.
I decided to take a more literal approach, and connect the transistor output directly to the input signal, without any filtering. That way as soon as the user pressed or released the button, there'd be an immediate reaction, which should feel the most similar to hardware flippers on older games. But I still didn't have an EOS switch, so I needed to control the power winding somehow, but I didn't want it to be a set pulse or anything. I figured that, when the flipper button is released, the flipper necessarily drops at a slower speed than when it raises, so I could conservatively simulate the timing of an EOS by counting how long the button is pressed and released for. Instead of a set 40ms pulse, I'll count up to 40ms when the button is closed, and if the button is released, I'll start counting down again. So a 1ms release+press of the button would result in only a 1ms pulse, which won't cause overheating issues. And since there's no logic of ensuring certain pulses or debouncing, there's a lower chance of bugs.
So I coded that up, and it seemed to work. Flippers flipped reliably, no dropping when pushed or non-flips. But the flippers were really weak! Looking at my logs, whenever I press my flipper button, I actually get 6-8 press-release cycles, just from dirty signal from the button contacts. So basically my flipper is dropping out 6 times during the flip. I knew that could happen, but I didn't expect it to be noticeable since each drop should be under half a millisecond. Turns out they're not! Suspiciously, every single drop is 4ms, and there's no way that's right. After some debugging, I found out that, any time the node board attempts to send a second message while the first message is still sending, it freezes the entire thing for 4ms. This had apparently been happening the entire time, but I never paid enough attention to my logs to notice the pattern until now. This turned out to be a bug in my message queuing logic, which was also causing some other corruption issues I hadn't had time to track down yet, so I rewrote that and added some checks.
One of the checks I added was one to make sure the queue wasn't full. I had figured the messages should get sent pretty much instantly, as I was running at a fairly high data rate, but maybe one or two could get backed up, so I'd made a 5 deep queue. Once I added a safety check to make sure the queue wasn't full, I found out it was actually filling up fairly often. This wasn't very noticeable since I also had the 4ms lag time when sending a message, so the board would be frozen, and thus couldn't check the switches to generate more messages! But without the 4ms lag time I found that I needed to increase the queue to 50 items to be safe.
50 messages at once? That sounds weird. How are 50 switch events being generated in a quarter second? I'm not even in multiball yet! Turns out that even just pressing the flipper button is sometimes managing to generate 40-50 close+open events as the contacts get dragged across each other, which seems crazy to me. I wanted the switch detection to be fast and give me all the data so I could do smart things with it but... I don't think I have any use for *this* much data. Even when I added some 1ms debouncing, it was still sometimes 20-30 events. The game definitely doesn't care about anything under 5-10ms, but the flipper logic could. If the flippers can be made to still work good with this level of noise, maybe it's fine to just spam the game code with all this data and let it filter it out? I've never worked with a data bus like this before so I'm not sure if I'm creating a headache for myself down the line if the amount of data here grows even further....
The amount of noise here still seems out of hand though, so I'm going to have to dig out my scope again and start doing some more verification of the signals