Earlier platforms came in a time when embedded processing was still in its infancy and the performance increase of newer parts was huge. WPC machines were pretty much on the limit of what the cpu performance made possible. Just look at how you can slow down attract mode DMD playback on a WPC machine by hammering the flipper buttons ^^
Technology has advanced big time since then, look at what cheap ARM based CPUs in your phones can do now or even 10 years ago..
If I remember correctly the WPC Motorola CPU ran at 2MHz with a few kB RAM at its side.
Spike 1 runs on a 400MHz CPU with 64MB RAM, Spike 2 offers much more since it has to do video processing. Linux eats more of that than a dedicated low level framework has. But for a pinball game that mainly has to read switches, process through a game statemachine and send out a few commands to node board that do the simpler low level stuff, this is plenty. Unless Stern plans to do onboard 3D rendering there is no reason why Spike should be replaced anytime soon.
The main reason for others to go PC based is probably availability and ease of use. Building an embedded OS and especially the hardware it requires (or many in the case of a distributed system such as Spike) is an immense effort and requires well trained specialists. Using a readily available PROC takes all that out of the process. Even building a custom serially controlled driver/interface board with a USB interface from scratch is much less effort. Last but not least using a PC platform allows for easier and faster prototyping/testing since you can program on the same architecture that is going to run your code.