Typically, you want to rebuild the flipper that you are having issues with. Though you can certainly rebuild them all at the same time as well, as they will all eventually get worn and dirty in spots.
Sometime you don't really need to rebuild it. You simply inspect it, clean the parts, then decide if you should swap out damaged parts (if any).
It is a case by case basis, and depends on what issue/symptoms you actually have.