In audio equipment, this is almost always the case. In a well designed circuit, externally injected noise shouldn't be much problem- not that it can't be in practice.

and/ or high frequency noise.

I think this depends on several things: size/ cost, difficulty in changing later, time you expect to own item/ purpose for being in there in the first place, and possibly most important, the possible consequences to other parts of the circuit should it fail. IOW, if the transistor that's going to die as a result is NLA, and not easily substituted (think germanium) then definitely.

For the most part, they're virtually free for smaller values. If it's already out to test, why not just replace? It's 25 years old, and old electrolytic WILL fail. A $50 power supply cap is another story.

This is something you don't want to experience unless you do it on purpose, I assure you!

I almost shit my pants.

I was testing an older DC/AC inverter that had been donated to a local Scout troop when out of nowhere...BOOM! There was "confetti" and metal slivers everywhere.

The cap was the only problem. Replaced it and she was up and running.

Lessons learned- always photograph, sketch, or have schematics before applying full power, and wear safety glasses when working on electronics. (You already knew the last one, but do you really do it?)