Various factors can affect the voltage. You say you set your power supply to 5v , do you measure that 5 volts coming out right at the PSU ? If you measure 5v right at the PSU and then measure again at the connector on the board did the voltage drop already? If so , the cable is dropping a bit because of its resistance. Once on the board, are there additional regulators into which your 5v is fed ? (You didn't say which board it is so I can't look it up) If there are regulators on board, they would require a bit more voltage than 5v because they will have a voltage drop in them. Are there reverse current protection diodes installed at the 5v line ? Which would also have a voltage drop and reduce the voltage.

Is your ground solid when taking measurements ? Do you have the same results using different ground sources on the board ?

The PSU can simply be too weak for the current demand of the board , and so the voltage will drop. You can compensate by raising the voltage on the PSU but that's not always a smart solution. If you raise the voltage so that it gets closer to 5v with the current drawn, and a component fails and that current is no longer pulled from the source , then the voltage will shoot back up and possibly go beyond component tolerances.. If you measure 4.4v at the PSU directly and it is set to 5v , it's really telling you that your PSU is too weak for this particular board. You should measure the current pulling through the board to see if things are out of place with a very high current or if it simply exceeds your PSU's capabilities. If you check the required fuse on the 5v for that board in the schematics , it will tell you vaguely how much current it's expected to pull.

Of course a high current could be caused by a shorted part somewhere and that voltage drop, if you know your PSU should be able to feed it, could be a sign that there is such a short.. But usually shorts come with smoke , sparks or at the very least, heat..

No you are not clear at all.. Now I think you're saying that on your bench PSU, the voltage remains at 5v and the board works fine. When you put the board back in the machine, that's when the 5v drops to 4.4v, and so you now think the original PSU needs its 5v section rebuilt. Do you have a schematic for that psu? Does it use a variable voltage regulator that you can increase the voltage output to get closer to the 5v? If it uses fixed components for regulation, capacitors are always a good place to start.. Can you tell us what psu this is, by looking at the schematic we could pinpoint components to focus on.