I think he means by five the hand because the man's hand has five fingers.

Assuming they are playing golf (as your other post indicated, although I was not completely sure), a handicap refers to the "numerical measure of a golfer's potential ability" … "used to calculate a net score from the number of strokes played during a competition, thus allowing players of different proficiency to play against each other on somewhat equal terms."

More information here:
https://en.wikipedia.org/wiki/Handicap_(golf)