I'm not sure about this, and couldn't find any resources, but it might have something to do with alternative or archaic spellings of férfi. In literature it's sometimes spelled as férfiú, so I guess it's similar to how fiú = boy becomes fiam = my son.