If your choices included both fir and fear, then either of them would be grammatically correct. Without an English translation provided to direct your choice, only a trick question would treat one of those two choices as being wrong.
Given that hata fir is a fragment (and it therefore has no grammatical context to provide guidance), I think "man's hat" should be correct as well as "a man's hat". Next time you get that question, try reporting it.
hata isn't in the genitive, only fhir is. In English, "'s" is the genitive in English, so "man's" indicates that "man" is in the genitive case. To say "I wear a man's hat", you need to use the genitive of fear, so Caithim hata fir is correct.