No, this isn't an insult. It could have a few meanings. One means something sort of like "he's a man that doesn't care too much for the complicated things in life". Another meaning would be a man like a farm labourer, a peddler, or someone of that sort, a man who probably hasn't had much of a chance in life at being educated, wears plain clothing, a simple man. If you wanted to say "he's an ordinary man", you would say הוא גבר רגיל.
Yes, it can also refer to females (although if you use the long form, you should say "בת אדם"). They both mean the same thing and both of them originate in the book of Genesis. "אדם" refers to Adam, the first man, and since we are all his descendants then we are all "בני אדם" or "sons of Adam" :)
I don't think there's a specific reason for "אדם", It's just a shorter form of that means the exact same thing.
By the way, another way to say the same thing is: "בני אנוש". Here I am not 100% sure, but I think that אנוש was the grandchild of Adam, from which humanity as we know it developed (correct me if i'm wrong).
I tried, "I think that he is just a man," but they marked it wrong. They had given as a clue for פשוט simple, simply or just. It's like the line from the famous Tammy Wynette song Stand By Your Man. She loves him and she forgives him because "after all, he's just a man" meaning he's only human; he's not a Superman.