It is not easy to answer this question. I would say when we are referring to a specific person as "ember", it usually means "man". When it is more of a generic term, then it can be any gender. But this is just a hunch.
"Ember" used to definitely mean "man". If you read old stories, tales, etc.... There was an old "ember" and he had a wife and four children. Etc.
A real neutral term is "személy" - person.

Same as English. When is "man" a male person and when any person?