Seriously, when is ember "man" and when is it "person"? It seems like no matter which one I pick I'm wrong...
It is not easy to answer this question. I would say when we are referring to a specific person as "ember", it usually means "man". When it is more of a generic term, then it can be any gender. But this is just a hunch.
"Ember" used to definitely mean "man". If you read old stories, tales, etc.... There was an old "ember" and he had a wife and four children. Etc.
A real neutral term is "személy" - person.