"men" can also mean "ember(ek)" . So "Hat ember áll az épület előtt" is right, too. At least in my opinion, as a native.
Using "man" to mean "a person of either sex" is old-fashioned and frowned upon these days. They would've used "people" then. On the other hand it's more likely to be still used to mean "the human race".
What's wrong with az épület előtt áll hat férfi?