How do we decide between using instrumental and genitive here ?
You wouldn't use the genitive here. You can use the nominative in the construction "X to Y" with both X and Y in the nominative, or "X jest Y" with Y being in the instrumental. These two options can be used ibterchangeably.
Can't "człowiek" be used interchangeably with "mężczynza", or does it lean more towards the term "human", rather than "male"?
Exactly this :)
why not "Dog is not a person."?
You need an article for 'dog' as well. Also, 'person' is not exactly 'człowiek'. For example in India they've recently decided, that dolphins are also persons. But they are definitely not people.
Why doesn't it accept "A dog isn't a person"?
Pies to nie Człowięk. Should apply just fine either or Pies nie jest człowiekiem.
Apart from a typo ("człowiek", normal 'e'), yes, both are correct and mean the same.