why not "Dog is not a person."?
You need an article for 'dog' as well. Also, 'person' is not exactly 'człowiek'. For example in India they've recently decided, that dolphins are also persons. But they are definitely not people.
Why is this not human as well???
Can't "człowiek" be used interchangeably with "mężczynza", or does it lean more towards the term "human", rather than "male"?
Exactly this :)
How do we decide between using instrumental and genitive here ?
You wouldn't use the genitive here. You can use the nominative in the construction "X to Y" with both X and Y in the nominative, or "X jest Y" with Y being in the instrumental. These two options can be used ibterchangeably.
Why doesn't it accept "A dog isn't a person"?