"This is the key to my big brother's house."
It is about context (as most things are).
兄（あに） is more natural because of the context that it was presented in. If the surrounding text uses polite/formal language it is natural to use polite/formal language for this word as well, no?
Reference to one's own brother to someone outside of family/intimate friends is usually 兄（あに）. Polite/formal language is also more likely to be used in these situations.
Direct address of one's own brother can be お兄さん、兄さん、兄ちゃん. Also, reference to one's own brother as such within the immediate family and among close friends is more natural.
So I use 兄 and not 兄さん in this sentence, because I am not talking to the older brother directly. And because I am probably talking with someone outside the "intimate circle", the sentence automaticly has a higher chance of beeing formal. Thank you for your help and explanation.