"This is the key to my big brother's house."
It is about context (as most things are).
兄（あに） is more natural because of the context that it was presented in. If the surrounding text uses polite/formal language it is natural to use polite/formal language for this word as well, no?
Reference to one's own brother to someone outside of family/intimate friends is usually 兄（あに）. Polite/formal language is also more likely to be used in these situations.
Direct address of one's own brother can be お兄さん、兄さん、兄ちゃん. Also, reference to one's own brother as such within the immediate family and among close friends is more natural.