"Big brother, welcome back."
Lol same I can barely read these sentences without thinking of some anime characters >_
As far as I understand, 兄さん is sufficiently respectful to address your own brother. The お is extra polite, and necessary for other people's brothers but optional for your own. Increased politeness implies a larger distance between you, perhaps in age, experience or how close you are as a family.
Ok im assuming that the big brother here is MY big brother, otherwise in english it would need to say 'his/her big bro etc, right?)
So isn't お兄さん usually reserved for the big brothers not within the immediate family? Wouldn't this be 兄 alone?
Its not. Because the o is used to when talking ABOUT a member of your own family. But since you're addressing them, youre being respectful
Isn't "okaerinasai" (sorry, I don't have kana on my phone yet) too formal for your own brother?
Different families have different relationships and levels of respect. For some it could be too formal, for others it's normal to be always polite. Also, it's considered safer to learn polite way of speaking first, you always can drop off extra part if you need.